Set a more permissive `Accept:` header for sitemap loading

[barjin]

Some sitemaps (e.g. https://docs.superjoin.ai/sitemap.xml) return 404 for the current utils/sitemap value of Accept: HTTP header.

crawlee/packages/utils/src/internals/sitemap.ts

Line 261 in 615c8f9

accept: 'text/plain, application/xhtml+xml, application/xml;q=0.9, */*;q=0.8',

Investigate why this was required in the first place (maybe it actually wasn't). If it wasn't, switch the accept header to something more permissive (e.g. */*) or remove it completely.

Additional context:

• https://apify.slack.com/archives/C051BSK664C/p1762868702710649
• https://apify.slack.com/archives/C05683VTD6J/p1762891856864289

[janbuchar]

crawlee/packages/utils/src/internals/sitemap.ts

Line 261 in 615c8f9

accept: 'text/plain, application/xhtml+xml, application/xml;q=0.9, */*;q=0.8',

Investigate why this was required in the first place (maybe it actually wasn't). If it wasn't, switch the accept header to something more permissive (e.g. */*) or remove it completely.

Here's your answer 🙂 #2619 (comment)

[barjin]

Thank you - I did find that, but it still left me wondering why we would add it in the first place (instead of just using got-scraping's default Accept).

[janbuchar]

Thank you - I did find that, but it still left me wondering why we would add it in the first place (instead of just using got-scraping's default Accept).

I see. IMO even an empty Accept header should be fine. Noone in their right mind would block access to a sitemap based on browser fingerprinting. Then again, what the aforementioned server is doing is not exactly sane either.