route() interception causes 'pragma' and 'cache-control' headers to be wrongly added on requests

[corford]

Not sure if this is camoufox-js or underlying camoufox but there seems to be a bug whereby if you setup a route (e.g. to block certain domains or assets), the camoufox browser adds pragma and cache-control headers (both with value no-cache) to all requests. This causes the spider to get blocked by Datadome.

If we swap to using stock google chrome binary rather than camoufox, everything works fine with route interception active (neither of these two headers are present on the requests). If we disable route interception, Camoufox also works fine (these two headers are not present).

In case it's interesting, this is our route logic:

export const addRequestFiltering = async ({
  page,
  filters,
}) => { 
  if (!filters.length) return;

  // Convert non-regex filters to regular expressions, emulating Puppeteer's
  // Network.setBlockedURLs() pattern behaviour
  const regexRequestFilters = filters.map(
    filter => (filter instanceof RegExp
      ? filter
      : new RegExp(`.*${filter.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}.*`)
    ),
  );

  const routeRegexStr = regexRequestFilters.length > 1
    ? regexRequestFilters.map(regex => `(${regex.source})`).join('|')
    : regexRequestFilters[0].source;

  await page.route(new RegExp(routeRegexStr), route => {
    route.abort('blockedbyclient');
  });
};

await addRequestFiltering({
  page, 
  [
        'cookielaw',
        'googlesyndication.com',
        'trustpilot.com',
  ]
});

[barjin]

Hello and thank you for your interest in this project!

Using the .route method in Playwright disables in-browser cache (see docs). This is mostly to ensure that the route-d paths return correct data (i.e. the data returned by the "interceptor" function) and not data cached from previous requests.

You can disable this behaviour by commenting out this line in Playwright code base (e.g. using patch-package, which allows you to edit your dependencies).

The difference between Chrome and Camoufox is likely in how Chromium / Firefox handle the in-browser cache.

I'll close this issue now, but feel free to ask additional questions if you have any.
Cheers!

[corford]

Thanks @barjin for the explanation and patch point. Makes sense (I'd forgotten route interception disables the cache!). Do you know if Playwright finally supports Network.setBlockedURLs for FF/Camoufox (which would allow us to side-step this issue without needing to patch the cache logic)?