Instagram & Tesco - Got HTML content but no text found (With 200 reply code) #2054

Pheggas · 2023-12-23T07:45:38Z

Pheggas
Dec 23, 2023

Hey! I love changedetection.io project but there are some websites that changedetection just can't check and always returns 200 reply code. One of them is Instagram and second potravinydomov.itesco.sk (basically Tesco website). Do you have any recommendations on workaround or help with these problematic websites?

Running on ubuntu server vm 22.04 x64; Hypervisor Truenas Scale 23.10.0.1;
Hardware:
CPU: Intel i5 8500T
RAM: 32 gb (16 gb reserved for the ubuntu vm)

docker-compose.yml

version: '3.2'
services:
    changedetection:
      image: ghcr.io/dgtlmoon/changedetection.io
      container_name: changedetection
      hostname: changedetection
      volumes:
        - /home/docker/docker_changedetection:/datastore
      environment:
        - PUID=1001
        - PGID=1001
        - PLAYWRIGHT_DRIVER_URL=ws://playwright-chrome:3000/?stealth=1&--disable-web-security=true
      ports:
        - 4080:5000
      restart: unless-stopped

    playwright-chrome:
        hostname: playwright-chrome
        image: browserless/chrome:1.60-chrome-stable
        restart: unless-stopped
        environment:
            - SCREEN_WIDTH=1920
            - SCREEN_HEIGHT=1024
            - SCREEN_DEPTH=16
            - ENABLE_DEBUGGER=false
            - PREBOOT_CHROME=true
            - CONNECTION_TIMEOUT=300000
            - MAX_CONCURRENT_SESSIONS=10
            - CHROME_REFRESH_TIME=600000
            - DEFAULT_BLOCK_ADS=true
            - DEFAULT_STEALTH=true
            - DEFAULT_IGNORE_HTTPS_ERRORS=true

changedetection container logs (right after boot):

>>> Datastore path is  /datastore/url-watches.json
Watching: 49adf95f-d714-4330-a1b4-d2b3e2eca104 redacted
Watching: da4cea58-55d5-4dde-864f-2eec7e8e7c73 redacted
Watching: 57f71fd1-4e26-48e6-b79b-bd89da6e32e9 redacted
Watching: 8dc80db4-5e9e-4533-b163-2e418ab023a8 redacted
Watching: c42ee4dd-8028-4648-8a34-36a8fb731c53 redacted
Watching: 628e480a-24cf-4c59-8680-31145a4c27ce redacted
Watching: d20f9f98-89b5-4666-b333-bb4b5c16d59e redacted
Watching: fd26ef0a-cd66-4607-8658-3c547eb1abb9 redacted
Watching: 86241d73-d24f-45e4-bc41-94fd02682f7b redacted
Watching: c8fad0d7-02ad-499f-a7e8-874d932cbb68 redacted
(1) wsgi starting up on http://0.0.0.0:5000
System env MINIMUM_SECONDS_RECHECK_TIME 20

playwright-chrome container logs (right after boot):

  "pauseOnConnect": false,
  "userDataDir": "/tmp/browserless-data-dir-ZbXE5s",
  "playwright": false,
  "stealth": false,
  "meta": null,
  "executablePath": "/usr/bin/google-chrome",
  "handleSIGINT": false,
  "handleSIGTERM": false,
  "handleSIGHUP": false
}
2023-12-23T10:12:56.557Z browserless:chrome-helper Chrome PID: 21
2023-12-23T10:12:56.559Z browserless:chrome-helper Finding prior pages
2023-12-23T10:12:56.579Z browserless:chrome-helper Chrome PID: 32
2023-12-23T10:12:56.580Z browserless:chrome-helper Finding prior pages
2023-12-23T10:12:56.590Z browserless:chrome-helper Chrome PID: 37
2023-12-23T10:12:56.590Z browserless:chrome-helper Finding prior pages
2023-12-23T10:12:56.606Z browserless:chrome-helper Chrome PID: 59
2023-12-23T10:12:56.609Z browserless:chrome-helper Finding prior pages
2023-12-23T10:12:56.632Z browserless:chrome-helper Chrome PID: 48
2023-12-23T10:12:56.632Z browserless:chrome-helper Finding prior pages
2023-12-23T10:12:56.680Z browserless:chrome-helper Chrome PID: 79
2023-12-23T10:12:56.681Z browserless:chrome-helper Finding prior pages
2023-12-23T10:12:56.686Z browserless:chrome-helper Found 1 pages
2023-12-23T10:12:56.687Z browserless:chrome-helper Setting up page Unknown
2023-12-23T10:12:56.687Z browserless:chrome-helper Patching file-chooser dialog
2023-12-23T10:12:56.687Z browserless:chrome-helper Injecting download dir "/usr/src/app/workspace"
2023-12-23T10:12:56.692Z browserless:system Chrome launched 1518ms
2023-12-23T10:12:56.693Z browserless:chrome-helper Setting up file:// protocol request rejection
2023-12-23T10:12:56.693Z browserless:chrome-helper Setting up page for ad-blocking
2023-12-23T10:12:56.702Z browserless:chrome-helper Found 1 pages
2023-12-23T10:12:56.704Z browserless:chrome-helper Setting up page Unknown
2023-12-23T10:12:56.704Z browserless:chrome-helper Patching file-chooser dialog
2023-12-23T10:12:56.706Z browserless:chrome-helper Injecting download dir "/usr/src/app/workspace"
2023-12-23T10:12:56.707Z browserless:system Chrome launched 1536ms
2023-12-23T10:12:56.710Z browserless:chrome-helper Setting up file:// protocol request rejection
2023-12-23T10:12:56.710Z browserless:chrome-helper Setting up page for ad-blocking
2023-12-23T10:12:56.728Z browserless:chrome-helper Found 1 pages
2023-12-23T10:12:56.728Z browserless:chrome-helper Setting up page Unknown
2023-12-23T10:12:56.728Z browserless:chrome-helper Patching file-chooser dialog
2023-12-23T10:12:56.728Z browserless:chrome-helper Injecting download dir "/usr/src/app/workspace"
2023-12-23T10:12:56.729Z browserless:system Chrome launched 1558ms
2023-12-23T10:12:56.748Z browserless:chrome-helper Setting up file:// protocol request rejection
2023-12-23T10:12:56.748Z browserless:chrome-helper Setting up page for ad-blocking
2023-12-23T10:12:56.750Z browserless:chrome-helper Found 1 pages
2023-12-23T10:12:56.750Z browserless:chrome-helper Setting up page Unknown
2023-12-23T10:12:56.750Z browserless:chrome-helper Patching file-chooser dialog
2023-12-23T10:12:56.750Z browserless:chrome-helper Injecting download dir "/usr/src/app/workspace"
2023-12-23T10:12:56.750Z browserless:system Chrome launched 1579ms
2023-12-23T10:12:56.756Z browserless:chrome-helper Chrome PID: 102
2023-12-23T10:12:56.756Z browserless:chrome-helper Finding prior pages
2023-12-23T10:12:56.761Z browserless:chrome-helper Setting up file:// protocol request rejection
2023-12-23T10:12:56.761Z browserless:chrome-helper Setting up page for ad-blocking
2023-12-23T10:12:56.765Z browserless:chrome-helper Chrome PID: 96
2023-12-23T10:12:56.765Z browserless:chrome-helper Finding prior pages
2023-12-23T10:12:56.791Z browserless:chrome-helper Chrome PID: 25
2023-12-23T10:12:56.791Z browserless:chrome-helper Finding prior pages
2023-12-23T10:12:56.803Z browserless:chrome-helper Chrome PID: 43
2023-12-23T10:12:56.805Z browserless:chrome-helper Finding prior pages
2023-12-23T10:12:56.814Z browserless:chrome-helper Found 1 pages
2023-12-23T10:12:56.814Z browserless:chrome-helper Setting up page Unknown
2023-12-23T10:12:56.814Z browserless:chrome-helper Patching file-chooser dialog
2023-12-23T10:12:56.816Z browserless:chrome-helper Injecting download dir "/usr/src/app/workspace"
2023-12-23T10:12:56.816Z browserless:system Chrome launched 1645ms
2023-12-23T10:12:56.817Z browserless:chrome-helper Found 1 pages
2023-12-23T10:12:56.818Z browserless:chrome-helper Setting up page Unknown
2023-12-23T10:12:56.818Z browserless:chrome-helper Patching file-chooser dialog
2023-12-23T10:12:56.818Z browserless:chrome-helper Injecting download dir "/usr/src/app/workspace"
2023-12-23T10:12:56.818Z browserless:system Chrome launched 1647ms
2023-12-23T10:12:56.822Z browserless:chrome-helper Setting up file:// protocol request rejection
2023-12-23T10:12:56.822Z browserless:chrome-helper Setting up page for ad-blocking
2023-12-23T10:12:56.824Z browserless:chrome-helper Setting up file:// protocol request rejection
2023-12-23T10:12:56.824Z browserless:chrome-helper Setting up page for ad-blocking
2023-12-23T10:12:56.837Z browserless:chrome-helper Found 1 pages
2023-12-23T10:12:56.837Z browserless:chrome-helper Setting up page Unknown
2023-12-23T10:12:56.837Z browserless:chrome-helper Patching file-chooser dialog
2023-12-23T10:12:56.837Z browserless:chrome-helper Injecting download dir "/usr/src/app/workspace"
2023-12-23T10:12:56.838Z browserless:system Chrome launched 1667ms
2023-12-23T10:12:56.838Z browserless:chrome-helper Found 1 pages
2023-12-23T10:12:56.838Z browserless:chrome-helper Setting up page Unknown
2023-12-23T10:12:56.838Z browserless:chrome-helper Patching file-chooser dialog
2023-12-23T10:12:56.839Z browserless:chrome-helper Injecting download dir "/usr/src/app/workspace"
2023-12-23T10:12:56.839Z browserless:system Chrome launched 1668ms
2023-12-23T10:12:56.840Z browserless:chrome-helper Setting up file:// protocol request rejection
2023-12-23T10:12:56.840Z browserless:chrome-helper Setting up page for ad-blocking
2023-12-23T10:12:56.841Z browserless:chrome-helper Setting up file:// protocol request rejection
2023-12-23T10:12:56.841Z browserless:chrome-helper Setting up page for ad-blocking
2023-12-23T10:12:56.842Z browserless:chrome-helper Found 1 pages
2023-12-23T10:12:56.842Z browserless:chrome-helper Setting up page Unknown
2023-12-23T10:12:56.842Z browserless:chrome-helper Patching file-chooser dialog
2023-12-23T10:12:56.842Z browserless:chrome-helper Injecting download dir "/usr/src/app/workspace"
2023-12-23T10:12:56.844Z browserless:system Chrome launched 1673ms
2023-12-23T10:12:56.847Z browserless:chrome-helper Setting up file:// protocol request rejection
2023-12-23T10:12:56.847Z browserless:chrome-helper Setting up page for ad-blocking
2023-12-23T10:12:56.851Z browserless:chrome-helper Found 1 pages
2023-12-23T10:12:56.859Z browserless:chrome-helper Setting up page Unknown
2023-12-23T10:12:56.859Z browserless:chrome-helper Patching file-chooser dialog
2023-12-23T10:12:56.861Z browserless:chrome-helper Injecting download dir "/usr/src/app/workspace"
2023-12-23T10:12:56.862Z browserless:system Chrome launched 1691ms
2023-12-23T10:12:56.878Z browserless:chrome-helper Setting up file:// protocol request rejection
2023-12-23T10:12:56.880Z browserless:chrome-helper Setting up page for ad-blocking

dgtlmoon · 2023-12-23T09:57:05Z

dgtlmoon
Dec 23, 2023
Maintainer

Does potravinydomov.itesco.sk work with browsersteps? can you try? (atleast if the page can load)

also again, no real information from you to solve it, platform, operating system, memory, etc etc

1 reply

Pheggas Dec 23, 2023
Author

I updated the original post so you can see my setup in more detailed way

dgtlmoon · 2023-12-23T09:57:22Z

dgtlmoon
Dec 23, 2023
Maintainer

you dont even say if you are using playwright, or selenium or anything, you dont give enough information

1 reply

Pheggas Dec 23, 2023
Author

I updated the original post so you can see my setup in more detailed way

dgtlmoon · 2023-12-23T12:00:46Z

dgtlmoon
Dec 23, 2023
Maintainer

Tested on our paid subscriptions, works really nice

You still dont give enough information, maybe you are doing some custom JS, or are waiting some seconds, or have some filter, or...

3 replies

Pheggas Dec 23, 2023
Author

I didn't change anything. Just entered the URL to watch, entered label for it and changed default check interval. Nothing other than that.

Are there any visible issues with my compose file? I think it's weird that changedetection don't log it uses playwright. Also, in browser container there's playwright set to false which I assume is not right.

dgtlmoon Dec 23, 2023
Maintainer

Did you select "chrome" browser and NOT "plain requests" ? it is a JS website...

Pheggas Dec 23, 2023
Author

It was set on "System settings default". So if the default value for fetch method was plaintext, then it probably couldn't fetch it as it is JS website as you wrote.

When i changed fetching method to "Playwright" on those two problematic examples, it started to work. Thank you for your help and time and wishing luck with the project!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Instagram & Tesco - Got HTML content but no text found (With 200 reply code) #2054

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 5 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Instagram & Tesco - Got HTML content but no text found (With 200 reply code) #2054

Pheggas Dec 23, 2023

Replies: 3 comments · 5 replies

dgtlmoon Dec 23, 2023 Maintainer

Pheggas Dec 23, 2023 Author

dgtlmoon Dec 23, 2023 Maintainer

Pheggas Dec 23, 2023 Author

dgtlmoon Dec 23, 2023 Maintainer

Pheggas Dec 23, 2023 Author

dgtlmoon Dec 23, 2023 Maintainer

Pheggas Dec 23, 2023 Author

Pheggas
Dec 23, 2023

Replies: 3 comments 5 replies

dgtlmoon
Dec 23, 2023
Maintainer

Pheggas Dec 23, 2023
Author

dgtlmoon
Dec 23, 2023
Maintainer

Pheggas Dec 23, 2023
Author

dgtlmoon
Dec 23, 2023
Maintainer

Pheggas Dec 23, 2023
Author

dgtlmoon Dec 23, 2023
Maintainer

Pheggas Dec 23, 2023
Author