Scrape at scale running multiple Headful Nodriver Instances (Like Selenium Grid)

[juanfrilla]

Hi everyone,

I'm looking for a way to run multiple headful nodriver instances concurrently, similar to how Selenium Grid works. In my current setup, I use Selenium Grid via Docker Compose, which is great because it allows me to manage and allocate resources efficiently for scalable scraping.

With nodriver, however, I’ve only been able to launch a single headful session at a time. Ideally, I’d like to spin up multiple nodriver instances, each with its own attached browser, to perform parallel scraping tasks at scale.

Has anyone successfully achieved this setup with nodriver, or could guide me in the right direction to do so?

Thanks in advance!

[OnlineProxy-io]

Hey!
Yep, it's definitely possible to run multiple headful nodriver instances concurrently, it just takes a bit of extra setup compared to Selenium Grid. Each instance needs its own user data directory and unique Chrome DevTools Protocol port to avoid conflicts. Since nodriver uses CDP under the hood, it works similarly to launching multiple standalone Chrome sessions. To scale, I containerized each browser + nodriver combo in Docker (with xvfb if you need GUI), and used something like Redis/Celery to dispatch scraping jobs across the containers. Think DIY Selenium Grid, just without the orchestration baked in.
You can go further with Kubernetes for autoscaling, and add a hub API to route tasks to free nodes. No official tooling for this yet, but it works well once set up. If you’re interested, I’d love to see an open-source orchestration layer built around this, the stealth from nodriver makes it worth the effort.
Hope that helps. Cheers!

[juanfrilla]

Thank you for your detailed response, it would be interesting to embark on that as a personal project, I feel capable of implementing it but currently I dont have time.
Let's see