Support for non-Basic Auth in scrapyd-deploy

[jwebb-va]

My team is working on a set of scrapy spiders which we want to deploy to a scrapyd server. Our scrapyd server is configured to use an oauth2 proxy to authenticate traffic.

On all of our requests to our Scrapyd API, we need the following header to authenticate our requests:

Authorization: Bearer 1/AbC123

where 1/AbC123 is a OAuth2 access token.

Currently the scrapyd-deploy utility only supports using Basic auth.

[jwebb-va]

I foresee a few ways to solve this issue...

scrapyd-deploy could support some kind of oauth2_bearer_token key in the scrapy.cfg file. This could add the HTTP Authorization header for us automatically.

[deploy]
url = https://scrapyd.example.com/
project = example
oauth2_bearer_token = 1/AbC123

A more generic solution might be to simply allow users to specify additional HTTP headers manually via command-line arguments. This would allow us to use OAuth2 authentication but would also allow other use cases which require custom headers.

scrapyd-deploy -h "Authorization: Bearer 1/AbC123", "Another-Header: foo"

[madzohan]

Currently the scrapyd-deploy utility only supports using Basic auth.

Can you explain how it can be done?

[Digenis]

@VlaGrishenko,

scrapyd-client/scrapyd-client/scrapyd-deploy

Lines 207 to 210 in 47a7c7a

	def _add_auth_header(request, target):
	if 'username' in target:
	u, p = target.get('username'), target.get('password', '')
	request.add_header('Authorization', basic_auth_header(u, p))

You need to add a username and password to your target in scrapy.cfg.
scrapyd doesn't support authorization
but you can configure it to listen only on 127.0.0.1
and then use apache to proxy connections from elsewhere
while requiring them to provide credentials.

You still however trust users of the same computer connecting on 127.0.0.1 without auth.
What makes sense to me is enabling scrapyd to listen on a uds
to which only the apache user has access and then proxy it as described above.

[kevalsakhiya]

@jpmckinney

I plan to work on this enhancement and wanted to share the full plan for clarity and feedback.

The main objective is incorporating Bearer token authentication into scrapyd-client. Currently, the tool only supports Basic Auth through username and password settings in scrapy.cfg. I plan to extend the _add_auth_header() method to also check for an authorization configuration key. This will allow users to include a Bearer token directly in the config, or alternatively set it via an environment variable like SCRAPYD_AUTHORIZATION, which is especially useful for CI/CD workflows. The implementation will remain backward compatible—if a username is specified, it will fall back to Basic Auth.

Since Scrapyd doesn't have native authentication, it’s typically secured using a reverse proxy. While @Digenis suggested to use Apache or Nginx with Basic Auth, I propose using OAuth2 Proxy for better flexibility and security. (Please share your thoughts on this @Digenis)

OAuth2 Proxy sits in front of Scrapyd (which should be bound to 127.0.0.1) and enforces OAuth2-based login flows with providers like Google or GitHub. It automatically handles token validation, login redirects, and user session management.

Access can be restricted by specifying either an authenticated-emails-file with a list of approved email addresses, or by using the email-domain flag to allow only users from a specific company domain.

This setup ensures that only authorized and verified users can reach Scrapyd via the proxy. Each scrapyd-deploy request would need to include a valid Authorization: Bearer <token> header, which this enhancement will support.

I’m looking forward to implementing this and will open a PR once the changes are ready. I’m also happy to hear any early feedback or considerations from maintainers or others interested in this use case.

[jpmckinney]

Hi @kevalsakhiya, there is no _add_auth_header anymore (removed in 58debf1 and further refactored since then). Please be sure you are looking at the latest commit from GitHub.

In any case, I am happy to support Bearer token authentication. I think it should be fairly straightforward.