Skip to content

enhance http_wrapper to be a better http citizen #1037

@willmurphyscode

Description

@willmurphyscode

There have been a couple of recent instances of automated traffic interception blocking Vunnel from downloading data.

When asking to be allowed to programmatically read the data we need, we want to be able to say that we respect the servers we get that data from.

Concretely:

  • always set a User-Agent header so we identify ourselves (some providers do this today)
  • respect 429s - don't retry until Retry-After or a reasonable default
  • respect Retry-After on 503s
  • respect Retry-After on 403s (I recently learned GitHub will sometimes do this)

Please comment if I missed something. Thanks!

Metadata

Metadata

Labels

Type

No type

Projects

Status

Ready

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions