-
Notifications
You must be signed in to change notification settings - Fork 29
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: App token refreshing #289
feat: App token refreshing #289
Conversation
👋 I ran some large extractions over the weekend using this PR's SHA and a single app key. The good news is that it ran smoothly for several hours at a time without any authenticator related failure. The longest job I ran took about 8.5 hours, and I can see the One slightly confusing find is that whenever the expiry time approached a refresh would happen, and then another refresh would happen say 3-6 minutes later. Log excerpt demonstrating it:
I was extracting the issues stream, and when I look at where GitHubTokenAuthenticator is initialized it looks like maybe the parent stream (repos) and child stream (issues) are each constructing and maintaining a GitHubTokenAuthenticator object, meaning there would be two installation access tokens at any given time and each would need to be updated periodically. Does that match your mental model @edgarrmondragon? As long as we understand why it's refreshing twice and believe it isn't due to a bug, I don't think this is a problem - multiple installation access tokens can exist for the same app and will just cumulatively contribute to the app's rate limit. |
If it helps in anyway, we've been using app tokens with this method without issues since last year, but i haven't refactored the code to address Meltano's team feedback: #201 |
That makes sense to me. We could always make it singleton in the future, but I don't think it causes issues at the moment? |
Agreed. 👍 |
…me, refresh it as part of has_calls_remaining.
for more information, see https://pre-commit.ci
7a168f9
to
fd6ed06
Compare
Quality Gate passedIssues Measures |
Thanks @TrishGillett, this is a monumental improvement! |
This PR implements app token refreshing! Thank you for your patience with the previous refactoring PRs that prepared for this one.
has_calls_remaining
, the amount of time left until estimated token expiry is checked, and tokens are refreshed if expiry is close.expiry_time_buffer
is added, controlling how many minutes before expiry the app token will be refreshed. This parallels howrate_limit_buffer
is used to ensure we don't push tokens all the way to their limits.