process.json not written, always restarting from scratch

[Frank-Steiner]

Hi,
I've just fetched the latest code and tried to download invocies from amazon (great idea to automate this!!!). My call is

docudigger scrape amazon -u xxx -p yyy -l ~/.docudigger --fileDestinationFolder ~/.docudigger/amazon

This works find and fetches the PDFs from 4 years. But when I call it again with --yearFilter 2025 --onlyNew I get these messages:

[info] [2025-10-24 17:10:28] [scrape:amazon]:   OnlyNew activated.
[info] [2025-10-24 17:10:28] [scrape:amazon]:   Getting last run
[warn] [2025-10-24 17:10:28] [scrape:amazon]:   process.json not found. Full run needed. OnlyNew deactivated. 

and there is indeed no process.json anywhere in ~/.docudigger or anywhere else in $HOME. Should this file be filled during the scrape, or is it written at once at the end?
Any idea why it could be missing?

cu,
Frank

[dosubot]

Hi @Frank-Steiner! I'm Dosu and I’m helping the docudigger team.

The process.json file is written at the end of a successful scrape run, specifically after all invoices have been processed. It's expected to be located inside the folder you specify with --fileDestinationFolder (so in your case, ~/.docudigger/amazon/process.json) and is essential for the --onlyNew flag to work as intended. If process.json is missing, docudigger will always do a full run and disable --onlyNew, as you observed.

Common reasons for process.json not being created include the scrape process not completing successfully—this can happen if there are errors (like navigation timeouts, missing selectors due to Amazon page changes), or if the process is interrupted before it finishes. In these cases, the code path that writes process.json is never reached, so the file isn't created or updated. If your initial scrape appeared to finish but process.json is still missing, check for any errors or warnings in the logs that might indicate an early exit or crash during the run. Also, make sure the destination folder exists and is writable.

After a successful full scrape, you should see process.json in ~/.docudigger/amazon, and subsequent runs with --onlyNew should work as expected. If not, reviewing the logs for errors during the scrape will help pinpoint the cause.
Relevant code and issue discussion

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[Frank-Steiner]

Sorry, but this is a non-working concept. The script downloads 500 invoices from 4 years and gets one or two errors at some point (timeout for the popup, even with timeout set to 10 seconds), and that prevents the process.json from being written. So the only way to download the next new invoices is going through the 4 years and 500 already-downloaded invoices again???
This can't work. A reasonable concept would be writing what has been downloaded already and not considering these parts again. Just a "yes/no" result is guaranteed to fail with a huge number of initial invoices.