Use GitHub GraphQL for Metadata fetching (with new metadata fields)

[Rabenherz112]

This is my best try on implement awesome-selfhosted/awesome-selfhosted-data#84, as well as some other nice-to-have metadata (which I personally would like to see, such as the current release and release date), which means fetching the metadata over the GrapSQL API instead of the Python package used.

New metadata preview:

name: Paperless-ngx
website_url: https://docs.paperless-ngx.com/
description: Scan, index, and archive all of your paper documents with an improved interface (fork of Paperless).
licenses:
  - GPL-3.0
platforms:
  - Python
  - Docker
tags:
  - Document Management
source_code_url: https://github.com/paperless-ngx/paperless-ngx
demo_url: https://demo.paperless-ngx.com/
stargazers_count: 17058
updated_at: '2024-05-07'
archived: false
current_release:
  tag: v2.8.1
  published_at: '2024-05-07'
commit_history:
  2024-05: 46

The new changes have been tested by running the full metadata processing on the awesome-selfhosted-data repo (last tested May 5). However I still have some open questions regarding how we should do some things, these things have been marked with TODO: in the code itself.

Additionally, there have been no tests written or modified (since I don't code normally or use python), so this is still open and would need to be done.

[nodiscc]

Thank you.
I will review this and #133 when I get some time, it might take a while, I will do it eventually but don't know when.

[Rabenherz112]

Pending Issues have been fixed, and a test has been done by running a full metadata processing on the awesome-selfhosted-data repository.

Some things in the code still have open Questions; see comments marked with TODO:.

[Rabenherz112]

Just a quick ping to remind you about this. 😊
No rush at all, though!

[Rabenherz112]

Hey there,
It’s been two months since the last update, so if you have any availability, I would greatly appreciate another review of the PR.

Separately, I would like to challenge the current status of whether we should include the commit history. I am currently finishing up the last touches for a new alternative interface for awesome-selfhosted, and plan to ask for feedback in the data repo, once I've ironed out a few bugs in the next coming days.

Maybe as a bit of a spoiler why I would appreciate, if he could include it, here are the interfaces resulting out of the datasets:

• Current Dataset: Automatically rebuilt daily from awesome-selfhosted-data
• New Dataset: Static snapshot where I ran this PR against (includes Commit History)

[nodiscc]

These web interfaces look absolutely awesome
I will reconsider adding the commit history in the yaml data files
Will try to review and merge this as soon as possible (possibly this week)
Is there any chance you could share the source code and deployment process for your web interface? We could host it alongside the current website for some time, and why not, make it the default

[Rabenherz112]

The current version is already open-source (under AGPL-3.0 license) (However this is not yet production ready - I only push to the repo, when I want to check the code against the live data from awesome-selfhosted).

For deployment see this repo (the github action there put's this already on my webserver): https://github.com/Rabenherz112/awesome-selfhosted-web

Quite important to mention also (will do this as well, when I open the discussion in awesome-selfhosted-data, since I find this rather important) that quite a lot of this code is AI generated / fixed and patched, since I am a Sysadmin not a web dev.

[Rabenherz112]

Little update on this PR.
Since I have been working a lot more with Python lately, I wanted to give this PR a slight rework and apply what I have learned (since initially when I did this PR, this was with one of my first real interactions with python).

The core functionality remains unchanged and has been tested successfully against the awesome-selfhosted-data repo.
Since I expect the review will probably still take a while (this could slowly become a meme 😆), I went ahead and re-added the commit_history key (maybe by the time this PR is actually merged, we'll have a first-party use case for it).

Some improvements I did were:

• Enhanced the commit_history key with better error handling
• Added support for initial runs to backfill complete history
• Refactored for improved readability
• Added error catching, especially when GraphQL randomly throws error 500, or just gives you empty data for no reason what so ever (it is crazy, how just retiring it with the same query, sometimes is already enough)

PS: I don't understand why the test is failing.. I didn't even touch utils.py or archive_webpages.py (So I assume, I can ignore those errors)

Edit: I use this new version in my pipeline for my own fronted for awesome-selfhosted.

[nodiscc]

Let's do this, reviewed, tested and works well.
I fixed the remaining pylint errors, I don't know why they didn't show earlier.
I will make a release, and then a PR in awesome-selfhosted to bump the hecat version there.
Thanks for your patience

Edit: other tests started failing in the archive_webpage, this is not related to this PR but it's expected that they start failing when websites against which the tests are run change things. I will fix these here as well.

[Rabenherz112]

Massive thanks, for reviewing and merging it, even though you are quite busy (really did expect that) <3