Help diagnosing Granian Problem?

[shamoon]

First of all just want to say thank you for this project, amazing work.

In fact, the project is so exciting that one of our maintainers @stumpylog of the FOSS tool we work on wanted to try and move us over to Granian, which at first seemed to go great! However, in testing a beta of our first release with Granian we received reports (and can reproduce) an issue in our docker image where the web server becomes unresponsive after some time. It happens with normal usage but can be reproduced quickly by making many requests (see video below).

Full disclaimer, I'm not our backend dev so I'm not very well-versed in this, he is AFK this week (I think on mobile) but I wanted to get the conversation started.

• Here's a video of the issue in case it helps, you'll see reloading a page that makes a lot of requests (~75 for thumbnails, hitting the API etc) eventually seizes up the web server. There is nothing obvious to me in the logs, nothing super obvious about resource utilization. After this happens the web server never comes back, no matter how long I wait: https://www.dropbox.com/scl/fi/u6aegez4s8ls9c03jxaz7/granian_again.mov?rlkey=08urnthg81d4g7xw0uyequual&st=4369bpxw&dl=0 (too large for GH upload)
  • At the end I try to run a django command which fails and says the db is locked (testing locally with sqlite but same happens with Postgres), which makes me wonder if all of the canceled requests are somehow floating around or something like that?
• We're running granian like this: https://github.com/paperless-ngx/paperless-ngx/blob/dev/docker/rootfs/etc/s6-overlay/s6-rc.d/svc-webserver/run

We've tried a bunch of random things to no avail including upgrading to 2.1.0, using uvloop, tweaking backpressure, etc. As I mentioned I usually work more on our frontend code so I dont really know what Im doing here, but wondering if anyone has suggestions for how to investigate / what might be the issue.

Again, thank you!

[gi0baro]

@shamoon thank you for trying Granian within your project, it definitely helps to have more behavioral data on existing large projects.

We've tried a bunch of random things to no avail including upgrading to 2.1.0, using uvloop, tweaking backpressure, etc.

A few things:

• 2.1.x is definitely the way to go, as includes several ASGI improvements compared to previous versions
• uvloop should generally helps in high-concurrent environments
• can you be more specific about the backpressure tweaks?

As you mentioned

At the end I try to run a django command which fails and says the db is locked (testing locally with sqlite but same happens with Postgres), which makes me wonder if all of the canceled requests are somehow floating around or something like that?

not a Django expert here, but feels like the Django ORM is running out of available connections.
So theoretically setting a backpressure value matching the maximum amount of available connections (or less) should improve the thing.
If that's not the case, maybe those connections are kept open forever and never released after the request completed. Are you using a connection pool in the ORM?

[gi0baro]

Well, we're off to a very n00b start here (sorry) where I think I made a mistake in trying out uvloop, when trying it now Im getting [...]

With Granian 2.x you need to manually install uvloop package (or switch the dependency to granian[uvloop])

I read through #406 and tried setting it low (8) and a few random other values less than the default. Setting it to 4 seemed to basically break things...

Interesting. So there's definitely something going on with concurrency. What happens if you set backpressure to 1 (basically making everything serial)?

Yea it does seem that way, what's confusing is that our app has always worked the way it does with respect to requests / the db, for example no we dont use pooling (never have)

Yeah, the point is Granian uses a custom asyncio.Future implementation for the ASGI send and receive methods, I fear this might be a similar case to #525, where Django expects some specific sequence of events happening, but in Granian there might be a slight difference in cancellation flow, that might cause the database connection not being closed at the end.

To have more context:

• which Django version is this?
• I see you explicitly set websockets, do you launch a websocket when you load the page and connect to the database from that as well? It might be worth trying to disable it (--no-ws) and see if anything changes (at least we can see whether this depends on websockets entirely or not)

Also, if you have indications on how to run paperless locally with some data already there, I might have some time in the weekend to debug this locally both on Granian and the Django side of things.

[shamoon]

Ahhhh using uvloop (properly) seems to do it!! And actually so does disabling websockets. Youre a hero!

So:

• Django 5.1.6
• uvloop (actually) fixes it!
• Setting backpressure to 1 makes the app basically immediately become unresponsive
• Disabling websockets also seems to fix it. Yes we launch a websocket connection on starting the app which will connect to the db (checks user perms etc). Is there a better way to do this?
• If you are interested in running paperless locally I can definitely help with that, seems like your insights have been super helpful already so lmk if you still want to do that.

Again, thank you so much.

[gi0baro]

uvloop (actually) fixes it!

I'm glad it fixed it. Were you using uvloop with the previous server implementation?

Yes we launch a websocket connection on starting the app which will connect to the db (checks user perms etc). Is there a better way to do this?

I'm lacking too much context to have suggestions in that regards. In general I tend to manually open/close DB connections in websockets around queries, just to prevent connections stealing. But I don't know how much is feasible with Django and/or your project.

If you are interested in running paperless locally I can definitely help with that, seems like your insights have been super helpful already so lmk if you still want to do that.

Yes, I'd still like to run some tests locally, as I'd like to find out what's the actual issue with Django websockets using the default loop implementation. Ideally this should led to open an issue in the Django project, but I need to collect a bunch of data/try few things before doing that.

[shamoon]

Amazing, thanks again.

Yea I’m going to experiment with more explicitly handling db connections in the websocket, I think you’ve hit the nail on the head that there’s a difference with closing those connections automagically with uvloop vs without. This wasn’t even really on my radar.

As far as getting setup with our project I’ve added you to a private repo we have (let me know if you didn’t get the invite) which has some dummy data. It’s nothing private or anything just purely for testing. So, options are:

• Use the devcontainer and add some simple data. I don’t think you’ll need much (any?) to test this. More details: https://github.com/paperless-ngx/paperless-ngx/blob/dev/.devcontainer/README.md
• Clone our main repository ( https://github.com/paperless-ngx/paperless-ngx/tree/dev ) and build the docker image and test against the “paperless-ngx-instance” repo I added you to. I dunno how familiar you are with docker, there’s a compose file you can just change the image tag to point to your local image.
• There are dev instructions here https://docs.paperless-ngx.com/development/ One note is that Django has a built-in dev server so one way or another you’ll want to test the docker image to check granian (at least I think that’s the easiest way).
• Reminder that granian is only in our dev/beta branches, eg not main or the latest docker image

I do think getting the project setup isn’t always the easiest for some, please let me know if you hit any snags.

In general so appreciative of your time and help, thank you!

[shamoon]

Pretty interesting, this seems to come down to 1 line of code (we do a similar thing in _is_owner_or_unowned also but that is only called when a websocket messge is sent, this line is called upon connecting):

https://github.com/paperless-ngx/paperless-ngx/blob/5d6cfa7349e65162d9cb446cb0e84a849ad730bf/src/paperless/consumers.py#L11

This is using channels auth middleware ( https://channels.readthedocs.io/en/latest/topics/authentication.html ) to access the user simply as self.scope["user"]. If I remove that line, problem fixed. What's interesting is that even when I try to aggresively close db connections, like this: paperless-ngx/paperless-ngx@76771e8 it still happens. Also, even if you remove that line, a db connection is still opened (but the problem doesnt happen):

2025-03-20 13:22:29 [2025-03-20 13:22:29,928] [DEBUG] [paperless.websockets] Connecting ws...
2025-03-20 13:23:51 [2025-03-20 13:23:51,194] [DEBUG] [paperless.websockets] Disconnecting ws...
2025-03-20 13:23:51 2025-03-20 13:23:51,198] [DEBUG] [paperless.websockets] Discarding 1 database connections...
...

When looking at the actual scope["user"] object, its a <channels.auth.UserLazyObject object at 0xffff7d1cb170>, essentially a Django LazyObject ( https://docs.djangoproject.com/en/5.0/_modules/django/utils/functional/ ), you can guess what that does.

So anyway, not sure what the conclusion is here, I'm still playing with it, but maybe you (or our own dev) will find this interesting so I'm just reporting here. Either way, we have the uvloop solution in-hand.

Thanks again