Should I report this as a bug? #853

Minglarn · 2021-02-27T10:45:30Z

Minglarn
Feb 27, 2021

Hi!
Running 0.8.4 on a ubuntu server.
Started to see this strange lines in the logs.
peewee.OperationalError: Connection already opened. 2021-02-27T09:31:28Z {'REMOTE_ADDR': '127.0.0.1', 'REMOTE_PORT': '44364', 'HTTP_HOST': '192.168.1.209', (hidden keys: 24)} failed with OperationalError
and when this occur the frigate stops the communications with my HA. After a restart FRIGATE docker runs for a while and then it starts to do this again.

Went back to 0.8.2 to see if it repeats. Any ideias?

Minglarn · 2021-02-27T12:25:19Z

Minglarn
Feb 27, 2021
Author

Update..
As this happens even on 0.8.2 i started to suspect the addon.

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/flask_sockets.py", line 40, in __call__
    handler, values = adapter.match()
  File "/usr/local/lib/python3.8/dist-packages/werkzeug/routing.py", line 1945, in match
    raise NotFound()
werkzeug.exceptions.NotFound: 404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/gevent/pywsgi.py", line 999, in handle_one_response
    self.run_application()
  File "/usr/local/lib/python3.8/dist-packages/geventwebsocket/handler.py", line 87, in run_application
    return super(WebSocketHandler, self).run_application()
  File "/usr/local/lib/python3.8/dist-packages/gevent/pywsgi.py", line 945, in run_application
    self.result = self.application(self.environ, self.start_response)
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2464, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python3.8/dist-packages/flask_sockets.py", line 48, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2450, in wsgi_app
    response = self.handle_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1867, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.8/dist-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.8/dist-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1948, in full_dispatch_request
    rv = self.preprocess_request()
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2242, in preprocess_request
    rv = func()
  File "/opt/frigate/frigate/http.py", line 89, in _db_connect
    database.connect()
  File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 3061, in connect
    raise OperationalError('Connection already opened.')
peewee.OperationalError: Connection already opened.
2021-02-27T12:21:47Z {'REMOTE_ADDR': '127.0.0.1', 'REMOTE_PORT': '34510', 'HTTP_HOST': '192.168.1.209', (hidden keys: 24)} failed with OperationalError

I restarted the integration (in HA) and the errors vanished. Seems that the addon is the bad guy here.

0 replies

blakeblackshear · 2021-02-27T15:14:35Z

blakeblackshear
Feb 27, 2021
Maintainer

This is probably a bug.

1 reply

neonpak Mar 1, 2021

I can confirm. I have the same problem after upgrading from v8.0.0 to v8.0.4. Frigate works after a restart and after some time it spills those errors and I can not access the web interface anymore. It constantly shows the spinner.

Minglarn · 2021-03-02T16:18:27Z

Minglarn
Mar 2, 2021
Author

Really don't know to find reproduce this bug. Today after almost 3 days of up and running it suddently did go bananas... The integration died and frigate web ui did stop responding. But the object detection did still run smoothly, restarted the docker and saw new object detection clips.

Edit: This even happens when running the docker in HA.

0 replies

Minglarn · 2021-03-08T10:58:33Z

Minglarn
Mar 8, 2021
Author

Update... I've changed my detection streams to only detect at 640x480 and it looks like it's working better. No errors so far. Up and running for 3 days.

1 reply

Minglarn Mar 17, 2021
Author

Well. It happens from time to time with sub streams as well.

mtweten · 2021-03-17T19:14:07Z

mtweten
Mar 17, 2021

I've seen this occasionally as well (running 0.8.4-armv7 docker image on rpi4). I hadn't seen it happen for quite awhile, but today I started trying out https://github.com/jakowenko/double-take to send frigate images to CompreFace and saw it happen again. I don't know if that's just coincidence yet. It does seem related to hitting the rest API - double-take hits the frigate API to download snapshots/latest images, and I also hit the API from home assistant to get snapshots when detection events occur.

6 replies

mtweten Mar 18, 2021

Based on coleifer/peewee#1651, I'm assuming it has to do with database connections not being closed somewhere (I say this with no python, flask, or peewee knowledge, OR how frigate uses it :P).

Only thing I could think of is the connection not being closed properly, but the connection management seems quite simple and teardown_request, where the connection is closed, sounds like it should trigger no matter what (even exception cases) based on the flask docs.

One interesting thing I saw in the peewee docs around connection management (http://docs.peewee-orm.com/en/latest/peewee/database.html#connection-management) is that you can specify reuse_if_open=True to connect() to avoid this type of error, but it doesn't seem like it should be necessary.

Only thing I could guess at at this point is some kind of threading issue, or database.is_closed() is returning an incorrect value in some cases and the connection isn't getting closed correctly (funnily enough I found coleifer/peewee#1330, but that was from a long time ago and presumably fixed).

thundergreen Mar 20, 2021

I've seen this occasionally as well (running 0.8.4-armv7 docker image on rpi4). I hadn't seen it happen for quite awhile, but today I started trying out https://github.com/jakowenko/double-take to send frigate images to CompreFace and saw it happen again. I don't know if that's just coincidence yet. It does seem related to hitting the rest API - double-take hits the frigate API to download snapshots/latest images, and I also hit the API from home assistant to get snapshots when detection events occur.

I also run double take and Yess seems like this is the root cause.will contact the Dev of double take

fl42 Mar 20, 2021

Same for me, the issue started when I setup HA to download snapshot using Frigate API on MQTT event

mtweten Mar 21, 2021

@thundergreen I wouldn't log an issue to doubletake, I don't think that's related. It's more just about anything that hits the API tends to trigger this.

jakowenko Mar 22, 2021

Double Take dev here, I don't think the issue is on DT side because the code is simply hitting Frigate's API to get the image data. I do try to add a random delay before requesting new images to prevent simultaneous requests from happening, but outside of things like that, I'm not sure there's much I can do.

I have seen this pop up from time to time and a docker restart frigate fixes the issue, but I also haven't had any problems for a few days now.

Hoping a future update of Frigate resolves this if it is indeed related to not closing a connection.

blakeblackshear · 2021-03-22T12:36:42Z

blakeblackshear
Mar 22, 2021
Maintainer

This is almost certainly a result of switching to gevent to add websockets support. I would suggest downgrading to 0.8.1 if you are running into this issue.

6 replies

Minglarn Mar 31, 2021
Author

Any more errors since you changed to 0.8.1?

fl42 Mar 31, 2021

No, downgrading to 0.8.1 solved the issue

thundergreen · 2021-03-26T19:20:11Z

thundergreen
Mar 26, 2021

Ok then I got it ☺️ will maybe look into it thanks for the hint

0 replies

thundergreen · 2021-03-31T20:15:50Z

thundergreen
Mar 31, 2021

Ok.. downgraded to 0.8.1 and testing it's working

2 replies

Minglarn Apr 11, 2021
Author

It happens to often so be reliable at the moment.
What will I lost if I go back to 0.8.1?
Cant find any important info about this.

blakeblackshear Apr 11, 2021
Maintainer

https://github.com/blakeblackshear/frigate/releases

Uh oh!

Should I report this as a bug? #853

Uh oh!

Replies: 8 comments · 16 replies

Uh oh!

Uh oh!

Minglarn Feb 27, 2021 Author

Uh oh!

blakeblackshear Feb 27, 2021 Maintainer

Uh oh!

Uh oh!

Uh oh!

Minglarn Mar 2, 2021 Author

Uh oh!

Minglarn Mar 8, 2021 Author

Uh oh!

Minglarn Mar 17, 2021 Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

blakeblackshear Mar 22, 2021 Maintainer

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Minglarn Mar 31, 2021 Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Minglarn Apr 11, 2021 Author

Uh oh!

blakeblackshear Apr 11, 2021 Maintainer

Replies: 8 comments 16 replies

Minglarn
Feb 27, 2021
Author

blakeblackshear
Feb 27, 2021
Maintainer

Minglarn
Mar 2, 2021
Author

Minglarn
Mar 8, 2021
Author

Minglarn Mar 17, 2021
Author

blakeblackshear
Mar 22, 2021
Maintainer

Minglarn Mar 31, 2021
Author

Minglarn Apr 11, 2021
Author

blakeblackshear Apr 11, 2021
Maintainer