Recorder still locking up on Purge even after update to 2025.2.4

[transmit-receive]

The problem

Recorder stops recording after purge action is called. CPU stays at 100% for hours.
Behaviour started two days ago on sunday 2025-02-16 around 4 am, when purge was called - maybe something to do with rebuild?
No version updates or changes in configuration around that time. The night before, everything was fine.

Maybe related to #117263 and #123348.

My Database is about 3 GB in size, i still have 70 GB of hard drive space left. Does not make a difference whether I choose to rebuild or not when calling purge.

I tried

sudo sqlite3 home-assistant_v2.db
sqlite3 > VACUUM main INTO "out.db"
and copied back the result
This shrinks my db by 100 MB and the new db works in home assistant, but does not cure the problem.

For the moment, i set auto_purge: false in configuration.yaml

@bdraco: You were extremely helpful for the above mentioned issues, any ideas for my incarnation of the issue?

What version of Home Assistant Core has the issue?

core-2025.2.4

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Container

Integration causing the issue

Recorder

Link to integration documentation on our website

https://www.home-assistant.io/integrations/recorder/#service-purge

Diagnostics information

No response

Example YAML snippet

Anything in the logs that might be useful for us?

Full log output with
logger:
  default: warning
  logs:
    homeassistant.components.recorder.purge: debug
after starting purge:

2025-02-18 23:27:27.471 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging states and events before target 2025-02-08 22:27:27+00:00
2025-02-18 23:27:27.471 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purge running in new format as there are NO states with event_id remaining
2025-02-18 23:27:27.524 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 4000 state ids and 696 attributes_ids to remove
2025-02-18 23:27:27.661 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated <sqlalchemy.engine.cursor.CursorResult object at 0x7fd25c6f1080> states to remove old_state_id
2025-02-18 23:27:27.723 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted <sqlalchemy.engine.cursor.CursorResult object at 0x7fd26acebbd0> states
2025-02-18 23:27:27.739 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 4000 state ids and 687 attributes_ids to remove
2025-02-18 23:27:27.835 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated <sqlalchemy.engine.cursor.CursorResult object at 0x7fd25c6f1080> states to remove old_state_id
2025-02-18 23:27:27.893 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted <sqlalchemy.engine.cursor.CursorResult object at 0x7fd26acebbd0> states
2025-02-18 23:27:27.910 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 4000 state ids and 715 attributes_ids to remove
2025-02-18 23:27:28.007 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated <sqlalchemy.engine.cursor.CursorResult object at 0x7fd25c6f1080> states to remove old_state_id
2025-02-18 23:27:28.063 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted <sqlalchemy.engine.cursor.CursorResult object at 0x7fd26acebbd0> states
2025-02-18 23:27:28.078 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 4000 state ids and 714 attributes_ids to remove
2025-02-18 23:27:28.177 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated <sqlalchemy.engine.cursor.CursorResult object at 0x7fd25c6f1080> states to remove old_state_id
2025-02-18 23:27:28.234 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted <sqlalchemy.engine.cursor.CursorResult object at 0x7fd26acebbd0> states
2025-02-18 23:27:28.250 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 4000 state ids and 671 attributes_ids to remove
2025-02-18 23:27:28.349 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated <sqlalchemy.engine.cursor.CursorResult object at 0x7fd25c6f1080> states to remove old_state_id
2025-02-18 23:27:28.406 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted <sqlalchemy.engine.cursor.CursorResult object at 0x7fd26acebbd0> states
2025-02-18 23:27:28.422 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 4000 state ids and 580 attributes_ids to remove
2025-02-18 23:27:28.522 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated <sqlalchemy.engine.cursor.CursorResult object at 0x7fd25c6f1080> states to remove old_state_id
2025-02-18 23:27:28.580 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted <sqlalchemy.engine.cursor.CursorResult object at 0x7fd26acebbd0> states
2025-02-18 23:27:28.595 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 4000 state ids and 478 attributes_ids to remove
2025-02-18 23:27:28.694 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated <sqlalchemy.engine.cursor.CursorResult object at 0x7fd25c6f1080> states to remove old_state_id
2025-02-18 23:27:28.752 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted <sqlalchemy.engine.cursor.CursorResult object at 0x7fd26acebbd0> states
2025-02-18 23:27:28.769 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 4000 state ids and 457 attributes_ids to remove
2025-02-18 23:27:28.870 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated <sqlalchemy.engine.cursor.CursorResult object at 0x7fd25c6f1080> states to remove old_state_id
2025-02-18 23:27:28.930 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted <sqlalchemy.engine.cursor.CursorResult object at 0x7fd26acebbd0> states
2025-02-18 23:27:28.946 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 4000 state ids and 450 attributes_ids to remove
2025-02-18 23:27:29.045 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated <sqlalchemy.engine.cursor.CursorResult object at 0x7fd25c6f1080> states to remove old_state_id
2025-02-18 23:27:29.105 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted <sqlalchemy.engine.cursor.CursorResult object at 0x7fd26acebbd0> states
2025-02-18 23:27:29.122 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 4000 state ids and 372 attributes_ids to remove
2025-02-18 23:27:29.220 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated <sqlalchemy.engine.cursor.CursorResult object at 0x7fd25c6f1080> states to remove old_state_id
2025-02-18 23:27:29.279 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted <sqlalchemy.engine.cursor.CursorResult object at 0x7fd26acebbd0> states
2025-02-18 23:27:29.295 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 4000 state ids and 399 attributes_ids to remove
2025-02-18 23:27:29.395 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated <sqlalchemy.engine.cursor.CursorResult object at 0x7fd260933d20> states to remove old_state_id
2025-02-18 23:27:29.456 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted <sqlalchemy.engine.cursor.CursorResult object at 0x7fd26acebbd0> states
2025-02-18 23:27:29.472 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 4000 state ids and 430 attributes_ids to remove
2025-02-18 23:27:29.572 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated <sqlalchemy.engine.cursor.CursorResult object at 0x7fd25c6f2cf0> states to remove old_state_id
2025-02-18 23:27:29.632 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted <sqlalchemy.engine.cursor.CursorResult object at 0x7fd2644df7e0> states
2025-02-18 23:27:29.649 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 4000 state ids and 476 attributes_ids to remove
2025-02-18 23:27:29.750 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated <sqlalchemy.engine.cursor.CursorResult object at 0x7fd2644dc050> states to remove old_state_id
2025-02-18 23:27:29.812 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted <sqlalchemy.engine.cursor.CursorResult object at 0x7fd2644df7e0> states
2025-02-18 23:27:29.828 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 4000 state ids and 457 attributes_ids to remove
2025-02-18 23:27:29.929 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated <sqlalchemy.engine.cursor.CursorResult object at 0x7fd2644df7e0> states to remove old_state_id
2025-02-18 23:27:29.988 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted <sqlalchemy.engine.cursor.CursorResult object at 0x7fd26acebbd0> states
2025-02-18 23:27:30.006 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 4000 state ids and 462 attributes_ids to remove
2025-02-18 23:27:30.105 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated <sqlalchemy.engine.cursor.CursorResult object at 0x7fd2644df7e0> states to remove old_state_id
2025-02-18 23:27:30.164 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted <sqlalchemy.engine.cursor.CursorResult object at 0x7fd26acebbd0> states
2025-02-18 23:27:30.180 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 4000 state ids and 435 attributes_ids to remove
2025-02-18 23:27:30.281 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated <sqlalchemy.engine.cursor.CursorResult object at 0x7fd2644df7e0> states to remove old_state_id
2025-02-18 23:27:30.341 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted <sqlalchemy.engine.cursor.CursorResult object at 0x7fd26acebbd0> states
2025-02-18 23:27:30.358 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 4000 state ids and 439 attributes_ids to remove
2025-02-18 23:27:30.460 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated <sqlalchemy.engine.cursor.CursorResult object at 0x7fd2644df7e0> states to remove old_state_id
2025-02-18 23:27:30.521 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted <sqlalchemy.engine.cursor.CursorResult object at 0x7fd27d753230> states
2025-02-18 23:27:30.539 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 4000 state ids and 507 attributes_ids to remove
2025-02-18 23:27:30.646 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated <sqlalchemy.engine.cursor.CursorResult object at 0x7fd27cb026d0> states to remove old_state_id
2025-02-18 23:27:30.707 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted <sqlalchemy.engine.cursor.CursorResult object at 0x7fd26642c1a0> states
2025-02-18 23:27:30.725 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 4000 state ids and 499 attributes_ids to remove
2025-02-18 23:27:30.828 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated <sqlalchemy.engine.cursor.CursorResult object at 0x7fd27cb026d0> states to remove old_state_id
2025-02-18 23:27:30.891 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted <sqlalchemy.engine.cursor.CursorResult object at 0x7fd26642c1a0> states
2025-02-18 23:27:30.912 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 4000 state ids and 493 attributes_ids to remove
2025-02-18 23:27:31.013 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated <sqlalchemy.engine.cursor.CursorResult object at 0x7fd27cb026d0> states to remove old_state_id
2025-02-18 23:27:31.074 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted <sqlalchemy.engine.cursor.CursorResult object at 0x7fd26642c1a0> states
2025-02-18 23:27:32.106 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 8996 shared attributes to remove
2025-02-18 23:27:32.356 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted <sqlalchemy.engine.cursor.CursorResult object at 0x7fd25c6f03d0> attribute states
2025-02-18 23:27:32.481 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted <sqlalchemy.engine.cursor.CursorResult object at 0x7fd25c6f16a0> attribute states
2025-02-18 23:27:32.515 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted <sqlalchemy.engine.cursor.CursorResult object at 0x7fd25c6f03d0> attribute states
2025-02-18 23:27:32.516 DEBUG (Recorder) [homeassistant.components.recorder.purge] After purging states and attributes_ids remaining=True
2025-02-18 23:27:32.537 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 4000 event ids and 1651 data_ids to remove

Additional information

No response

[home-assistant]

Hey there @home-assistant/core, mind taking a look at this issue as it has been labeled with an integration (recorder) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of recorder can trigger bot actions by commenting:

• @home-assistant close Closes the issue.
• @home-assistant rename Awesome new title Renames the issue.
• @home-assistant reopen Reopen the issue.
• @home-assistant unassign recorder Removes the current integration label and assignees on the issue, add the integration domain after the command.
• @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue.
• @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

---

recorder documentation
recorder source
_{^{(message by IssueLinks)}}

[bdraco]

If you send me a copy of the database, I'll load it up in a test system and try to purge it to empty to see if I can reproduce the issue

my gdrive is bdraco@gmail.com or my Dropbox is nick@koston.org

[transmit-receive]

Thanks, I shared the DB with your gmail account, I hope that worked.

[bdraco]

Looks like you have some corruption in the database file

% sqlite3 home-assistant_v2.db 
SQLite version 3.43.2 2023-10-10 13:08:14
Enter ".help" for usage hints.
sqlite> pragma integrity_check;
*** in database main ***
Tree 11 page 310656 cell 0: invalid page number 832838
Tree 11 page 310656 cell 395: Child page depth differs
Tree 39 page 778928 cell 142: invalid page number 833025
Tree 39 page 778928 cell 160: invalid page number 832848
Tree 39 page 735976 cell 84: invalid page number 833042
Tree 39 page 769519 cell 42: invalid page number 832866
Tree 39 page 431682 cell 140: invalid page number 832836
Tree 39 page 814757 cell 30: invalid page number 832786
Tree 39 page 425742 cell 100: invalid page number 832817
Tree 39 page 679502 cell 65: invalid page number 832950
Tree 39 page 422112 cell 126: invalid page number 832782
Tree 39 page 421122 cell 126: 2nd reference to page 421039
Tree 39 page 420132 cell 120: invalid page number 832806
Tree 39 page 683522 cell 27: invalid page number 832985
Tree 39 page 773122 cell 75: invalid page number 832889
Tree 39 page 679439 cell 26: invalid page number 832972
Tree 39 page 408996 cell 82: invalid page number 832914
Tree 39 page 689566 cell 134: invalid page number 832895
Tree 39 page 679514 cell 152: invalid page number 832843
Tree 39 page 777012 cell 67: invalid page number 832893
Tree 39 page 738097 cell 139: invalid page number 832809
Tree 39 page 754327 cell 132: invalid page number 832901
Tree 39 page 392656 cell 64: invalid page number 832933
Tree 39 page 763449 cell 65: invalid page number 833007
Tree 39 page 387840 cell 76: invalid page number 832885
Tree 39 page 743817 cell 82: invalid page number 832980
Tree 39 page 679468 cell 29: invalid page number 832935
Tree 37 page 832621 cell 0: invalid page number 833097
Tree 37 page 832621 cell 124: invalid page number 833089
Tree 37 page 832621 cell 123: invalid page number 833081
Tree 37 page 832621 cell 122: invalid page number 833073
Tree 37 page 832621 cell 121: invalid page number 833067
Tree 37 page 832621 cell 120: invalid page number 833060
Tree 37 page 832621 cell 119: invalid page number 833051
Tree 37 page 832621 cell 118: invalid page number 833040
Tree 37 page 832621 cell 117: invalid page number 833031
Tree 37 page 832621 cell 116: invalid page number 833020
Tree 37 page 832621 cell 115: invalid page number 833014
Tree 37 page 832621 cell 114: invalid page number 833004
Tree 37 page 832621 cell 113: invalid page number 832996
Tree 37 page 832621 cell 112: invalid page number 832988
Tree 37 page 832621 cell 111: invalid page number 832979
Tree 37 page 832621 cell 110: invalid page number 832966
Tree 37 page 832621 cell 109: invalid page number 832959
Tree 37 page 832621 cell 108: invalid page number 832954
Tree 37 page 832621 cell 107: invalid page number 832945
Tree 37 page 832621 cell 106: invalid page number 832939
Tree 37 page 832621 cell 105: invalid page number 832927
Tree 37 page 832621 cell 104: invalid page number 832920
Tree 37 page 832621 cell 103: invalid page number 832910
Tree 37 page 832621 cell 102: invalid page number 832903
Tree 37 page 832621 cell 101: invalid page number 832891
Tree 37 page 832621 cell 100: invalid page number 832883
Tree 37 page 832621 cell 99: invalid page number 832876
Tree 37 page 832621 cell 98: invalid page number 832870
Tree 37 page 832621 cell 97: invalid page number 832862
Tree 37 page 832621 cell 96: invalid page number 832851
Tree 37 page 832621 cell 95: invalid page number 832841
Tree 37 page 832621 cell 94: invalid page number 832832
Tree 37 page 832621 cell 93: invalid page number 832825
Tree 37 page 832621 cell 92: invalid page number 832815
Tree 37 page 832621 cell 91: invalid page number 832807
Tree 37 page 832621 cell 90: invalid page number 832798
Tree 37 page 832621 cell 89: invalid page number 832792
Tree 37 page 832621 cell 88: invalid page number 832783
Tree 29 page 814026 cell 0: invalid page number 833091
Tree 29 page 814026 cell 204: invalid page number 832929
Tree 29 page 814026 cell 88: invalid page number 832905
Tree 29 page 774600 cell 103: invalid page number 833059
Tree 29 page 774600 cell 102: invalid page number 832791
Tree 29 page 546374 cell 31: invalid page number 832941
Tree 29 page 542853 cell 203: invalid page number 832913
Tree 29 page 806969 cell 39: invalid page number 832899
Tree 29 page 679458 cell 147: invalid page number 832799
Tree 29 page 536561 cell 85: invalid page number 832940
Tree 29 page 534383 cell 157: invalid page number 832842
Tree 29 page 532689 cell 192: invalid page number 832864
Tree 29 page 531479 cell 77: invalid page number 832816
Tree 29 page 529543 cell 95: invalid page number 832928
Tree 29 page 819335 cell 95: invalid page number 832922
Tree 28 page 832526 cell 0: invalid page number 833087
Tree 28 page 832526 cell 170: invalid page number 833068
Tree 28 page 832526 cell 169: invalid page number 833046
Tree 28 page 832526 cell 168: invalid page number 833018
Tree 28 page 832526 cell 167: invalid page number 832998
Tree 28 page 832526 cell 166: invalid page number 832974
Tree 28 page 832526 cell 165: invalid page number 832955
Tree 28 page 832526 cell 164: invalid page number 832937
Tree 28 page 832526 cell 163: invalid page number 832911
Tree 28 page 832526 cell 162: invalid page number 832887
Tree 28 page 832526 cell 161: invalid page number 832869
Tree 28 page 832526 cell 160: invalid page number 832845
Tree 28 page 832526 cell 159: invalid page number 832822
Tree 28 page 832526 cell 158: invalid page number 832800
Tree 28 page 832526 cell 157: invalid page number 832778
Tree 27 page 832446 cell 0: invalid page number 833092
Tree 27 page 832446 cell 144: invalid page number 833079
Tree 27 page 832446 cell 143: invalid page number 833070
Tree 27 page 832446 cell 142: invalid page number 833057
Tree 27 page 832446 cell 141: invalid page number 833045
sqlite> 

[bdraco]

To repair:

Stop Home Assistant

Backup home-assistant_v2.db

Run sqlite3 home-assistant_v2.db .recover | sqlite3 home-assistant_v2_repaired.db

Copy the repaired file home-assistant_v2_repaired.db over home-assistant_v2.db
using cp -f home-assistant_v2_repaired.db home-assistant_v2.db

Restart Home Assistant

[bdraco]

I've done this locally and will now try to purge the db to see if there is another issue

[transmit-receive]

Thanks for the patience and work you're putting in.

I'm a bit at loss, I checked both the version i uploaded and the one that I kept running in Home Assistant for the last three days, both came back clean:

SQLite version 3.37.2 2022-01-06 13:25:41
Enter ".help" for usage hints.
sqlite> pragma integrity_check;
ok
sqlite>

Are we both talking about the same db?

[bdraco]

Definitely the mkandroiddummy db, original size 3412414464 bytes.

I'm checking with SQLite version 3.43.2 so it might be a bit better at discovering issues

However even after repairing I can still hit the 100% cpu issue while deleting events

[bdraco]

Its stuck on this query

2025-02-21 13:27:34.956 INFO (Recorder) [sqlalchemy.engine.Engine] DELETE FROM events WHERE events.event_id IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
2025-02-21 13:27:34.956 INFO (Recorder) [sqlalchemy.engine.Engine] [generated in 0.00095s] (344064, 344065, 344066, 344067, 344068, 344069, 344070, 344071, 344072, 344073, 344074, 344075, 344076, 344077, 344078, 344079, 344080, 344081, 344082, 344083, 344084, 344085, 344086, 344087, 344088, 344089, 344090, 344091, 344092, 344093, 344094, 344095, 344096, 344097, 344098, 344099, 344100, 344101, 344102, 344103, 344104, 344105, 344106, 344107, 344108, 344109, 344110, 344111, 344112, 344113 ... 3900 parameters truncated ... 344014, 344015, 344016, 344017, 344018, 344019, 344020, 344021, 344022, 344023, 344024, 344025, 344026, 344027, 344028, 344029, 344030, 344031, 344032, 344033, 344034, 344035, 344036, 344037, 344038, 344039, 344040, 344041, 344042, 344043, 344044, 344045, 344046, 344047, 344048, 344049, 344050, 344051, 344052, 344053, 344054, 344055, 344056, 344057, 344058, 344059, 344060, 344061, 344062, 344063)

[transmit-receive]

Correct, i bumped sqlite to 3.45.1 and now I'm also getting the invalid pages - even if that seems not to be the core issue.

[bdraco]

sqlite> delete from events where event_id in (340612, 340613, 340614, 340615, 340616, 340617, 340618, 340619, 340620, 340621, 340622, 340623, 340624, 340625, 340626, 340627, 340628, 340629, 340630, 340631, 340632, 340633, 340634, 340635, 340636, 340637, 340638, 340639, 340640, 340641, 340642, 340643, 340644, 340645, 340646, 340647, 340648, 340649, 340650, 340651, 340652, 340653, 340654, 340655, 340656, 340657, 340658, 340659, 340660, 340661);

This works fine on the CLI but seems to hang when executed from sqlalchemy

[bdraco]

If I drop it down to max 10 per delete it does work

but its still very very slow

2025-02-21 13:36:14.683 INFO (Recorder) [sqlalchemy.engine.Engine] [cached since 36.41s ago] (1740166538.138723, 10, 0)
2025-02-21 13:36:14.683 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 10 event ids and 2 data_ids to remove
2025-02-21 13:36:14.683 INFO (Recorder) [sqlalchemy.engine.Engine] DELETE FROM events WHERE events.event_id IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
2025-02-21 13:36:14.683 INFO (Recorder) [sqlalchemy.engine.Engine] [cached since 36.41s ago] (340722, 340723, 340724, 340725, 340726, 340727, 340728, 340729, 340730, 340731)
2025-02-21 13:36:15.592 DEBUG (MainThread) [snitun.multiplexer.core] Receive pong from peer / reset healthy
2025-02-21 13:36:20.828 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted <sqlalchemy.engine.cursor.CursorResult object at 0x341f73460> events
2025-02-21 13:36:20.828 INFO (Recorder) [sqlalchemy.engine.Engine] SELECT events.event_id, events.data_id 
FROM events 
WHERE events.time_fired_ts < ?
 LIMIT ? OFFSET ?
2025-02-21 13:36:20.828 INFO (Recorder) [sqlalchemy.engine.Engine] [cached since 42.56s ago] (1740166538.138723, 10, 0)
2025-02-21 13:36:20.828 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 10 event ids and 4 data_ids to remove
2025-02-21 13:36:20.829 INFO (Recorder) [sqlalchemy.engine.Engine] DELETE FROM events WHERE events.event_id IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
2025-02-21 13:36:20.829 INFO (Recorder) [sqlalchemy.engine.Engine] [cached since 42.56s ago] (340736, 340737, 340738, 340739, 340740, 340741, 340732, 340733, 340734, 340735)

[bdraco]

If I turn off foreign_keys checks they delete instantly.

So likely a schema problem. How did you create this database?

diff --git a/homeassistant/components/recorder/const.py b/homeassistant/components/recorder/const.py
index c91845e8436..7b4bd89bf55 100644
--- a/homeassistant/components/recorder/const.py
+++ b/homeassistant/components/recorder/const.py
@@ -32,7 +32,7 @@ MIN_AVAILABLE_MEMORY_FOR_QUEUE_BACKLOG = 256 * 1024**2
 
 # The maximum number of rows (events) we purge in one delete statement
 
-DEFAULT_MAX_BIND_VARS = 4000
+DEFAULT_MAX_BIND_VARS = 10
 
 DB_WORKER_PREFIX = "DbWorker"
 
diff --git a/homeassistant/components/recorder/util.py b/homeassistant/components/recorder/util.py
index a686c7c6498..7ee98110b5c 100644
--- a/homeassistant/components/recorder/util.py
+++ b/homeassistant/components/recorder/util.py
@@ -497,7 +497,7 @@ def setup_connection_for_dialect(
         execute_on_connection(dbapi_connection, f"PRAGMA synchronous={synchronous}")
 
         # enable support for foreign keys
-        execute_on_connection(dbapi_connection, "PRAGMA foreign_keys=ON")
+        #execute_on_connection(dbapi_connection, "PRAGMA foreign_keys=ON")
 
     elif dialect_name == SupportedDialect.MYSQL:
         execute_on_connection(dbapi_connection, "SET session wait_timeout=28800")

[transmit-receive]

This db was created by Home Assistant, I never touched it manually afaik.