Skip to content

Understanding why a new generation is created after restoration #627

Open
@aallrd

Description

@aallrd

I am using litestream to replicate the local database of my Python application running in a container to an SFTP server.

/etc/litestream.yml

dbs:
  - path: database.sqlite3
    replicas:
      - type:     sftp
        host:     sftp.fr.company.com
        user:     XXX
        password: XXX
        path:     /backup/app/${LITESTREAM_REPLICA_PATH}
        retention: 168h # 7 days
        snapshot-interval: 24h
        # The 33m is to prevent snapshot and validations from happening concurrently.
        # https://github.com/benbjohnson/litestream/issues/253
        validation-interval: 12h33m

Here is my application service startup log:

+ litestream restore -if-replica-exists database.sqlite3
time=2025-01-13T18:57:39.516+01:00 level=INFO msg="restoring snapshot" db=/home/app/database.sqlite3 replica=sftp generation=9d7f0b28cf1c6440 index=0 path=/home/app/database.sqlite3.tmp
time=2025-01-13T18:57:39.524+01:00 level=INFO msg="restoring wal files" db=/home/app/database.sqlite3 replica=sftp generation=9d7f0b28cf1c6440 index_min=0 index_max=0
time=2025-01-13T18:57:39.528+01:00 level=INFO msg="downloaded wal" db=/home/app/database.sqlite3 replica=sftp generation=9d7f0b28cf1c6440 index=0 elapsed=4.470316ms
time=2025-01-13T18:57:39.530+01:00 level=INFO msg="applied wal" db=/home/app/database.sqlite3 replica=sftp generation=9d7f0b28cf1c6440 index=0 elapsed=2.073276ms
time=2025-01-13T18:57:39.530+01:00 level=INFO msg="renaming database from temporary location" db=/home/app/database.sqlite3 replica=sftp
+ alembic upgrade head
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [sqlalchemy.engine.Engine] BEGIN (implicit)
2025-01-13 18:57:41,348 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2025-01-13 18:57:41,348 INFO sqlalchemy.engine.Engine PRAGMA main.table_info("alembic_version")
2025-01-13 18:57:41,348 INFO sqlalchemy.engine.Engine [raw sql] ()
INFO  [sqlalchemy.engine.Engine] PRAGMA main.table_info("alembic_version")
INFO  [sqlalchemy.engine.Engine] [raw sql] ()
2025-01-13 18:57:41,349 INFO sqlalchemy.engine.Engine SELECT alembic_version.version_num 
FROM alembic_version
INFO  [sqlalchemy.engine.Engine] SELECT alembic_version.version_num 
FROM alembic_version
2025-01-13 18:57:41,349 INFO sqlalchemy.engine.Engine [generated in 0.00015s] ()
INFO  [sqlalchemy.engine.Engine] [generated in 0.00015s] ()
2025-01-13 18:57:41,354 INFO sqlalchemy.engine.Engine ROLLBACK
INFO  [sqlalchemy.engine.Engine] ROLLBACK
+ exec litestream replicate -exec ./run.py
time=2025-01-13T18:57:41.481+01:00 level=INFO msg=litestream version=v0.3.13
time=2025-01-13T18:57:41.482+01:00 level=INFO msg="initialized db" path=/home/app/database.sqlite3
time=2025-01-13T18:57:41.482+01:00 level=INFO msg="replicating to" name=sftp type=sftp sync-interval=1s host=sftp.fr.company.com user=XXX path=/backup/app/staging/demo
time=2025-01-13T18:57:42.485+01:00 level=INFO msg="sync: new generation" db=/home/app/database.sqlite3 generation=e82baa425de42b74 reason="no generation exists"
time=2025-01-13T18:57:42.501+01:00 level=INFO msg="write snapshot" db=/home/app/database.sqlite3 replica=sftp position=e82baa425de42b74/00000000:4152
time=2025-01-13T18:57:42.509+01:00 level=INFO msg="snapshot written" db=/home/app/database.sqlite3 replica=sftp position=e82baa425de42b74/00000000:4152 elapsed=7.576832ms sz=2840
time=2025-01-13T18:57:42.511+01:00 level=INFO msg="write wal segment" db=/home/app/database.sqlite3 replica=sftp position=e82baa425de42b74/00000000:0
time=2025-01-13T18:57:42.517+01:00 level=INFO msg="wal segment written" db=/home/app/database.sqlite3 replica=sftp position=e82baa425de42b74/00000000:0 elapsed=5.298369ms sz=4152

Litestream successfully restores the database with generation 9d7f0b28cf1c6440, but then it creates a new generation e82baa425de42b74 when the replication process starts (reason="no generation exists").

Since the migration process between the restore / replicate steps is not doing anything here, I was expecting the same initial generation 9d7f0b28cf1c6440 to be used as the database file is the same.

Is my understanding correct ?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions