proper upgrade procedure

[az-z]

@alexandrevilain :
what should be a proper update/upgrade procedure fo the operator?

The operator was running without errors for a few weeks and earlier today i deleted the "frontend" pod. It got restarted but started to throw a ton of errors. I removed and reinstalled the operator. I did not remove the PG database that was installed originally.
Later today the "frontend" crashed again ( due to space pressure on a node) and after it restarted:
"...

[
Fx] PROVIDE    config.Persistence <= go.temporal.io/server/temporal.ApplyClusterMetadataConfigProvider()                                                                   │
│ [Fx] PROVIDE    *pprof.PProfInitializerImpl <= go.temporal.io/server/common/pprof.NewInitializer()                                                                          │
│ [Fx] PROVIDE    []trace.SpanExporter <= go.temporal.io/server/temporal.glob..func2()                                                                                        │
│ [Fx] SUPPLY    []temporal.ServerOption                                                                                                                                      │
│ [Fx] RUN    supply: stub([]temporal.ServerOption)                                                                                                                           │
│ [Fx] RUN    provide: go.temporal.io/server/temporal.ServerOptionsProvider()                                                                                                 │
│ [Fx] Error returned: received non-nil error from function "go.temporal.io/server/temporal".ServerOptionsProvider                                                            │
│     /home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:183:                                                                                              │
│ sql schema version compatibility check failed: unable to read DB schema version keyspace/database: temporal error: pq: relation "schema_version" does not exist             │
│ [Fx] ERROR        Failed to initialize custom logger: could not build arguments for function "go.uber.org/fx".(*module).constructCustomLogger.func2                         │
│     /home/runner/go/pkg/mod/go.uber.org/[email protected]/module.go:292:                                                                                                           │
│ failed to build fxevent.Logger:                                                                                                                                             │
│ could not build arguments for function "go.temporal.io/server/temporal".glob..func8                                                                                         │
│     /home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:1009:                                                                                             │
│ failed to build log.Logger:                                                                                                                                                 │
│ received non-nil error from function "go.temporal.io/server/temporal".ServerOptionsProvider                                                                                 │
│     /home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:183:                                                                                              │
│ sql schema version compatibility check failed: unable to read DB schema version keyspace/database: temporal error: pq: relation "schema_version" does not exist             │
│ Unable to create server. Error: could not build arguments for function "go.uber.org/fx".(*module).constructCustomLogger.func2 (/home/runner/go/pkg/mod/go.uber.org/[email protected] │
│ stream closed EOF for demo/prod-frontend-6d8b45759-v2hb8 (service)

"

Please advise.

[az-z]

it appears that the database lost all the tables:

psql -U temporal
psql (13.20 (Debian 13.20-1.pgdg120+1))
Type "help" for help.

select distinct(table_catalog) from information_schema.tables ;
 table_catalog 
---------------
 temporal
(1 row)

temporal-# \c temporal_visibility
You are now connected to database "temporal_visibility" as user "temporal".
temporal_visibility-# \l
                                      List of databases
        Name         |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges   
---------------------+----------+----------+------------+------------+-----------------------
 postgres            | temporal | UTF8     | en_US.utf8 | en_US.utf8 | 
 template0           | temporal | UTF8     | en_US.utf8 | en_US.utf8 | =c/temporal          +
                     |          |          |            |            | temporal=CTc/temporal
 template1           | temporal | UTF8     | en_US.utf8 | en_US.utf8 | =c/temporal          +
                     |          |          |            |            | temporal=CTc/temporal
 temporal            | temporal | UTF8     | en_US.utf8 | en_US.utf8 | 
 temporal_visibility | temporal | UTF8     | en_US.utf8 | en_US.utf8 | =Tc/temporal         +
                     |          |          |            |            | temporal=CTc/temporal
(5 rows)

temporal_visibility-# \dt
Did not find any relations.

How do i (re) init the database?

P.S. The current "fix" is to uninstall (delete) and install (apply) the temporal operator AND the DB. Only after that db objects get installed.

[alexandrevilain]

Hi!

Looks like this is not an operator bug but an issue on your database.

How did you managed to create the temporal database ? I hope you're not using example database manifests in the ./example directory, they are clearly not production ready as they use emptydir.

[az-z]

Alex,
this is a development (or as I call it "swamp") env.
And, you are correct, I ran the pg deployment from the examples directory.