Relations public.Withdrawal and public.Epoch does not exist

[mkocur-ledger]

Summary

Hello!

I have restored a snapshot with db-sync:
(https://update-cardano-mainnet.iohk.io/cardano-db-sync/13.5/db-sync-snapshot-schema-13.5-block-10823207-x86_64.tgz)

[db-sync-node:Info:83] [2024-09-19 00:51:47.84 UTC] Reached EpochNo 508
[db-sync-node:Info:92] [2024-09-19 00:56:57.60 UTC] Asynchronously wrote a ledger snapshot to /config/state/134092758-670ca68c3d-507.lstate in 323.408766108s.
[db-sync-node:Info:83] [2024-09-19 01:02:12.81 UTC] Removing old epoch boundary file /config/state/130636777-c1ecf5af89-499.lstate
[db-sync-node:Info:83] [2024-09-19 01:02:28.43 UTC] Reached EpochNo 509
[db-sync-node:Info:83] [2024-09-19 01:02:30.04 UTC] Received block which is not in the db with HeaderFields {headerFieldSlot = SlotNo 134525754, headerFieldBlockNo = BlockNo 10823208, headerFieldHash = 1fbfe5fd0f3e99d6e1f4ccffd8ad7202e36e57c18cf8863bf6aec57eb1062db9}. Time to restore consistency.
[db-sync-node:Info:83] [2024-09-19 01:02:30.05 UTC] Starting at epoch 509
[db-sync-node:Info:83] [2024-09-19 01:02:32.80 UTC] Insert Conway Block: epoch 509, slot 134525754, block 10823208, hash 1fbfe5fd0f3e99d6e1f4ccffd8ad7202e36e57c18cf8863bf6aec57eb1062db9
[db-sync-node:Info:83] [2024-09-19 01:02:32.85 UTC] Setting ConsistencyLevel to Consistent
[db-sync-node:Info:92] [2024-09-19 01:05:35.24 UTC] Asynchronously wrote a ledger snapshot to /config/state/134524753-29011cc132-508.lstate in 202.425990791s.
[db-sync-node:Info:83] [2024-09-19 01:08:29.04 UTC] Insert Conway Block: epoch 509, slot 134563412, block 10825000, hash e6f71ef9d62932dd429b61051e63fd97929b605f9c33612817cc5205eb6b4ca2
[db-sync-node:Info:83] [2024-09-19 01:23:30.51 UTC] Inserted 1335280 EpochStake for EpochNo 510
[db-sync-node:Info:83] [2024-09-19 01:31:00.87 UTC] Insert Conway Block: epoch 509, slot 134664823, block 10830000, hash ec59c8053d5f30299ebbfb272083ead1e968f4aab60ab868203a9c8b93cd7201
[db-sync-node:Info:83] [2024-09-19 01:31:00.87 UTC] Voting Anchor Offchain metadata fetch: 0 results, 1000 fetch errors
[db-sync-node:Info:83] [2024-09-19 01:31:01.69 UTC] Pool Offchain metadata fetch: 0 results, 1000 fetch errors
[db-sync-node:Info:83] [2024-09-19 01:41:25.76 UTC] Insert Conway Block: epoch 509, slot 134767193, block 10835000, hash 11cace016758192b7459071aade17f575b305ebf374f6970c78f4ae936bead5e
[db-sync-node:Warning:83] [2024-09-19 01:47:59.59 UTC] prepareTxMetadata: Column 'json' in table 'metadata' was recorded as null, due to a Unicode NUL character found when trying to parse the json.
[db-sync-node:Warning:83] [2024-09-19 01:47:59.79 UTC] prepareTxMetadata: Column 'json' in table 'metadata' was recorded as null, due to a Unicode NUL character found when trying to parse the json.
[db-sync-node:Info:83] [2024-09-19 01:49:47.05 UTC] Insert Conway Block: epoch 509, slot 134867805, block 10840000, hash 4a858262fe88e24fabffd4f7e2b4ef6b51699cbd3aea350e63f748f98f78d175

hasura/graphql-engine:v2.43.0.cli-migrations-v2 is reporting public.Withdrawal and public.Epoch relations missing:

{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.10.97","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database q
uery error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \
"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM  (SELECT  row_
to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.base\".\"number\" AS \"number\"       ) AS \"_e\"      ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM  (SELECT  *  FROM \"public\".\"
Epoch\"  WHERE ('true')   ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\"    ORDER BY \"root.pg.number\" DESC NULLS FIRST  ) AS \"_root\"      "},"path":"$"},"query":{"type":null},"query_
execution_time":3.047279e-3,"request_id":"fc74d816-c063-47d9-ae71-cf1ac17e8c2b","request_mode":"error","request_read_time":6.125e-6,"response_size":826,"uncompressed_response_size":826,"user_vars":{"x-has
ura-role":"cardano-graphql"}},"request_id":"fc74d816-c063-47d9-ae71-cf1ac17e8c2b"},"level":"error","span_id":"db6b71dd35db4c48","timestamp":"2024-09-19T06:25:06.376+0000","trace_id":"4753ed8ad279d7b47c8bf
4e4ac8a7194","type":"http-log"}

{"timestamp":"2024-09-19T06:19:34.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"env var HASURA_GRAPHQL_METADATA_DIR is not set, defaulting to /hasura-metadata"}}
{"timestamp":"2024-09-19T06:19:34.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-apply","info":"directory /hasura-migrations does not exist, skipping migrations"}}
{"timestamp":"2024-09-19T06:19:34.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-apply","info":"directory /hasura-metadata does not exist, skipping metadata"}}
{"timestamp":"2024-09-19T06:19:34.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-shutdown","info":"killing temporary server"}}
{"timestamp":"2024-09-19T06:19:34.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-shutdown","info":"graphql-engine will now start in normal mode"}}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.10.97","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database q
uery error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \
"public.Withdrawal\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  json_build_object('aggregate', json_build_object('sum', json_build_object('amount', sum(\"_root\".\"root.pg
.amount\" ) ) ) ) AS \"root\" FROM  (SELECT  \"_root.base\".\"amount\" AS \"root.pg.amount\" FROM  (SELECT  *  FROM \"public\".\"Withdrawal\"  WHERE ('true')     ) AS \"_root.base\"      ) AS \"_root\"
   "},"path":"$"},"query":{"type":null},"query_execution_time":265.310693138,"request_id":"06d7e070-be6b-4a1a-98a5-5f10d6330822","request_mode":"error","request_read_time":5.408e-6,"response_size":669,"un
compressed_response_size":669,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"06d7e070-be6b-4a1a-98a5-5f10d6330822"},"level":"error","span_id":"4c832a518ef4b722","timestamp":"2024-09-19T06:
25:05.375+0000","trace_id":"fdb208a2ea0746a898ea78d82148a64d","type":"http-log"}

The sync status:

curl \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"query": "{ cardanoDbMeta { initialized syncPercentage }}"}' \
  http://localhost:3100/graphql
{"data":{"cardanoDbMeta":{"initialized":true,"syncPercentage":100}},"extensions":{"tracing":{"version":1,"startTime":"2024-09-19T06:49:12.102Z","endTime":"2024-09-19T06:49:12.115Z","duration":13500570,"execution":{"resolvers":[{"path":["cardanoDbMeta"],"parentType":"Query","fieldName":"cardanoDbMeta","returnType":"CardanoDbMeta!","startOffset":502588,"duration":12826503},{"path":["cardanoDbMeta","initialized"],"parentType":"CardanoDbMeta","fieldName":"initialized","returnType":"Boolean!","startOffset":13396134,"duration":15571},{"path":["cardanoDbMeta","syncPercentage"],"parentType":"CardanoDbMeta","fieldName":"syncPercentage","returnType":"Percentage!","startOffset":13420908,"duration":5172}]}}}}
 curl \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"query": "{ cardano { tip { number slotNo epoch { number } } } }"}' http://localhost:3100/graphql
{"data":{"cardano":{"tip":{"number":10854265,"slotNo":"135162299","epoch":{"number":510}}}},"extensions":{"tracing":{"version":1,"startTime":"2024-09-19T06:49:55.759Z","endTime":"2024-09-19T06:49:55.779Z","duration":19672084,"execution":{"resolvers":[{"path":["cardano"],"parentType":"Query","fieldName":"cardano","returnType":"Cardano!","startOffset":2899698,"duration":16592105},{"path":["cardano","tip"],"parentType":"Cardano","fieldName":"tip","returnType":"Block!","startOffset":19533689,"duration":15467},{"path":["cardano","tip","number"],"parentType":"Block","fieldName":"number","returnType":"Int","startOffset":19563055,"duration":9800},{"path":["cardano","tip","slotNo"],"parentType":"Block","fieldName":"slotNo","returnType":"String","startOffset":19582450,"duration":3407},{"path":["cardano","tip","epoch"],"parentType":"Block","fieldName":"epoch","returnType":"Epoch","startOffset":19597505,"duration":5842},{"path":["cardano","tip","epoch","number"],"parentType":"Epoch","fieldName":"number","returnType":"Int!","startOffset":19617981,"duration":4217}]}}}}

I have restarted graphql, tmr, hasura, background components after the sync was completed to face the same issue.

db-sync: "13.5.0.2"
node: "9.1.1"
ogmios-server: "6.6.2"
graphql-engine: "v2.43.0.cli-migrations-v2"
graphql-server: "8.2.2"
graphql-background: "8.2.2-mainnet"
cf-token-metadata-registry-api: "1.0.2"
postgres: 15.6

Steps to reproduce the bug

No response

Actual Result

Some db data missing after snapshot restore.

Expected Result

No errors

Environment

Kubernetes
{
os: {
arch: 'x64',
cpus: { count: 48, model: 'Intel(R) Xeon(R) Gold 6312U CPU @ 2.40GHz' },
release: '5.15.0-117-generic',
totalMem: 269877899264,
type: 'Linux'
},
version: '8.0.0'
}

Platform

• Linux (Ubuntu)
• Linux (Other)
• macOS
• Windows

Platform version

Ubuntu 22.04.4 LTS

Runtime

• Node.js
• Docker

Runtime version

No response

[Kartiiyer12]

Hi @mkocur-ledger I think the issue for the Epoch and withdrawal table missing could be issue with the hasura migration and not the db-sync-snapshot. Can you try to see if there are errors on the graphql-background & hasura while applying the migrations?

For the token metadata registry server, are you running the default container as it is from the repo? Table missing seems improbable.

[mkocur-ledger]

Hi @Kartiiyer12, thanks for your interest in my bugreport.
About metadata, I have edit my report and removed it. It was fixed here: cardano-foundation/cf-token-metadata-registry#19

About the migration, I didn't create any of my own. These are related log messages:

{"timestamp":"2024-09-24T06:38:27.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"database url for migrations is set by HASURA_GRAPHQL_D
ATABASE_URL"}}
{"timestamp":"2024-09-24T06:38:27.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"migrations server port env var is not set, defaulting
to 9691"}}
{"timestamp":"2024-09-24T06:38:27.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"server timeout is not set, defaulting to 30 seconds"}}
{"timestamp":"2024-09-24T06:38:27.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"starting graphql engine temporarily on port 9691"}}
{"timestamp":"2024-09-24T06:38:27.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"waiting 30 for 9691 to be ready"}}
{"detail":{"info":"Already at the latest catalog version (48); nothing to do.","kind":"catalog_migrate"},"level":"info","timestamp":"2024-09-24T06:38:27.784+0000","type":"startup"}
{"detail":{"info":"Schema sync enabled. Polling at Refined (Milliseconds {milliseconds = 1s})","kind":"schema-sync"},"level":"info","timestamp":"2024-09-24T06:38:27.784+0000","type":"startup"}
{"detail":{"info":{"instance_id":"4d2b28e4-c896-465b-88f0-3120b600e68c","message":"listener thread started","thread_id":"ThreadId 72"},"kind":"schema-sync"},"level":"info","timestamp":"2024-09-24T06:38:27.784+0000","type":"startup"}
{"detail":{"info":"EE client credentials not present in the metadata database. Hasura EE features are disabled.","kind":"license_info"},"level":"warn","timestamp":"2024-09-24T06:38:27.784+0000","type":"startup"}
{"timestamp":"2024-09-24T06:38:29.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"env var HASURA_GRAPHQL_MIGRATIONS_DIR is not set, defaulting to /hasura-migrations"}}
{"timestamp":"2024-09-24T06:38:29.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"env var HASURA_GRAPHQL_METADATA_DIR is not set, defaulting to /hasura-metadata"}}
{"timestamp":"2024-09-24T06:38:29.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-apply","info":"directory /hasura-migrations does not exist, skipping migrations"}}
{"timestamp":"2024-09-24T06:38:29.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-apply","info":"directory /hasura-metadata does not exist, skipping metadata"}}
{"timestamp":"2024-09-24T06:38:29.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-shutdown","info":"killing temporary server"}}
{"timestamp":"2024-09-24T06:38:29.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-shutdown","info":"graphql-engine will now start in normal mode"}}

[Kartiiyer12]

These are migration folder that should be applied - https://github.com/cardano-foundation/cardano-graphql/tree/16ed6d71dbe95ff9e7d959ea51a8476da668684c/packages/api-cardano-db-hasura/hasura/project/migrations/default/1589369664961_init.
This happens from the background-graphql side. Are you running everything as it is from the repo or are there any customizations?

[mkocur-ledger]

I have made a fresh installation and after applying this: https://github.com/cardano-foundation/cf-token-metadata-registry/blob/47f6cf08515f45c3dc11968a7e15a791875cf59c/api/src/main/resources/db/migration/postgresql/V0__metadata_server_db_init.sql

I see no errors. Created an issue that it is not applied automatically:
cardano-foundation/cf-token-metadata-registry#19

About your question, there are customizations, yes, but I can see messages in background pod that it is actually inserting assets and the assets were synced in graphql to 100%, so I assume it works.

[Kammerlo]

I'll check the issue in the token-registry next week and fix this. Thanks for finding it! And I'm glad it worked for you.
I close this issue since it's probably a topic for the Token Registry. If there is anything else, let us know.

[mkocur-ledger]

Actually @Kammerlo please reopen this. I have disabled INFO messages and I found those errors again (I have applied the sql file you mentioned on my database but it didn't help):

"relation \"public.Epoch\" does not exist"
relation \"public.Withdrawal\" does not exist"
{"error":{"code":"validation-failed","error":"no mutations exist","path":"$"}
{"code":"validation-failed","error":"field 'epochs' not found in type: 'query_root'","path":"$.selectionSet.epochs"}

Full log:

{"detail":{"info":"EE client credentials not present in the metadata database. Hasura EE features are disabled.","kind":"license_info"},"level":
"warn","timestamp":"2024-10-01T08:14:36.932+0000","type":"startup"}                                                                             {"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"
operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not
 exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM  (SELECT  row_to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.base\".\"number\" AS \"number\"       ) AS \"_e\"      ) ) A
S \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM  (SELECT  *  FROM \"public\".\"Epoch\"  WHERE ('true')   ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\"    ORDER BY \"root.pg.number\" DESC NULLS FIRST  ) AS \"_root\"      "},"path":"$"},"query":{"type":n
ull},"query_execution_time":59.342454474,"request_id":"858e9742-dc65-4e28-8906-a6259378869e","request_mode":"error","request_read_time":2.06e-6,"response_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"858e9742-dc65-4e28-8906-a625
9378869e"},"level":"error","span_id":"1856f494e02ce310","timestamp":"2024-10-01T12:55:28.255+0000","trace_id":"f070f1320c096d4104734fc7e6db5fa0","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"ca
rdano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Withdrawal\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  json_build_object('aggregate', json_build_object('sum', json_build_obje
ct('amount', sum(\"_root\".\"root.pg.amount\" ) ) ) ) AS \"root\" FROM  (SELECT  \"_root.base\".\"amount\" AS \"root.pg.amount\" FROM  (SELECT  *  FROM \"public\".\"Withdrawal\"  WHERE ('true')     ) AS \"_root.base\"      ) AS \"_root\"      "},"path":"$"},"query":{"type":null},"query_e
xecution_time":65.628226835,"request_id":"ffdf5de9-e978-4c5c-9cb6-c41b2d4dc766","request_mode":"error","request_read_time":1.54e-6,"response_size":669,"uncompressed_response_size":669,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"ffdf5de9-e978-4c5c-9cb6-c41b2d4dc766"},"l
evel":"error","span_id":"93a4c54f8be46366","timestamp":"2024-10-01T12:55:28.255+0000","trace_id":"0a44ad3329a1255d3af62717187eee0a","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"
operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not
 exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM  (SELECT  row_to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.base\".\"number\" AS \"number\"       ) AS \"_e\"      ) ) A
S \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM  (SELECT  *  FROM \"public\".\"Epoch\"  WHERE ('true')   ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\"    ORDER BY \"root.pg.number\" DESC NULLS FIRST  ) AS \"_root\"      "},"path":"$"},"query":{"type":n
ull},"query_execution_time":1.759961e-3,"request_id":"1ddc327d-36dc-48e7-b3d1-4e3fedb441c0","request_mode":"error","request_read_time":2.38e-6,"response_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"1ddc327d-36dc-48e7-b3d1-4e3fe
db441c0"},"level":"error","span_id":"415b0c75b73d3f9b","timestamp":"2024-10-01T12:55:28.255+0000","trace_id":"7221d21491e389cadda32b981deeddc9","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"ca
rdano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[
]' ) AS \"root\" FROM  (SELECT  row_to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.base\".\"number\" AS \"number\"       ) AS \"_e\"      ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM  (SELECT  *  FROM \"public\".\"Epoch\"  WHERE ('true')   ORDER BY \"number\" DE
SC NULLS FIRST LIMIT 1 ) AS \"_root.base\"    ORDER BY \"root.pg.number\" DESC NULLS FIRST  ) AS \"_root\"      "},"path":"$"},"query":{"type":null},"query_execution_time":1.466746e-3,"request_id":"f6f2695f-647f-4ac8-b8c4-8174d98d7b67","request_mode":"error","request_read_time":8.1e-7,"r
esponse_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"f6f2695f-647f-4ac8-b8c4-8174d98d7b67"},"level":"error","span_id":"f0fc2276bc19e1ad","timestamp":"2024-10-01T12:55:29.256+0000","trace_id":"f13c31a79f12fd3a95fea03cca50d56f","
type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"ca
rdano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[
]' ) AS \"root\" FROM  (SELECT  row_to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.base\".\"number\" AS \"number\"       ) AS \"_e\"      ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM  (SELECT  *  FROM \"public\".\"Epoch\"  WHERE ('true')   ORDER BY \"number\" DE
SC NULLS FIRST LIMIT 1 ) AS \"_root.base\"    ORDER BY \"root.pg.number\" DESC NULLS FIRST  ) AS \"_root\"      "},"path":"$"},"query":{"type":null},"query_execution_time":1.23195e-3,"request_id":"dc69b318-34f9-4282-becb-0e487414eac3","request_mode":"error","request_read_time":1.01e-6,"r
esponse_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"dc69b318-34f9-4282-becb-0e487414eac3"},"level":"error","span_id":"83934b4c0740d43f","timestamp":"2024-10-01T12:55:29.256+0000","trace_id":"d4b99c418de9e6fcca17a14586011e69","
type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"ca
rdano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[
]' ) AS \"root\" FROM  (SELECT  row_to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.or.adaPots\".\"adaPots\" AS \"adaPots\"       ) AS \"_e\"      ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM  (SELECT  *  FROM \"public\".\"Epoch\"  WHERE ('true')   ORDER BY \"num
ber\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\" LEFT OUTER JOIN LATERAL (SELECT  row_to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.or.adaPots.base\".\"reserves\" AS \"reserves\"       ) AS \"_e\"      ) ) AS \"adaPots\" FROM  (SELECT  *  FROM \"public\".\"ada_pots\"  WHERE ((\"_root
.base\".\"number\") = (\"epoch_no\"))    LIMIT 1 ) AS \"_root.or.adaPots.base\"      ) AS \"_root.or.adaPots\" ON ('true')    ORDER BY \"root.pg.number\" DESC NULLS FIRST  ) AS \"_root\"      "},"path":"$"},"query":{"type":null},"query_execution_time":1.229809e-3,"request_id":"5f82fa48-a
433-4b54-af2a-ef488244b533","request_mode":"error","request_read_time":1.07e-6,"response_size":1194,"uncompressed_response_size":1194,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"5f82fa48-a433-4b54-af2a-ef488244b533"},"level":"error","span_id":"ca4807fd288b05b5","timest
amp":"2024-10-01T12:55:28.255+0000","trace_id":"33c6aa0f5dda79245bc95b61916ef6e5","type":"http-log"}                                            {"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"
operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not
 exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM  (SELECT  row_to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.base\".\"number\" AS \"number\"       ) AS \"_e\"      ) ) A
S \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM  (SELECT  *  FROM \"public\".\"Epoch\"  WHERE ('true')   ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\"    ORDER BY \"root.pg.number\" DESC NULLS FIRST  ) AS \"_root\"      "},"path":"$"},"query":{"type":n
ull},"query_execution_time":1.657673e-3,"request_id":"2fc31b05-0f3c-4dd6-9192-60204fbb4c8d","request_mode":"error","request_read_time":1.72e-6,"response_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"2fc31b05-0f3c-4dd6-9192-60204
fbb4c8d"},"level":"error","span_id":"95066c5648e8a6ec","timestamp":"2024-10-01T12:55:32.259+0000","trace_id":"671b908ec3a31d38c24227731b82c72e","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.122","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"validation-failed","error":"no mutations exist","path":"$"},"query":{"type":null},"query_execution_time":1.67017e-
4,"request_id":"362d856b-1c3c-4587-ab1e-a5650317c48a","request_mode":"error","request_read_time":1.66637e-4,"response_size":68,"uncompressed_response_size":68,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"362d856b-1c3c-4587-ab1e-a5650317c48a"},"level":"error","span_id":"
e2a38977c75f3546","timestamp":"2024-10-01T12:55:33.259+0000","trace_id":"ba50a6997116f6e3be10e397d77dc9d6","type":"http-log"}                   {"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"
operation":{"error":{"code":"validation-failed","error":"field 'epochs' not found in type: 'query_root'","path":"$.selectionSet.epochs"},"query":{"type":null},"query_execution_time":2.50166e-4,"request_id":"1fa181ec-87d2-427f-8a4b-6143d23bae55","request_mode":"error","request_read_time":
2.36e-6,"response_size":116,"uncompressed_response_size":116,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"1fa181ec-87d2-427f-8a4b-6143d23bae55"},"level":"error","span_id":"e8d20c1563873073","timestamp":"2024-10-01T12:55:56.281+0000","trace_id":"2fe4e636804b493c12da50d98
4004e3c","type":"http-log"}                                                                                                                     {"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"
operation":{"error":{"code":"validation-failed","error":"field 'epochs' not found in type: 'query_root'","path":"$.selectionSet.epochs"},"query":{"type":null},"query_execution_time":2.43936e-4,"request_id":"0ed47e3e-4e7a-41af-a66e-3dc3c854cb95","request_mode":"error","request_read_time":
2.06e-6,"response_size":116,"uncompressed_response_size":116,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"0ed47e3e-4e7a-41af-a66e-3dc3c854cb95"},"level":"error","span_id":"5d374fc3c09d8b7a","timestamp":"2024-10-01T12:55:56.281+0000","trace_id":"0416d3597fb9ebc592dbaffe7
483147d","type":"http-log"}

[Kammerlo]

Could you provide logs from hasura service? Because this service should apply automatically the sql files and create these tables.

[Kammerlo]

@mkocur-ledger I further investigated what could be the problem. I found an error within the sql file, which hindert hasura to finish the migration. You can find the PR here: #898
I'll finish testing today, merge and build a new fix release.

Could you check if this solves the problems for your issue as well?

[mkocur-ledger]

@Kammerlo I have seen this, I was actually supposed to create a PR or bugreport, but didn't find time yet.

Actually, this is not going to help, I have run those SQL queries already against the database (fixing the missing comma).

// full logs as requested:

{"timestamp":"2024-10-01T08:14:35.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"database url for migrations is set by HASURA_GRAPHQL_DATABASE_URL"}}
{"timestamp":"2024-10-01T08:14:35.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"migrations server port env var is not set, defaulting to 9691"}}
{"timestamp":"2024-10-01T08:14:35.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"starting graphql engine temporarily on port 9691"}}
{"timestamp":"2024-10-01T08:14:35.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"waiting 60 for 9691 to be ready"}}
{"detail":{"info":"WARNING: No admin secret provided","kind":"no_admin_secret"},"level":"warn","timestamp":"2024-10-01T08:14:35.949+0000","type":"startup"}
{"timestamp":"2024-10-01T08:14:36.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"port 9691 is ready"}}
{"timestamp":"2024-10-01T08:14:36.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"env var HASURA_GRAPHQL_MIGRATIONS_DIR is not set, defaulting to /hasura-migrations"}}
{"timestamp":"2024-10-01T08:14:36.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"env var HASURA_GRAPHQL_METADATA_DIR is not set, defaulting to /hasura-metadata"}}
{"timestamp":"2024-10-01T08:14:36.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-apply","info":"directory /hasura-migrations does not exist, skipping migrations"}}
{"timestamp":"2024-10-01T08:14:36.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-apply","info":"directory /hasura-metadata does not exist, skipping metadata"}}
{"timestamp":"2024-10-01T08:14:36.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-shutdown","info":"killing temporary server"}}
{"timestamp":"2024-10-01T08:14:36.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-shutdown","info":"graphql-engine will now start in normal mode"}}
{"detail":{"info":"WARNING: No admin secret provided","kind":"no_admin_secret"},"level":"warn","timestamp":"2024-10-01T08:14:36.932+0000","type":"startup"}
{"detail":{"info":"EE client credentials not present in the metadata database. Hasura EE features are disabled.","kind":"license_info"},"level":"warn","timestamp":"2024-10-01T08:14:35.949+0000","type":"startup"}
{"detail":{"info":"EE client credentials not present in the metadata database. Hasura EE features are disabled.","kind":"license_info"},"level":"warn","timestamp":"2024-10-01T08:14:36.932+0000","type":"startup"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM  (SELECT  row_to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.base\".\"number\" AS \"number\"       ) AS \"_e\"      ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM  (SELECT  *  FROM \"public\".\"Epoch\"  WHERE ('true')   ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\"    ORDER BY \"root.pg.number\" DESC NULLS FIRST  ) AS \"_root\"      "},"path":"$"},"query":{"type":null},"query_execution_time":59.342454474,"request_id":"858e9742-dc65-4e28-8906-a6259378869e","request_mode":"error","request_read_time":2.06e-6,"response_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"858e9742-dc65-4e28-8906-a6259378869e"},"level":"error","span_id":"1856f494e02ce310","timestamp":"2024-10-01T12:55:28.255+0000","trace_id":"f070f1320c096d4104734fc7e6db5fa0","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Withdrawal\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  json_build_object('aggregate', json_build_object('sum', json_build_object('amount', sum(\"_root\".\"root.pg.amount\" ) ) ) ) AS \"root\" FROM  (SELECT  \"_root.base\".\"amount\" AS \"root.pg.amount\" FROM  (SELECT  *  FROM \"public\".\"Withdrawal\"  WHERE ('true')     ) AS \"_root.base\"      ) AS \"_root\"      "},"path":"$"},"query":{"type":null},"query_execution_time":65.628226835,"request_id":"ffdf5de9-e978-4c5c-9cb6-c41b2d4dc766","request_mode":"error","request_read_time":1.54e-6,"response_size":669,"uncompressed_response_size":669,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"ffdf5de9-e978-4c5c-9cb6-c41b2d4dc766"},"level":"error","span_id":"93a4c54f8be46366","timestamp":"2024-10-01T12:55:28.255+0000","trace_id":"0a44ad3329a1255d3af62717187eee0a","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM  (SELECT  row_to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.base\".\"number\" AS \"number\"       ) AS \"_e\"      ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM  (SELECT  *  FROM \"public\".\"Epoch\"  WHERE ('true')   ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\"    ORDER BY \"root.pg.number\" DESC NULLS FIRST  ) AS \"_root\"      "},"path":"$"},"query":{"type":null},"query_execution_time":1.759961e-3,"request_id":"1ddc327d-36dc-48e7-b3d1-4e3fedb441c0","request_mode":"error","request_read_time":2.38e-6,"response_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"1ddc327d-36dc-48e7-b3d1-4e3fedb441c0"},"level":"error","span_id":"415b0c75b73d3f9b","timestamp":"2024-10-01T12:55:28.255+0000","trace_id":"7221d21491e389cadda32b981deeddc9","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM  (SELECT  row_to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.base\".\"number\" AS \"number\"       ) AS \"_e\"      ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM  (SELECT  *  FROM \"public\".\"Epoch\"  WHERE ('true')   ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\"    ORDER BY \"root.pg.number\" DESC NULLS FIRST  ) AS \"_root\"      "},"path":"$"},"query":{"type":null},"query_execution_time":1.466746e-3,"request_id":"f6f2695f-647f-4ac8-b8c4-8174d98d7b67","request_mode":"error","request_read_time":8.1e-7,"response_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"f6f2695f-647f-4ac8-b8c4-8174d98d7b67"},"level":"error","span_id":"f0fc2276bc19e1ad","timestamp":"2024-10-01T12:55:29.256+0000","trace_id":"f13c31a79f12fd3a95fea03cca50d56f","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM  (SELECT  row_to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.base\".\"number\" AS \"number\"       ) AS \"_e\"      ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM  (SELECT  *  FROM \"public\".\"Epoch\"  WHERE ('true')   ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\"    ORDER BY \"root.pg.number\" DESC NULLS FIRST  ) AS \"_root\"      "},"path":"$"},"query":{"type":null},"query_execution_time":1.23195e-3,"request_id":"dc69b318-34f9-4282-becb-0e487414eac3","request_mode":"error","request_read_time":1.01e-6,"response_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"dc69b318-34f9-4282-becb-0e487414eac3"},"level":"error","span_id":"83934b4c0740d43f","timestamp":"2024-10-01T12:55:29.256+0000","trace_id":"d4b99c418de9e6fcca17a14586011e69","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM  (SELECT  row_to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.or.adaPots\".\"adaPots\" AS \"adaPots\"       ) AS \"_e\"      ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM  (SELECT  *  FROM \"public\".\"Epoch\"  WHERE ('true')   ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\" LEFT OUTER JOIN LATERAL (SELECT  row_to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.or.adaPots.base\".\"reserves\" AS \"reserves\"       ) AS \"_e\"      ) ) AS \"adaPots\" FROM  (SELECT  *  FROM \"public\".\"ada_pots\"  WHERE ((\"_root.base\".\"number\") = (\"epoch_no\"))    LIMIT 1 ) AS \"_root.or.adaPots.base\"      ) AS \"_root.or.adaPots\" ON ('true')    ORDER BY \"root.pg.number\" DESC NULLS FIRST  ) AS \"_root\"      "},"path":"$"},"query":{"type":null},"query_execution_time":1.229809e-3,"request_id":"5f82fa48-a433-4b54-af2a-ef488244b533","request_mode":"error","request_read_time":1.07e-6,"response_size":1194,"uncompressed_response_size":1194,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"5f82fa48-a433-4b54-af2a-ef488244b533"},"level":"error","span_id":"ca4807fd288b05b5","timestamp":"2024-10-01T12:55:28.255+0000","trace_id":"33c6aa0f5dda79245bc95b61916ef6e5","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM  (SELECT  row_to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.base\".\"number\" AS \"number\"       ) AS \"_e\"      ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM  (SELECT  *  FROM \"public\".\"Epoch\"  WHERE ('true')   ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\"    ORDER BY \"root.pg.number\" DESC NULLS FIRST  ) AS \"_root\"      "},"path":"$"},"query":{"type":null},"query_execution_time":1.657673e-3,"request_id":"2fc31b05-0f3c-4dd6-9192-60204fbb4c8d","request_mode":"error","request_read_time":1.72e-6,"response_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"2fc31b05-0f3c-4dd6-9192-60204fbb4c8d"},"level":"error","span_id":"95066c5648e8a6ec","timestamp":"2024-10-01T12:55:32.259+0000","trace_id":"671b908ec3a31d38c24227731b82c72e","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.122","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"validation-failed","error":"no mutations exist","path":"$"},"query":{"type":null},"query_execution_time":1.67017e-4,"request_id":"362d856b-1c3c-4587-ab1e-a5650317c48a","request_mode":"error","request_read_time":1.66637e-4,"response_size":68,"uncompressed_response_size":68,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"362d856b-1c3c-4587-ab1e-a5650317c48a"},"level":"error","span_id":"e2a38977c75f3546","timestamp":"2024-10-01T12:55:33.259+0000","trace_id":"ba50a6997116f6e3be10e397d77dc9d6","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"validation-failed","error":"field 'epochs' not found in type: 'query_root'","path":"$.selectionSet.epochs"},"query":{"type":null},"query_execution_time":2.50166e-4,"request_id":"1fa181ec-87d2-427f-8a4b-6143d23bae55","request_mode":"error","request_read_time":2.36e-6,"response_size":116,"uncompressed_response_size":116,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"1fa181ec-87d2-427f-8a4b-6143d23bae55"},"level":"error","span_id":"e8d20c1563873073","timestamp":"2024-10-01T12:55:56.281+0000","trace_id":"2fe4e636804b493c12da50d984004e3c","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"validation-failed","error":"field 'epochs' not found in type: 'query_root'","path":"$.selectionSet.epochs"},"query":{"type":null},"query_execution_time":2.43936e-4,"request_id":"0ed47e3e-4e7a-41af-a66e-3dc3c854cb95","request_mode":"error","request_read_time":2.06e-6,"response_size":116,"uncompressed_response_size":116,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"0ed47e3e-4e7a-41af-a66e-3dc3c854cb95"},"level":"error","span_id":"5d374fc3c09d8b7a","timestamp":"2024-10-01T12:55:56.281+0000","trace_id":"0416d3597fb9ebc592dbaffe7483147d","type":"http-log"}

//edit
forgot I have warn level. Here's debug from start:

{"timestamp":"2024-10-02T12:37:14.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"database url for migrations is set by HASURA_GRAPHQL_DATABASE_URL"}}
{"timestamp":"2024-10-02T12:37:14.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"migrations server port env var is not set, defaulting to 9691"}}
{"timestamp":"2024-10-02T12:37:14.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"starting graphql engine temporarily on port 9691"}}
{"timestamp":"2024-10-02T12:37:14.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"waiting 60 for 9691 to be ready"}}
{"detail":{"info":"WARNING: No admin secret provided","kind":"no_admin_secret"},"level":"warn","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":{"admin_secret_set":false,"auth_hook":null,"auth_hook_mode":null,"console_assets_dir":null,"console_sentry_dsn":null,"cors_config":{"allowed_origins":"*","disabled":false,"ws_read_cookie":null},"enable_allowlist":false,"enable_console":true,"enable_maintenance_mode":false,"enable_metadata_query_logging":false,"enable_telemetry":true,"enabled_apis":["metadata"],"enabled_log_types":["startup","http-log","jwk-refresh-log","webhook-log","websocket-log"],"events_fetch_batch_size":100,"experimental_features":[],"graceful_shutdown_timeout":60,"infer_function_permissions":true,"jwt_secret":[],"live_query_options":{"batch_size":100,"refetch_delay":1},"log_level":"debug","port":9691,"remote_schema_permissions":false,"server_host":"HostAny","stringify_numeric_types":false,"transaction_isolation":"ISOLATION LEVEL READ COMMITTED","unauth_role":null,"use_prepared_statements":true,"v1-boolean-null-collapse":false,"websocket_compression_options":"NoCompression","websocket_connection_init_timeout":"Refined (Seconds {seconds = 3s})","websocket_keep_alive":"KeepAliveDelay {unKeepAliveDelay = Refined (Seconds {seconds = 5s})}"},"kind":"server_configuration"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":{"database_url":"postgres://cardano-mainnet:...@10.99.2.57:5432/cardano-mainnet","retries":1},"kind":"postgres_connection"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"timestamp":"2024-10-02T12:37:15.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"port 9691 is ready"}}
{"timestamp":"2024-10-02T12:37:15.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"env var HASURA_GRAPHQL_MIGRATIONS_DIR is not set, defaulting to /hasura-migrations"}}
{"timestamp":"2024-10-02T12:37:15.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"env var HASURA_GRAPHQL_METADATA_DIR is not set, defaulting to /hasura-metadata"}}
{"timestamp":"2024-10-02T12:37:15.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-apply","info":"directory /hasura-migrations does not exist, skipping migrations"}}
{"timestamp":"2024-10-02T12:37:15.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-apply","info":"directory /hasura-metadata does not exist, skipping metadata"}}
{"timestamp":"2024-10-02T12:37:15.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-shutdown","info":"killing temporary server"}}
{"timestamp":"2024-10-02T12:37:15.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-shutdown","info":"graphql-engine will now start in normal mode"}}
{"detail":{"info":"WARNING: No admin secret provided","kind":"no_admin_secret"},"level":"warn","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":{"admin_secret_set":false,"auth_hook":null,"auth_hook_mode":null,"console_assets_dir":null,"console_sentry_dsn":null,"cors_config":{"allowed_origins":"*","disabled":false,"ws_read_cookie":null},"enable_allowlist":false,"enable_console":true,"enable_maintenance_mode":false,"enable_metadata_query_logging":false,"enable_telemetry":true,"enabled_apis":["metrics","metadata","config","graphql"],"enabled_log_types":["startup","http-log","jwk-refresh-log","webhook-log","websocket-log"],"events_fetch_batch_size":100,"experimental_features":[],"graceful_shutdown_timeout":60,"infer_function_permissions":true,"jwt_secret":[],"live_query_options":{"batch_size":100,"refetch_delay":1},"log_level":"debug","port":9090,"remote_schema_permissions":false,"server_host":"HostAny","stringify_numeric_types":false,"transaction_isolation":"ISOLATION LEVEL READ COMMITTED","unauth_role":null,"use_prepared_statements":true,"v1-boolean-null-collapse":false,"websocket_compression_options":"NoCompression","websocket_connection_init_timeout":"Refined (Seconds {seconds = 3s})","websocket_keep_alive":"KeepAliveDelay {unKeepAliveDelay = Refined (Seconds {seconds = 5s})}"},"kind":"server_configuration"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":{"database_url":"postgres://cardano-mainnet:...@10.99.2.57:5432/cardano-mainnet","retries":1},"kind":"postgres_connection"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":"Thread runCronEventsGenerator (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Thread processScheduledTriggers (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":{"info":"Unlocking all locked scheduled events on `hdb_scheduled_events` and `hdb_cron_events` tables","kind":"scheduled_triggers"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":"Thread checkForUpdates (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Thread websocketConnectionReaper (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":{"info":"Already at the latest catalog version (48); nothing to do.","kind":"catalog_migrate"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":"Schema sync enabled. Polling at Refined (Milliseconds {milliseconds = 1s})","kind":"schema-sync"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":{"instance_id":"b580b3d4-6fb3-4180-bc52-22dfc2baba95","message":"listener thread started","thread_id":"ThreadId 72"},"kind":"schema-sync"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":"EE client credentials not present in the metadata database. Hasura EE features are disabled.","kind":"license_info"},"level":"warn","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":{"message":"source \"default\" has not been initialized yet.","source":"default"},"kind":"source_catalog_migrate"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":{"instance_id":"b580b3d4-6fb3-4180-bc52-22dfc2baba95","message":"processor thread started","thread_id":"ThreadId 95"},"kind":"schema-sync"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":"Starting in eventing enabled mode","kind":"server"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":"Starting workers","kind":"event_triggers"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":"Help us improve Hasura! The graphql-engine server collects anonymized usage stats which allows us to keep improving Hasura at warp speed. To read more or opt-out, visit https://hasura.io/docs/latest/graphql/core/guides/telemetry.html","kind":"telemetry"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":{"message":"Starting API server","time_taken":0.394567301},"kind":"server"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":"Thread SchemeUpdate.listener (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Thread SchemeUpdate.processor (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Thread processEventQueue (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Thread sourcePingPoller (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Thread runTelemetry (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"metrics_info: {\"server_metrics\":{\"resource_cpu\":4096,\"resource_memory\":10737418240,\"resource_checker_error_code\":null}}","level":"debug","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Shutting websockets server down","level":"debug","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Stopping immortal updateJWK thread","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Stopping immortal runTelemetry thread","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Stopping immortal websocketConnectionReaper thread","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Stopping immortal sourcePingPoller thread","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Stopping immortal checkForUpdates thread","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":{"info":"All in-flight events have finished processing","kind":"scheduled_events"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":"Thread ourIdleGC (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Thread asyncActionsProcessor (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Thread updateJWK (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":{"info":"Gracefully shutting down server","kind":"server"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":"Thread asyncActionSubscriptionsProcessor (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Thread runCronEventsGenerator (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread processScheduledTriggers (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread checkForUpdates (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread sourcePingPoller (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":{"info":"Unlocking all locked scheduled events on `hdb_scheduled_events` and `hdb_cron_events` tables","kind":"scheduled_triggers"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":"Already at the latest catalog version (48); nothing to do.","kind":"catalog_migrate"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":"Schema sync enabled. Polling at Refined (Milliseconds {milliseconds = 1s})","kind":"schema-sync"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":{"instance_id":"e9c63890-18ac-4a6f-a0d1-623c32926af0","message":"listener thread started","thread_id":"ThreadId 72"},"kind":"schema-sync"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":"EE client credentials not present in the metadata database. Hasura EE features are disabled.","kind":"license_info"},"level":"warn","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":{"message":"source \"default\" has not been initialized yet.","source":"default"},"kind":"source_catalog_migrate"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":{"instance_id":"e9c63890-18ac-4a6f-a0d1-623c32926af0","message":"processor thread started","thread_id":"ThreadId 94"},"kind":"schema-sync"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":"Starting in eventing enabled mode","kind":"server"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":"Starting workers","kind":"event_triggers"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":"Help us improve Hasura! The graphql-engine server collects anonymized usage stats which allows us to keep improving Hasura at warp speed. To read more or opt-out, visit https://hasura.io/docs/latest/graphql/core/guides/telemetry.html","kind":"telemetry"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":{"message":"Starting API server","time_taken":0.443591001},"kind":"server"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":"Thread SchemeUpdate.listener (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread SchemeUpdate.processor (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread asyncActionSubscriptionsProcessor (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread websocketConnectionReaper (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread runTelemetry (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"metrics_info: {\"server_metrics\":{\"resource_cpu\":4096,\"resource_memory\":10737418240,\"resource_checker_error_code\":null}}","level":"debug","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread ourIdleGC (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread processEventQueue (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread updateJWK (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread asyncActionsProcessor (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"http_success: {\"error\":\"invalid_payload\"}","level":"debug","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":{"http_error":{"http_exception":null,"response":"{\"error\":\"invalid_payload\"}","status_code":400,"url":"https://telemetry.hasura.io/v1/http"},"message":"failed to post telemetry","type":"http_error"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"telemetry-log"}
{"detail":"metrics_info: {\"topic\":\"server_metrics_v2\",\"data\":{\"metadata_db_uid\":\"156d95bd-4660-480c-9b89-a7a37d9dd9ff\",\"instance_uid\":\"e9c63890-18ac-4a6f-a0d1-623c32926af0\",\"hasura_version\":\"v2.43.0\",\"ci\":null,\"source_metadata\":{\"db_uid\":\"156d95bd-4660-480c-9b89-a7a37d9dd9ff\",\"backend_type\":\"postgres\",\"db_kind\":\"postgres\",\"db_version\":\"150007\"},\"metrics\":{\"tables\":6,\"views\":27,\"enum_tables\":0,\"relationships\":{\"manual\":82,\"auto\":0},\"permissions\":{\"select\":32,\"insert\":1,\"update\":1,\"delete\":1,\"roles\":1},\"event_triggers\":0,\"functions\":0,\"remote_schemas\":0,\"service_timings\":{\"collectionTag\":1727872637,\"serviceTimingMetrics\":[]},\"actions\":{\"synchronous\":0,\"asynchronous\":0,\"query_actions\":0,\"type_relationships\":0,\"custom_types\":0},\"native_queries\":{\"with_parameters\":0,\"without_parameters\":0},\"stored_procedures\":{\"with_parameters\":0,\"without_parameters\":0},\"logical_models\":{\"count\":0}},\"experimental_features\":[]}}","level":"debug","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"http_success: {\"message\":\"ok\"}","level":"debug","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Stopping immortal processScheduledTriggers thread","level":"info","timestamp":"2024-10-02T12:37:25.275+0000","type":"unstructured"}
{"detail":{"num_cron_events_fetched":0,"num_fetches":1,"num_one_off_scheduled_events_fetched":0},"level":"info","timestamp":"2024-10-02T12:37:25.275+0000","type":"scheduled-trigger-process"}
{"detail":"Stopping immortal runCronEventsGenerator thread","level":"info","timestamp":"2024-10-02T12:37:25.275+0000","type":"unstructured"}
{"detail":"Stopping immortal asyncActionSubscriptionsProcessor thread","level":"info","timestamp":"2024-10-02T12:37:25.275+0000","type":"unstructured"}
{"detail":{"info":"All in-flight events have finished processing","kind":"async_actions"},"level":"info","timestamp":"2024-10-02T12:37:25.275+0000","type":"startup"}
{"detail":"Stopping immortal asyncActionsProcessor thread","level":"info","timestamp":"2024-10-02T12:37:26.276+0000","type":"unstructured"}
{"detail":{"info":"All in-flight events have finished processing","kind":"event_triggers"},"level":"info","timestamp":"2024-10-02T12:37:26.276+0000","type":"startup"}
{"detail":"Stopping immortal processEventQueue thread","level":"info","timestamp":"2024-10-02T12:37:27.277+0000","type":"unstructured"}
{"detail":"Stopping immortal SchemeUpdate.processor thread","level":"info","timestamp":"2024-10-02T12:37:27.277+0000","type":"unstructured"}
{"detail":"Stopping immortal SchemeUpdate.listener thread","level":"info","timestamp":"2024-10-02T12:37:27.277+0000","type":"unstructured"}
{"detail":{"num_events_fetched":{},"num_fetches":13},"level":"info","timestamp":"2024-10-02T12:37:27.277+0000","type":"event-trigger-process"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"51.91.233.11","method":"GET","status":200,"url":"/healthz"},"operation":{"query":{"type":null},"request_id":"14b2edc0-899c-40b8-b84c-dd350104184b","request_mode":"non-graphql","response_size":2,"uncompressed_response_size":2},"request_id":"14b2edc0-899c-40b8-b84c-dd350104184b"},"level":"info","timestamp":"2024-10-02T12:37:38.258+0000","type":"http-log"}