Skip to content

Issue with Vended Credential Renewal with Iceberg REST Catalog #25827

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ashwinsangem opened this issue May 20, 2025 · 4 comments
Open

Issue with Vended Credential Renewal with Iceberg REST Catalog #25827

ashwinsangem opened this issue May 20, 2025 · 4 comments
Labels
iceberg Iceberg connector

Comments

@ashwinsangem
Copy link

ashwinsangem commented May 20, 2025

Hi Team,

I'm integrating Trino with an external Iceberg REST Catalog and have enabled vended credentials using the setting:
iceberg.rest-catalog.vended-credentials-enabled=true
Basic queries on Iceberg tables are working as expected. However, when executing long-running queries that exceed the STS token's expiry duration (30 minutes), the query fails.

Upon reviewing the Trino logs, I observed that the client does not invoke the /credentials endpoint to refresh the vended credentials. It only calls the oauth/tokens endpoint.

I've attached a relevant log snippet highlighting the failure for your reference.

TIMELINE: Query 20250519_171345_00011_uzse9 :: FAILED (ICEBERG_CURSOR_ERROR) :: elapsed 1803662ms :: planning 22ms :: waiting 3266ms :: scheduling 3363ms :: running 1800315ms :: finishing 0ms :: begin 2025-05-20T07:07:37.337Z :: end 2025-05-20T07:37:40.999Z

Could you please help confirm if Trino is expected to refresh credentials during long-running queries, or if there’s additional configuration required?

logs2.txt

Regards,
Ashwin

@ebyhr
Copy link
Member

ebyhr commented May 20, 2025

Could you share Trino version and catalog config properties? You can redact confidential info.
cc: @mayankvadariya

@ashwinsangem
Copy link
Author

I'm using docker image for trino with version trinodb/trino:latest sha256:0d15d7aca701172beb4ca0ec2481ad1680a6cac4d1c7d989e9c3ec8346cf9fb0 FROM redhat/ubi9-minimal:9.5-1731593028

and here are the catalog config properties:

iceberg.rest-catalog.vended-credentials-enabled=true
connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=https://tableflow.us-west-2.aws.devel.cpdev.cloud/iceberg/catalog/organizations/<org>/environments/<env>
iceberg.rest-catalog.oauth2.credential=<userid>:<password>
iceberg.rest-catalog.security=OAUTH2
fs.hadoop.enabled=false
fs.native-s3.enabled=true
s3.endpoint=https://s3.us-west-2.amazonaws.com
s3.region=us-west-2
s3.path-style-access=true

@ebyhr
Copy link
Member

ebyhr commented May 22, 2025

Could you share the full stackrace? The provided log doesn't contain failed place in Trino.

@ashwinsangem
Copy link
Author

ashwinsangem commented May 25, 2025

I have enabled debug logs on Trino by adding log.properties with io.trino=DEBUG to Trino containere and shared the entire log initially. Here's the error in the trino cli, FYI, I was trying to run a full table scan query that spans more than 30 mins.
Image

Could you share the full stackrace?

Other than the CLI and debug logs, do you need any other things?

@kumiDa kumiDa added the iceberg Iceberg connector label May 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
iceberg Iceberg connector
Development

No branches or pull requests

3 participants