Proxy Settings for catalog REST API client

[sfc-gh-dtriana]

Feature Request / Improvement

Despite configuring a proxy for the driver or executor

spark.driver.extraJavaOptions="-Dhttp.proxyHost= -Dhttp.proxyPort= -Dhttps.proxyHost= -Dhttps.proxyPort="
spark.executor.extraJavaOptions="-Dhttp.proxyHost= -Dhttp.proxyPort= -Dhttps.proxyHost= -Dhttps.proxyPort="

Calls made by the Spark engine using Iceberg REST API do not follow the above proxy configurations,

This is a feature request for the ability to configure proxy settings when connecting to an Iceberg Rest API Catalog via Spark. This is especially critical for organizations where network access to catalog services is restricted and must go through a corporate proxy.

Query engine

Spark

Willingness to contribute

• I can contribute this improvement/feature independently
• I would be willing to contribute this improvement/feature with guidance from the Iceberg community
• I cannot contribute this improvement/feature at this time

[flyrain]

Thanks @sfc-gh-dtriana for raising it. We could enable this feature on Java REST clients(class RESTSessionCatalog), ensuring that all Java-based engines, including Spark, can benefit from it seamlessly.