Add a DB based session property manager #24995

infvg · 2025-04-28T11:34:34Z

Description

Add a DB based session property manager.

Motivation and Context

Will add the option of adding a new session property manager

Impact

A new db-based session property manager. Will be off by default but can be turned on using the config session-property-config.configuration-manager=db

Test Plan

Unit tests

Contributor checklist

Please make sure your submission complies with our contributing guide, in particular code style and commit standards.
PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
Documented new properties (with its default value), SQL syntax, functions, or other functionality.
If release notes are required, they follow the release notes guidelines.
Adequate tests were added if applicable.
CI passed.

Release Notes

== RELEASE NOTES ==

General Changes
* Added a new db-based session property manager. :doc:`/admin/session-property-managers`

pratyakshsharma

Thank you for raising this PR @infvg. One high level comment - please update the docs section for "Session Property Managers" with file and db based managers that will be supported post this change.
I am still going over the changes.

steveburnett

Thanks for the documentation! A few suggestions.

presto-docs/src/main/sphinx/admin/session-property-managers.rst

steveburnett

Minor phrasing suggestion.

presto-docs/src/main/sphinx/admin/session-property-managers.rst

steveburnett

LGTM! (docs)

Pull updated branch, new local doc build, looks good. Thanks!

imjalpreet

@infvg, thank you, I finished my first pass, please have a look.

imjalpreet · 2025-07-15T07:29:56Z

presto-docs/src/main/sphinx/admin/session-property-managers.rst

+    session-property-manager.db.url=url
+    session-property-manager.db.username=username
+    session-property-manager.db.password=password
+    session-property-manager.db.refresh-period=50s


This is missing one config. Please add session-property-config.configuration-manager=db as that is required to enable the DB-based session property manager.

@infvg I think we should also add a brief description of every new property introduced. @steveburnett, do you think that would be useful?

@imjalpreet yes, I agree. All properties should be documented. See Designing Your Code section iii in CONTRIBUTING.md.

Added an explanation for session-property-manager.db.url and session-property-manager.db.refresh-period. I also added the session-property-config.configuration-manager=db

imjalpreet · 2025-07-15T07:31:06Z

presto-docs/src/main/sphinx/admin/session-property-managers.rst

+
+.. code-block:: text
+
+    presto:tpch> DESCRIBE session_specs;


This table is present in the MySQL DB. The CLI prompt presto:tpch> is a little confusing.

Maybe we should show output from MySQL CLI rather than Presto CLI.

imjalpreet · 2025-07-15T07:31:57Z

presto-docs/src/main/sphinx/admin/session-property-managers.rst

+
+.. code-block:: text
+
+    presto:tpch> DESCRIBE session_client_tags;


This table is present in the MySQL DB. The CLI prompt presto:tpch> is a little confusing.

imjalpreet · 2025-07-15T07:32:07Z

presto-docs/src/main/sphinx/admin/session-property-managers.rst

+
+.. code-block:: text
+
+    presto:tpch> DESCRIBE session_property_values;


This table is present in the MySQL DB. The CLI prompt presto:tpch> is a little confusing.

imjalpreet · 2025-07-15T07:39:08Z

presto-docs/src/main/sphinx/admin/session-property-managers.rst

+    session-property-manager.db.password=password
+    session-property-manager.db.refresh-period=50s
+
+This database consists of three tables: session specs, client tags and properties.


We should also note in the documentation that these tables will be automatically created in the configured database on the first server startup after the configurations are added, if they do not already exist.

Added a note

imjalpreet · 2025-07-15T08:04:21Z

...-managers/src/test/java/com/facebook/presto/session/file/TestFileSessionPropertyManager.java

+import static com.facebook.presto.session.file.FileSessionPropertyManager.CODEC;
+import static org.testng.Assert.assertEquals;
+
+public class TestFileSessionPropertyManager


Can we add a similar test for the DB-based session property manager?

Added tests

imjalpreet · 2025-07-15T08:10:32Z

presto-main/src/main/java/com/facebook/presto/server/testing/TestingPrestoServer.java

@@ -659,6 +666,11 @@ public Optional<InternalResourceGroupManager<?>> getResourceGroupManager()
        return resourceGroupManager;
    }

+    public SessionPropertyDefaults getSessionPropertyDefaults()


Do we need the changes in this file? I don't see any usage.

imjalpreet · 2025-07-15T08:12:23Z

presto-session-property-managers/pom.xml

+        <dependency>
+            <groupId>io.airlift</groupId>
+            <artifactId>testing-mysql-server</artifactId>
+            <version>8.0.12-2</version>
+            <exclusions>
+                <exclusion>
+                    <groupId>mysql</groupId>
+                    <artifactId>mysql-connector-java</artifactId>
+                </exclusion>
+            </exclusions>
+            <scope>test</scope>
+        </dependency>


I don't see any tests added yet that are using this dependency.

Changed to similar mysql dependencies that are used in the new tests.

imjalpreet · 2025-07-15T08:40:58Z

...property-managers/src/main/java/com/facebook/presto/session/db/DbSessionPropertyManager.java

+    @Override
+    public SystemSessionPropertyConfiguration getSystemSessionProperties(SessionConfigurationContext context)
+    {
+        List<SessionMatchSpec> sessionMatchSpecs = specsProvider.get();
+
+        // later properties override earlier properties
+        Map<String, String> defaultProperties = new HashMap<>();
+        Set<String> overridePropertyNames = new HashSet<String>();
+        for (SessionMatchSpec sessionMatchSpec : sessionMatchSpecs) {
+            Map<String, String> newProperties = sessionMatchSpec.match(context);
+            defaultProperties.putAll(newProperties);
+            if (sessionMatchSpec.getOverrideSessionProperties().orElse(false)) {
+                overridePropertyNames.addAll(newProperties.keySet());
+            }
+        }
+
+        // Once a property has been overridden it stays that way and the value is updated by any rule
+        Map<String, String> overrideProperties = new HashMap<>();
+        for (String propertyName : overridePropertyNames) {
+            overrideProperties.put(propertyName, defaultProperties.get(propertyName));
+        }
+
+        return new SystemSessionPropertyConfiguration(ImmutableMap.copyOf(defaultProperties), ImmutableMap.copyOf(overrideProperties));
+    }
+
+    @Override
+    public Map<String, Map<String, String>> getCatalogSessionProperties(SessionConfigurationContext context)
+    {
+        // NOT IMPLEMENTED YET
+        return ImmutableMap.of();
+    }
+}


We shouldn't override these. Based on what I understand, we can just override the abstract method: getSessionMatchSpecs in all implementations of AbstractSessionPropertyManager

imjalpreet

@infvg, please rebase on the latest master as well to trigger all tests.

imjalpreet

@infvg I did another pass, mostly good, a few suggestions.

Looks like there are some test failures. Please look into them. Thanks.

imjalpreet · 2025-07-22T22:16:39Z

presto-docs/src/main/sphinx/admin/session-property-managers.rst

+``session-property-manager.db.refresh-period`` should be set to how often presto refreshes to
+match the database.
+
+This database consists of three tables: session specs, client tags and properties.


nit: let's mention the actual table names like session_specs, session_client_tags, and session_property_values

imjalpreet · 2025-07-22T22:22:37Z

...perty-managers/src/main/java/com/facebook/presto/session/AbstractSessionPropertyManager.java

+    public static final String DEFAULT_PROPERTIES = "defaultProperties";
+    public static final String OVERRIDE_PROPERTIES = "overrideProperties";


nit: can be private

imjalpreet · 2025-07-22T22:27:41Z

...y-managers/src/main/java/com/facebook/presto/session/db/DbSessionPropertyManagerFactory.java

+    public SessionPropertyConfigurationManager create(Map<String, String> config, SessionPropertyConfigurationManagerContext context)
+    {
+        try {
+            Bootstrap app = new Bootstrap(new JsonModule(), new JsonModule(), new DbSessionPropertyManagerModule());


nit: repetitive JsonModule() argument

imjalpreet · 2025-07-22T22:35:53Z

...ty-managers/src/main/java/com/facebook/presto/session/db/DbSessionPropertyManagerModule.java

+        binder.bind(DbSessionPropertyManager.class).in(Scopes.SINGLETON);
+        binder.bind(SessionPropertiesDao.class).toProvider(SessionPropertiesDaoProvider.class).in(Scopes.SINGLETON);
+        binder.bind(DbSpecsProvider.class).to(RefreshingDbSpecsProvider.class).in(Scopes.SINGLETON);
+        newExporter(binder).export(DbSpecsProvider.class).withGeneratedName();


Are we planning to expose any JMX metrics? Currently, I don't see anything exposed by RefreshingDbSpecsProvider/DbSpecsProvider. If not, we don't need this exporter.

imjalpreet · 2025-07-22T22:50:50Z

...erty-managers/src/test/java/com/facebook/presto/session/db/TestDbSessionPropertyManager.java

+    public TestDbSessionPropertyManager()
+            throws Exception
+    {
+        this.mysqlServer = new TestingMySqlServer("testuser", "testpass", ImmutableList.of(), MY_SQL_OPTIONS);


nit: let's add an AfterClass tearDown method to close/stop this.

Co-authored-by: Ashish Tadose <ashishtadose@gmail.com> Co-authored-by: Ariel Weisberg <aweisberg@fb.com> Co-authored-by: Jalpreet Singh Nanda (:imjalpreet) <jalpreetnanda@gmail.com>

imjalpreet

Mostly LGTM, one last nit.

imjalpreet · 2025-07-24T20:43:37Z

presto-docs/src/main/sphinx/admin/session-property-managers.rst

+.. code-block:: none
+
+    session-property-config.configuration-manager=db
+    session-property-manager.db.url=url


nit: instead of url let's mention jdbc:mysql://localhost:3306/session_properties to give an example of what is expected in this config

prestodb-ci added the from:IBM label Apr 28, 2025

infvg force-pushed the db-session-property branch from b9191b4 to a405418 Compare April 28, 2025 11:37

infvg force-pushed the db-session-property branch 2 times, most recently from 064f35e to 9f7283f Compare May 25, 2025 08:30

infvg marked this pull request as ready for review May 25, 2025 10:25

infvg requested review from shrinidhijoshi and a team as code owners May 25, 2025 10:25

infvg requested a review from hantangwangd May 25, 2025 10:25

prestodb-ci requested review from a team, wanglinsong and ShahimSharafudeen and removed request for a team May 25, 2025 10:25

infvg force-pushed the db-session-property branch 2 times, most recently from c335922 to b29057b Compare May 26, 2025 09:01

pratyakshsharma requested changes Jun 5, 2025

View reviewed changes

infvg force-pushed the db-session-property branch 2 times, most recently from f174a51 to fd53f8b Compare June 18, 2025 09:05

infvg requested review from steveburnett and elharo as code owners June 18, 2025 09:05

steveburnett requested changes Jun 18, 2025

View reviewed changes

infvg force-pushed the db-session-property branch 3 times, most recently from 78ac74c to 30a553c Compare June 23, 2025 08:01

steveburnett requested changes Jun 23, 2025

View reviewed changes

presto-docs/src/main/sphinx/admin/session-property-managers.rst Outdated Show resolved Hide resolved

infvg force-pushed the db-session-property branch from 30a553c to 4c6495f Compare June 25, 2025 06:07

steveburnett previously approved these changes Jun 25, 2025

View reviewed changes

infvg requested a review from imjalpreet July 11, 2025 14:12

imjalpreet requested changes Jul 15, 2025

View reviewed changes

imjalpreet reviewed Jul 15, 2025

View reviewed changes

infvg dismissed steveburnett’s stale review via dff7591 July 17, 2025 15:19

infvg force-pushed the db-session-property branch 3 times, most recently from 6fb43f1 to 323cf08 Compare July 21, 2025 08:21

infvg requested a review from imjalpreet July 21, 2025 08:24

infvg force-pushed the db-session-property branch 2 times, most recently from 86eb966 to 3a398c7 Compare July 22, 2025 17:48

imjalpreet requested changes Jul 22, 2025

View reviewed changes

infvg force-pushed the db-session-property branch 2 times, most recently from 884d625 to a122789 Compare July 23, 2025 09:39

Add a DB based session property manager

77c666d

Co-authored-by: Ashish Tadose <ashishtadose@gmail.com> Co-authored-by: Ariel Weisberg <aweisberg@fb.com> Co-authored-by: Jalpreet Singh Nanda (:imjalpreet) <jalpreetnanda@gmail.com>

infvg force-pushed the db-session-property branch from a122789 to 77c666d Compare July 23, 2025 22:26

imjalpreet reviewed Jul 24, 2025

View reviewed changes


		.. code-block:: text

		presto:tpch> DESCRIBE session_client_tags;


		.. code-block:: text

		presto:tpch> DESCRIBE session_property_values;

		public static final String DEFAULT_PROPERTIES = "defaultProperties";
		public static final String OVERRIDE_PROPERTIES = "overrideProperties";

Add a DB based session property manager #24995

Are you sure you want to change the base?

Add a DB based session property manager #24995

Conversation

infvg commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

Impact

Test Plan

Contributor checklist

Release Notes

Uh oh!

pratyakshsharma left a comment

Choose a reason for hiding this comment

Uh oh!

steveburnett left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

steveburnett left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

steveburnett left a comment

Choose a reason for hiding this comment

Uh oh!

imjalpreet left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

imjalpreet left a comment

Choose a reason for hiding this comment

Uh oh!

imjalpreet left a comment

infvg commented Apr 28, 2025 •

edited

Loading