HIVE-28576: Add jdbc tests for tpcds queries #5510

soumyakanti3578 · 2024-10-14T21:23:39Z

What changes were proposed in this pull request?

Add a new test driver for jdbc tests TestMiniLlapLocalJdbcCliDriver to create tpcds tables in dockerized postgres, and create corresponding external tables. Generate explain plans for tpcds queries.

Why are the changes needed?

To improve JDBC test coverage

Does this PR introduce any user-facing change?

No

Is the change a dependency upgrade?

No

How was this patch tested?

mvn test -pl itests/qtest  -Pitests -Dtest=TestMiniLlapLocalJdbcCliDriver -Dtest.output.overwrite=true

deniskuzZ · 2024-10-24T10:28:31Z

@soumyakanti3578, isn't that already covered by TestTezTPCDS30TBPerfCliDriver?

soumyakanti3578 · 2024-10-24T17:49:42Z

@deniskuzZ No, TestTezTPCDS30TBPerfCliDriver generates logical plans and uses a stats dump. This is different because here we actually create tpcds tables in Postgres, create external tables in Hive corresponding to the tables in Postgres, and then generate the jdbc plans.

deniskuzZ · 2024-10-28T19:45:32Z

@deniskuzZ No, TestTezTPCDS30TBPerfCliDriver generates logical plans and uses a stats dump. This is different because here we actually create tpcds tables in Postgres, create external tables in Hive corresponding to the tables in Postgres, and then generate the jdbc plans.

hi @soumyakanti3578 , why not simply parameterize the db to run tests against? default is derby, which could be overridden. why create a separate driver for that?

sonarqubecloud · 2024-10-28T20:42:18Z

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

soumyakanti3578 · 2024-10-29T22:40:04Z

@deniskuzZ

why not simply parameterize the db to run tests against?

Only option I saw was to add something like --!qt:database:postgres:<script-to-run-in-postgres>. Did you have this in mind as well? I didn't go with this approach because I need to run all tpcds queries on external tables. There are 2 ways to do this:

Have a single qfile - test_all_tpcds_queries_postgres.q with --!qt:database:postgres:q_test_tpcds_tables.postgres.sql (q_test_tpcds_tables.postgres.sql) on the top of the file to create all tpcds queries in postgres. Then we would need the contents of q_test_external_tpcds_tables_postgres.q in test_all_tpcds_queries_postgres.q to create all external tables. And then, we will need all tpcds queries too. This would be a huge file, and the resulting qout file could be very difficult to maintain too.
Other approach that I can think of is to have individual q files for each tpcds query, but then we will need to create <script-to-run-in-postgres> for each one of them if we want to only create the required tables in postgres. We can also just create all tpcds tables for each q file but that would be inefficient I guess. We would still need to create external tables in each q file. For example, for cbo_query3.q, we would need to create 3 jdbc tables (date_dim, store_sales, item) in <script-to-run-in-postgres-for-cbo_query3>, and 3 external tables in cbo_query3_postgres.q, before the actual query. This could become very difficult to maintain too.

With the new driver, we can configure CliConfigs.java to run 1 file to create all tpcds tables once with

setJdbcInitScript("q_test_tpcds_tables.postgres.sql");

and run 1 file to create all external tables once with

setExternalTablesForJdbcInitScript("q_test_external_tpcds_tables_postgres.q");

and in CoreJdbcCliDriver.java, we can launch the docker container before starting the tests with beforeClass(), run all the tests, then stop the docker container in shutdown() method.

I think this approach is easier to maintain.

But please let me know what your thoughts are regarding this, and if you have something else in mind too!

Thanks!

deniskuzZ · 2024-10-31T11:06:25Z

hi @soumyakanti3578, sorry for not replying, please give me some time, I am overwhelmed with diff requests atm

soumyakanti3578 · 2024-10-31T17:24:31Z

@deniskuzZ No problem at all. Thanks for looking into this :)

github-actions · 2024-12-31T00:25:26Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the [email protected] list if the patch is in need of reviews.

deniskuzZ · 2025-02-09T14:03:59Z

hi @soumyakanti3578, sorry, I didn't grasp the meaning of those tests at first. I'll reopen the PR.
Basically, that would cover the tpcds queries on external JDBC tables, right?

Would it make sence to rename init script like:

q_test_tpcds_schema.postgres.sql, 
q_test_tpcds_extDB_schema-postgres.sql

deniskuzZ · 2025-02-09T14:33:41Z

itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreJdbcCliDriver.java

+        LOG.info("No jdbc init script detected. Skipping");
+        return;
+      }
+      externalDB = QTestDatabaseHandler.DatabaseType.valueOf("POSTGRES").create();


can we reuse QTestDatabaseHandler here?

Unfortunately I don't think we can reuse QTestDatabaseHandler here.
We want to create a DB in @BeforeClass method so that the tables are available for all 200 tests. Otherwise we will have to create tpcds tables and corresponding hive external tables again and again for each test file.

QTestDatabaseHandler is set up to create DB and launch docker for each individual test, as it is done in beforeTest method.

Please let me know if I am missing something, and if there is a better way to handle this.

QTestDatabaseHandler dbHandler = qt.getDatabaseHandler(); externalDB = dbHandler.initDb("postgres", scriptFile)

sonarqubecloud · 2025-02-09T16:24:55Z

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

soumyakanti3578 · 2025-02-11T05:19:55Z

Thanks @deniskuzZ for reopening the PR. I will look into your suggestions, and also the failure, and revive this PR soon. :)

deniskuzZ · 2025-03-04T10:44:03Z

hi @soumyakanti3578, please ping me directly, I might miss GitHub notifications.

sonarqubecloud · 2025-03-13T12:39:55Z

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

deniskuzZ · 2025-03-28T09:58:58Z

itests/util/src/main/java/org/apache/hadoop/hive/cli/control/AbstractCliConfig.java

@@ -345,6 +347,22 @@ protected void setInitScript(String initScript) {
      this.initScript = initScript;
    }
  }
+
+  public String getJdbcInitScript() {


can't we reuse the existing initScript?

github-actions · 2025-05-28T00:29:46Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the [email protected] list if the patch is in need of reviews.

github-actions bot requested a review from pgaref October 14, 2024 21:23

asf-ci-hive added tests pending tests unstable tests passed and removed tests pending tests unstable labels Oct 14, 2024

soumyakanti3578 force-pushed the HIVE-28576 branch from 98542c2 to eb4f642 Compare October 21, 2024 19:09

asf-ci-hive added tests pending tests passed and removed tests passed tests pending labels Oct 21, 2024

deniskuzZ closed this Oct 28, 2024

deniskuzZ reopened this Oct 28, 2024

asf-ci-hive added tests pending and removed tests passed labels Oct 28, 2024

asf-ci-hive added tests passed and removed tests pending labels Oct 28, 2024

github-actions bot added the stale label Dec 31, 2024

github-actions bot closed this Jan 7, 2025

deniskuzZ reopened this Feb 9, 2025

deniskuzZ removed the stale label Feb 9, 2025

asf-ci-hive added tests pending and removed tests passed labels Feb 9, 2025

deniskuzZ reviewed Feb 9, 2025

View reviewed changes

asf-ci-hive added tests failed and removed tests pending labels Feb 9, 2025

soumyakanti3578 force-pushed the HIVE-28576 branch from eb4f642 to 600367c Compare March 13, 2025 00:46

asf-ci-hive added tests pending tests failed and removed tests failed tests pending labels Mar 13, 2025

soumyakanti3578 added 4 commits March 12, 2025 23:50

HIVE-28576: Add jdbc tests for tpcds queries

f4cfe8a

update conf and q.outs

d577d13

Refactor and rebase

59fadac

plan changes due to rebase

e336caf

soumyakanti3578 force-pushed the HIVE-28576 branch from 600367c to e336caf Compare March 13, 2025 07:00

asf-ci-hive added tests pending and removed tests failed labels Mar 13, 2025

asf-ci-hive added tests unstable and removed tests pending labels Mar 13, 2025

deniskuzZ reviewed Mar 28, 2025

View reviewed changes

github-actions bot added the stale label May 28, 2025

github-actions bot closed this Jun 4, 2025

HIVE-28576: Add jdbc tests for tpcds queries #5510

HIVE-28576: Add jdbc tests for tpcds queries #5510

Uh oh!

Conversation

soumyakanti3578 commented Oct 14, 2024

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

Is the change a dependency upgrade?

How was this patch tested?

Uh oh!

deniskuzZ commented Oct 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

soumyakanti3578 commented Oct 24, 2024

Uh oh!

deniskuzZ commented Oct 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sonarqubecloud bot commented Oct 28, 2024

Quality Gate passed

Uh oh!

soumyakanti3578 commented Oct 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

deniskuzZ commented Oct 31, 2024

Uh oh!

soumyakanti3578 commented Oct 31, 2024

Uh oh!

github-actions bot commented Dec 31, 2024

Uh oh!

deniskuzZ commented Feb 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

deniskuzZ Feb 9, 2025

Choose a reason for hiding this comment

Uh oh!

soumyakanti3578 Mar 13, 2025

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Mar 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Feb 9, 2025

Quality Gate passed

Uh oh!

soumyakanti3578 commented Feb 11, 2025

Uh oh!

deniskuzZ commented Mar 4, 2025

Uh oh!

sonarqubecloud bot commented Mar 13, 2025

Quality Gate passed

Uh oh!

deniskuzZ Mar 28, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented May 28, 2025

Uh oh!

Uh oh!

deniskuzZ commented Oct 24, 2024 •

edited

Loading

deniskuzZ commented Oct 28, 2024 •

edited

Loading

soumyakanti3578 commented Oct 29, 2024 •

edited

Loading

deniskuzZ commented Feb 9, 2025 •

edited

Loading

deniskuzZ Mar 23, 2025 •

edited

Loading