Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Hikari Connection Pool for handling Clickhouse connections #893

Merged
Merged
Changes from 1 commit
Commits
Show all changes
118 commits
Select commit Hold shift + click to select a range
eb46e45
Use Hikari Connection Pool for handling Clickhouse connections
subkanthi Oct 31, 2024
68dad63
Fixed imports
subkanthi Oct 31, 2024
af91740
Changes to use Hikari connection pool, Replaced ClickHouseConnection …
subkanthi Nov 8, 2024
c47d513
Set connection timeout in hikari pool to 50 seconds
subkanthi Nov 8, 2024
83cfcd4
Set connection pool to 100 in hikari
subkanthi Nov 8, 2024
25b85a6
Updated version of upload-artifact to v4.
subkanthi Jan 23, 2025
03beb53
Merged conflicts with 2.5.1 branch.
subkanthi Jan 26, 2025
b8285ef
Merged conflicts with 2.5.1 branch.
subkanthi Jan 26, 2025
a5a2bb4
Changes to all integration tests to move the common logic of creating…
subkanthi Jan 27, 2025
ec2d634
Changes to all integration tests to move the common logic of creating…
subkanthi Jan 27, 2025
62ce23d
Changes to all integration tests to move the common logic of creating…
subkanthi Jan 27, 2025
00032c2
Changes to support one HikariDbSource for one database.
subkanthi Jan 27, 2025
67dc11c
Fix integration tests.
subkanthi Feb 3, 2025
538428f
Fix integration tests.
subkanthi Feb 3, 2025
a47ea48
Fix integration tests.
subkanthi Feb 4, 2025
19b242e
Fix integration tests.
subkanthi Feb 4, 2025
65299f0
Fix integration tests.
subkanthi Feb 4, 2025
fd103ef
DBWriter - Add step to create destination database.
subkanthi Feb 5, 2025
4b7eb6b
revert docker-compose changes.
subkanthi Feb 5, 2025
b435571
Add prometheus metrics for Hikari connection pool.
subkanthi Feb 5, 2025
3f2b7c4
Set connection pool name and updated grafana dashboard.
subkanthi Feb 5, 2025
5c36c44
Modified integration tests to close hikari connection pool.
subkanthi Feb 6, 2025
a4e070c
Modified integration tests to close hikari connection pool.
subkanthi Feb 6, 2025
c4477c8
Modified integration tests to close hikari connection pool.
subkanthi Feb 6, 2025
6c1cc81
add connection.pool.max.size to lightweight
Feb 6, 2025
9b4f5d0
Modified integration tests to close hikari connection pool.
subkanthi Feb 6, 2025
f354e45
Merge remote-tracking branch 'origin/867-potential-connection-leak-in…
subkanthi Feb 6, 2025
96fca9c
Modified integration tests to close hikari connection pool.
subkanthi Feb 6, 2025
79d666a
Modified integration tests to close hikari connection pool.
subkanthi Feb 6, 2025
8d3a241
Modified integration tests to close hikari connection pool.
subkanthi Feb 6, 2025
a5ac62f
Modified integration tests to close hikari connection pool.
subkanthi Feb 6, 2025
b9783a6
fixed integration tests.
subkanthi Feb 6, 2025
a6c5794
fixed integration tests.
subkanthi Feb 6, 2025
9557020
fixed integration tests.
subkanthi Feb 6, 2025
3af4ed6
fixed integration tests.
subkanthi Feb 6, 2025
fc7cb34
fixed integration tests.
subkanthi Feb 6, 2025
31a4363
fixed integration tests.
subkanthi Feb 7, 2025
ce16a4e
fixed integration tests.
subkanthi Feb 7, 2025
00372c4
try different values for `connection.pool.max.size`
Feb 7, 2025
1bdf393
Added logic to pause/resume thread pool instead of shutting it down.
subkanthi Feb 7, 2025
3588ce3
Merge remote-tracking branch 'origin/867-potential-connection-leak-in…
subkanthi Feb 7, 2025
a5251ab
Remove connection.close in tests.
subkanthi Feb 7, 2025
42e1f9d
Fixed select with db name.
subkanthi Feb 7, 2025
f803a87
Fixed select with db name.
subkanthi Feb 7, 2025
7035494
Fixed select with db name.
subkanthi Feb 7, 2025
ef8db8f
Fixed select with db name.
subkanthi Feb 7, 2025
ad26764
Fixed select with db name.
subkanthi Feb 7, 2025
db49449
Fixed select with db name.
subkanthi Feb 7, 2025
779a82b
Fixed select with db name.
subkanthi Feb 7, 2025
530795c
Fixed select with db name.
subkanthi Feb 7, 2025
d933913
Fixed select with db name.
subkanthi Feb 7, 2025
abdf30f
Fixed select with db name.
subkanthi Feb 7, 2025
2ca0cd8
Fixed select with db name.
subkanthi Feb 7, 2025
c541b38
Disabled MongoDBIT
subkanthi Feb 8, 2025
1373505
Added support for drop database.
subkanthi Feb 10, 2025
e687771
Fix integration tests.
subkanthi Feb 10, 2025
bc0e842
Merge branch 'develop' of github.com:Altinity/clickhouse-sink-connect…
subkanthi Feb 10, 2025
643ec80
Added keepalive timeout as jdbc parameters
subkanthi Feb 10, 2025
f9ee5b7
Removed mlruns folder
subkanthi Feb 10, 2025
0de836f
Removed mlartifacts
subkanthi Feb 10, 2025
d9e65f8
Removed ipynb
subkanthi Feb 10, 2025
e7d7b51
Merge pull request #988 from Altinity/add_jdbc_params
subkanthi Feb 10, 2025
e197366
Update Troubleshooting.md
subkanthi Feb 10, 2025
e3097cb
Merge pull request #989 from Altinity/subkanthi-patch-4
subkanthi Feb 10, 2025
07076c2
mvoe to debug mode
Feb 12, 2025
91fd5c9
Refactor, so that all the Connection objects are co-located in DBMeta…
subkanthi Feb 12, 2025
4d43308
Merge remote-tracking branch 'origin/867-potential-connection-leak-in…
subkanthi Feb 12, 2025
3608f0c
Fix integration tests.
subkanthi Feb 12, 2025
97ce8d7
Fixed version call and moved it to initialization logic.
subkanthi Feb 12, 2025
b667df8
Added logic to create database.
subkanthi Feb 13, 2025
318a46f
Added logic to recreate connection incase of failure.
subkanthi Feb 13, 2025
fd0eb8b
Added logic to retry for all Connection calls.
subkanthi Feb 13, 2025
271073b
Added break for retries.
subkanthi Feb 13, 2025
d0dc7b9
update clickhouse version to 23.8
Feb 13, 2025
9e5fe44
use MAX_RETRIES config and moved insert query preparedstatement to DB…
subkanthi Feb 14, 2025
2899ee8
Merge remote-tracking branch 'origin/867-potential-connection-leak-in…
subkanthi Feb 14, 2025
debe848
Refactor DebeziumChangeEventCapture to retain one single connection.
subkanthi Feb 14, 2025
88822bc
Refactor DebeziumChangeEventCapture to retain one single connection.
subkanthi Feb 14, 2025
87798ad
remove pool max size setting for check
Feb 17, 2025
b99519b
Removed set idle timeout from 10 secs to default 10 mins.
subkanthi Feb 18, 2025
e1dac7d
Merge remote-tracking branch 'origin/867-potential-connection-leak-in…
subkanthi Feb 18, 2025
fa836a6
temporarily comment out the test that causes mysql restart
Feb 18, 2025
474d6d0
Merge remote-tracking branch 'origin/867-potential-connection-leak-in…
Feb 18, 2025
c6cd999
Merge branch 'develop' of github.com:Altinity/clickhouse-sink-connect…
subkanthi Feb 18, 2025
5a202ef
print connection info and fixed sink connector integration tests.
subkanthi Feb 18, 2025
63baddd
Merge remote-tracking branch 'origin/867-potential-connection-leak-in…
subkanthi Feb 18, 2025
4404c73
mount log4js
Feb 19, 2025
5829cdd
Set Max connections to 100
subkanthi Feb 19, 2025
2ce080c
Set Max connections to 100
subkanthi Feb 19, 2025
af01738
use custom http client okhttp.
subkanthi Feb 19, 2025
0df4b02
Set http connection provider to HTTP_URL_CONNECTION
subkanthi Feb 19, 2025
75784a0
Set http connection provider to HTTP_URL_CONNECTION
subkanthi Feb 19, 2025
dc559fe
Set http connection provider to HTTP_URL_CONNECTION
subkanthi Feb 20, 2025
b722ad8
Added support to disable connection pool.
subkanthi Feb 20, 2025
8925a98
Fixed default value for offset storage table
subkanthi Feb 20, 2025
55c069c
update with new runs that have hikari pool enabled and disable
Feb 21, 2025
859f019
try to trigger ci/cd
Feb 21, 2025
406e0b5
fix inputs
Feb 21, 2025
db9f026
fix inputs
Feb 21, 2025
91e83e5
fix arguments
Feb 21, 2025
c762010
fix arguments
Feb 21, 2025
a96b9bf
integration test changes.
subkanthi Feb 21, 2025
a9e6216
Merge remote-tracking branch 'origin/disable_connection_pool' into di…
subkanthi Feb 21, 2025
4691b5b
Ignore creating offset database if its running in kafka connect.
subkanthi Feb 21, 2025
1d710d3
Merge branch '867-potential-connection-leak-in-createconnection-metho…
subkanthi Feb 21, 2025
80979de
pass logger to print connector type exception.
subkanthi Feb 21, 2025
5105d0e
Added configuration for connector.class.
subkanthi Feb 21, 2025
c22c6f7
Update configurations.py
Selfeer Feb 21, 2025
5a1d10e
increase heap size to 5gb
Feb 22, 2025
d4f24e1
disable pool
Feb 22, 2025
28275f8
update heap size to 5gb and disable pool
Selfeer Feb 22, 2025
e8c352e
Merge pull request #992 from Altinity/disable_connection_pool
subkanthi Feb 23, 2025
6e7e444
Fix integration tests and added connection.pool.disable to kafka.
subkanthi Feb 23, 2025
b23529d
Merge remote-tracking branch 'origin/867-potential-connection-leak-in…
subkanthi Feb 23, 2025
b865e9e
Added logic to not retry hikari when pool is disabled.
subkanthi Feb 24, 2025
38094de
Fixed Integration tests.
subkanthi Feb 24, 2025
fd9b58f
Fixed Integration tests.
subkanthi Feb 24, 2025
3f876df
Merge branch '2.5.1' into 867-potential-connection-leak-in-createconn…
subkanthi Feb 24, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fixed imports
subkanthi committed Oct 31, 2024
commit 68dad637e880b9d75c31b2b8f4efdf4ec735f81d
Original file line number Diff line number Diff line change
@@ -7,26 +7,37 @@
import java.sql.Connection;
import java.sql.SQLException;

// Singleton class to manage the Hikari connection pool
// Singleton class.
public class HikariDbSource {

private static HikariDbSource instance;

private HikariDataSource dataSource;

// private constructor
private HikariDbSource(ClickHouseDataSource dataSource) {
this.createConnectionPool(dataSource);
}

public HikariDbSource(HikariDataSource dataSource, String jdbcUrl, String username, String password) {
public Connection getConnection() throws SQLException {
return this.dataSource.getConnection();
}

this.dataSource = dataSource;
public static HikariDbSource getInstance(ClickHouseDataSource dataSource) {
if (instance == null) {
instance = new HikariDbSource(dataSource);
}
return instance;
}
public void createConnectionPool(ClickHouseDataSource dataSource) {
// pass the clickhouse config to create the datasource


HikariConfig poolConfig = new HikariConfig();
poolConfig.setConnectionTimeout(5000L);
poolConfig.setMaximumPoolSize(20);
poolConfig.setMaxLifetime(300_000L);
poolConfig.setDataSource(new ClickHouseDataSource(dataSource.getJdbcUrl(), dataSource.getUsername(), dataSource.getPassword()));
poolConfig.setDataSource(dataSource);

HikariDataSource ds = new HikariDataSource(poolConfig);
}

public Connection getConnection() throws SQLException {
return this.dataSource.getConnection();
}
this.dataSource = new HikariDataSource(poolConfig);
}
}
Loading