Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bulk load for MySQL #501

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

bulk load for MySQL #501

wants to merge 1 commit into from

Conversation

murfffi
Copy link
Contributor

@murfffi murfffi commented Dec 2, 2024

The PR implements bulk loading for MySQL using the "LOAD DATA from io.Reader" feature of github.com/go-sql-driver/mysql - https://github.com/go-sql-driver/mysql?tab=readme-ov-file#load-data-local-infile-support .
As expected, bulk loading this way is significantly faster. 1 mln. rows in the "staff" table from the test schema are inserted for 15 sec vs. 120 sec using INSERT: 8x improvement. Note that LOAD DATA INFILE LOCAL is disabled by default on MySQL 8+ servers and must be enabled using SET GLOBAL local_infile = ON beforehand. MySQL doesn't seem to have any remote bulk loading options that are enabled by default.

The PR also extends TestCopy in drivers_test.go with comparison of copied data to ensure MySQL bulk loading is safe across data types.

Testing Done: tests in drivers_test.go

@murfffi murfffi marked this pull request as draft December 2, 2024 16:39
@murfffi murfffi force-pushed the fix/458-mysql branch 3 times, most recently from 6f73721 to 2e5d24c Compare December 3, 2024 07:02
@murfffi murfffi marked this pull request as ready for review December 5, 2024 09:45
@murfffi
Copy link
Contributor Author

murfffi commented Dec 5, 2024

This is ready for review.

Background: I recently resumed working on #458 and while looking for a database to experiment with generic bulk copy, I saw that MySQL supports bulk load similar to Postgres. I decided to implement that first.

@murfffi murfffi changed the title [in progress] Bulk load for MySQL bulk load for MySQL Dec 5, 2024
@murfffi murfffi force-pushed the fix/458-mysql branch 2 times, most recently from cae603a to f391e8f Compare December 6, 2024 15:16
The PR implements bulk loading for MySQL using the "LOAD DATA from io.Reader" feature of github.com/go-sql-driver/mysql - https://github.com/go-sql-driver/mysql?tab=readme-ov-file#load-data-local-infile-support . As expected,
bulk loading this way is significantly faster. 1 mln. rows in the "staff" table from the test schema are inserted for 15 sec vs. 120 sec using INSERT: 8x improvement. Note that LOAD DATA INFILE LOCAL is disabled by default on
MySQL 8+ servers and must be enabled using SET GLOBAL local_infile = ON beforehand. MySQL doesn't seem to have any remote bulk loading options that are enabled by default.

The PR also extends TestCopy in drivers_test.go with comparison of copied data to ensure MySQL bulk loading is safe across data types.

Testing Done: tests in drivers_test.go#
@murfffi
Copy link
Contributor Author

murfffi commented Dec 7, 2024

The PR checks failure seems unrelated. The checks on master fail the same way. I got them to pass in #503 but that PR probably can't be merged as is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant