Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(disableEval): add static parsers #3365

Open
wants to merge 19 commits into
base: master
Choose a base branch
from

Conversation

thomasgauvin
Copy link

@thomasgauvin thomasgauvin commented Jan 29, 2025

I'm hoping to help push for getting the interpreting parser available (this makes it possible to use mysql2 as an alternative to mysql on Cloudflare Workers).

I opened this as a new PR because I have some bandwidth to address further comments in the PR if they come up. This code merely applies the recommended fixes as noted by @shiyuhang0 to the original PR by @sidorares #2099 (the original commit history is maintained and the branch was rebased onto master).

This could also be created as a PR to deoptimise-large-rows branch, however the maintainers recommend moving forward.

Copy link

codecov bot commented Jan 29, 2025

Codecov Report

Attention: Patch coverage is 97.08738% with 12 lines in your changes missing coverage. Please review.

Project coverage is 88.70%. Comparing base (f2cc820) to head (61ca972).

Files with missing lines Patch % Lines
lib/parsers/static_binary_parser.js 95.26% 10 Missing ⚠️
lib/commands/query.js 95.23% 1 Missing ⚠️
lib/connection_config.js 94.73% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3365      +/-   ##
==========================================
+ Coverage   88.45%   88.70%   +0.25%     
==========================================
  Files          83       85       +2     
  Lines       13067    13446     +379     
  Branches     1393     1531     +138     
==========================================
+ Hits        11558    11927     +369     
- Misses       1509     1519      +10     
Flag Coverage Δ
compression-0 88.70% <97.08%> (+0.25%) ⬆️
compression-1 88.70% <97.08%> (+0.25%) ⬆️
static-parser-0 86.28% <18.20%> (?)
static-parser-1 87.09% <96.84%> (?)
tls-0 88.11% <97.08%> (+0.26%) ⬆️
tls-1 88.47% <97.08%> (+0.25%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@wellwelwel
Copy link
Collaborator

wellwelwel commented Jan 29, 2025

Thanks, @thomasgauvin! And sorry, january was a hectic month and I didn't really pay attention to your comment in #2289 (comment).

lib/parsers/static_text_parser.js 11.26% 126 Missing ⚠️

I'll add the label needs tests, but I don't intend to block this PR progress because of this. Probably tomorrow I'll go through the tests that currently use evaluation and adapt them for the new parser.

Perhaps a simple approach with parameterized tests can make things easier to avoid duplicate tests.

Copy link
Collaborator

@wellwelwel wellwelwel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@thomasgauvin, I added some security points that were resolved after PR #2099 was created.

If it's not a problem, can I commit directly to your branch (including the typeCast / nestTables question and to add the tests I mentioned yesterday)?

lib/connection_config.js Outdated Show resolved Hide resolved
lib/parsers/static_text_parser.js Outdated Show resolved Hide resolved
lib/parsers/static_text_parser.js Show resolved Hide resolved
@wellwelwel
Copy link
Collaborator

Also, in #2289 it was decided to use disableEval instead of useStaticParser (I'm happy with both, to be honest):

@wellwelwel wellwelwel changed the title Add "interpreting" parser v2 feat: add static parser Jan 30, 2025
@wellwelwel wellwelwel changed the title feat: add static parser feat(query): add text static parser Jan 30, 2025
@wellwelwel wellwelwel changed the title feat(query): add text static parser feat(query): add static parser Jan 30, 2025
@thomasgauvin
Copy link
Author

@thomasgauvin, I added some security points that were resolved after PR #2099 was created.

If it's not a problem, can I commit directly to your branch (including the typeCast / nestTables question and to add the tests I mentioned yesterday)?

Yes, please feel free to commit anything to the branch directly. I've added you as collaborator on the fork so you should have commit access.

@thomasgauvin
Copy link
Author

Also, in #2289 it was decided to use disableEval instead of useStaticParser (I'm happy with both, to be honest):

Missed that, will adjust

thomasgauvin and others added 2 commits January 30, 2025 15:44
@wellwelwel

This comment was marked as resolved.

@wellwelwel
Copy link
Collaborator

wellwelwel commented Jan 31, 2025

@sidorares, as the query (and eventually execute) method is one of the most critical in the entire project, I chose not to create an exclusive test for the static parser, but rather to test the entire test suite against this option, including all Node.js, Bun, and Deno versions.

@thomasgauvin, I won't be committing until Saturday. Feel free to continue or wait 🙋🏻‍♂️

  • For the reasons in my comment above, naturally all CI using this option will fail at the moment.
  • I'll wait to add this variation to the coverage CI when all the tests have passed.

@wellwelwel

This comment was marked as resolved.

@wellwelwel wellwelwel changed the title feat(query): add static parser feat(disableEval): add static parser Feb 2, 2025
@wellwelwel
Copy link
Collaborator

wellwelwel commented Feb 2, 2025

Just inspired on an Sunday dawn 🌌

Static binary parser implementation

I've implemented the new binary static parser preserving the code as faithful as possible to the original, except for two things:

Unified/simplified the typeCast source logic (globally or via execute method):

const typeCast =
  options.typeCast !== undefined ? options.typeCast : config.typeCast;

A minimal memory improvement, by pre-allocating the length of null bitmaps instead of []:

const nullBitmapLength = Math.floor((fields.length + 7 + 2) / 8);
const nullBitmaskBytes = new Array(nullBitmapLength); // ⬅️

for (let i = 0; i < nullBitmapLength; i++) {
  nullBitmaskBytes[i] = packet.readInt8();
}

Testing

As previously mentioned, there are no new tests due to new parsers. Instead, every single test, including the matrixes (CI), platforms and their respective versions (Node.js, Bun, Deno), as well as the entire test suite are tested against the new parsers globally.

Note

Although this is a rather intensive approach, it ensures that a modification in one parser will cause failures if the other parser is not updated accordingly, as well as being easier to maintain in the long term by allowing the same test to be run dynamically for each variation.


Coverage

As coverage is tested on different matrixes (CI), naturally the percentage when testing only standard or static parsers will reduce the total coverage count. For this reason, I adapted the limits to support this difference.


As far as I'm concerned, it's ready to merge, so please feel free to review 🙋🏻‍♂️

@wellwelwel wellwelwel changed the title feat(disableEval): add static parser feat(disableEval): add static parsers Feb 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants