Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exponential growth in precision causing memory usage to spike as of 3.1.2 #222

Open
smidwap opened this issue Feb 25, 2022 · 11 comments · May be fixed by #240
Open

Exponential growth in precision causing memory usage to spike as of 3.1.2 #222

smidwap opened this issue Feb 25, 2022 · 11 comments · May be fixed by #240
Assignees
@mrkn
Labels
bug

Comments

@smidwap
Copy link

smidwap commented Feb 25, 2022

I upgraded to BigDecimal 3.1.2 to pull in the fix to #220, which was causing crashes in our Rails app. However, I believe the fix applied in 127a1b5 created the potential for exponential growth in the number of digits a BigDecimal can have. The result is that certain calculations can eat up all available memory (several gigs) and crash the process.

Take the following as an example, which outputs the number of digits from the calculation:

Array.new(10) { BigDecimal("1.03") }.inject(1.0) { |result, num| result / num }.to_digits.length

In 3.1.1, this calculation produces 119 digits.

In 3.1.2, this calculation produces 23,033 digits.

Moreover, if Array.new(10) is grown to Array.new(20), 3.1.1 produces 209 digits, whereas 3.1.2 produces 23,592,953.

The runaway growth in digits and resulting memory spike could be solved by rounding the result in inject's block like so:

Array.new(10) { BigDecimal("1.03") }.inject(1.0) { |result, num| (result / num).round(3) }.to_digits.length

But others may run into this same surprise. We've also run into PG::NumericValueOutOfRange exceptions from postgres since upgrading to 3.1.2, and my hunch is the same root issue is at play.

Let me know if I can provide any more details.
@mrkn mrkn added the bug label Mar 6, 2022
@mrkn mrkn self-assigned this Mar 6, 2022
@mrkn
Copy link
Member

mrkn commented Mar 6, 2022

@smidwap Thank you for reporting this issue. I'll investigate the details.

@mrkn mrkn linked a pull request Nov 13, 2022 that will close this issue
Draft
@Confusion
Copy link

We are running into this issue; would it be possible to merge and release this fix?

@mrkn
Copy link
Member

mrkn commented Apr 15, 2023

@Confusion I'm working on this issue in #240. But the work is temporarily paused now because I'm too busy. Could you please use 3.1.1 until it's fixed? I'm sorry for the inconvenience.

@TomNaessens
Copy link

Downgrading to 3.1.1 will re-introduce the segfaults from #220 (which in my opinion is less preferred than memory spikes). We'll try to downgrade to 3.1.0 until this is fixed.

This was referenced Dec 11, 2023
Closed
Open
@israelb israelb mentioned this issue Dec 27, 2023
Open
@owst
Copy link

owst commented Jan 15, 2024

Hi @mrkn - have you made any further progress in #240? This issue and #271 seem quite serious bugs - especially given Ruby 3.2.2 and 3.3.0 both include an affected version as a default gem.

@broksonic21
Copy link

I don't usually +1 but any chance there's a fix coming for this? We've had to avoid all future patches until this is fixed and would love to keep up

@Confusion
Copy link

I have some ideas for workarounds that I haven't been able to test yet:

  • instrument your calculations to report the precisions of the bigdecimal values that are used when using bigdecimal 3.1.1. Then set BigDecimal.limit to the highest precision you find. The calculation should then give the same result in 3.1.2+ without blowing the memory?

  • determine a suitable value for BigDecimal.limit by trial-and-error: if you set it to some value and your results don't change, you're good?

  • if you have a very good understanding of exactly what your calculations do, you could calculate the maximum precision you should need to prevent rounding errors from influencing your final result

@alexshagov
Copy link

Future patches are blocked by this issue. Kindly asking to re-assess the severity

@mrkn
Copy link
Member

mrkn commented Nov 16, 2024

@alexshagov I'm sorry for the inconvenience. I'm still too busy to make progress on #240 in my personal time. Now, I can only handle minor bug fixes and review pull requests to decide whether to merge or reject them. This situation is likely to continue for more than a year.

I'd be eager to review and incorporate the patch if someone else could fix this bug.

@Kallin
Copy link

Kallin commented Nov 27, 2024

@mrkn, I'm sorry to hear that you're unable to make any progress here. I hope everything is OK in your personal life and that you have just adjusted your priorities. Do you think another maintainer, perhaps @nobu, could be made the assignee of this issue?

@mrkn
Copy link
Member

mrkn commented Nov 27, 2024

@Kallin I don't mind if @nobu wants.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mrkn mrkn

Labels
bug
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix divison ruby/bigdecimal
8 participants
@mrkn @owst @Kallin @Confusion @smidwap @TomNaessens @alexshagov @broksonic21