Benchmark revamp + run benchmark as part of CI #3176
Conversation
stefannibrasil
force-pushed
the
sb-3159-benchmark-revamp
branch
9 times, most recently
from
8a7e4f2 to
a937aac
Compare
| @@ -0,0 +1,40 @@ | |||
| # frozen_string_literal: true | |||
|
|
|||
There was a problem hiding this comment.
Moved from the previous benchmark rake task
stefannibrasil
force-pushed
the
sb-3159-benchmark-revamp
branch
2 times, most recently
from
0d24c5e to
ca35c75
Compare
Inspired by ruby/json#606
Having these in a folder helps because we can document experiment results in it as well. And we can edit the require script to raise an error if it takes longer than a threshold to load faker. Co-Authored-By: Thiago Araujo <[email protected]>
stefannibrasil
force-pushed
the
sb-3159-benchmark-revamp
branch
from
5fee76f to
a109a55
Compare
| branches: [ main ] | ||
| pull_request: | ||
| branches: [ main ] | ||
|
|
There was a problem hiding this comment.
maybe
| permissions: | |
| contents: read | |
| eval("Faker::#{subclass}.public_methods(false) - Faker::Base.public_methods(false)").sort.map do |method| | ||
| "Faker::#{subclass}.#{method}" | ||
| end.sort | ||
| end |
There was a problem hiding this comment.
maybe we don't need eval here - we could loop over these constants, do a const_get to get the class, then list its public methods to get generators.
def subclass_methods(subclass)
klass = Faker.const_get(subclass)
public_methods = klass.public_methods(false) - Faker::Base.public_methods(false)
generators = public_methods.sort.map do |method|
[klass, method]
end
end
| x.report("Number of generators: #{all_generators.count}") do | ||
| all_generators.each { |generator| eval(generator) } | ||
| end |
There was a problem hiding this comment.
we should build the list of generators outside so that we're only benchmarking generator execution
| x.report("Number of generators: #{all_generators.count}") do | |
| all_generators.each { |generator| eval(generator) } | |
| end | |
| generators = all_generators | |
| x.report("Number of generators: #{all_generators.count}") do | |
| generators.each { |klass, generator| klass.send(generator) } | |
| end |
| x.report('JSON') { JSON.load_file("#{File.dirname(__FILE__)}/../test/fixtures/locales/es-MX.json") } | ||
|
|
||
| x.compare!(order: :baseline) | ||
| end |
There was a problem hiding this comment.
Do we still want to keep this? Benchmarking json vs yaml load times is not really relevant to Faker, this was just an experiment.
| Description: The use of eval represents a serious security risk. | ||
| Exclude: | ||
| - 'lib/faker/default/json.rb' | ||
| - 'benchmark/generators.rb' |
There was a problem hiding this comment.
maybe we don't need eval on generators, see previous comment
thdaraujo
left a comment
thdaraujo
left a comment
There was a problem hiding this comment.
Nice work on this! do you mind adding to the description the machine you're using so we can compare apples to apples?
e.g. Apple M1 16GB memory on MacOS X.Y.Z
Motivation / Background
We want to run some experiments to improve the library's performance needs and before making any changes, we need to have baseline stats to guarantee new code does not degrade performance. Plus, we want to have benchmark scripts to eventually be part of our CI.
To keep everything in a single place, I moved the previous benchmark tasks to a folder. The goal is to use the folder to document experiment results.
Closes #3159, #3160
Inspired by ruby/json#606.
Results from running the scripts (January 12th, 2026):
Require
Load locales - YML vs JSON
Generators