Add option to balance jobs by example instead of by file #105
Conversation
The purpose of this change is to allow more optimal distribution when the suite contains outlier files. With enough cores on a CI run we noticed that the lower bound of the duration of the full run was the runtime of the slowest file. In order to be able to split by example, ExampleJobBuilder needs to load all spec files and extract examples from them. As a side effect, this unlocks more filtering options (such as by tag) which previously would silently not work as expected.
| # Load spec files in a fork to avoid polluting the parent process, | ||
| # otherwise the actual execution will return warnings for redefining constants | ||
| # and shared example groups. | ||
| @examples_to_run = within_forked_process { load_examples_to_run(configuration) } |
There was a problem hiding this comment.
This was a bit tricky - I need to access RSpec.world.ordered_example_groups which only works after configuration.load_spec_files, but calling that loads rails_helper (etc) which can define some constants, and then the actual execution in workers would load these files producing warnings.
Would be great if there was another way to do it, but for now the least-bad idea I had was to get the list of examples from a forked process.
grk
force-pushed
the
balance-examples
branch
from
8c6a505 to
47f4371
Compare
grk
force-pushed
the
balance-examples
branch
from
47f4371 to
e2bee92
Compare
|
This is very cool. "near-perfect distribution" is exciting! Before this goes to master I'd like to see a couple changes.
|
|
Up for this pull request seems very promising 👍🏻 |
3 participants
Hi,
I'm opening this PR to start a discussion, but we've been using this branch for a few weeks without any issues.
The initial motivation for this PR was our suite having large spec files that were the lower bound of the time needed for a complete CI run, and we couldn't scale out any further. With switching to balancing by example, we achieve near-perfect distribution of work per worker and finish the run in optimal time.
As a bonus, this allows us to run that large single file spec on multiple processes in local development without having to use something like parallel_split_tests.
There's a new flag added to the rspec command -
--job-builderwhich by default usesFileJobBuilder(the current implementation) and optionally allows the user to setExampleJobBuilder, the new implementation.