Change SimulationResultsWriter to write more directly to file #1763
Conversation
Changes SimulationResultsWriter to write output JSON more directly to the chosen output file (or stdout). It no longer builds in-memory JsonValue nor String representations of large chunks of the output. Instead, the resultsGenerator is connected directly to the chosen output stream, and build* subroutines to build in-memory JsonValue objects are replaced by corresponding write* subroutines to write that section of the output directly to file. When using a ResourceFileStreamer, this means opening only one resource temp file at a time and streaming data from that file to the final output file, never loading the full resource profile into memory. This makes stateless Aerie capable of handling very large simulation results. This was tested with a Clipper plan that generated a nearly 14GB output file.
When rewriting the SimulationResultsWriter for performance, I also corrected the way it handles topics and events to maintain parity with the database-based simulation results.
DavidLegg
force-pushed
the
feature/stateless-direct-to-file
branch
from
d6fe5db to
3f5b90a
Compare
|
|
||
| // Append to the array builder | ||
| simulatedActivitiesBuilder.add(actBuilder); | ||
| resultsGenerator |
There was a problem hiding this comment.
Minor, nonblocking: the initial order of fields for directives were chosen deliberately so that all of the ID fields were next to each other at the top of the entry. Could you restore this order?
| // Sadly, this requires reading the object into a JsonValue, just to write it back out (!) | ||
| try (final JsonReader jr = Json.createReader(new StringReader(s))) { | ||
| segmentsBuilder.add(jr.readObject()); | ||
| resultsGenerator.write(jr.readValue()); |
There was a problem hiding this comment.
Just a comment, no changes: Shame there's no writeRaw method like in Jackson's implementation. That said, the reader and JSON Value only exist for the length of the loop, so memory should be fine.
| } | ||
|
|
||
| /* | ||
| Json Schema for Sim results: |
There was a problem hiding this comment.
Could you update the JSON Schema to reflect the new pattern for topics and events?
Mythicaeda
left a comment
Mythicaeda
left a comment
There was a problem hiding this comment.
Thanks for improving the memory use! I think we can revisit parallelization/streaming as we go once that starts being an issue with stateless sim.
I'm only holding off on approval because of the schema docs. It's not the best place to have them, I know, but they ought to stay up-to-date while we find them a new home.
Additionally for @dandelany: while it can still only sim, the change to events/topics is the one change that I really wanted done before we "announced" Stateless Aerie/took it out of "beta"/whatever the appropriate phrase is.
Description
Changes SimulationResultsWriter to write output JSON more directly to the chosen output file (or stdout).
It no longer builds in-memory JsonValue nor String representations of large chunks of the output.
Instead, the resultsGenerator is connected directly to the chosen output stream, and build* subroutines to build in-memory JsonValue objects are replaced by corresponding write* subroutines to write that section of the output directly to file.
When using a ResourceFileStreamer, this means opening only one resource temp file at a time and streaming data from that file to the final output file, never loading the full resource profile into memory.
This makes stateless Aerie capable of handling very large simulation results.
Verification
This was tested with a Clipper plan that generated a nearly 14GB output file.
Documentation
N/A - purely a performance-oriented refactor
Future work
There are potentially more performance gains to be had by changing the output format to one that can be written incrementally. Something like a JSON Lines file where each line is a result, in time order, and a result can be any update from the simulation: activity start or end, event on a topic, or change in resource dynamics.
Such an output file could be written as the simulation runs. Combined with the results-streaming interface that's already in place, and we could write the final output file in parallel with simulation, with no need to run a post-processing step.