Backport performance improvements to Foxy

[emersonknapp]

Description

Discussion overview at https://discourse.ros.org/t/fast-forward-merging-rosbag2-master-api-to-foxy/18927

Summary: major performance improvements on master make rosbag2 usable, compared to the Foxy release. This ticket tracks getting the performance improvement onto the Foxy branch (one way or another).

The https://github.com/ros2/rosbag2/tree/foxy-future branch exists with all development up to mid-March 2021, and can be built from source for users who want all improvements

For the default user who installs from APT, however, we want to do the best we can to provide them some of the performance improvements that were discovered.

Related Issues

• [foxy backport] Latest rosbag2 #625
• Performance testing of rosbag2 #435
• "Futurize" foxy-future branch #687

Completion Criteria

Backport the following PRs, without breaking API/ABI, to Foxy and get it into a sync release. Some may prove difficult and need re-writing on the Foxy branch, this is OK to target a more minimal patch improvement into the stable distro.

• SQlite config optimized by default SQLite storage optimized by default #568
• Double buffering for async writes
  • feature Sqlite storage double buffering #546
  • Thread sync fixes Mutex protection for db writing and stl collections in writer & storage #603
• User configuration of SQlite3
  • add storage_config_uri #493
  • read yaml config file #497

Testing Notes / Suggestions

All existing test suites should run and the build should have no test failures or compiler warnings on the build farm.

[emersonknapp]

Updated issue description with a concrete list of PRs to backport

[ros-discourse]

This issue has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/fast-forward-merging-rosbag2-master-api-to-foxy/18927/49

[emersonknapp]

Closed with foxy-future branch