Class StatesDocument (#3902)

* Added class StatesDocument

* Update StatesDocument.cpp

- Modified the names of the Trajectory methods to match the latest names in Component.h.
- Reintroduced the precision argument in SimTK::Xml::Elment::setValueAs<T>.

* Create testStatesDocument.cpp

Added a place holder for the Catch2 unit test for class StatesDocument.

testStatesDocument.cpp is now being incorporated in the build, and the test case runs when "build" executed on the target "RUN_TESTS_PARALLEL".

* Update StatesDocument.cpp

- Moved all utility methods into a struct with local linkage.
- Streamlined code by adding `getEltValue()`. Use of this method reduced some code repetition.

* Update StatesDocument.cpp

- Minor changes to line formatting.

* Update testComponentInterface.cpp

- Minor corrections/additions to comments

* Update testStatesDocument.cpp

- Added code borrowed from testComponentInterface.cpp to generate a dummy model with a variety of states.

The code compiles and runs with all test cases passing, but this is just a starting point. De/serialization using class StatesDocument is not yet being tested.

* Update testStatesDocument.cpp

- Removed code borrowed from testComponentInterface.cpp. That code was not the way to go. A legitimate model is needed.

I'm starting over.

* Update StatesDocument.h

- Made small improvements and corrections in the documentation.

* Update testStatesDocument.cpp

- Now building a simple model and running a simulation.

* Update testStatesDocument.cpp

- Added code to serialize, deserialize, and then re-serialize the state trajectory recorded during a simulation.

The code is functioning as expected!

* Removed trailing whitespace in StatesTrajectory.h

* Update StatesTrajectory.h

- Changed member variable `m_states` from type std::vector<SimTK::State> to SimTK::Array_<SimTK::State>
- Added `exportToStatesDocument()`

* Update StatesTrajectory.h

- Added a `get` method that provides access to the underlying SimTK::Array_<SimTK::State> member variable.

* Update testStatesDocument.cpp

- Added a method that tests equality of 2 state trajectories.

The method does not yet test equality for discrete variables or modeling options, but Qs, Us, and Zs are passing.

* Update testStatesDocument.cpp

- Added a static method that computes maximum rounding error based on the specified precision and value that is rounded.

* Update testStatesDocument.cpp

- Cleaned up the computation of max_eps.

* Update testStatesDocument.cpp

- Added equality comparisons for discrete variables and modeling options.

* Update testStatesDocument.cpp

- Added ExtendedPointToPointSpring.

* Merge branch 'opensim-org:main' into ostates_2

* Removed trailing whitespace from testStatesTrajectory.cpp

* Altered testBoundsCheck()

Since the trajectory was changed from std::vector<State> to SimTK::Array_<State>, bounds checking behavior has changed.

SimTK::Array_<T> only checks bounds in DEBUG. In other words, an IndexOutOfRange exception will not be thrown in a release build. To account for this, I inserted an #ifdef to skip bounds checking unless in DEBUG.

* Minor: Added a missing space.

* Added discrete variables to the test model.

Added the following types:
- bool
- int
- double
- Vec2
- Vec3
- Vec4
- Vec5
- Vec6

All tests are passing, and the .ostates file looks right.

* Revamped ExtendedPointToPointSpring

So that discrete variables can be updated during a simulation, class ExtendedPointToPointSpring is now built on top of class ExtendedTwoPointLinearSpring. ExtendedTwoPointLinearSpring allocates auto-update discrete variables in its own `realizeTopology()` and those discrete variables are updated in its own `realizePosition()`.

* Reverted to a simpler ExtendedPointToPointSpring class.

The discrete variables can be added in extendRealizeTopology().
And, they can be changed in computeForce().

* Discrete variables are now being altered during simulation

- Changed the allocation for discrete variables to the DefaultSystemSubsystem.
- Primarily using the subsystem index instead of the pointer to the subsystem.
- Discrete variables are being changed during simulation in ExtendedPointToPointSpring::computeForce().

All checks are now passing!

* Added ability to set a note for a states document.

The note is serialized and deserialized.

All checks are passing.

* Merge branch 'opensim-org:main' into ostates_2

* Update testStatesDocument.cpp

1. Now looping over over output precision from precision = 2 to precision = 22.
2. Added the ability to change the name of a discrete state for the purpose of testing exceptions when a discrete state is not found.

* Update testStatesDocument.cpp

- Cleaned up the code a little bit.
- Added code to deserialize with a model that has one discrete variable with a changed name. This was done to check exception handling.

* Check that model names match

* Update testStatesDocument.cpp

Cleaned up the code.

The following checks are now performed:
- Serialization / deserialization for precision 1 to 21.
- Check that exception is thrown when model names don't match.
- Check that exception is thrown when a discrete state is not found.

* Update testStatesDocument.cpp

- code cleanup
- added the ability to omit a discrete state

When a discrete state is omitted, an exception should be thrown but it is not.  Need to look into this.

Previous checks are still running properly.

* All checks passing.

* Removed old code

* Added read-only accessor for low-level state array

const SimTK::Array_<SimTK::State>& StatesTrajectoryReporter::getStateArray();

* Code cleanup and refinement

- Removed StatesDocument::parseDoc().  No reason to have it.
- Simplified the name of StatesTrajectory::getUnderlyingStateArray() to getStateArray().
- Refined documentation comments and checked them for accuracy.

* Merge branch 'opensim-org:main' into ostates_2

* Minor code cleanup

- Deleted unused code.
- Updated copyright dates.

* Update StatesTrajectory.h

- Improved documentation comments.

* Fixed typo in StatesDocument.h

An error was being issued for Ubuntu and Mac builds because "OPENSIM_STATES_DOCUMENT_H_" was not commented out on line 609. I added the comment token.

* Corrected some type compatibilities in StatesDocument.cpp

In a number of instances, I changed the type from `int` to `size_t`.

* Using std::string::c_str() for strings in SimTK_ASSERT macros

Windows can handle a std::string, but Mac and Ubuntu builds are failing.

* Update testStatesDocument.cpp

- Addressed some type warnings.
- Removed the method `ComputeMaxRoundingError()`.

* Preventing auto generation of a default constructor

* Update StatesDocument.h

I temporarily added a default constructor to see if that would resolve some errors that are occurring in the Continuous Integration.

* Merge branch 'opensim-org:main' into ostates_2

* Update testStatesDocument.cpp

It looks like different return types of size() methods are assumed across operating systems. I inserted a (size_t) cast to address compiler errors on Ubuntu

* Add bindings for StatesDocument class

* Merge branch 'opensim-org:main' into ostates_2

* Reverted StatesTrajectory to use std::vector(SimTK::State)

For reasons of performance and compatibility with Simbody, OpenSim::StatesDocument uses SimTK::Array_<>, instead of std::vector<>, as the container for states trajectories.

However, some user-facing classes, like OpenSim::StatesTrajectory, have opted to use std::vector<> as the container. I migrated OpenSim::StatesTrajectory to use SimTK::Array_<>. Unfortunately, this caused errors when bindings for Python and perhaps Java and Matlab are generated.

To maintain compatibility with existing code and simplify binding generation, this commit reverts to using std::vector as the container for state trajectories.

In addition, several new interface methods were introduced to the StatesDocument API to allow either SimTK::Array_<> or std::vector<> to be used as the container. Internally, however, all state trajectory operations are mediated by SimTK::Array_<>. This flexibility is gained at almost no computational cost by using a shallow copy constructor to convert std::vector<State> objects to SimTK::Array_<State> objects.

* Update StatesDocument.cpp

Now settings for `note` and `precision` are updated in the body of the constructor instead of in the initializer list.

* Restored commented-out code in testStatesTrajectory.cpp

* typo corrected

* Update StatesDocument.h

Updated the documentation to reflect the fact that std::vector<SimTK::State> is also an acceptable form of the state trajectory for interfacing with OpenSim.

* Update StatesTrajectoryReporter.cpp

Migrated StatesTrajectoryReporter::getStateArray() to getVectorOfStateObjects()

* Update StatesTrajectoryReporter.h

Migrated StatesTrajectoryReporter::getStateArray() to getVectorOfStateObjects().

* Update StatesDocument.cpp

I added comments for developers should a demand additional supported variable types emerge.

* Update testStatesDocument.cpp

Corrected a compile error.

* Updated CHANGELOG.md to include the new StatesDocument class (#3902)

Co-Authored-By: aymanhab <[email protected]>

Bindings/OpenSimHeaders_simulation.h

@@ -148,6 +148,7 @@
 #include <OpenSim/Simulation/OpenSense/IMU.h>
 #include <OpenSim/Simulation/OpenSense/OpenSenseUtilities.h>
 
+#include <OpenSim/Simulation/StatesDocument.h>
 #include <OpenSim/Simulation/StatesTrajectory.h>
 #include <OpenSim/Simulation/StatesTrajectoryReporter.h>
 

Bindings/simulation.i

@@ -253,6 +253,7 @@ OpenSim::ModelComponentSet<OpenSim::Controller>;
 
 %template(StdVectorIMUs) std::vector< OpenSim::IMU* >;
 
+%include <OpenSim/Simulation/StatesDocument.h>
 %include <OpenSim/Simulation/StatesTrajectory.h>
 // This enables iterating using the getBetween() method.
 %template(IteratorRangeStatesTrajectoryIterator)

CHANGELOG.md

@@ -12,8 +12,8 @@ v4.6
   the case where provided string is just the name of the value, rather than a path to it (#3782)
 - Fixed bugs in `MocoStepTimeAsymmetryGoal::printDescriptionImpl()` where there were missing or incorrect values printed. (#3842)
 - Added `ModOpPrescribeCoordinateValues` which can prescribe motion of joints in a model given a table of data. (#3862)
-- Fixed bugs in `convertToMocoTrajectory()` and `MocoTrajectory::resampleWithFrequency()` by updating `interpolate()` to 
-  allow extrapolation using the `extrapolate` flag. Combined with the `ignoreNaNs` flag, this prevents NaNs from 
+- Fixed bugs in `convertToMocoTrajectory()` and `MocoTrajectory::resampleWithFrequency()` by updating `interpolate()` to
+  allow extrapolation using the `extrapolate` flag. Combined with the `ignoreNaNs` flag, this prevents NaNs from
   occurring in the output. (#3867)
 - Added `Output`s to `ExpressionBasedCoordinateForce`, `ExpressionBasedPointToPointForce`, and `ExpressionBasedBushingForce` for accessing force values. (#3872)
 - `PointForceDirection` no longer has a virtual destructor, is `final`, and its `scale` functionality
@@ -23,12 +23,24 @@ v4.6
   components. The `ForceProducer` API was also rolled out to a variety of existing `Force` components, which
   means that API users can now now ask many `Force` components what forces they produce (see #3891 for a
   comprehensive overview).
-- Made improvements to `MocoUtilities::createExternalLoadsTableForGait()`: center of pressure values are now set to zero, rather 
-  than NaN, when vertical force is zero, and the vertical torque is returned in the torque columns (rather than the sum of the 
+- Made improvements to `MocoUtilities::createExternalLoadsTableForGait()`: center of pressure values are now set to zero, rather
+  than NaN, when vertical force is zero, and the vertical torque is returned in the torque columns (rather than the sum of the
   sphere torques) to be consistent with the center of pressure GRF representation.
 - Fixed an issue where a copy of an `OpenSim::Model` containing a `OpenSim::ExternalLoads` could not be
   finalized (#3926)
-- Updated all code examples to use c++14 (#3929) 
+- Updated all code examples to use c++14 (#3929)
+- Added class `OpenSim::StateDocument` as a systematic means of serializing and deserializing a complete trajectory
+  (i.e., time history) of all states in the `SimTK::State` to and from a single `.ostates` file. Prior to `StatesDocument`,
+  only the trajectories of continuous states (e.g., joint angles, joint speeds, muscle forces, and the like) could be systematically
+  written to file, either in the form of a `Storage` file or as a `TimeSeriesTable` file, leaving discrete states (e.g., muscle
+  excitations and contact model anchor points) and modeling options (e.g., joint clamps) unserialized. `StatesDocument`, on the
+  other hand, serializes all continuous states, discrete states, and modeling options registered with `OpenSim::Component`.
+  Whereas neither `Storage` files nor `TimeSeriesTable` files are currently able to handle mixed variable types, `StatesDocument`
+  is able to accommodate variables of type `bool`, `int`, `double`, `Vec2`, `Vec3`, `Vec4`, `Vec5`, and `Vec6` all in the same
+  `.ostates` file. `.ostate` files are written in `XML` format and internally carry the name of the OpenSim model to which the
+  states belong, a date/time stamp of when the file was written, and a user-specified note. `.ostate` files also support a
+  configurable output precision. At the highest ouput precsion (~20 significant figures), serialization/deserialization is
+  a lossless process. (#3902)
 
 v4.5.1
 ======

OpenSim/Common/Test/testComponentInterface.cpp

@@ -390,7 +390,7 @@ class Bar : public Component {
         // and to access a dv that is not type double.
         // The following calls put the mo and dv into the maps used to contain
         // all mo's and dv's exposed in OpenSim. When Stage::Topology is
-        // realized, they will allocated in class Bar's override of
+        // realized, they will be allocated in class Bar's override of
         // extendRealizeTopology(). See below.
         bool allocate = false;
         int maxFlagValue = 1;
@@ -401,6 +401,8 @@ class Bar : public Component {
     // Manually allocate and update the index and subsystem for
     // a discrete variable and a modeling option as though they were
     // natively allocated in Simbody and brought into OpenSim.
+    // Note, as of May 2024, this is also what one would need to do in order
+    // to add a discrete variable that is a type other than double.
     void extendRealizeTopology(SimTK::State& state) const override {
         Super::extendRealizeTopology(state);
 
@@ -2057,7 +2059,7 @@ TEST_CASE("Component Interface State Trajectories")
     }
 
     // Create a new state trajectory (as though deserializing)
-    // newTraj must be must the expected size before any set calls.
+    // newTraj must be the expected size before any set calls.
     SimTK::Array_<SimTK::State> newTraj;
     for (int i = 0; i < nsteps; ++i) newTraj.emplace_back(s);
     // state variables