CHANGELOG.md

Changelog

All notable changes to this project will be documented in this file. See Conventional Commits for commit guidelines.

4.33.0 (2023-05-17)

Features

• commands: add --force-from-scratch cli arg when running their evaluation to force delete any previous mission information (ae2793c)
• metrics: delete evaluation metrics checkpoint at the end of a run to prevent accidentally resuming (6e77b7a)
• metrics: if the action outputs dir is empty and we are not resuming, delete the checkpoint file (b88c03d)
• metrics: support creating and restoring checkpoints of evaluation metrics if resuming runs (bef8c1f)
• wandb: automatically determine whether we should start a new run instead of using an environment variable (82d6ace)
• wandb: include the offline inference version used in the run config (6505c7f)
• wandb: track session ids and success per mission across runs (1260998)

Bug Fixes

• just manually point to the service registry path (612315b)
• metrics: log the step number instead of trying to modify the config (871ca3f)
• prevent type coercion for the CDFs (within MissionTrajectory) (90b9613)
• scripts: symlink commands in the prepare-user-data script (e5af1e5)
• settings: boolean whether we resume the run or not (99d3dbc)
• wandb: allow config value changes during a run (bb9499c)
• wandb: column name for the per mission group success rate (e732ee8)
• wandb: saving output files during the evaluation run (ade01ff)
• wandb: set the default success rate to 0 if there are nan's (eec3a00)
• wandb: step needs to be an int (b65310a)

4.32.0 (2023-05-15)

Features

• create flag to enforce/ignore successful preparation steps (4af06aa)
• script: create script that prepares user area for running the offline inference (951080a)

Bug Fixes

• guard against InterruptedByNewCommandBatch (6b8c239)
• scripts: automatically setup necessary symlinks (b45cff7)

4.31.1 (2023-05-12)

Bug Fixes

• raycast missed exception handling (#40) (7a3d3c4)

4.31.0 (2023-04-28)

Features

• add setting for fastmode (b891a94)
• added flag randomise start position (#41) (741abc8)

Bug Fixes

• arena source cdfs (be81cb8)
• restore support for running the T1 trajectories (75d545c)

4.30.0 (2023-04-26)

Features

• T3: add mission 1 (455d023)

4.29.9 (2023-04-26)

Bug Fixes

• operate microwave challenges (59b8b44)

4.29.8 (2023-04-25)

Bug Fixes

• coffee unmaker: improve plans, add pot to beans, and iterate layouts (30a6a0d)

4.29.7 (2023-04-25)

Bug Fixes

• ensure the microwave is empty (a4c454a)
• remove the boss coffee mug from the colour changer missions (eee16b4)

4.29.6 (2023-04-25)

Bug Fixes

• disabling color variants with microwave challenges (0d8ad3a)

4.29.5 (2023-04-25)

Bug Fixes

• prep condition for pickup objects from containers (ab9a60c)
• prep goal condition for place object in container challenge (40769c4)

4.29.4 (2023-04-25)

Bug Fixes

• disable start colour variants for the colour changer challenges (a1e39cb)
• remove objects that cannot be color-changed (66451f1)
• update plans for the challenges (b4269d1)

4.29.3 (2023-04-25)

Bug Fixes

• sink-related challenges (e44408c)
• sink-related goal conditions and plans (ed27162)

4.29.2 (2023-04-23)

Bug Fixes

• target object was not flagged as ambiguous in key (5579cda)

4.29.1 (2023-04-23)

Bug Fixes

• high level keys and plans (b1dfde9)
• include from_receptacle in breaking challenges (0f236ba)
• remove the from receptacle from high level key in break challenges (8db7588)

4.29.0 (2023-04-23)

Features

• break objects on random desks (#39) (4e2afe9)

Bug Fixes

• desks should not be unique (9f3f1f2)
• do not use the colour changer to make the object the same color (80d9a90)

4.28.0 (2023-04-23)

Features

• place bowl on plate on gravity pad (#38) (e058d15)

Bug Fixes

• check that the gravity pad contains the plate during prep subgoal (6ec8b31)
• disable color variants for fridge/freezer challenges (1778d72)
• update room/colour was failing because there was no 0th index (a57bb0c)

4.27.0 (2023-04-23)

Features

• added color variants flag (#37) (6823c47)

4.26.0 (2023-04-23)

Features

• breaking things with the hammer (#36) (0bed877)

4.25.0 (2023-04-23)

Features

• pickup object from printer (#35) (b5f1cb3)

Bug Fixes

• add breakroom table to required objects list (4e6d8b6)
• dont make the spawned object into a required one (987a287)
• replace the spawned object with the printer cartridge (172fb7a)

4.24.1 (2023-04-22)

Bug Fixes

• expliticly ensure at least the first subgoal has been completed (f61f810)
• return after logging the failed trajectory (940d400)

4.24.0 (2023-04-22)

Features

• pickup target among distractors (#34) (8264204)

4.23.0 (2023-04-22)

Features

• add goal for prep steps (f3db7ec)

4.22.0 (2023-04-22)

Features

• add goal for prep steps (7742128)
• add goals to ensure the "objects in containers" missions are setup correctly (606a4c3)
• mark the run as failed if the preparation steps did not succeed (4830023)
• mark the run as failed if the subgoal success is 0 (f3659b3)

Bug Fixes

• disable color variants usage for pickup and place missions separately (b941892)

4.21.0 (2023-04-22)

Features

• add the target-object-is-ambiguous high-level key (f534502)

Bug Fixes

• update high-level key config within wandb tracker (3bbc069)

4.20.0 (2023-04-21)

Features

• place object onto plate in container (#33) (ae16713)

4.19.1 (2023-04-21)

Bug Fixes

• wrong receptacle for target object (c45a898)

4.19.0 (2023-04-21)

Features

• make it easier to generate trajectories in a different folder (4215037)

Bug Fixes

• make the output dir for generated trajectories (fcedc65)

4.18.0 (2023-04-21)

Features

• enable carrot machine challenges (321d7a6)

Bug Fixes

• kill the unity instance before trying to restart it (1274ae2)

4.17.0 (2023-04-21)

Features

• stack food plate (#32) (2ef14db)

4.16.1 (2023-04-21)

Bug Fixes

• always remove objects from microwave (ea0dbcd)

4.16.0 (2023-04-21)

Features

• include stacked objects in the high level key (3f937e7)

Bug Fixes

• remove the change_color instruction action (d53cbaa)

4.15.1 (2023-04-21)

Bug Fixes

• color changer preparation step (cc548ec)

4.15.0 (2023-04-21)

Features

• Add container challenges for the warehouse sink (#30) (460dbe1)

Bug Fixes

• use color variants for both boss mug and normal mug (a7b5371)

4.14.2 (2023-04-21)

Bug Fixes

• incorrect function call (af355fd)

4.14.1 (2023-04-20)

Bug Fixes

• attempt to handle the 408 connection error (467f5c7)

4.14.0 (2023-04-20)

Features

• include unity logs with each run (bdea65e)

Bug Fixes

• preparation steps for operate microwave missions (e23eb24)
• Remove final pick ups (#29) (fa71470)

4.13.3 (2023-04-20)

Bug Fixes

• Color changer interaction object in High Level Key (#28) (c0468d9)

4.13.2 (2023-04-20)

Bug Fixes

• goal conditions for the operate printer challenges (151647c)

4.13.1 (2023-04-19)

Bug Fixes

• disable the carrot machine challenges (4badcef)

4.13.0 (2023-04-19)

Features

• add more helper methods to structures (893f0b6)

Bug Fixes

• change starting room to robotics lab (518425a)
• let object instance ID's end in a * (f3d7713)
• make the argument name be more descriptive to what is needed (93e1760)
• operate printer challenges (82ce622)
• operate printer: make sure robotic arm is out the way (1dca91b)
• operate time machine with carrots (1f85703)
• printer cartridge name in the preparation plan (b8250fa)
• typo (0355636)
• validator condition for object instance id suffix (d8fb10f)

4.12.1 (2023-04-19)

Bug Fixes

• Initial room for carrot maker challenge (#27) (7221c32)

4.12.0 (2023-04-19)

Features

• add printer challenges (#26) (f895cd4)

4.11.0 (2023-04-19)

Features

• Add carrot machine challenges (#23) (dabbc47)

4.10.0 (2023-04-19)

Features

• Add time machine on carrots (#25) (7d91e43)

4.9.0 (2023-04-19)

Features

• coffee unmaker with additional objects (#22) (7f284f9)

4.8.0 (2023-04-19)

Features

• Add microwave challenge (#21) (f62b996)
• object transformations (#19) (f76c4c2)

4.7.0 (2023-04-19)

Features

• support setting wandb group name from run command (9b72cc8)

Bug Fixes

• kill command for the experience hub (9852ce8)
• use the preparation plan to toggle the sink (4d3c266)

4.6.0 (2023-04-18)

Features

• Add more pickable objects for fridge/freezer (#18) (baf6850)
• save the mission trajectory file and the output file to wandb (211bd8c)

4.5.0 (2023-04-18)

Features

• log the experience hub version with the wandb run (45c17fb)

4.4.1 (2023-04-17)

Bug Fixes

• command that starts the experience hub (f187fef)
• command used to kill the experience hub (5e8bd3d)
• if failed to go to a random viewpoint, just go to the first one in the room (bb3d652)
• use 2 workers to hopefully stop the experience hub from crashing (daa17a0)

4.4.0 (2023-04-17)

Features

• also send subgoal completion success rate to wandb per session (3a789f0)
• set the session id as the run name (1960040)

4.3.0 (2023-04-17)

Features

• use subprocess to run the experience hub because its easier to consistently kill (bfaeea2)

Bug Fixes

• disable fast mode (d894ef2)
• disable look actions in random walk (6060c33)
• increase number of healthcheck attempts for experience hub (a6737a2)
• plan for placing objects in container (f522c26)
• remove initial contained items from various containers (d53b7a0)

4.2.0 (2023-04-16)

Features

• track progress of generated trajectories on wandb (cb5c1ab)

Bug Fixes

• set experience hub timeout to be stupidly high (9302cca)

4.1.0 (2023-04-16)

Features

• store any remaining utterances that have not been sent to the action outputs (2bf68b7)

Bug Fixes

• break out the loop if all the goals are complete (5b8a18f)
• break out the loop if the goals have been completed (5abe3b1)

4.0.1 (2023-04-16)

Bug Fixes

• make sure the breakroom table exists, and has space for the preparation items (34a10cd)
• preparation plan for fill object in sink challenge (86a96b0)

4.0.0 (2023-04-16)

⚠ BREAKING CHANGES

• migrate challenges to using state conditions

Features

• migrate challenges to using state conditions (615b37c)
• turn on fastmode in the arena (3325143)

Bug Fixes

• "and" is no longer allowed (1ca4655)
• ensure the reimported CDF is same as the original CDF (df7a6b8)
• go to a random viewpoint that actually exists in the current scene (f4a21cb)
• high level key action for "fill object in sink" (af4829b)
• remove the duplicated running of preparation steps (4db9b7e)
• use join and close when killing the experience hub process (3c18b58)

3.2.0 (2023-04-14)

Features

• do not send utterances to the arena if all the goals are complete (b8cc13c)
• make it easier to kill the experience hub when something goes wrong (7e375fe)

Bug Fixes

• fill the object challenge goals (8543bb2)
• include a preparation plan to be able to pickup objects (#16) (d86301a)

3.1.2 (2023-04-13)

Bug Fixes

• remove the trajectory batching (a467340)

3.1.1 (2023-04-12)

Bug Fixes

• do not randomise trajectory run order by default (3e16279)
• get rid of the progress bar (133b049)
• remove old "kill arena" command (23e4cda)

3.1.0 (2023-04-12)

Features

• restart the arena after every 10 sessions (0f9fcdd)

Bug Fixes

• improve the styling of the progress bar (3191a34)
• update the progress bar after sending the utterance (5d698c3)
• update the progress bar more (4ecce70)

3.0.0 (2023-04-12)

⚠ BREAKING CHANGES

• use a new structure for the high-level keys

Features

• do not re-run missions that have already been run (1720664)
• use a new structure for the high-level keys (2116539)

Bug Fixes

• add more state names to the arena constants (79365b2)
• use kebab-case when converting high-level key to string (2e2d3e8)

2.13.0 (2023-04-12)

Features

• print a table of num challenges per high level key (4364dbd)

2.12.0 (2023-04-12)

Features

• separate the trajectory generation from the trajectory running (4b9cd0d)

2.11.0 (2023-04-12)

Features

• challenge: convert coffee into beans using the coffee unmaker (#15) (8e36065)

2.10.0 (2023-04-12)

Features

• ensure each object-related key is a 'readable name' (in the HighLevelKey (0cb022e)

2.9.1 (2023-04-12)

Bug Fixes

• explicitly forbid unsupported keys from the HighLevelKey (aca0079)

2.9.0 (2023-04-11)

Features

• fill trajectories (#13) (420313c)

2.8.0 (2023-04-11)

Features

• add challenge to clean a plate in sink (392518a)

Bug Fixes

• object instance ids for the sink and plate (9354196)
• use deepcopy on the sink and create a trajectory for every layout (0b004a5)

2.7.0 (2023-04-11)

Features

• lowercase the session ids (except for the prefix) (eac8202)
• separate the readable name from the object key when building challenges (35fbde2)
• use readable names for the keys (9c30b00)

2.6.1 (2023-04-11)

Bug Fixes

• add more error types to ignore when randomising start position (f866cdb)
• go back to using str for str-based structures (8a62425)

2.6.0 (2023-04-11)

Features

• add challenges for broken bowls and coloured bowls (43cb76c)
• improve the progress bar for the challenge validator (9e9e2d2)
• only use the colour changer colors (4775e46)
• shuffle the order trajectories are generated in (33ea2fa)
• support using RNG for the CDF scenes (with floor_plan) (b7e1b4d)
• validate cdfs from generated missions (062178c)

Bug Fixes

• required_objects key within the CDFScene (fe3bcd1)
• add __str__ methods for the ObjectId and ObjectInstanceId (9b764be)
• change CDF floor_plan validation to allow for "-1" (d12bfe2)
• object ids for broken cords and computer monitors (33aaffa)
• remove the duplicated object state (3d92f22)
• send actions to randomise start position one-by-one (8b31f76)

2.5.0 (2023-04-10)

Features

• generate all the pickup from fridge/freezer missions (e263c93)

2.4.0 (2023-04-10)

Features

• add insert and vendingmachine actions to the high-level key structure (a6486a2)
• add command to print the high-levels keys that we have challenges for (051043e)

2.3.0 (2023-04-09)

Features

• randomise start position before challenge begins (f78a9fb)

2.2.0 (2023-04-09)

Features

• add 'pickup apple from fridge' mission, with coloured variants (f431757)
• register challenge to pick up (coloured) apples from an open fridge (43fd6ed)
• support generating challenges from other challenges with minor modifications (1ffbc1e)

Bug Fixes

• make sure the isColorChanged key doesn't exist already to prevent duplicates (9555241)
• replace property setters with explicit functions (2d4038f)
• using *-is-container when parsing high-level keys from string (6c78d32)

2.1.0 (2023-04-09)

Features

• improve generalisability of challenge builders for less boilerplate (fe24100)

Bug Fixes

• add tests and make sure the generation process works (1a6f6ac)

2.0.0 (2023-04-08)

⚠ BREAKING CHANGES

• be able to generate missions from high-level keys

Features

• add more missions for other layouts (1c5615d)
• add option to send dummy actions when validating cdfs (0de8f4b)
• be able to generate missions from high-level keys (1772f92)
• include progress to more clearly know the overall progress (e135740)
• optionally add randomness to the session id name (c0c43e2)
• upload trajectory results to s3 (77d1532)
• use cloudpathlib to upload all the metrics to S3 (986b1ce)

Bug Fixes

• create all parents for the metric output file (41d9372)
• created session id needs to not have slashes (644268b)
• created session id to be in the form T.DATE/KEY-UUID (e94d2ff)
• env var key to enable the offline evaluation mode (5380a4b)
• generated session id that is valid as a path and uri (c2a033d)
• lint issues (ad3527d)
• set a long timeout for experience hub, which gets overridden by the settings client timeout (30b45c1)
• simplify the provided session id prefix (70c5971)

1.5.0 (2023-04-06)

Features

• add command to validate cdfs within a dir (8b65c55)
• add flag to enable offline evaluation mode in experience hub (4514467)
• add missions for picking up from freezer (8108cf1)
• clone the experience hub into the storage dir (22b91a7)
• create new structures for the challenges and trajectories (1e12c0b)
• improve how generating trajectories are to be run (d3b8979)

Bug Fixes

• improve the first attempt to make the CDFs (c19be24)
• just use a single high level key for each mission (788ea34)
• kill command for the arena (605863b)
• missions for pickup from fridge (b74601c)
• settings need to exist before being able to run things (951aa38)
• the service registry path (eb8c5c5)

1.4.0 (2023-04-05)

Features

• be able to validate cdfs (faa6722)

Bug Fixes

• if the arena/experience hub are running, do not try and start it again (cde1077)
• make sure the experience hub dies (5094187)
• run command with the new experience hub version (6ad167b)

1.3.1 (2023-04-04)

Bug Fixes

• context managers for the controllers/orchestrators (fc27d12)

1.3.0 (2023-04-02)

Features

• simplify run commands and add in the backend for the web tool (4b4ee95)

1.2.0 (2023-04-01)

Features

• add options to run for subset (2391594)
• create separate module/script to run the background services (ec19cd8)
• dump all metrics so they can be stitched together again (d8366ce)
• improve metrics calculating (1bfa793)
• include CDFs (d22fe14)
• just everything from running the eval before report submission (07b000e)
• only evaluate missions that have not been evaluated yet (453ceb9)
• run eval on single gpu (bf16694)
• set instance range in settings and send to s3 when done (b126ffc)

Bug Fixes

• improve logs and healthchecks and responses (849ba61)
• just make loads of changes to make it actually work properly (041d40c)
• just make loads of changes to make it actually work properly (2a88df5)
• make sure we do all instances and dont miss any (f4901b5)
• use the new arena executable (c8a287d)

1.1.0 (2023-03-19)

Features

• handle lightweight dialogs within the context of the actions (8647fe7)

1.0.1 (2023-01-24)

Bug Fixes

• only download and prepare T2 validation data (54f716e)

1.0.0 (2023-01-23)

Features

• add arena wrapper from ml toolbox (9837f96)
• add code to orchestrate the test (bc17922)
• add experience hub dir to settings (d292d1b)
• add log points (bac27af)
• add metric logging for the evaluation (c0f8ee7)
• add run command (0830cc5)
• add script to launch xserver (21389ce)
• add scripts to download the mission data (ef568c6)
• add terraform config for creating the instance (4ab0e6c)
• automatically prepare the file system for the evaluation (397018b)
• automatically update permissions and start xserver (37a39fe)
• create the settings file (1b45fcd)
• disable client timeouts on experience hub (ed5551d)
• install multiprocessing logging (b767f43)
• set ssh key to the ec2 key (e4a98bd)
• setup repo (39949b5)
• setup venv and prepare trajectory data in user-data script (4ccf28d)
• stop docker containers on exit too (a369b42)
• use experience hub for access to the storage/docker configs (9249bbe)
• use loguru for logging in arena_orchestrator (c9fc3ba)
• use rich logging (557b006)

Bug Fixes

• add catch for timeout on healthcheck (0bbab99)
• allow pickle when loading data (7ac9ec0)
• change port for the experience hub to run on (f211885)
• copy the arena deps as ubuntu (91f8c90)
• create session id directory for auxiliary metadata (214ccb6)
• creating the storage dir for cloning the experience hub (8cc6dbf)
• disable multiprocess logging (f8cdc4a)
• do not create a symlink (70cb36f)
• do not force download the models if they exist (5b8949e)
• do not run process as a daemon (7ec2f2c)
• do not start xserver within the user-data (700f440)
• do not try to setup the python env on launch - it wont play nice (9380128)
• explicitly define the args to run the controller api (a97ddb4)
• explicitly disable observability and production (79b43cd)
• formatting (b21c550)
• improve orchestrators start order (defb686)
• lint errors (ada1332)
• method order in class (4a49a24)
• model storage dir (ced8940)
• only need about 10 retries before it should be running (61f6e20)
• order of setting orchestrators up (f4949ef)
• re-able running as a daemon (eceb2f4)
• remove dialog actions from the experience hub response actions (e7f1d50)
• remove the xserver module - its not needed (f1836b4)
• send dummy actions when loading the game (c152998)
• set arena env vars within the run (5edb809)
• set the appconfig to a dataclass so that it hopefully loads the env vars (31624c9)
• set the envvars outside the function (4192e36)
• type error because im pickling so we dont care (5f0c3a3)
• types for paths in case they dont exist already because we create them (05d565d)
• unzip path for cdf data (a035e7c)
• use the httpx client when making the request (5897d15)
• use the settings to hopefully run the thing (db07a39)