Virtualship package restructure and defining the Python public API

[VeckoTheGecko]

The package structure currently looks like this:

src
└── virtualship
    ├── __init__.py
    ├── _version_setup.py
    ├── cli
    │   ├── __init__.py
    │   ├── _creds.py
    │   ├── _fetch.py
    │   ├── commands.py
    │   └── main.py
    ├── expedition
    │   ├── __init__.py
    │   ├── checkpoint.py
    │   ├── do_expedition.py
    │   ├── expedition_cost.py
    │   ├── input_data.py
    │   ├── instrument_type.py
    │   ├── schedule.py
    │   ├── ship_config.py
    │   ├── simulate_measurements.py
    │   ├── simulate_schedule.py
    │   ├── space_time_region.py
    │   └── waypoint.py
    ├── instruments
    │   ├── __init__.py
    │   ├── adcp.py
    │   ├── argo_float.py
    │   ├── ctd.py
    │   ├── drifter.py
    │   ├── ship_underwater_st.py
    │   └── xbt.py
    ├── location.py
    ├── make_realistic
    │   ├── __init__.py
    │   ├── adcp_make_realistic.py
    │   └── ctd_make_realistic.py
    ├── spacetime.py
    ├── static
    │   ├── __init__.py
    │   ├── schedule.yaml
    │   └── ship_config.yaml
    └── utils.py

7 directories, 35 files

The cli code is siloed off (which is great), but the rest of the code could be structured better:

• configuration objects are all living in the expedition package. This makes it difficult to locate the configuration classes compared to the code that contains the logic. The config subcomponents which build into the ShipConfig, for example, are split across multiple files making it difficult to see that indeed they belong to the ShipConfig object.
• The __init__.py files do a lot of imports, which increases the likelihood of running into circular import issues.
• a lot of separate files (almost one per class) which is not necessary, and instead we could group them by functionality into themed modules

Defining the Python public API

From a user POV, are there any cases where users will run VirtualShip via Python itself (e.g., import virtualship; virtualship.do_expedition(....))?

Since the main goal of VirtualShip is education, I would opt for making the public API for the project being the CLI and have the Python part of things be internal. I think this reflects the current development, provides clear expectations to users, and gives developers the flexibility to make changes. If users want to run VirtualShip programmatically they can use Python to run command line commands.

Keen to know your thoughts and plans for future development.

cc @erikvansebille @ammedd @iuryt @j-atkins let me know if you have opinions on the public API

[erikvansebille]

Thanks for starting this discussion, @VeckoTheGecko!

Note that while the primary goal of the VirtualShip is indeed in education, I also see much power for reanalysing ocean model data as if taken from a ship.

For example, last year we had a project with @vesnaber and a student to compare the CROCO model data against observations taken from the Pelagia. It would have been great to be able to deploy a set of instruments at a set of locations, so that they can be easily compared.

Now, it could be that these use cases would also be through the CLI, but I do want to highlight that there's more scope for VirtualShip than just education.

[j-atkins]

I agree that it could be useful to keep the option open for running VirtualShip via Python itself, in case it allows e.g. research-related users to integrate it directly into their existing (Python) workflows. It may well be valuable to leave these kinds of doors open for future VirtualShip use cases...

However, is it realistic/practical to support both CLI and direct Python options? Or does it make communicating how to use VirtualShip quite convoluted if there are say two 'routes' for using it?

EDIT: Scratch that, I have just chatted with Nick, he has convinced me that having the CLI as the public API for the project could be the way to go...

[ammedd]

I've just spend a day at EGU promoting the take up of the tool by researchers, so let's please make that easy as well! Though I am sure the people who will use it from that perspective are proficient enough to figure out how to use internal Python code for that. So I would opt for making it easy and having a tutorial on it, but not full on catering for it. (Hope you get what I mean)

I don't oversee all the code structure improvements you're suggesting, but probably agree. It was for example quite tedious to add the XBT to the ship after Iury created it as a file in -instruments.

[VeckoTheGecko]

Now, it could be that these use cases would also be through the CLI, but I do want to highlight that there's more scope for VirtualShip than just education.

I agree, and I think that the CLI would be suitable for such a usecase.

I've just spend a day at EGU promoting the take up of the tool by researchers, so let's please make that easy as well! Though I am sure the people who will use it from that perspective are proficient enough to figure out how to use internal Python code for that.

I think here its important to identify what sort of research questions they would want to ask, and how those would be achieved via the CLI. I think if we promote that users use the internal Python code, then we have to provide some sort of assurance that the Python code is stable and not going to change (which is an assurance I am quite hesitant to make - especially since I don't see a clear reason why one would opt for using the Python code instead of the CLI). The narrower the interface we expose to users, the less "promises" that we have to make as developers.

[iuryt]

I can see several exciting research applications for this package:

• Cruise Planning
  Planning observational campaigns by simulating various sampling routes and timelines ahead of deployment.

• Real-time Adaptive Sampling
  Dynamically adjusting the sampling strategy in response to model forecasts or real-time data.
  Example: Modifying ship transects based on updated ocean current predictions.

• Sampling Optimization
  Investigating optimal deployment strategies to capture specific ocean features.
  Example: Evaluating how many drifters are needed and their ideal spatial distribution through comparative simulations.

• Sampling Automation
  Designing and testing control algorithms for autonomous platforms such as gliders, REMUS vehicles, Saildrones, or even semi-autonomous systems like Argo floats.

• Instrument Design
  Simulating how different instrument specifications (e.g., sampling frequency, sensor range, mobility) influence data quality and mission outcomes. This could inform design trade-offs for new oceanographic tools. (we could provide a tutorial on how users could add their own instrument)

• Data-Driven Mission Design
  Integrating with machine learning or statistical models to inform deployment decisions based on historical data or near-real-time forecasts.

Although this has not yet been explored, I can see huge potential for this tool for both research and operational oceanography.
In this sense, I think that keeping the support for direct import inside Python scripts is very important.

[erikvansebille]

Thanks @iuryt, for providing this extensive list; I fully agree that our VirtualShip tool would also be very useful in these cases!

The question, though, is whether we can now already define a stable API for users who want to work on the aspects above. Perhaps we can postpone developing a public API until a next major release? So for the coming months/version focus on the CLI and the internals of the code and then revisit this Issue later? Would that be a good compromise, @VeckoTheGecko?

[VeckoTheGecko]

I think that sounds good @erikvansebille . In the meantime, @iuryt, if you know people those wanting to use virtualship for those usecases they can:

• pin their version of VirtualShip
• work with the existing developper API (https://virtualship.readthedocs.io/en/latest/api/index.html)

Although this API wouldn't be stable, having them use VirtualShip for their research usecase would provide us with valuable feedback on which endpoints or functionality they could benefit from.