Skip to content

Home

Jump to bottom
Ashley Smith edited this page Jan 26, 2019 · 17 revisions

Written by Ashley Smith

The aim of this wiki is to collate information on development of VirES, focussing on viresclient, and to provide guidance on contribution. This is the dev counterpart to the user-focused documentation on readthedocs.

VirES comprises three main components: the server; the web client; the python client. There is overlap between these, so discussion of development on viresclient will also involve the other components. Much of the work to be done is in response to formal requirements to EOX from ESA/DISC, so this will be carried out by EOX and will be identified as such. Other work is suitable for the wider community to provide - if there is a missing feature which you would like, please raise it on the GitHub issue tracker or otherwise get in touch. We can then discuss how to proceed with implementing it.

The SunPy project is a good example of how to manage this project as it grows and I will continue to investigate management processes we could adopt.

Follow progress within HelioPython (http://heliopython.org/meetings/), in particular about the possibility of viresclient as a data source for https://github.com/heliophysicsPy/helio_downloader and recommendation of using pycdf (https://github.com/spacepy/spacepy)

See Python paper "Snakes on a Spaceship" (https://doi.org/10.1029/2018JA025877)

Vision of viresclient

I would like viresclient itself to not be specific to individual missions and research domains, i.e. that it supports any VirES server - which today means "VirES for Swarm" and "VirES for Aeolus" (not yet implemented). This means that additional features should be kept minimal, focusing on generic data access and simple plots to inspect the data. Dependencies should be kept minimal so as to limit the complexity and promote maintainability. Other packages will be built on top of viresclient to provide domain-specific functionality, so it is important to keep viresclient simple and reliable. I expect that viresclient will frequently be used within Jupyter notebooks, but also within processing chains and Python-based web services, so this should be kept in mind when designing interfaces.

Aims

  • Users shouldn't need to know about file formats & original data/model organisation/implementation
  • Provide access to data structures which support analysis immediately (i.e. pandas etc.)
  • Make it easy to share analyses with others (i.e. share reproducible notebooks)
  • Flexible to easily work together with other libraries
  • Save researchers' time!

See also: blog post, Swarm DQW presentation

viresclient dependencies

  • Jinja2: renders XML templates that form WPS requests
  • tqdm: for progress bars
  • cdflib: pure python access to CDF files
  • pandas, xarray, tables

Wider usage of viresclient

To demonstrate usage of viresclient and to help spread ideas and code recipes within the research community, I have set up a repository of Jupyter notebooks to act as a kind of cookbook. I hope people will submit modifications and additions, or maintain their own repositories which I will gladly link to from the documentation. This should also serve to prototype and identify new features that could be added to VirES (in the client, the server, or web client).

There are many possibilities for applications that can be built using viresclient to provide easy data access. To start with:

  • a geomagnetic field modelling library
    • typical data filtering routines
    • multiple options for SHA
    • tools to analyse field models, e.g. power spectra plots etc.
    • eoxmagmod, SHTOOLS, ChaosMagPy are good places to start
      • it would be useful to refactor eoxmagmod to make it easily installable, and to document it
  • FAC calculation

These packages would be separate from but affiliated with viresclient: they depend on viresclient, but can also use other tools.

Related projects

Clone this wiki locally