Reorganise sections?

[dougiesquire]

I've been doing a full read-through of the book to try and decide where a new section on "recommendations when using conda" might best fit. I'm wondering if some reorganisation of sections might help users and contributors. I've had a stab at an example draft outline below to start some discussion.

Please note that I'm not wedded to this proposal at all, but I thought I should mention to the team that as the current structure evolves, I'm starting to find it difficult to know where to look for specific things.

---

Overview - as is, but add table-of-contents providing details of what each section aims to do.

Introduction - new section pulling info from a few existing sections and providing context for what's to come. Very high-level concepts like "there are lot's of computation and storage resources in Australia", "for Big Data, data storage and compute must be considered together", "for Big Data it's best to utilise compute close to the data"... Include some of the list of platforms here (https://acdguide.github.io/BigData/platforms/platforms-intro.html), but save the "analysis environments" (e.g. OOD, ARE) for the Computation section below

Data storage - overview taken from https://acdguide.github.io/BigData/data_storage.html#about-large-scale-data

• Data standards - new subsection giving high level overview of CF conventions (and others?) and pointing to other resources
• Chunking - taken from https://acdguide.github.io/BigData/chunking.html
• Data formats - includes subsubsections on netcdf (https://acdguide.github.io/BigData/data_storage.html#netcdf, https://acdguide.github.io/BigData/format_metadata.html, https://acdguide.github.io/BigData/data_storage.html#netcdf-zarr), zarr (https://acdguide.github.io/BigData/data_storage.html#zarr), etc
• Analysis-ready data - taken from https://acdguide.github.io/BigData/data_storage.html#advice-on-writing-datasets-for-efficient-use, also subsubsection on Pangeo Forge from https://acdguide.github.io/BigData/data_storage.html#pangeo-forge-an-open-source-framework-for-extraction-transformation-and-loading-of-scientific-data
• Tools for streamlining data access - taken from https://acdguide.github.io/BigData/accessing_data.html#methods-of-accessing-data-and-metadata

Computation - generic overview taken from https://acdguide.github.io/BigData/computations.html#general-tools, https://acdguide.github.io/BigData/computations.html#command-line-tools and https://acdguide.github.io/BigData/computations.html#other-languages-matlab-r-etc

• Analysis environments - New section including tools like conda, Jupyter and platforms like ARE, OOD (taken from https://acdguide.github.io/BigData/platforms/platforms-nci-ood.html), EasyHub?...
• Chunking - "Some tools provide explicitly chunked data models that enable you to distribute your computations across multiple chunks in parallel". Different from the Data storage/Chunking section in that it provides a computational perspective. Show Scott's great examples using dask (https://acdguide.github.io/BigData/computations.html#common-tasks)
• Tools for efficient computation - taken from https://acdguide.github.io/BigData/tools/intro.html#available-tools-and-their-best-use
• Examples - New section for including example notebooks/scripts??

Resources - taken from https://acdguide.github.io/BigData/resources.html#resources

---

I think this includes all the existing sections/information (and adds a few). I'm not sure about the division into "Data storage" and "Computation" sections because they're so heavily related. I'm interested to hear what people think (@paigem, @hot007, @paolap). Please do feel free to tell me that this is not needed.

P.S. There are also a number of "hints" and "asides" scattered throughout the book. I wonder if putting these boxes would help with clarity (e.g. https://acdguide.github.io/BigData/accessing_data.html#working-with-authorised-catalogues)?

[dougiesquire]

Just noticed that some of this may already have been addressed in #65

[paolap]

We should probably merge that, we were waiting for Paige review, than we can use it at starting point for further changes, the structure can definitely be improved and as you said as we add content some of the initial structure might not make sense anymore.

[paolap]

Now that I'm reading this more carefully, formatting was addressing inconsistencies at notebook level not such a restricting, I like the idea of restructuring and I'm also finding difficult to separate some of the new sections from the data storage and the computation part. I'll like to have a go at a draft roughly following Dougie suggestions in a separate branch so we can then share it at the meeting on Thursday otherwise it might be tricky to visualise this.

[paolap]

After starting the process I'm wondering if we should have 3 broad categories

Data structure including:

• data formats,
• metadata, for this we should relate to the Governance book for conventions and general information and write here only about implication of following/breaking the standards
• chunking introduction,
• anything else that describe the data and/or conversion between data formats

Analysis/computations including:

• common analysis tasks (broadly speaking the computations notebook)
• working in parallel (NB there's an entire training from Scott on this, we might be able to leverage on)
• dask (we mentioned it already but more advanced examples, including chunking aspects)
• timeseries specific examples
• Machine Learning / gpu ?? it would be empty now, but there's definitely interest and new project going around
• ...

** Platforms/tools** basically anything that defines a working environment, the platform available, the intake, pangeo, etc data collections, the packages and pre-defined software environments. Including:

• Computing platform, I think it work as it is I'm not convinced we need to separate OOD and ARE, we are discussing jupyter, jupyterlab etc in tools but these are specific examples of working environments
• Analysis ready data (as in Dougie's comment)
• tools for streamlining data access (as in Dougie's comment)
• software environments (we could put it the last addition from Dougie, I currently placed it in tools but possibly that and the Community section from tools: (https://acdguide.github.io/BigData/tools/tools-python1.html#community) should be moved here
• tools, NB I added in my example a Julia section as it is gaining popularity

Resources section to close, as it is but making sure it includes all the materials we are listing elsewhere, there's already a section for example workflows in there.

As said before I'm trying to get an example of this before Thursday, as it might help us visualise the final product and make it easier moving sections around

[dougiesquire]

Thanks @paolap. Yes, it could be more extensible to have the 3 categories you suggest. One thing I notice with your new proposal is that some highly-related sections are now quite separated (e.g. analysis tasks and software environments). So we'd want to make sure we link things clearly in the text. Also it's not clear to me what constitutes a "tool". For example, the section on dask sits in "Analysis/computations" in your proposal, not in "Platforms/tools". Maybe we can come up with some clear definitions to help future contributors?

The big difference I see between my and your suggestions is that mine includes a "platforms/tools" subsection within each of the "data" and "computation" sections, whereas yours breaks this out into a new section. I see pros and cons to both. Perhaps we could see what others think in our upcoming meeting?

[paolap]

Yes more clarity would be great, it is probably the terms I'm using that are making the two approaches look more different than they are.

Also my approach is constantly shifting the more I try to fit what we got so far into some sort of structure. So what I'm currently trialling is a bit different from what I've written which stemmed from an attempt to apply your suggestions :-)

The "tools" section as it is currently it is meant as a list of useful software that can be linked (mostly they're in a glossary form) from other part of the books. So while there are some comparisons between software falling in the same category, there are not actual examples on how to use any of them.
For example, in my approach, dask would have an introduction in the "tools" section, (as it has currently) but also a page with "dask in practice" examples and tips in the analysis/computation section.

Similarly chunking appears in data format as an introduction to the topic, but then it will be expanded/demonstrated in the computations (as Scott basically as already done in his notebook) and in the dask examples section.

It will take a while to find a good structure, I'm aware that the changes I'm trying to get together in a separate branch might not work, and will end up in a potential waste of time, but I'm finding really hard to think of a different structure without actually moving files around or even sections of text from one file to another.

[dougiesquire]

Great - thanks for having a stab at something. I think that's a great way to start and we can iterate from there if we want to

[paigem]

I really like the ideas here for reorganizing this book! Thank you @dougiesquire and @paolap for pushing these ideas forward!

I think it will be easier for me (and others) to give feedback if we can see the updates that @paolap is making in the book, so I'll hold off on comments until then. Thanks for getting a working example of this going @paolap!

[Thomas-Moore-Creative]

Thanks for pushing your restructured branch @paolap > https://github.com/ACDguide/BigData/tree/restructure_paola
The instructions and local build worked fine for me.

Can we recap what we think the next steps are? A further discussion of the proposed new structure here in this issue #69 ?

[paolap]

Steps from here are:

• try to come up with your own restructuring locally from the main branch
• alternatively you can start from my branch restructure_paola
• if you are short on time just add comments to either of them
  Then we can discuss this again at the next meeting and hopefully come out with at least an initial restructure we all feel could work. As Dougie said we're not going to get it right.

One thing we all seems to agree is that we need clear overviews at the start of the book and at the start of each chapter. So potential users can come up to speed.
So far we individuated 3 possible kind of users:

• new to climate data analysis
• experienced looking for specific advice
• familiar with climate science analysis moving to a different system/language/institution etc.

Might be nice to show were possible two approaches for examples we are showing, one maybe less efficient but simpler to adopt, and more advanced example for experienced users.

Other comments on building a book and my branch that I sent via email:
For help building the jupyter book locally:

https://github.com/pabloinsente/jupyter-book-tutorial
and
https://jupyterbook.org/en/stable/start/your-first-book.html

There’s a few warnings that will pop up the first time you build the book, they can usually be ignored, subsequently warnings are repeated only for the files you actually modified.
If you think the books isn’t showing what you expect, then clear up previous builds, basically in BigData/BigData folder:

rm -rf _build/

Finally, I tried not to remove any content, just move it around, but I might have missed out something and there are a few things I added:

• Machine learning and dask skeleton notebooks
• Content on compression in the data section
• Julia to the newly renamed software part

[Thomas-Moore-Creative]

@paolap - FYI just subbed a trivial PR for typos, largely as a test of my GitNoob skills PRing from a patched non-main branch in a fork.

[dougiesquire]

Just noting that I am planning to have a stab at a reorganised structure early next week (probably building off @paolap's branch) - sorry for the delay in getting to this!

[dougiesquire]

I started going through and trying to reorganise according to my initial comment in this issue and now I feel your pain @paolap! I'm not sure it's sensible for us to all try and reorganise the book, as this is very fiddly/time-consuming and it will be very difficult to consolidate our attempts. Instead, perhaps it's more feasible for us to all go through the current main structure and take notes on what we like/don't like, what could be added/clarified/moved, etc. Then we can compare notes in the next meeting, see where we overlap, and try to come up with a structure that best addresses everyones comments. I'd be happy to implement whatever we arrive at after the meeting.

I think the structure has improved substantially since opening this issue thanks to @paolap's effort. My notes are:

• Overview - remove "Overview" from navbar, add some more introductory content around who the target audience is
• Computing Platforms - add paragraph to intro recommending utilising compute close to the data, rather than downloading data
• Computing Platforms - Add details about storage options to all relevant platforms (as is done already for NCI)
• Methods of accessing data and metadata - Some of this is really written as a generic description of tools at the moment. Could we split out the stuff that is relevant to users wanting to access data on the computing platforms (i.e. the NCI Intake Catalogue atm) and move the rest to Available tools?
• Metadata - I’m not sure this heading is necessary. Could we move Data formats, variables and metadata into Data Storage?
• Computations - I think this could benefit from some reorganising. My opinion is that the concept of chunking needs to be introduced earlier on. Currently we show lot’s of in-depth computations that use chunking before we talk about what chunking is. Something like the structure I propose in the Computation section of my original post could work?
• Resources - I think the expandable headings are unnecessary.

I think this restructure exercise will be most effective/easiest if we can get multiple people comparing what works or doesn’t for them. @hot007, @paigem, @Thomas-Moore-Creative, @AliciaTak, might you guys have time to make your own notes prior to our next meeting on Sept 8th (your notes might simply be "I like it exactly as it is", which would be great).