Merge pull request #41 from hschellman/Nov25-update

Nov25 update - table of contents and new tokens information .

Author: hschellman

_episodes/01-introduction.md

@@ -12,6 +12,8 @@ keypoints:
 - This tutorial is brought to you by the DUNE Computing Consortium.
 - The goals are to give you the computing basis to work on DUNE.
 ---
+
+{% include 01-introduction.toc.md %}
 ## DUNE Computing Consortium
 
 The DUNE Computing Consortium works to establish a global computing network that will handle the massive data streams produced by distributing these across the computing grid.

_episodes/01.5-documentation.md

@@ -11,6 +11,8 @@ keypoints:
 - There is documentation somewhere!
 ---
 
+{% include 01.5-documentation.toc.md %}
+
 ## Documentation access 
 
 Much of DUNE's computing documentation is public and hosted in github
@@ -77,4 +79,5 @@ Many repositories have wikis or associated dune.github.io pages.
 
 [Computing FAQ](https://github.com/orgs/DUNE/projects/19)
 
-Lists of common connection problems and issues with running jobs. 
+Lists of common connection problems and issues with running jobs. 
+

_episodes/02-storage-spaces.md

@@ -10,6 +10,7 @@ keypoints:
 - CVMFS distributes software and related files without installing them on the target computer (using a VM, Virtual Machine).
 ---
 
+{% include 02.3-cvmfs.toc.md %}
 
 ## CVMFS  
 **What is CVMFS and why do we need it?**  

_episodes/02.3-cvmfs.md

@@ -13,7 +13,9 @@ keypoints:
 - Xrootd allows user to stream data files. 
 ---
 
-#### Session Video
+{% include 03-data-management.toc.md %}
+
+## Session Video
 
 <!--The session will be captured on video a placed here after the workshop for asynchronous study.-->
 
@@ -77,7 +79,11 @@ If you want to process data using the full power of DUNE computing, you should t
 
 ## How to find and access official data
 
-### What is metacat?
+{% include OfficialDatasets_include.md %}
+
+You can also query the catalogs yourself using [metacat][metacat] and [rucio][rucio] catalogs.  Metacat contains information about file content and official datasets, rucio stores the physical location of those files.  Files should have entries in both catalogs.  Generally you ask metacat first to find the files you want and then ask rucio for their location.
+
+## What is metacat?
 
 Metacat is a file and dataset catalog - it allows you to search for files and datasets that have particular attributes and understand their provenance, including details on all of their processing steps. 
 It also allows for querying jointly the file catalog and the DUNE conditions database.
@@ -94,9 +100,9 @@ DUNE runs multiple experiments (far detectors, protodune-sp, protodune-dp hd-pro
 
 To find your data you need to specify at the minimum 
 
-- `core.run_type`  (the experiment)
+- `core.run_type`  (the experiment:  fardet-vd, hd-protodune ...)
 - `core.file_type` (mc or detector)
-- `core.data_tier` (the level of processing raw, full-reconstructed, root-tuple)
+- `core.data_tier` (the level of processing raw, full-reconstructed, root-tuple ...)
 
 and when searching for specific types of data
 
@@ -145,7 +151,8 @@ First get metacat if you have not already done so
  token authentication. 
 {: .callout} -->
 
-### then do queries to find particular sets of files
+### then do queries to find particular groups of files
+
 
 ~~~
 metacat query "files from dune:all where core.file_type=detector and core.run_type=hd-protodune and core.data_tier=raw and core.runs[any]=27331 limit 1"
@@ -240,10 +247,9 @@ Total size:   17553648200600 (17.554 TB)
 {: .output}
 
 
-
 <!-- To look at all the files in that run you need to use XRootD - **DO NOT TRY TO COPY 4 TB to your local area!!!*** -->
 
-## Official datasets <a name="Official_Datasets"></a>
+<!-- ## Official datasets <a name="Official_Datasets"></a>
 
 The production group make official datasets which are sets of files which share important characteristics such as experiment, data_tier, data_stream, processing version and processing configuration. 
 
@@ -335,11 +341,29 @@ fardet-vd:fardet-vd__full-reconstructed__v09_81_00d02__reco2_dunevd10kt_anu_1x8x
 You can also do keyword/value queries like the ones above using the Other tab on the web-based Data Catalog.
 
 ![Full query search](../fig/otherquery.png){: .image-with-shadow }
+ -->
+
+### find out how much data there is in a dataset
+
+Do a query using the `-s` or `--summary` option
+
+~~~
+metacat query -s "files from fardet-vd:fardet-vd__full-reconstructed__v09_81_00d02__reco2_dunevd10kt_anu_1x8x6_3view_30deg_geov3__prodgenie_anu_numu2nue_nue2nutau_dunevd10kt_1x8x6_3view_30deg__out1__v2_official"
+~~~
+{: .language-bash}
+
+~~~
+Files:        20648
+Total size:   34550167782531 (34.550 TB)
+~~~
+{: .output}
 
+this may take a while as that is a big dataset. 
 
+ 
 ### What describes a dataset?
 
-Let's look at the metadata describing that anti-neutrino dataset: the -j means json output
+Let's look at the metadata describing an anti-neutrino dataset: the -j means json output
 
 ~~~
 metacat dataset show -j fardet-vd:fardet-vd__full-reconstructed__v09_81_00d02__reco2_dunevd10kt_anu_1x8x6_3view_30deg_geov3__prodgenie_anu_numu2nue_nue2nutau_dunevd10kt_1x8x6_3view_30deg__out1__v2_official
@@ -386,7 +410,7 @@ You can use any of those keys to refine dataset searches as we did above. You pr
 
 ### What files are in that dataset and how do I use them?
 
-You can either click on a dataset in the web data catalog or:
+You can either locate and click on a dataset in the [web data catalog](https://dune-tech.rice.edu/dunecatalog/) or use the[metacat web interface](https://metacat.fnal.gov:9443/dune_meta_prod/app/gui)  or use the command line:
 
 ~~~
 metacat query  "files from  fardet-vd:fardet-vd__full-reconstructed__v09_81_00d02__reco2_dunevd10kt_anu_1x8x6_3view_30deg_geov3__prodgenie_anu_numu2nue_nue2nutau_dunevd10kt_1x8x6_3view_30deg__out1__v2_official limit 10"
@@ -398,7 +422,7 @@ will list the first 10 files in that dataset (you probably don't want to list al
 You can also use a similar query in your batch job to get the files you want. 
 
 
-### Finding those files on disk
+## Finding those files on disk
 
 To find your files, you need to use [Rucio](#Rucio) directly or give the [justIN](https://dunejustin.fnal.gov/docs/tutorials.dune.md) batch system your query and it will locate them for you. 
 
@@ -417,7 +441,8 @@ export SAM_EXPERIMENT=dune
 -->
 ## Getting file locations using Rucio
 
-### What is Rucio? <a name="Rucio"></a>
+### What is Rucio? 
+<!-- <a name="Rucio"></a> -->
 Rucio is the next-generation Data Replica service and is part of DUNE's new Distributed Data Management (DDM) system that is currently in deployment. 
 Rucio has two functions:
 1. A rule-based system to get files to Rucio Storage Elements around the world and keep them there.
@@ -427,7 +452,7 @@ As of the date of the 2025 tutorial:
 - The Rucio client is available in CVMFS and Spack
 - Most DUNE users are now enabled to use it. New users may not automatically be added. 
 
-### You will need to authenticate to use read files
+### You will need to authenticate to read files
 
 > #### For SL7 use justin to get a token
 {:.callout}
@@ -498,7 +523,7 @@ which  the locations of the file on disk and tape. We can use this to copy the f
 > Try to access the file at manchester using the command:
 > ~~~
 > root -l root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/fardet-vd/fd/a6/prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250217T033222Z_gen_004122_supernova_g4stage1_g4stage2_detsim_reco.root
-> _file0->ls
+> _file0->ls()
 > ~~~
 > {: .language-bash}
 {: .challenge}

_episodes/03-data-management.md

@@ -17,7 +17,7 @@ keypoints:
 > You need to be in the Apptainer to use it. 
 > UPS is being replaced by a new [spack][Spack Documentation] system for Alma9.  We will be adding a Spack tutorial soon but for now, you need to use SL7/UPS to use the full DUNE code stack.   
 > 
-> Go back and look at the [SL7/Apptainer]({{ site.baseurl }}/setup.html#SL7_setup) instructions to get an SL7 container for this section. 
+> Go back and look at the [SL7/Apptainer]({{ site.baseurl }}/sl7_setup) instructions to get an SL7 container for this section. 
 {: .challenge}
 
 An important requirement for making valid physics results is computational reproducibility. You need to be able to repeat the same calculations on the data and MC and get the same answers every time. You may be asked to produce a slightly different version of a plot for example, and the data that goes into it has to be the same every time you run the program. 

_episodes/03.2-UPS.md

@@ -10,6 +10,8 @@ keypoints:
 - Spack is a tool to deliver well defined software configurations
 - CVMFS distributes software and related files without installing them on the target computer (using a VM, Virtual Machine).
 ---
+
+{% include 04-Spack.toc.md %}
 ## What is Spack and why do we need it?
 
 > ## Note 

_episodes/04-Spack.md

@@ -10,11 +10,13 @@ keypoints:
 - CPU, memory, and build time optimizations are possible when good code practices are followed.
 ---
 
-#### Session Video
+## Improve your Code efficiency
+
+### Session Video
 
 The session will be captured on video a placed here after the workshop for asynchronous study.
 
-#### Live Notes
+### Live Notes
 
 <!-- Participants are encouraged to monitor and utilize the [Livedoc for May. 2023](https://docs.google.com/document/d/19XMQqQ0YV2AtR5OdJJkXoDkuRLWv30BnHY9C5N92uYs/edit?usp=sharing) to ask questions and learn.  For reference, the [Livedoc from Jan. 2023](https://docs.google.com/document/d/1sgRQPQn1OCMEUHAk28bTPhZoySdT5NUSDnW07aL-iQU/edit?usp=sharing) is provided.
 --> 

_episodes/05.1-improve-code-efficiency.md

@@ -5,37 +5,40 @@ keypoints:
 - Errors that people report in doing the tutorial
 --- 
 
-- ####  `/usr/bin/xauth: unable to write authority file`
-  #### `disk quota exceeded error with metacat auth login`
+## Common Error Messages 
 
-    These likely means your kerberos ticket was not forwarded and you can't access your home are without it.  do a kinit in your terminal session.  Or possibly you really have filled your home area. 
 
-- #### `bash: setup: command not found`
+{% include Common-Error-Messages.toc.md %}
 
-  setup is a UPS command.  You need to be running in the Apptainer and setup the DUNE ups system - check out the instructions in [SL7 setup]
-  ({{ site.baseurl }}/sl7_setup)
+###  Error: /usr/bin/xauth: unable to write authority file
 
+These likely means your kerberos ticket was not forwarded and you can't access your home are without it.  do a kinit in your terminal session.  Or possibly you really have filled your home area. 
 
-- #### `SyntaxError: future feature annotations is not defined` 
+### bash: setup: command not found
 
-    This looks like a bad python version, try doing `which python` if it isn't > 3.9 you don't have a modern python version.
+setup is a UPS command.  You need to be running in the Apptainer and setup the DUNE ups system - check out the instructions in [SL7 setup]({{ site.baseurl }}/sl7_setup)
 
-    - On SL7 we suggest setting up the dunesw as shown in the example setup. alternatively you can 
 
-        ~~~
-        setup root -v v6_28_12 -q e26:p3915:prof
-        ~~~
-        {: .language-bash} 
+### SyntaxError: future feature annotations is not defined
 
-    - On AL9 we suggest loading ROOT which brings in a modern version of python and allows xrootd access to data. 
+This looks like a bad python version, try doing `which python` if it isn't > 3.9 you don't have a modern python version.
 
-        ~~~
-        spack load root@6.28.12
-        ~~~
-        {: .language-bash} 
+- On SL7 we suggest setting up the dunesw as shown in the example setup. alternatively you can 
 
+~~~
+setup root -v v6_28_12 -q e26:p3915:prof
+~~~
+{: .language-bash} 
 
-- #### Spack ==> `Error: somecode matches multiple packages
+- On AL9 we suggest loading ROOT which brings in a modern version of python and allows xrootd access to data. 
+
+~~~
+spack load root@6.28.12
+~~~
+{: .language-bash} 
+
+
+### Spack : Error: somecode matches multiple packages
   ~~~
   Matching packages:
     jhpj2js somecode@6.28.06%gcc@12.2.0 arch=linux-almalinux9-x86_64_v2

_extras/Common-Error-Messages.md

@@ -12,13 +12,17 @@ keypoints:
 - It is also something almost all people who get paid to program are expected to know well
 ---
 
-## 0. Back up your machine
+## Computer setup
+
+{% include ComputerSetup.toc.md %}
+
+### Back up your machine
 
 We are going to be messing with your operating system at some level so it is extremely wise to do a complete backup of your machine to an external drive right now.
 
 Also turn off automatic updates.  Operating system updates can mess with your setup.  Generally, back up before doing updates so you can revert if necessary.
 
-## 1. Open a unix terminal window
+### Open a unix terminal window
 
 First figure out how to open a terminal on your system.  The Carpentries Shell Training has a [section that explains this][New Shell]
 
@@ -35,7 +39,7 @@ On Windows it's a bit more complicated as the underlying operating system is not
 
 
 
-## 2. Learn how to use the Unix Shell
+### Learn how to use the Unix Shell
 
 <!-- First figure out [how to open a terminal on your system][New Shell]
 -->
@@ -47,7 +51,7 @@ It tells you how to start a terminal session in Windows, Mac OSX and Unix system
 Please do that [unix shell tutorial][Unix Shell Basics] to learn about the basic command line.
 
 
-## 3. Install an x-windows emulator
+### Install an x-windows emulator
 
 #### MacOS
 
@@ -88,7 +92,7 @@ See the information about [Windows]({{ site.baseurl }}/Windows.html) terminal co
 > You should now be ready to go for the ({{ site.baseurl }}/setup) 
 {: .callout}
 
-## Extra - Get a compiler/code editor
+### Extra - Get a code editor
 
 Although you will mainly be using python to code to begin with, most HEP code is actually C++ and it is good to have access to a C++ compiler.  Bonus is that you normally get a good editor as well.
 
@@ -108,7 +112,7 @@ You can also use vim or emacs if you are old school.
 Likely you should load up the full [Visual Studio][Visual Studio] as it has a nice C++ compiler
 
 
-### Useful Links
+## Useful Links
 
 [HSF Training Center][HSF Training Center]