01_HEPandBigData.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center>\n",
    "<h1><b><span style=\"color:blue\">Introduction to the Scikit-HEP<br>Big Data Python ecosystem for analysis in Particle Physics</span></b></h1>\n",
    "<h3>Eduardo Rodrigues<br>University of Liverpool</h3>\n",
    "\n",
    "<h3><span style=\"color:gray\"><a href=\"https://indico.in2p3.fr/event/20306/\">ESCAPE Summer School 2021</a><br>Data Science for Astronomers, Astroparticle and Particle Physics, 7-18 June 2021</span></h3>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "<table>\n",
    "<tr style=\"background: white;\">\n",
    "    <td>\n",
    "    <b>\n",
    "    The School is organized in the framework and with the support of the European Science Cluster of Astronomy\n",
    "    &amp; Particle physics ESFRI research infrastructures (ESCAPE),\n",
    "    funded by the European Union's Horizon 2020 - Grant N. 824064.\n",
    "    </b>\n",
    "    </td>\n",
    "</tr>\n",
    "<tr style=\"background: white;\">\n",
    "    <td align=\"center\"><img src=\"images/logo_Escape.png\" width=\"15%\"><img src=\"images/logo_flag-of-europe.webp\" width=\"15%\"></td>\n",
    "</tr>\n",
    "</table>\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Abstract\n",
    "\n",
    "Data analysis in High Energy Physics (HEP) has evolved considerably in recent years. In particular, the role of Python has gained\n",
    "much momentum, sharing at present the show with C++ as a language of choice.\n",
    "Several (community) domain-specific projects have seen the day, providing (HEP) data analysis packages that profit from, and talk to well with,\n",
    "the huge Python scientific ecosystem, which navigates around NumPy and friends.\n",
    "This lecture introduces the Scikit-HEP project, which I started in late 2016 with a few colleagues from various backgrounds\n",
    "and domains of expertise.\n",
    "Scikit-HEP is a community-driven and community-oriented project with the aim of providing Particle Physics at large\n",
    "with a Big Data ecosystem for analysis in Python. It has developed considerably in the past couple of years,\n",
    "and is now part of the official software stack of the experiments ATLAS, Belle II, CMS and KM3NeT.\n",
    "In this lecture ample time will be provided to \"play around\" with the material, in Jupyter notebooks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Lecture material\n",
    "\n",
    "- Notebooks GitHub repository https://github.com/eduardo-rodrigues/2021-06-14_ESCAPESummerSchool.\n",
    "- Can be nicely viewed [with nbviewer](https://nbviewer.jupyter.org/github/eduardo-rodrigues/2021-06-14_ESCAPESummerSchool/tree/master/).\n",
    "\n",
    "(Launching with Binder from the Github repository? Please be patient, the repository may take of order 5 minutes to load completely on Binder.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## **1.1 Particle Physics and Big Data**\n",
    " \n",
    "A lot of what has happened in the HEP Python ecosystem can be thought of as trying to **bridge the Particle Physics and Big Data worlds** and **profit from what the Data Science scientific software stack has to offer**.\n",
    "\n",
    "We will come back to software, but what about the data itself? Is that \"Big Data\"?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**The CERN [ROOT team](https://root.cern/) advertises that of the order of 1 EB - 1 million TB! - of data exists right now in the `.root` format.**\n",
    "\n",
    "<center><img src=\"images/intro_ROOT.png\" width=\"70%\"/></center>\n",
    "\n",
    "Impressive. **We are already in the exascale era!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "For the record, the CERN Data Centre had accumulated more than 200 PB of data back in 2017 already!\n",
    "[[CERN news, July 6, 2017](https://home.cern/news/news/computing/cern-data-centre-passes-200-petabyte-milestone)]:<br>\n",
    "<center><img src=\"images/intro_CERN_200PB.png\" width=\"40%\"/></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "And in just an extra 1.5 years, 50% more data got saved in CERN's Data Centre.\n",
    "Citing the [CERN news, March 1, 2019](https://home.cern/news/news/computing/lhc-pushing-computing-limits),<br>\n",
    "> \"*The CERN Advanced Storage system (CASTOR), which relies on a tape-based backend for permanent data archiving, reached **330 PB of data (equivalent to 330 million gigabytes) stored on tape, an equivalent of over 2000 years of 24/7 HD video recording**. In November 2018 alone, a record-breaking 15.8 PB of data were recorded on tape, a remarkable achievement given that it corresponds to more than what was recorded during the first year of the LHC’s Run 1.*\"\n",
    "\n",
    "In fact, in 2018, over 115 PB of data in total (including about 88 PB of LHC data) were recorded on tape."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The accummulation of data generated by the [LHC](https://home.cern/science/accelerators/large-hadron-collider) experiments alone, over a decade-ish, speaks for itself, as seen by this graph on CERN computing:<br> \n",
    "data (in terabytes) recorded on tapes at CERN month-by-month (2010–2018) [[ref](https://cds.cern.ch/images/CERN-HOMEWEB-PHO-2019-003-1)]:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><img src=\"images/intro_CERN_Graphique_DataTapes_web_1920x1080p.png\" width=\"75%\"/></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**And what about the outside world?**\n",
    "\n",
    "Let's look at Amazon for the sake of argument [[ref](https://aws.amazon.com/snowmobile/)]:\n",
    ">\"*AWS Snowmobile is an Exabyte-scale data transfer service used to move extremely large amounts of data to AWS. You can transfer up to 100PB per Snowmobile, a 45-foot long ruggedized shipping container, pulled by a semi-trailer truck.*\"\n",
    "<center><img src=\"images/intro_aws-snowmobile.jpg\" width=\"50%\"/></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "> \"*Each Snowmobile includes a network cable connected to a high-speed switch capable of supporting 1Tbps of data transfer spread across multiple 40Gbps connections.*\"\n",
    "\n",
    "This is to be compared with the throughput of 1-2 Tbps the [LHCb experiment](http://lhcb-public.web.cern.ch/)'s first High-Level Trigger (HLT1, partial reconstruction on GPUs) will put to buffer while the real-time calibration and alignment is run, which is needed to digest the data in the second stage, HLT2 (full reconstruction) ... next year!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### **Is it relevant and useful to learn non-HEP tools?**\n",
    "\n",
    "We've just quickly recalled that data requirements for Particle Physics match those of the Big Data world. Huge amounts of data are in fact used by companies worldwide for just about any business, see for example the report \"State of Data Science and Analytics, IDC InfoBrief, April 2019\":\n",
    "<center><img src=\"images/intro_data_in_business.png\" width=\"50%\"/></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "International surveys indicate **over 50M data workers worldwide! Can we really compete in terms of tools for data analysis?\n",
    "... Or should we rather try and profit from, and even contribute to, the huge ecosystem available to do Data Science?**\n",
    "\n",
    "Anyway, what are data scientists, data engineers, etc., using for their daily work? That largely involves to some (larger and larger) extent Machine Learning, statistics, and even AI. International surveys give a hint ... the **tools are dominated by Python tools ...!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "From the 2019 Figure Eight report \"The State of AI and Machine Learning\",\n",
    "> \"*Some popular frameworks and tools technical practitioners prefer in different stages of the ML pipeline are: Numpy and Pandas for loading data; Matplotlib for visualization; Scikit-learn and TensorFlow (including Keras) for ML models.*\":<br>\n",
    "<center><img src=\"images/intro_popular_python_packages.png\" width=\"40%\"/></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### **The reign of Python**\n",
    "\n",
    "One may say that 2019 was the year for Python. Now it simply is more popular than anything else:\n",
    "\n",
    "<center><img src=\"images/intro_python_popularity.png\" width=\"55%\"/></center>\n",
    "\n",
    "The trend is only more clear when checking the popularity of programming languages for Machine Learning, BTW."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Particle Physics is clearly following this same trend - Python ever more popular for/in analysis. Here is an example study from CMS:\n",
    "\n",
    "<center><img src=\"images/intro_python_adoption_CMS.png\" width=\"55%\"/></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "\n",
    "<h4><b>Take aways</b></h4>\n",
    "\n",
    "- Particle Physics and Data Science deal with Big Data and share requirements.\n",
    "- The Data Science world has over the years built an extensive, powerful, well maintained and documented software ecosystem.\n",
    "- It would be real shame for Particle Physics not to profit from it, as user but potentially also as a contributor.\n",
    "- Python is the programming language of choice.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## **1.2 The \"Python in HEP\" Big Data ecosystem**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "So, we concluded that:\n",
    "- The Data Science world relies on an extensive, powerful, well maintained and documented software ecosystem in Python.\n",
    "- Particle Physics has much to gain if engaging and contributing.\n",
    "\n",
    "It's a lot about making it possible, natural and efficient to perform HEP analysis in the NumPy ecosystem ...!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### **The central role of NumPy**\n",
    "\n",
    "The (vast) scientific Python ecosystem can be organised, schematically, as a layered set of libraries and packages ever more specialised, atop a series of building blocks. The [NumPy library](https://numpy.org/) is a central pillar."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><img src=\"images/PyHEP_scientific_python_ecosystem.png\" alt=\"Scikit-HEP project motivation\" width=\"65%\"/></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "This fact has been very recently acknowledged by Nature. Quoting [ZDNet (25/09/2020)](https://www.zdnet.com/article/numpys-contribution-to-python-is-remarkable-but-where-it-goes-next-could-be-even-more-so/),\n",
    "> \"Something fascinating happened in the world of scientific publishing last week: The prestigious journal Nature featured an overview of a 15-year-old programming library for the language Python. The widely popular library, called NumPy, gives Python the ability to perform scientific computing functions. \""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The Nature article, [Harris, C.R., Millman, K.J., van der Walt, S.J. *et al.*, Array programming with NumPy, Nature 585, 357–362 (2020)](https://www.nature.com/articles/s41586-020-2649-2), features in particular the following diagram,\n",
    "<center><img src=\"images/PyHEP_NumPy_ecosystem.png\" width=\"650px\"/></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    ", which provides a view of the contemporary Python scientific ecosystem, from foundational libraries to domain-specific and application-specific libraries. \n",
    "\n",
    "BTW, if you look carefully, you will see above a single HEP \"library\", namely Scikit-HEP!\n",
    "Aside the fact that the authors misinterpreted the scope of the [Scikit-HEP project](https://scikit-hep.org/), which rather is a *domain-specific project*, this is excellent news for us (HEP), to show up in such a publication, since the appearance acknowledges our position out there! Scikit-HEP will be the focus of the next notebooks ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### **Intermezzo - The HEP Software Foundation**\n",
    "\n",
    "The goal of the [HEP Software Foundation (HSF)](https://hepsoftwarefoundation.org/) is to *facilitate coordination and common efforts in software and computing across HEP in general*.\n",
    "It was founded in 2014, explicitly to address current and future computing & software challenges in common."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><img src=\"images/PyHEP_HSF_raison-d-etre.png\" width=\"60%\"/></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### **PyHEP - \"Python in HEP\"**\n",
    "\n",
    "The HSF comprises a series of Working Groups and (less formal) Activity Areas ... among which the [**“Python in HEP” WG**](https://hepsoftwarefoundation.org/workinggroups/pyhep.html).\n",
    "\n",
    "<center><img src=\"images/PyHEP_within_the_HSF.png\" width=\"400px\"/></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "As the WG homepage says,\n",
    "\n",
    "> *The PyHEP working group brings together a community of developers and users of Python in Particle Physics, with the aim of improving the sharing of knowledge and expertise. It embraces the broad community, from HEP to the Astroparticle and Intensity Frontier communities.*\n",
    "\n",
    "Several of the [HSF Gitter channels](https://gitter.im/HSF/) relate to PyHEP topics, and are the means by which everyone communicates and exchanges informations, experiences, and asks questions ...\n",
    "\n",
    "Feel free to join. It's trivial!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### **The PyHEP workshops**\n",
    "\n",
    "These are major events for the WG. Check out the forthcoming [PyHEP 2021 workshop](https://indico.cern.ch/e/PyHEP2021) ...\n",
    "\n",
    "<table align=\"left\" border=\"0\">\n",
    "    <tr style=\"background: white;\">\n",
    "        <td style=\"width:100px\">&nbsp;</td>\n",
    "        <td align=\"center\"><img src=\"images/logo_PyHEP.png\" width=\"100px\"/></td>\n",
    "        <td style=\"width:40px\">&nbsp;</td>\n",
    "        <td align=\"center\"><img src=\"images/PyHEP_workshops.png\" width=\"500px\"/></td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### **\"PyHEP community projects\"**\n",
    "\n",
    "A series of Python projects have seen the light in recent years, where by *projects* I select endeavours that provide one or more Python libraries *with a community around it*. ROOT and SWAN (both are CERN projects) presently share the arena with a few community projects that build around the general Python scientific ecosystem instead of on ROOT:\n",
    "\n",
    "<table align=\"left\" border=\"0\">\n",
    "    <tr style=\"background: white;\">\n",
    "        <td style=\"width:150px\">&nbsp;</td>\n",
    "        <td>\n",
    "            <h4><a href=\"https://github.com/CoffeaTeam\">Coffea</a></h4>\n",
    "            <h4><a href=\"https://github.com/FAST-HEP\">FAST-HEP</a></h4>\n",
    "            <h4><a href=\"https://root.cern\">ROOT</a></h4>\n",
    "            <h4><a href=\"https://scikit-hep.org\">Scikit-HEP</a></h4>\n",
    "            <h4><a href=\"https://cern.ch/swan\">SWAN</a></h4>\n",
    "            <h4><a href=\"https://github.com/zfit\">zfit</a></h4>\n",
    "        </td>\n",
    "        <td style=\"width:50px\">&nbsp;</td>\n",
    "        <td align=\"center\"><img src=\"images/PyHEP_projects.png\" width=\"250px\"/></td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**Scikit-HEP** is:\n",
    "- The one I co-founded in late 2016 with a few colleagues, hence intimately involved with.\n",
    "- The oldest of such projects.\n",
    "- The one with more libraries provided.\n",
    "- The project on which all of the others (i.e. non-CERN) depend on (they depend on at least one of Scikit-HEP libraries).\n",
    "\n",
    "For these reasons the next few notebooks will focus on Scikit-HEP packages. The last notebook will work with the 2 main packages of the `zfit` project."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div class=\"alert alert-info\">\n",
    "\n",
    "<h4><b>On these PyHEP community projects and libraries</b></h4>\n",
    "\n",
    "1) The ecosystem that is going to be showcased in these tutorial notebooks is already advanced enough to perform analysis in it. All the packages are written and maintained by other physicists or scientists in general. As any other library, these packages are neither perfect (bug free) nor do they have all the features that you may want today. They are, however, built for simple contributions: if there are missing features or bugs, please report or directly contribute with code and become part of the ecosystem!\n",
    "\n",
    "2) Several of the libraries of this \"HEP ecosystem\" are in fact NOT domain specific! Do feel free to use them :-) (and anything you may find interesting or useful, it \"goes without saying\").\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### **For reference - some useful links collected in one place:**\n",
    "\n",
    "#### HSF PyHEP Working Group\n",
    "Information on the [HEP Software Foundation](hepsoftwarefoundation.org/) PyHEP (\"Python in HEP\") Working Group.\n",
    "- Webpage: https://hepsoftwarefoundation.org/workinggroups/pyhep.html\n",
    "- Gitter channel: https://gitter.im/HSF/PyHEP\n",
    "\n",
    "#### Scikit-HEP project\n",
    "- Website: https://scikit-hep.org/\n",
    "- GitHub: https://github.com/scikit-hep/\n",
    "- Gitter channel: https://gitter.im/Scikit-HEP/community"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### **How to explore these lectures**\n",
    "\n",
    "- **The notebooks are topical and self-consistent for you to run though them at your own pace and leisure.\n",
    "Run what topics sound appealing to you ...**\n",
    "\n",
    "- You will also find some corresponding exercises notebooks. As the names indicate, they suggest exercises beyond the introduction to the various topics and related packages. Again, feel free to decide what to try out.\n",
    "\n",
    "- Be warned, I am <i>no</i> expert on all packages! But I will try and help as much as possible.\n",
    "\n",
    "- You liked these tutorials? Consider dropping a line and/or giving the GitHub repository a star, as that's a trivial way to convey positive feedback."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}