Merge pull request #164 from csdms/egp/add-envs-lesson

Add intro lesson on package managers

Author: ethan-pierce

lessons/conda/environments.ipynb

@@ -0,0 +1,174 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Python environments and package management\n",
+    "\n",
+    "Python was first released in 1991. Our current version, Python 3.0, was first released in 2008. This means we have 30+ years of development to lean on, including a robust ecosystem of packages for scientific computing. But, not all packages play nicely with one another. In particular, packages have *dependencies*, which are *other packages* they rely on (and those probably also have dependencies, and so forth). When we want to update something or install something new, what happens if that package brings dependencies that conflict with our other packages? What if I have some code that works, with a certain set of packages, but I want to share that with my advisor, who doesn't have everything installed already? What if I return to some software I wrote a year ago, but I've updated/installed/uninstalled packages since then? \n",
+    "\n",
+    "This is why we need tools for *package management*.\n",
+    "\n",
+    "Objectives:\n",
+    "In this lesson, you'll learn...\n",
+    "- What a package manager is, and why it's useful,\n",
+    "- How to organize Python packages into virtual environments, and\n",
+    "- How to set up a new environment using *conda*, *venv*, or *poetry*."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# What do we need in a package manager?\n",
+    "### 1. Python environments\n",
+    "An environment is an isolated group of packages and dependencies **that all work together**. We can define environments in plain text files, just by delineating the exact version of Python and each package we want to use. Environments are ephemeral. Create them, use them, break them, throw them out, and then create them again. (As long as you have the text file with the environment specification.)\n",
+    "\n",
+    "### 2. Package installation\n",
+    "We could install all of our packages by hand, by going to each repository individually, picking a version, and downloading that software. But, why do all of the hard work ourselves? A good package manager will let us specify constraints that we know (e.g., I want *numpy* 2.X), but will make reasonable choices in the absence of any specific requirement. It should go get all of the necessary dependencies for our packages. And, it will resolve conflicts that arise between different versions of packages - or at the very least, flag those conflicts and ask us to weigh in and resolve things. \n",
+    "\n",
+    "### 3. Reproducibility\n",
+    "Our environments are ephemeral, so our package manager needs to be able to save a specification file, share it with others, and make new environments from files that others give us. Each one will be opinionated about how the specification file should be formatted, but the underlying functionality is the same. In general, **reproducibility** requires our package manager to behave in expected, predictable ways. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Conda"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Downloading conda\n",
+    "Ironically, there are lots of different versions, or *distributions*, of the conda software. If you are going to use conda, **we highly recommend the [miniforge distribution](https://github.com/conda-forge/miniforge)**. It's about as minimal as a conda distribution can get, and it connects by default to a *channel*, called **conda-forge**, which has most of the packages you'll ever need for scientific computing.\n",
+    "\n",
+    "Note: *mamba* is a faster version of conda, written in C++. They are used in exactly the same way (99% of the time), so pick whichever one you like. \n",
+    "\n",
+    "### Making a new conda environment\n",
+    "If conda is not enabled in your shell, you may need to run:\n",
+    "```\n",
+    "conda init\n",
+    "```\n",
+    "Once you have conda installed, you can make a new environment with:\n",
+    "```\n",
+    "conda create --name my_project\n",
+    "```\n",
+    "Now, run\n",
+    "```\n",
+    "conda env list\n",
+    "```\n",
+    "to see all of your environments. It should return a list with (base) and (my_project) or whatever you called your new environment. Before doing anything else, let's *activate* our new environment, using:\n",
+    "```\n",
+    "conda activate my_project\n",
+    "```\n",
+    "Now, you should see (my_project) at the left side of your shell prompt, before the $. This indicates that our new conda environment is activated and ready to go.\n",
+    "\n",
+    "**Important note: do not install packages into base.**\n",
+    "\n",
+    "**Seriously, don't do it.**\n",
+    "\n",
+    "### Installing packages\n",
+    "To install a new package, let's say *numpy*, run:\n",
+    "```\n",
+    "conda install numpy\n",
+    "```\n",
+    "and then follow the prompts. Conda will attempt to *solve* your dependency tree, which is a fancy way of saying that it will make sure there aren't any conflicts. If you want a specific version, you can instead run:\n",
+    "```\n",
+    "conda install numpy=2.0\n",
+    "```\n",
+    "for exactly numpy 2.0, or:\n",
+    "```\n",
+    "conda install numpy>=2.0\n",
+    "```\n",
+    "for *at least* numpy 2.0, if you're okay with more recent versions. You can combine =, <, > to specify different generations of packages.\n",
+    "\n",
+    "To list all of the packages in your current environment, run:\n",
+    "```\n",
+    "conda list\n",
+    "```\n",
+    "And to update packages, run:\n",
+    "```\n",
+    "conda update foo\n",
+    "```\n",
+    "where *foo* can be a package name, or even *conda* or *python*.\n",
+    "\n",
+    "### Using environment files\n",
+    "The last thing we need is a way to save, share, and rebuild our environments. Conda uses YAML files to specify environments. To export your current environment to a YAML file, you can use:\n",
+    "```\n",
+    "conda env export > environment.yaml\n",
+    "```\n",
+    "which basically just exports a list of the current packages and versions in your environment to a text file. Making a new environment from a file is very similar to what we did above, with one addition:\n",
+    "```\n",
+    "conda env create --file environment.yaml\n",
+    "```\n",
+    "By default, this will make a new environment named \"environment.\" But you can also use the --name flag to name it whatever you want."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Venv"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We can also make ourselves a DIY conda in base Python using a combination of *venv* and *pip*. To make a new virtual environment in python, run:\n",
+    "```\n",
+    "python3 -m venv [FOLDER]\n",
+    "```\n",
+    "where [FOLDER] will be the directory where your environment is installed. Note that this should be a sub-directory of your project directory, if you are following the convention of having one environment for one project. It's really common to name this folder .venv so it is hidden by default and doesn't conflict with other directories. Now, when you run\n",
+    "```\n",
+    "source .venv/bin/activate\n",
+    "```\n",
+    "an indicator should appear at the left of your shell prompt with (.venv) (or whatever the name of your venv is). \n",
+    "\n",
+    "The default installation tool in Python is *pip*. We want to be careful with Python and pip, because your operating system probably has a default Python installation. If we call pip from the wrong location, we might accidentally install packages into your system Python directory (which is annoying but not catastrophic). Check this now with:\n",
+    "```\n",
+    "which pip\n",
+    "```\n",
+    "You should see a path to the pip installed in your venv directory. If not, make sure to activate the venv. Next, we can install packages using:\n",
+    "```\n",
+    "pip install numpy\n",
+    "```\n",
+    "or whatever other package we want. Similar to conda, we can use numpy==2.0 or >, <, >=, and <=. Notice that the exact version is denoted with '==' instead of '='. \n",
+    "\n",
+    "Pip also gives us an easy way to save environment files. Try:\n",
+    "```\n",
+    "pip freeze > requirements.txt\n",
+    "```\n",
+    "Requirements files are basically the same as environment.yaml files, with slightly different formatting. You can also install environments from a requirements file using:\n",
+    "```\n",
+    "pip install -r requirements.txt\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Poetry"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Right now, my favorite package manager is [poetry](https://python-poetry.org/docs/basic-usage/). It does the same things as conda or venv for Python packages, but also has some advanced functionality built in."
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}