notebooks/example-list-test-files.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6c2bd4a5-464a-48fb-a72b-600602d5fdb2",
   "metadata": {
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "# Example of how to use the S3List Utility\n",
    "\n",
    "This notebook demonstrates the `helper.links.S3Links` class to get S3 links for the test files "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a1c42585-068d-4a12-8613-e8f4bf8d5405",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "import os\n",
    "import pprint\n",
    "\n",
    "current = os.path.abspath('..')\n",
    "sys.path.append(current)\n",
    "\n",
    "#from helpers.dataset_lists import BEAM_GROUP\n",
    "from helpers.links import S3Links, glob_s3bucket"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cdcf7f1-cc0f-453c-aca2-7e8a481151c2",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "Instantiate the class.  This loads paths into the S3 class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "33dab034-fd99-4c68-ae88-0eb4dea65c9a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "s3links = S3Links()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "993daf66-7017-4b53-8860-15a91a7e017f",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "The different formats available are listed using the `formats` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "cd86995e-db4b-4a70-85b1-4f3186366fe4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['flatgeobuf',\n",
       " 'flatgeobuf_sindex',\n",
       " 'geoparquet',\n",
       " 'h5repack',\n",
       " 'kerchunk-original',\n",
       " 'kerchunk-repacked',\n",
       " 'original']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s3links.formats"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14837ff1-d79d-4387-9716-d857b10e329e",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "S3 links for test files for a given format are returned as a list using the `get_links_by_format` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5aec5cba-723a-46bd-be1a-38c81c380296",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['s3://nasa-cryo-scratch/h5cloud/h5repack/ATL03_20181120182818_08110112_006_02_repacked.h5',\n",
       " 's3://nasa-cryo-scratch/h5cloud/h5repack/ATL03_20190219140808_08110212_006_02_repacked.h5',\n",
       " 's3://nasa-cryo-scratch/h5cloud/h5repack/ATL03_20200217204710_08110612_006_01_repacked.h5',\n",
       " 's3://nasa-cryo-scratch/h5cloud/h5repack/ATL03_20211114142614_08111312_006_01_repacked.h5',\n",
       " 's3://nasa-cryo-scratch/h5cloud/h5repack/ATL03_20230211164520_08111812_006_01_repacked.h5']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s3links.get_links_by_format('h5repack')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a6dffd0-dd98-49cb-9671-4dce6a86d4c3",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "The S3 link for a given file can be found using `get_link_by_name`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5a8541dc-00e7-47de-bdf0-a73c3e1db4c4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'s3://nasa-cryo-scratch/h5cloud/h5repack/ATL03_20181120182818_08110112_006_02_repacked.h5'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s3links.get_link_by_name('ATL03_20181120182818_08110112_006_02_repacked.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2406498-2164-490d-9d22-9c7bf2d57687",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "Links can also be returned for a given format by id."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "fd622ec6-c33f-4e64-8d6e-46504e565336",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'s3://nasa-cryo-scratch/h5cloud/original/ATL03_20181120182818_08110112_006_02.h5'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s3links.get_link_by_fileid('original', 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a3841a0-dbaf-4966-b52c-41192aa2b1a5",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "The links can be updated using `update_links`.  Links are updated directly from the S3 bucket with the test files.  This can be used as is but the `s3filelinks.json` file is updated - just answer `y` to the prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f4daf48b-20c0-41f7-aac6-a2752ce7ee95",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Differences between self.table and S3 buckets: updating self.table\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "Update ../helpers/s3filelinks.json (y or n)? y\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Updating ../helpers/s3filelinks.json\n"
     ]
    }
   ],
   "source": [
    "s3links.update_links()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cc7d86c-1e24-4ff5-a2ab-5a5e432d2298",
   "metadata": {},
   "source": [
    "Running `s3links.formats` shows the updated files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "efc6f670-951f-466a-9f1a-ed05c9bc7c94",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['flatgeobuf',\n",
       " 'flatgeobuf_no_sindex',\n",
       " 'geoparquet',\n",
       " 'h5repack',\n",
       " 'kerchunk-original',\n",
       " 'kerchunk-repacked',\n",
       " 'original']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s3links.formats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b79f77e9-5bdc-4ae0-b2bb-28abdc649433",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'flatgeobuf': {'ATL03_20181120182818_08110112_006_02.fgb': 's3://nasa-cryo-scratch/h5cloud/flatgeobuf/ATL03_20181120182818_08110112_006_02.fgb',\n",
       "  'ATL03_20190219140808_08110212_006_02.fgb': 's3://nasa-cryo-scratch/h5cloud/flatgeobuf/ATL03_20190219140808_08110212_006_02.fgb',\n",
       "  'ATL03_20200217204710_08110612_006_01.fgb': 's3://nasa-cryo-scratch/h5cloud/flatgeobuf/ATL03_20200217204710_08110612_006_01.fgb',\n",
       "  'ATL03_20211114142614_08111312_006_01.fgb': 's3://nasa-cryo-scratch/h5cloud/flatgeobuf/ATL03_20211114142614_08111312_006_01.fgb',\n",
       "  'ATL03_20230211164520_08111812_006_01.fgb': 's3://nasa-cryo-scratch/h5cloud/flatgeobuf/ATL03_20230211164520_08111812_006_01.fgb'},\n",
       " 'flatgeobuf_no_sindex': {'ATL03_20181120182818_08110112_006_02_no_sindex.fgb': 's3://nasa-cryo-scratch/h5cloud/flatgeobuf_no_sindex/ATL03_20181120182818_08110112_006_02_no_sindex.fgb',\n",
       "  'ATL03_20190219140808_08110212_006_02_no_sindex.fgb': 's3://nasa-cryo-scratch/h5cloud/flatgeobuf_no_sindex/ATL03_20190219140808_08110212_006_02_no_sindex.fgb',\n",
       "  'ATL03_20200217204710_08110612_006_01_no_sindex.fgb': 's3://nasa-cryo-scratch/h5cloud/flatgeobuf_no_sindex/ATL03_20200217204710_08110612_006_01_no_sindex.fgb',\n",
       "  'ATL03_20211114142614_08111312_006_01_no_sindex.fgb': 's3://nasa-cryo-scratch/h5cloud/flatgeobuf_no_sindex/ATL03_20211114142614_08111312_006_01_no_sindex.fgb',\n",
       "  'ATL03_20230211164520_08111812_006_01_no_sindex.fgb': 's3://nasa-cryo-scratch/h5cloud/flatgeobuf_no_sindex/ATL03_20230211164520_08111812_006_01_no_sindex.fgb'},\n",
       " 'geoparquet': {'ATL03_20181120182818_08110112_006_02.h5.gpq': 's3://nasa-cryo-scratch/h5cloud/geoparquet/ATL03_20181120182818_08110112_006_02.h5.gpq',\n",
       "  'ATL03_20190219140808_08110212_006_02.h5.gpq': 's3://nasa-cryo-scratch/h5cloud/geoparquet/ATL03_20190219140808_08110212_006_02.h5.gpq',\n",
       "  'ATL03_20200217204710_08110612_006_01.h5.gpq': 's3://nasa-cryo-scratch/h5cloud/geoparquet/ATL03_20200217204710_08110612_006_01.h5.gpq',\n",
       "  'ATL03_20211114142614_08111312_006_01.h5.gpq': 's3://nasa-cryo-scratch/h5cloud/geoparquet/ATL03_20211114142614_08111312_006_01.h5.gpq',\n",
       "  'ATL03_20230211164520_08111812_006_01.h5.gpq': 's3://nasa-cryo-scratch/h5cloud/geoparquet/ATL03_20230211164520_08111812_006_01.h5.gpq',\n",
       "  \"['ATL03_20200217204710_08110612_006_01.h5'].gpq\": \"s3://nasa-cryo-scratch/h5cloud/geoparquet/['ATL03_20200217204710_08110612_006_01.h5'].gpq\"},\n",
       " 'h5repack': {'ATL03_20181120182818_08110112_006_02_repacked.h5': 's3://nasa-cryo-scratch/h5cloud/h5repack/ATL03_20181120182818_08110112_006_02_repacked.h5',\n",
       "  'ATL03_20190219140808_08110212_006_02_repacked.h5': 's3://nasa-cryo-scratch/h5cloud/h5repack/ATL03_20190219140808_08110212_006_02_repacked.h5',\n",
       "  'ATL03_20200217204710_08110612_006_01_repacked.h5': 's3://nasa-cryo-scratch/h5cloud/h5repack/ATL03_20200217204710_08110612_006_01_repacked.h5',\n",
       "  'ATL03_20211114142614_08111312_006_01_repacked.h5': 's3://nasa-cryo-scratch/h5cloud/h5repack/ATL03_20211114142614_08111312_006_01_repacked.h5',\n",
       "  'ATL03_20230211164520_08111812_006_01_repacked.h5': 's3://nasa-cryo-scratch/h5cloud/h5repack/ATL03_20230211164520_08111812_006_01_repacked.h5'},\n",
       " 'kerchunk-original': {'original_ATL03_20181120182818_08110112_006_02.json': 's3://nasa-cryo-scratch/h5cloud/kerchunk-original/original_ATL03_20181120182818_08110112_006_02.json',\n",
       "  'original_ATL03_20190219140808_08110212_006_02.json': 's3://nasa-cryo-scratch/h5cloud/kerchunk-original/original_ATL03_20190219140808_08110212_006_02.json',\n",
       "  'original_ATL03_20200217204710_08110612_006_01.json': 's3://nasa-cryo-scratch/h5cloud/kerchunk-original/original_ATL03_20200217204710_08110612_006_01.json',\n",
       "  'original_ATL03_20211114142614_08111312_006_01.json': 's3://nasa-cryo-scratch/h5cloud/kerchunk-original/original_ATL03_20211114142614_08111312_006_01.json',\n",
       "  'original_ATL03_20230211164520_08111812_006_01.json': 's3://nasa-cryo-scratch/h5cloud/kerchunk-original/original_ATL03_20230211164520_08111812_006_01.json'},\n",
       " 'kerchunk-repacked': {'h5repack_ATL03_20181120182818_08110112_006_02_repacked.json': 's3://nasa-cryo-scratch/h5cloud/kerchunk-repacked/h5repack_ATL03_20181120182818_08110112_006_02_repacked.json',\n",
       "  'h5repack_ATL03_20190219140808_08110212_006_02_repacked.json': 's3://nasa-cryo-scratch/h5cloud/kerchunk-repacked/h5repack_ATL03_20190219140808_08110212_006_02_repacked.json',\n",
       "  'h5repack_ATL03_20200217204710_08110612_006_01_repacked.json': 's3://nasa-cryo-scratch/h5cloud/kerchunk-repacked/h5repack_ATL03_20200217204710_08110612_006_01_repacked.json',\n",
       "  'h5repack_ATL03_20211114142614_08111312_006_01_repacked.json': 's3://nasa-cryo-scratch/h5cloud/kerchunk-repacked/h5repack_ATL03_20211114142614_08111312_006_01_repacked.json',\n",
       "  'h5repack_ATL03_20230211164520_08111812_006_01_repacked.json': 's3://nasa-cryo-scratch/h5cloud/kerchunk-repacked/h5repack_ATL03_20230211164520_08111812_006_01_repacked.json'},\n",
       " 'original': {'ATL03_20181120182818_08110112_006_02.h5': 's3://nasa-cryo-scratch/h5cloud/original/ATL03_20181120182818_08110112_006_02.h5',\n",
       "  'ATL03_20190219140808_08110212_006_02.h5': 's3://nasa-cryo-scratch/h5cloud/original/ATL03_20190219140808_08110212_006_02.h5',\n",
       "  'ATL03_20200217204710_08110612_006_01.h5': 's3://nasa-cryo-scratch/h5cloud/original/ATL03_20200217204710_08110612_006_01.h5',\n",
       "  'ATL03_20211114142614_08111312_006_01.h5': 's3://nasa-cryo-scratch/h5cloud/original/ATL03_20211114142614_08111312_006_01.h5',\n",
       "  'ATL03_20230211164520_08111812_006_01.h5': 's3://nasa-cryo-scratch/h5cloud/original/ATL03_20230211164520_08111812_006_01.h5'}}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s3links.table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b64b798-702f-401f-8bff-1899d45e6ee5",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}