sdk/python/assets/data/working_with_mltable.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#Working with MLTable\n",
    "\n",
    "In this notebook you will learn how to:\n",
    "\n",
    "1. Read `mltable` in a job\n",
    "1. Register an `mltable` as a data asset in Azure Machine Learning\n",
    "1. Consume registered `mltable` assets in a job\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "You will need to install the following Python package dependencies:\n",
    "\n",
    "```bash\n",
    "pip uninstall azure-ai-ml\n",
    "pip install --pre azure-ai-ml\n",
    "pip uninstall mltable\n",
    "pip install --pre mltable\n",
    "pip install pandas\n",
    "```\n",
    "\n",
    "## Connect to Azure Machine Learning Workspace\n",
    "\n",
    "To connect to a workspace, we need identifier parameters - a subscription, resource group and workspace name. We will use these details in the `MLClient` from `azure.ai.ml` to get a handle to the required Azure Machine Learning workspace. We use the [DefaultAzureCredential](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python) for this tutorial. More advanced connection methods can be found [here](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity?view=azure-python).\n",
    "..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.ai.ml import MLClient\n",
    "from azure.identity import DefaultAzureCredential\n",
    "\n",
    "# Enter details of your AML workspace\n",
    "subscription_id = \"<SUBSCRIPTION_ID>\"\n",
    "resource_group = \"<RESOURCE_GROUP>\"\n",
    "workspace = \"<AML_WORKSPACE_NAME>\"\n",
    "\n",
    "# get a handle to the workspace\n",
    "ml_client = MLClient(\n",
    "    DefaultAzureCredential(), subscription_id, resource_group, workspace\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MLTable definition file from local data path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat ./sample_data/MLTable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can look at the contents of the `mltable` file in the notebook, using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mltable\n",
    "\n",
    "tbl = mltable.load(\"./sample_data\")\n",
    "df = tbl.to_pandas_dataframe()\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MLTable definition file from datastore uri path\n",
    "1. get datastore uri path from registered data asset with uri_file type\n",
    "2. construct mltable yaml with the datastore uri path and load it to pandas dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get datastore uri path from registered data asset with uri_file type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get datastore uri from local data path\n",
    "from azure.ai.ml.entities import Data\n",
    "from azure.ai.ml.constants import AssetTypes\n",
    "\n",
    "my_parquet_data = Data(\n",
    "    path=\"./sample_data/data.parquet\",\n",
    "    type=AssetTypes.URI_FILE,\n",
    "    description=\"Parquet data\",\n",
    "    name=\"v2_parquet_urifile\",\n",
    ")\n",
    "\n",
    "my_parquet_data = ml_client.data.create_or_update(my_parquet_data)\n",
    "print(my_parquet_data.path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Construct MLTable definition file from the datastore uri path(taking parquet file data source as example) and load it into pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# helper function to create MLTable datastore uri path for parquet file\n",
    "# uri_path is the datastore uri path in the format of long form datastore uri format: azureml://subscriptions/<sub-id>/resourcegroups/<resource-group>/workspaces/<workspace>/datastores/{datastore_name}/paths/{relative_data_path}\n",
    "# mltable_folder is where to save the MLTable yaml\n",
    "def mltable_from_parquet(uri_path, mltable_folder):\n",
    "    import yaml\n",
    "    import mltable\n",
    "\n",
    "    # MLTable yaml dictionary\n",
    "    mltable_from_parquet = {\n",
    "        \"paths\": [{\"file\": f\"{uri_path}\"}],\n",
    "        \"transformations\": [\"read_parquet\"],\n",
    "    }\n",
    "\n",
    "    temp_dir = tempfile.gettempdir()\n",
    "    with open(f\"{mltable_folder}/MLTable\", \"w\") as mltable_yaml:\n",
    "        yaml.dump(mltable_from_parquet, mltable_yaml, default_flow_style=False)\n",
    "\n",
    "    return mltable.load(mltable_folder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tempfile\n",
    "\n",
    "temp_dir = tempfile.gettempdir()\n",
    "\n",
    "# get datastore uri path from the registered asset\n",
    "# save MLTable file to temp folder\n",
    "mlt = mltable_from_parquet(my_parquet_data.path, temp_dir)\n",
    "mlt.to_pandas_dataframe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading `mltable` in a job\n",
    "\n",
    "Below we show how you can consume an `mltable` in a job. This job just prints the first 10 records of the table:\n",
    "\n",
    "```python\n",
    "# read_mltable.py\n",
    "import argparse\n",
    "import mltable\n",
    "import pandas\n",
    "\n",
    "parser = argparse.ArgumentParser()\n",
    "parser.add_argument(\"--input_data\", type=str)\n",
    "args = parser.parse_args()\n",
    "\n",
    "tbl = mltable.load(args.input_data)\n",
    "df = tbl.to_pandas_dataframe()\n",
    "print(df.head(10))\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.ai.ml import Input, command\n",
    "from azure.ai.ml.entities import Data, Environment\n",
    "from azure.ai.ml.constants import AssetTypes\n",
    "\n",
    "env = Environment(\n",
    "    image=\"mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04\",\n",
    "    conda_file=\"environments/mltable_environment.yaml\",\n",
    ")\n",
    "\n",
    "my_job_inputs = {\"input_data\": Input(type=AssetTypes.MLTABLE, path=\"./sample_data\")}\n",
    "\n",
    "job = command(\n",
    "    code=\"./src\",  # local path where the code is stored\n",
    "    command=\"python read_mltable.py --input_data ${{inputs.input_data}}\",\n",
    "    inputs=my_job_inputs,\n",
    "    environment=env,\n",
    "    compute=\"cpu-cluster\",\n",
    ")\n",
    "\n",
    "# submit the command job\n",
    "returned_job = ml_client.jobs.create_or_update(job)\n",
    "# get a URL for the status of the job\n",
    "returned_job.studio_url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Understanding the code\n",
    "\n",
    "When the job has executed, you will see in the log files a print out of the first 10 records of the titanic sample data. The cell above you can see the inputs to the job were defined using a `dict`:\n",
    "\n",
    "```python\n",
    "my_job_inputs = {\n",
    "    \"input_data\": Input(\n",
    "        type=AssetTypes.MLTABLE, \n",
    "        path='./sample_data'\n",
    "    )\n",
    "}\n",
    "```\n",
    "\n",
    "The `Input` class allow you to define data inputs where:\n",
    "\n",
    "- `type` can be a `uri_file` (a specific file), `uri_folder` (a folder location) or `mltable` (an abstraction over tabular data)\n",
    "- `path` can be a local path or a cloud path. Azure Machine Learning supports `https://`, `abfss://`, `wasbs://` and `azureml://` URIs. As you saw above, if the path is local but your compute is defined to be in the cloud, Azure Machine Learning will automatically upload the data to cloud storage for you.\n",
    "\n",
    "The `Input` defaults the `mode` - how the input will be exposed during job runtime - to `InputOutputModes.RO_MOUNT` (read-only mount). Put another way, Azure Machine Learning will mount the file or folder to the compute and set the file/folder to read-only. By design, you cannot *write* to `Inputs` only `JobOutputs`.\n",
    "\n",
    "#### Accessing data already in the cloud\n",
    "\n",
    "As mentioned above, the `path` in Input supports `https://`, `abfss://`, `wasbs://` and `azureml://` protocols. Therefore, you can simply change the `path` in the above cell to a cloud-based URI.\n",
    "\n",
    "## Registering an `mltable` as an asset in Azure Machine Learning\n",
    "\n",
    "You can register an `mltable` as a data asset in Azure Machine Learning. The benefits of registering data are:\n",
    "\n",
    "- Easy to share with other members of the team (no need to remember file locations)\n",
    "- Versioning of the metadata (location, description, etc)\n",
    "\n",
    "Below we show an example of versioning the sample data in this repo. The data is uploaded to cloud storage and registered as an asset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.ai.ml.entities import Data\n",
    "from azure.ai.ml.constants import AssetTypes\n",
    "\n",
    "my_data = Data(\n",
    "    path=\"./sample_data\",\n",
    "    type=AssetTypes.MLTABLE,\n",
    "    description=\"Titanic Data\",\n",
    "    name=\"titanic-mltable-example\",\n",
    ")\n",
    "\n",
    "my_mltable = ml_client.data.create_or_update(my_data)\n",
    "mltable_version = my_mltable.version\n",
    "print(mltable_version)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Note: Whilst the above example shows a local file. Remember that `path` supports cloud storage (`https`, `abfss`, `wasbs` protocols). Therefore, if you want to register data in a cloud location just specify the path with any of the supported protocols.\n",
    "\n",
    "### Consume data assets in an Azure Machine Learning Job\n",
    "\n",
    "Below we use the previously registered data asset in the job by refering to the long-form ID in the `path`:\n",
    "\n",
    "```txt\n",
    "/subscriptions/XXXXX/resourceGroups/XXXXX/providers/Microsoft.MachineLearningServices/workspaces/XXXXX/datasets/titanic-mltable/versions/1\n",
    "```\n",
    "\n",
    "This long-form URI is accessed using:\n",
    "\n",
    "```python\n",
    "registered_data_asset = ml_client.data.get(name='titanic-mltable-example', version = mltable_version)\n",
    "registered_data_asset.id\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.ai.ml import Input, command\n",
    "from azure.ai.ml.entities import Data, Environment\n",
    "from azure.ai.ml.constants import AssetTypes\n",
    "\n",
    "env = Environment(\n",
    "    image=\"mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04\",\n",
    "    conda_file=\"environments/mltable_environment.yaml\",\n",
    ")\n",
    "\n",
    "registered_data_asset = ml_client.data.get(\n",
    "    name=\"titanic-mltable-example\", version=mltable_version\n",
    ")\n",
    "\n",
    "my_job_inputs = {\"input_data\": Input(path=registered_data_asset.id)}\n",
    "\n",
    "job = command(\n",
    "    code=\"./src\",\n",
    "    command=\"python read_mltable.py --input_data ${{inputs.input_data}}\",\n",
    "    inputs=my_job_inputs,\n",
    "    environment=env,\n",
    "    compute=\"cpu-cluster\",\n",
    ")\n",
    "\n",
    "# submit the command job\n",
    "returned_job = ml_client.create_or_update(job)\n",
    "# get a URL for the status of the job\n",
    "returned_job.studio_url"
   ]
  }
 ],
 "metadata": {
  "description": {
   "description": "Read, write and register a data asset"
  },
  "interpreter": {
   "hash": "914e05ec6db097c6058fb02e940550a6dcea163440b917ff82e4b46d48329739"
  },
  "kernelspec": {
   "display_name": "Python 3.10 - SDK V2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}