sdk/python/jobs/pipelines/1h_automl_in_pipeline/automl-text-ner-named-entity-recognition-in-pipeline/automl-text-ner-named-entity-recognition-in-pipeline.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AutoML TextNer in pipeline\n",
    "\n",
    "**Requirements** - In order to benefit from this tutorial, you will need:\n",
    "- A basic understanding of Machine Learning\n",
    "- An Azure account with an active subscription - [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F)\n",
    "- An Azure ML workspace with computer cluster - [Configure workspace](../../../configuration.ipynb)\n",
    "- A python environment\n",
    "- Installed Azure Machine Learning Python SDK v2 - [install instructions](../../../../README.md) - check the getting started section\n",
    "\n",
    "**Learning Objectives** - By the end of this tutorial, you should be able to:\n",
    "- Create a pipeline with TextNer AutoML task.\n",
    "\n",
    "**Motivations** - This notebook explains how to use TextNer AutoML task inside pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Connect to Azure Machine Learning Workspace\n",
    "\n",
    "The [workspace](https://docs.microsoft.com/en-us/azure/machine-learning/concept-workspace) is the top-level resource for Azure Machine Learning, providing a centralized place to work with all the artifacts you create when you use Azure Machine Learning. In this section we will connect to the workspace in which the job will be run.\n",
    "\n",
    "## 1.1 Import the required libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import required libraries\n",
    "from azure.identity import DefaultAzureCredential, InteractiveBrowserCredential\n",
    "\n",
    "from azure.ai.ml import MLClient, Input, command, Output\n",
    "from azure.ai.ml.dsl import pipeline\n",
    "from azure.ai.ml.automl import text_ner\n",
    "from azure.ai.ml.entities import Environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 Configure credential\n",
    "\n",
    "We are using `DefaultAzureCredential` to get access to workspace. \n",
    "`DefaultAzureCredential` should be capable of handling most Azure SDK authentication scenarios. \n",
    "\n",
    "Reference for more available credentials if it does not work for you: [configure credential example](../../../configuration.ipynb), [azure-identity reference doc](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity?view=azure-python)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    credential = DefaultAzureCredential()\n",
    "    # Check if given credential can get token successfully.\n",
    "    credential.get_token(\"https://management.azure.com/.default\")\n",
    "except Exception as ex:\n",
    "    # Fall back to InteractiveBrowserCredential in case DefaultAzureCredential not work\n",
    "    credential = InteractiveBrowserCredential()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 Get a handle to the workspace\n",
    "\n",
    "We use config file to connect to a workspace. The Azure ML workspace should be configured with computer cluster. [Check this notebook for configure a workspace](../../../configuration.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get a handle to workspace\n",
    "ml_client = MLClient.from_config(credential=credential)\n",
    "\n",
    "# Retrieve an already attached Azure Machine Learning Compute.\n",
    "cluster_name = \"gpu-cluster\"\n",
    "print(ml_client.compute.get(cluster_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Basic pipeline job with TextNer task\n",
    "\n",
    "## 2.1 Build pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env_docker_conda = Environment(\n",
    "    image=\"mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04\",\n",
    "    conda_file=\"./environment/preprocessing_env.yaml\",\n",
    "    name=\"pipeline-custom-environment\",\n",
    "    description=\"Environment created from a Docker image plus Conda environment.\",\n",
    ")\n",
    "ml_client.environments.create_or_update(env_docker_conda)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# note that the used docker image doesn't suit for all size of gpu compute. Please use the following command to create gpu compute if experiment failed\n",
    "# !az ml compute create -n gpu-cluster --type amlcompute --min-instances 0 --max-instances 4 --size Standard_NC12"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define pipeline\n",
    "@pipeline(\n",
    "    description=\"AutoML TextNer Pipeline\",\n",
    ")\n",
    "def automl_text_ner(text_ner_train_data, text_ner_validation_data):\n",
    "    # define command function for preprocessing\n",
    "    preprocessing_command_func = command(\n",
    "        inputs=dict(\n",
    "            train_data=Input(type=\"mltable\"),\n",
    "            validation_data=Input(type=\"mltable\"),\n",
    "        ),\n",
    "        outputs=dict(\n",
    "            preprocessed_train_data=Output(type=\"mltable\"),\n",
    "            preprocessed_validation_data=Output(type=\"mltable\"),\n",
    "        ),\n",
    "        code=\"./preprocess.py\",\n",
    "        command=\"python preprocess.py \"\n",
    "        + \"--train_data ${{inputs.train_data}} \"\n",
    "        + \"--validation_data ${{inputs.validation_data}} \"\n",
    "        + \"--preprocessed_train_data ${{outputs.preprocessed_train_data}} \"\n",
    "        + \"--preprocessed_validation_data ${{outputs.preprocessed_validation_data}}\",\n",
    "        environment=\"pipeline-custom-environment@latest\",\n",
    "    )\n",
    "    preprocess_node = preprocessing_command_func(\n",
    "        train_data=text_ner_train_data, validation_data=text_ner_validation_data\n",
    "    )\n",
    "\n",
    "    # define the automl text_ner task with automl function\n",
    "    text_ner_node = text_ner(\n",
    "        training_data=preprocess_node.outputs.preprocessed_train_data,\n",
    "        validation_data=preprocess_node.outputs.preprocessed_validation_data,\n",
    "        primary_metric=\"accuracy\",\n",
    "        target_column_name=\"xxxxx\",  # REMOVE: This should be optional (BUG:1721836)\n",
    "        # currently need to specify outputs \"mlflow_model\" explictly to reference it in following nodes\n",
    "        outputs={\"best_model\": Output(type=\"mlflow_model\")},\n",
    "    )\n",
    "    text_ner_node.set_limits(timeout_minutes=120)\n",
    "\n",
    "    # define command function for registering the model\n",
    "    command_func = command(\n",
    "        inputs=dict(\n",
    "            model_input_path=Input(type=\"mlflow_model\"),\n",
    "            model_base_name=\"text_named_entity_recognition_example_model\",\n",
    "        ),\n",
    "        code=\"./register.py\",\n",
    "        command=\"python register.py \"\n",
    "        + \"--model_input_path ${{inputs.model_input_path}} \"\n",
    "        + \"--model_base_name ${{inputs.model_base_name}}\",\n",
    "        environment=\"azureml://registries/azureml/environments/sklearn-1.5/labels/latest\",\n",
    "    )\n",
    "    register_model = command_func(model_input_path=text_ner_node.outputs.best_model)\n",
    "\n",
    "\n",
    "pipeline = automl_text_ner(\n",
    "    text_ner_train_data=Input(path=\"./training-mltable-folder/\", type=\"mltable\"),\n",
    "    text_ner_validation_data=Input(path=\"./validation-mltable-folder/\", type=\"mltable\"),\n",
    ")\n",
    "\n",
    "# set pipeline level compute\n",
    "pipeline.settings.default_compute = \"gpu-cluster\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2.2 Submit pipeline job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# submit the pipeline job\n",
    "pipeline_job = ml_client.jobs.create_or_update(\n",
    "    pipeline, experiment_name=\"pipeline_samples\"\n",
    ")\n",
    "pipeline_job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Wait until the job completes\n",
    "ml_client.jobs.stream(pipeline_job.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Next Steps\n",
    "You can see further examples of running a pipeline job [here](../)"
   ]
  }
 ],
 "metadata": {
  "description": {
   "description": "Create pipeline with automl node"
  },
  "kernelspec": {
   "display_name": "Python 3.10 - SDK V2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.7"
  },
  "vscode": {
   "interpreter": {
    "hash": "a3e1ce86190527341b095dce2d981b591205330162e59d5b85eea3038817dc05"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}