Merge pull request #29 from Azure/NotebookSetupUpdate

Notebook setup update

Author: ianhelle

Entity Explorer - Account.ipynb

@@ -415,8 +415,7 @@
     "ExecuteTime": {
      "end_time": "2020-03-20T17:28:10.859462Z",
      "start_time": "2020-03-20T17:28:10.280107Z"
-    },
-    "scrolled": false
+    }
    },
    "outputs": [],
    "source": [
@@ -610,10 +609,10 @@
  ],
  "metadata": {
   "hide_input": false,
-  "kernelspec": {
+"kernelspec": {
    "display_name": "Python 3.6",
-   "language": "python36",
-   "name": "python36764bit5248c27440094c529a61105b4d9945f7"
+   "language": "python",
+   "name": "python36"
   },
   "language_info": {
    "codemirror_mode": {
@@ -625,7 +624,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.6"
+   "version": "3.6.7"
   },
   "latex_envs": {
    "LaTeX_envs_menu_present": true,
@@ -686,6 +685,13 @@
     "_Feature"
    ],
    "window_display": false
+  },
+  "widgets": {
+   "application/vnd.jupyter.widget-state+json": {
+    "state": {},
+    "version_major": 2,
+    "version_minor": 0
+   }
   }
  },
  "nbformat": 4,

Entity Explorer - Domain & URL.ipynb

@@ -0,0 +1,291 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Notebook Title\n",
+    " <details>\n",
+    "     <summary>&nbsp;<u>Details...</u></summary>\n",
+    "\n",
+    " **Notebook Version:** 1.0<br>\n",
+    " **Python Version:** Python 3.6 (including Python 3.6 - AzureML)<br>\n",
+    " **Required Packages**: kqlmagic, msticpy, pandas, pandas_bokeh, numpy, matplotlib, networkx, seaborn, datetime, ipywidgets, ipython, dnspython, ipwhois, folium, maxminddb_geolite2<br>\n",
+    " **Platforms Supported**:\n",
+    " - Azure Notebooks Free Compute\n",
+    " - Azure Notebooks DSVM\n",
+    " - OS Independent\n",
+    "\n",
+    " **Data Sources Required**:\n",
+    " - Log Analytics/Azure Sentinel - Syslog, Secuirty Alerts, Auditd, Azure Network Analytics.\n",
+    " - (Optional) - AlienVault OTX (requires account and API key)\n",
+    " </details>\n",
+    "\n",
+    "Notebook description...."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "toc": true
+   },
+   "source": [
+    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
+    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Notebook-Title\" data-toc-modified-id=\"Notebook-Title-1\">Notebook Title</a></span><ul class=\"toc-item\"><li><span><a href=\"#Notebook-Initialization\" data-toc-modified-id=\"Notebook-Initialization-1.1\">Notebook Initialization</a></span><ul class=\"toc-item\"><li><span><a href=\"#Get-WorkspaceId-and-Authenticate-to-Log-Analytics\" data-toc-modified-id=\"Get-WorkspaceId-and-Authenticate-to-Log-Analytics-1.1.1\">Get WorkspaceId and Authenticate to Log Analytics</a></span></li></ul></li><li><span><a href=\"#Example-query\" data-toc-modified-id=\"Example-query-1.2\">Example query</a></span></li></ul></li></ul></div>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "### Notebook initialization\n",
+    "The next cell:\n",
+    "- Checks for the correct Python version\n",
+    "- Checks versions and optionally installs required packages\n",
+    "- Imports the required packages into the notebook\n",
+    "- Sets a number of configuration options.\n",
+    "\n",
+    "This should complete without errors. If you encounter errors or warnings look at the following two notebooks:\n",
+    "- [TroubleShootingNotebooks](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/TroubleShootingNotebooks.ipynb)\n",
+    "- [ConfiguringNotebookEnvironment](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/ConfiguringNotebookEnvironment.ipynb)\n",
+    "\n",
+    "If you are running in the Azure Sentinel Notebooks environment (Azure Notebooks or Azure ML) you can run live versions of these notebooks:\n",
+    "- [Run TroubleShootingNotebooks](./TroubleShootingNotebooks.ipynb)\n",
+    "- [Run ConfiguringNotebookEnvironment](./ConfiguringNotebookEnvironment.ipynb)\n",
+    "\n",
+    "You may also need to do some additional configuration to successfully use functions such as Threat Intelligence service lookup and Geo IP lookup. \n",
+    "There are more details about this in the `ConfiguringNotebookEnvironment` notebook and in these documents:\n",
+    "- [msticpy configuration](https://msticpy.readthedocs.io/en/latest/getting_started/msticpyconfig.html)\n",
+    "- [Threat intelligence provider configuration](https://msticpy.readthedocs.io/en/latest/data_acquisition/TIProviders.html#configuration-file)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2020-05-16T02:00:38.505687Z",
+     "start_time": "2020-05-16T02:00:31.727307Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "from pathlib import Path\n",
+    "import os\n",
+    "import sys\n",
+    "import warnings\n",
+    "from IPython.display import display, HTML, Markdown\n",
+    "\n",
+    "REQ_PYTHON_VER=(3, 6)\n",
+    "REQ_MSTICPY_VER=(0, 5, 0)\n",
+    "\n",
+    "display(HTML(\"<h3>Starting Notebook setup...</h3>\"))\n",
+    "if Path(\"./utils/nb_check.py\").is_file():\n",
+    "    from utils.nb_check import check_python_ver, check_mp_ver\n",
+    "\n",
+    "    check_python_ver(min_py_ver=REQ_PYTHON_VER)\n",
+    "    try:\n",
+    "        check_mp_ver(min_msticpy_ver=REQ_MSTICPY_VER)\n",
+    "    except ImportError:\n",
+    "        !pip install --user --upgrade msticpy\n",
+    "        if \"msticpy\" in sys.modules:\n",
+    "            importlib.reload(msticpy)\n",
+    "        else:\n",
+    "            import msticpy\n",
+    "        check_mp_ver(MSTICPY_REQ_VERSION)\n",
+    "            \n",
+    "from msticpy.nbtools import nbinit\n",
+    "nbinit.init_notebook(\n",
+    "    namespace=globals(),\n",
+    "    extra_imports=[\"ipwhois, IPWhois\"]\n",
+    ");\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a></a>[Contents](#toc)\n",
+    "### Get WorkspaceId and Authenticate to Log Analytics \n",
+    "&lt;details&gt;\n",
+    "    <summary> <u>Details...</u></summary>\n",
+    "If you are using user/device authentication, run the following cell. \n",
+    "- Click the 'Copy code to clipboard and authenticate' button.\n",
+    "- This will pop up an Azure Active Directory authentication dialog (in a new tab or browser window). The device code will have been copied to the clipboard. \n",
+    "- Select the text box and paste (Ctrl-V/Cmd-V) the copied value. \n",
+    "- You should then be redirected to a user authentication page where you should authenticate with a user account that has permission to query your Log Analytics workspace.\n",
+    "\n",
+    "Use the following syntax if you are authenticating using an Azure Active Directory AppId and Secret:\n",
+    "```\n",
+    "%kql loganalytics://tenant(aad_tenant).workspace(WORKSPACE_ID).clientid(client_id).clientsecret(client_secret)\n",
+    "```\n",
+    "instead of\n",
+    "```\n",
+    "%kql loganalytics://code().workspace(WORKSPACE_ID)\n",
+    "```\n",
+    "\n",
+    "Note: you may occasionally see a JavaScript error displayed at the end of the authentication - you can safely ignore this.<br>\n",
+    "On successful authentication you should see a ```popup schema``` button.\n",
+    "To find your Workspace Id go to [Log Analytics](https://ms.portal.azure.com/#blade/HubsExtension/Resources/resourceType/Microsoft.OperationalInsights%2Fworkspaces). Look at the workspace properties to find the ID.\n",
+    "&lt;/details&gt;"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# List Workspaces available\n",
+    "# WorkspaceConfig().list_workspaces()\n",
+    "\n",
+    "# To use a specific workspace create a WorkspaceConfig using the\n",
+    "# workspace parameter\n",
+    "# ws_config = WorkspaceConfig(workspace='MyWorkspace')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2020-05-16T02:01:11.022700Z",
+     "start_time": "2020-05-16T02:00:49.394760Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "# Authentication\n",
+    "ws_config = WorkspaceConfig()\n",
+    "qry_prov = QueryProvider(data_environment=\"LogAnalytics\")\n",
+    "qry_prov.connect(connection_str=ws_config.code_connect_str)\n",
+    "table_index = qry_prov.schema_tables"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2020-05-16T02:03:12.112983Z",
+     "start_time": "2020-05-16T02:03:12.055984Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "query_scope = nbwidgets.QueryTime(auto_display=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Example query"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2020-05-16T02:03:25.227614Z",
+     "start_time": "2020-05-16T02:03:21.291120Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "qry_prov.SecurityAlert.list_alerts(query_scope)"
+   ]
+  }
+ ],
+ "metadata": {
+  "hide_input": false,
+  "kernelspec": {
+   "display_name": "Python 3.6",
+   "language": "python",
+   "name": "python36"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.6"
+  },
+  "latex_envs": {
+   "LaTeX_envs_menu_present": true,
+   "autoclose": false,
+   "autocomplete": true,
+   "bibliofile": "biblio.bib",
+   "cite_by": "apalike",
+   "current_citInitial": 1,
+   "eqLabelWithNumbers": true,
+   "eqNumInitial": 1,
+   "hotkeys": {
+    "equation": "Ctrl-E",
+    "itemize": "Ctrl-I"
+   },
+   "labels_anchors": false,
+   "latex_user_defs": false,
+   "report_style_numbering": false,
+   "user_envs_cfg": false
+  },
+  "toc": {
+   "base_numbering": 1,
+   "nav_menu": {},
+   "number_sections": false,
+   "sideBar": true,
+   "skip_h1_title": false,
+   "title_cell": "Table of Contents",
+   "title_sidebar": "Contents",
+   "toc_cell": true,
+   "toc_position": {},
+   "toc_section_display": true,
+   "toc_window_display": true
+  },
+  "varInspector": {
+   "cols": {
+    "lenName": 16,
+    "lenType": 16,
+    "lenVar": 40
+   },
+   "kernels_config": {
+    "python": {
+     "delete_cmd_postfix": "",
+     "delete_cmd_prefix": "del ",
+     "library": "var_list.py",
+     "varRefreshCmd": "print(var_dic_list())"
+    },
+    "r": {
+     "delete_cmd_postfix": ") ",
+     "delete_cmd_prefix": "rm(",
+     "library": "var_list.r",
+     "varRefreshCmd": "cat(var_dic_list()) "
+    }
+   },
+   "types_to_exclude": [
+    "module",
+    "function",
+    "builtin_function_or_method",
+    "instance",
+    "_Feature"
+   ],
+   "window_display": false
+  },
+  "widgets": {
+   "application/vnd.jupyter.widget-state+json": {
+    "state": {},
+    "version_major": 2,
+    "version_minor": 0
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}