notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "619f3292-b0a9-4491-9feb-419e265e685d",
   "metadata": {},
   "source": [
    "# Visualised: How SARS-CoV-2 compares to other infectious diseases\n",
    "## LUH ZQS Data Literacy Seminar"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5518f5d3-3620-4734-88c4-0e2978eb537a",
   "metadata": {},
   "source": [
    "In this exercise we use a few data sources to compare SARS-CoV-2 to other infectious diseases using basic data processing and visualization in Python. The exercise is inspired by a 2014 [The Guardian visualization](https://www.theguardian.com/news/datablog/ng-interactive/2014/oct/15/visualised-how-ebola-compares-to-other-infectious-diseases) that compared Ebola with other infectious diseases.\n",
    "\n",
    "The data sources include:\n",
    "* The Microbe Scope data published by The Guardian available at [bit.ly/KIB_Microbescope](https://bit.ly/KIB_Microbescope). A copy of this dataset is included (see `data.csv`).\n",
    "* [ORKG](https://orkg.org) Comparisons for SARS-CoV-2 [basic reproductive rate](https://www.orkg.org/orkg/comparison/R44930) and [case fatality rate](https://www.orkg.org/orkg/comparison/R41466)\n",
    "\n",
    "To make this exercise interactive and fun, you'll need to fill the occasional `[BLANKS]` in the code before you can execute it.\n",
    "\n",
    "Let's get started ..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "740d68b2-4ce7-4a44-96de-ea5be994fe46",
   "metadata": {},
   "source": [
    "First we need to install and import a few libraries we will need in our code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa525aab-59b5-4712-a217-18f937b79507",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install orkg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72e467f3-6376-4f9e-b4e5-2fd5c3ef7535",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from orkg import ORKG\n",
    "from scipy.spatial.distance import squareform, pdist"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32ca3e8b-91f3-48e7-8028-08c90ffc1d5c",
   "metadata": {},
   "source": [
    "Now let's read the Microbe Scope `data.csv` and get its shape, i.e. the number of rows and columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0ed2b43-8a59-497c-bbc5-71ac9011786d",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_microbe_scope = pd.read_csv([FILENAME], skiprows=[1]) # Replace [FILENAME] accordingly\n",
    "df_microbe_scope.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bd6a434-3652-43f9-9b5c-76a7cc9e06c4",
   "metadata": {},
   "source": [
    "... and take a look at the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12f6d90b-7ca5-4ff5-b609-0929e21681c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_microbe_scope.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca3aca0b-5cdd-4ba8-88a4-8c9916696cd8",
   "metadata": {},
   "source": [
    "Of particular interest are the first three columns including the disease name, the case fatality rate and the average basic reproductive rate. \n",
    "\n",
    "Let's select and rename them. \n",
    "\n",
    "Take a look at the shape and explain what happenend: Why is the second number different compared to before?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98c3fa20-d729-44a6-8f35-0cd34d23f646",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_microbe_scope = df_microbe_scope.iloc[:, 0:[TO_COLUMN]] # Replace [TO_COLUMN]; Hint: index starts with zero and the ending index is excluded\n",
    "df_microbe_scope.columns = ['disease', 'case_fatality_rate', 'basic_reproductive_rate']\n",
    "df_microbe_scope.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c7fc0f3-6c8f-4894-8b36-42ebf7f395fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_microbe_scope.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4599f67-c4e9-4992-85f9-ef6b3f8798bc",
   "metadata": {},
   "source": [
    "Next, we compute some descriptive statistics for case fatality and basic reproductive rates.\n",
    "\n",
    "Let's do mean and max case fatality rate first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74ca85c9-12c9-471f-9d14-4401d4db8eee",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_microbe_scope['case_fatality_rate'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ecce75ea-393a-4e4b-8582-e1e7f6dc867e",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_microbe_scope['case_fatality_rate'].max()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f7106b5-9c22-4139-998d-e93dac11d2e6",
   "metadata": {},
   "source": [
    "Now try the same for basic reproductive rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f781eb20-f00a-4bbe-ad13-e16ba5705da2",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f26454bb-f1f4-433c-9b95-134d55773681",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e7f611c5-750d-4a3d-bbc6-3a59d07d9a89",
   "metadata": {},
   "source": [
    "The following is a shorthand for descriptive statistics.\n",
    "\n",
    "Try it for both basic reproductive and case fatality rates and compare the values. They should be same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "180b4a85-1ef3-494c-80fe-2baee047cd5a",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_microbe_scope['case_fatality_rate'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1092fdf8-551c-4394-9ceb-628864981ce0",
   "metadata": {},
   "source": [
    "Next, let's plot the data and use visualizations to understand it better.\n",
    "\n",
    "Try this for both basic reproductive and case fatality rates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e321287-4c3a-46fe-b90e-ca98f1b11499",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_microbe_scope.plot(kind='barh', x='disease', y='basic_reproductive_rate', figsize=(10,10))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1709a62e-a096-4a8b-a92e-22fd4cc9a478",
   "metadata": {},
   "source": [
    "That's great, but sorted would be better to quickly see the top three and how they compare to the bottom three."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4cb0829-4f01-4623-bfd4-644c375a036e",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_microbe_scope.sort_values(by='basic_reproductive_rate').plot(kind='barh', x='disease', y='basic_reproductive_rate', figsize=(10,10))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4eaf6e3a-3f35-4f49-8b1e-0990f792ca88",
   "metadata": {},
   "source": [
    "The disease data is not univariate but multivariante, for each disease with have two variables (the two rates). \n",
    "\n",
    "It makes thus sense to visualize the diseases along both dimensions, using a scatter plot.\n",
    "\n",
    "With this we can easily identify a few characteristics:\n",
    "* Most diseases cluster in the lower left corner, meaning that most diseases tend to be on the lower ends of case fatality and basic reproductive rates\n",
    "* A few diseases are exceptional with respect to one dimension, either very high case fatality rate or very high basic reproductive rate\n",
    "* No disease has both high case fatality rate and basic reproductive rate\n",
    "\n",
    "Hint: With the additional parameters `logy=True` and `loglog=True` you can modify the axes to log scaling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11a5f158-a920-4a06-84e1-664378bf1e99",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_microbe_scope.plot.scatter(x='basic_reproductive_rate', y='case_fatality_rate', figsize=(10,10))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef887390-af55-4bf1-9bf2-400b6f493736",
   "metadata": {},
   "source": [
    "So far we have only inspected the Microbe Scope dataset, which does not include SARS-CoV-2.\n",
    "\n",
    "The ORKG provides data extracted from the scholarly literature for SARS-CoV-2 basic reproductive rate and case fatality rate.\n",
    "\n",
    "Take a look at the two ORKG comparisons:\n",
    "* SARS-CoV-2 [basic reproductive rate](https://www.orkg.org/orkg/comparison/R44930)\n",
    "* SARS-CoV-2 [case fatality rate](https://www.orkg.org/orkg/comparison/R41466)\n",
    "\n",
    "Both comparisons include numerous published articles, each with information about these rates (see the `has value` property in the comparisons).\n",
    "\n",
    "Let's load the two comparisons into data frames. The basic reproductive rate comparison has ID R44930 while the case fatality rate has ID R41466 (see the URL of the comparisons in the browser). We need these to load the corresponding data.\n",
    "\n",
    "First, let's connect to ORKG."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06c4d82e-9056-40ce-b9ee-45a96630ab28",
   "metadata": {},
   "outputs": [],
   "source": [
    "orkg = ORKG(host='https://orkg.org/orkg', simcomp_host='https://orkg.org/orkg/simcomp')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e51c08c-2f5e-4283-9f20-a7ed737639f9",
   "metadata": {},
   "source": [
    "Let's retrieve the data for the basic reproductive rate comparison first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "968418c8-fee2-417e-8cd5-40dfc9a7acb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_orkg_basic_reproductive_rate = orkg.contributions.compare_dataframe(comparison_id='R44930')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d61ef8a8-8478-4994-9c88-7b24b167c3c2",
   "metadata": {},
   "source": [
    "Now we do the same for the case fatality rate comparison."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc87ec2a-3eeb-4c42-891d-3c550a8c05e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_orkg_case_fatality_rate = orkg.contributions.compare_dataframe(comparison_id=[ID]) # Replace [ID] accordingly"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18a68b64-5983-4ae5-94fd-dae0f46d2ab4",
   "metadata": {},
   "source": [
    "Use the following code cell to take a look at this data. Where are the titles of the compared articles and can you find the `Has value` property?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d99d2ee-1115-448e-bac1-6fabbefbcf97",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "95a1cacf-b46e-45eb-843d-248f3b025780",
   "metadata": {},
   "source": [
    "Now, what we actually want is the mean estimates for SARS-CoV-2 basic reproductive rate and case fatality rate as published by the numerous articles. That's easy to compute, right? \n",
    "\n",
    "Try to understand what is going on in the following line of code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1946f6c7-35fa-4cd2-89a2-0c26dfc38b6d",
   "metadata": {},
   "outputs": [],
   "source": [
    "orkg_basic_reproductive_rate = np.mean(df_orkg_basic_reproductive_rate.loc['Has value', :].to_numpy(dtype=np.float32))\n",
    "\n",
    "orkg_basic_reproductive_rate"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6cefb33-b77a-4c26-aae9-7e18a72a63b0",
   "metadata": {},
   "source": [
    "... and do the same for case fatality rate, which is a bit more involved because for some articles the property `Has value` has two values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54673014-ede7-4ad9-99c3-9d4a467fc23e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def try_join(l):\n",
    "    try:\n",
    "        return ','.join(map(str, l))\n",
    "    except TypeError:\n",
    "        return str(l)\n",
    "    \n",
    "df = df_orkg_case_fatality_rate.T\n",
    "df['Has value'] = [try_join(l) for l in df['Has value']]\n",
    "df['Has value'] = df['Has value'].str.extract('([0-9]*[.]?[0-9]+)')\n",
    "df['Has value'] = pd.to_numeric(df['Has value'], errors='coerce')\n",
    "orkg_case_fatality_rate = df['Has value'].mean()\n",
    "\n",
    "orkg_case_fatality_rate"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbc8d5ab-fbd4-4a79-ac2f-7e87395be872",
   "metadata": {},
   "source": [
    "Now that we also have mean values for SARS-CoV-2 basic reproductive and case fatality rates, we can create a new dataset that combines the rates for all infectious diseases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06826139-7808-40a8-b9b5-cf42744492d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_sars_cov_2 = pd.DataFrame([['SARS-CoV-2', orkg_basic_reproductive_rate, orkg_case_fatality_rate]], columns=['disease', 'case_fatality_rate', 'basic_reproductive_rate'])\n",
    "df_diseases = pd.concat([df_microbe_scope, df_sars_cov_2])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "112113cf-5720-460e-99b0-138a3089f38f",
   "metadata": {},
   "source": [
    "Finally, let's compute the top N diseases that are closests to SARS-CoV-2 in terms of basic reproductive and case fatality rates. For this, we first compute the euclidean distance matrix and then extract the N smallest diseases. Note that SARS-CoV-2 is included because the euclidean distance with itself is zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53ce093c-583f-4286-a830-dd0b6e4fd15e",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_distance_matrix = pd.DataFrame(squareform(pdist(df_diseases.iloc[:, 1:])), columns=df_diseases['disease'].unique(), index=df_diseases['disease'].unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0549031c-3052-45ec-87c0-3509ca662181",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df_distance_matrix.nsmallest([N], 'SARS-CoV-2').index.tolist()) # Replace [N] with a number for top-N"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}