09_main.qmd

# 9. Plotting and Visualization

## Learning Objectives

::: incremental
-   We are going to learn the basic data visualization technique using matplotlib, pandas and seaborn.
:::

# Introduction

Making informative visualizations is one of the most important tasks in every exploratory data analysis process and this can be done using **matplotlib.** It may be a part of the exploratory process for example, to help identify **outliers** or needed data transformations, or as a way of generating ideas for models. For others, building an interactive visualization for the web may be the end goal. Python has many add-on libraries for making static or dynamic visualizations, but I'll be mainly focused on **matplotlib** and libraries that build on top of it.

## import the necessary library

```{python}
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns

```

::: panel-tabset
## Demo_Dataset

```{python}
#| echo: fenced
#| eval: false
data = np.arange(10)
```

## Print

```{python}
#| eval: true
#| echo: false
data = np.arange(10)

data
```
:::

::: panel-tabset
## Code

```{python}
#| echo: fenced
#| eval: false
plt.plot(data)
```

## Output

```{python}
#| eval: true
#| echo: false
plt.plot(data)
```
:::

## We can use plt.show() function to display the plot in quarto

```{python}
plt.show()
```

When we are in jupyter notebook we can use **%matplotlib notebook** so that we can display the plot, but when we are in Ipython we can use **%matplotlib** to display the plot.

## Customization of the visualization

While libraries like seaborn and pandas's built-in plotting functions will deal with many of the mundane details of making plots, should you wish to customize them beyond the function options provided, you will need to learn a bit about the matplotlib API.

## Figures and Subplots

Plots in **matplotlib** reside within a **Figure** object. You can create a new figure with **plt.figure ()**

```{python}
fig = plt.figure()
```

**plt.figure** has a number of options; notably, **figsize** will guarantee the figure has a certain size and aspect ratio if saved to disk.

You can't make a plot with a blank figure. You have to create one or more subplots using add_subplot

## Add Subplot

```{python}
ax1 = fig.add_subplot(2,2, 1)

ax1

```

This means that the figure should be 2 × 2, and we're selecting the first of four subplots (numbered from 1). We can add more subplot

## We can add more subplot

```{python}
ax2 = fig.add_subplot(2, 2, 2)

ax3 = fig.add_subplot(2, 2, 3)

ax2

ax3
```

## Adding axis methods to the plot

These plot axis objects have various methods that create different types of plots, and it is preferred to use the axis methods over the top-level plotting functions like **plt.show()**. For example, we could make a line plot with the plot method.

```{python}
fig = plt.figure()

ax1 = fig.add_subplot(2, 2, 1)

ax2 = fig.add_subplot(2, 2, 2)

ax3 = fig.add_subplot(2, 2, 3)

ax3.plot(np.random.standard_normal(50).cumsum(), color="black",
linestyle="dashed")

```

We may notice output like **matplotlib.lines.Line2D** at when we are creating our visualization. matplotlib returns objects that reference the plot subcomponent that was just added. A lot of the time you can safely ignore this output, or you can put a **semicolon** at the end of the line to suppress the output.

The additional options instruct matplotlib to plot a black dashed line. The objects returned by **fig.add_subplot** here are **AxesSubplot** objects, on which you can directly plot on the other empty subplots by calling each one's instance method.

```{python}
ax1.hist(np.random.standard_normal(100),bins=20,color="black", alpha=0.3);

ax2.scatter(np.arange(30), np.arange(30) + 3*np.random.standard_normal(30));
```

To make creating a grid of subplots more convenient, matplotlib includes a **plt.subplots** method that creates a new figure and returns a NumPy array containing the created subplot objects:

```{python}
axes = plt.subplots(2, 3)

axes

```

The axes array can then be indexed like a two-dimensional array; for example, **axes\[0, 1\]** refers to the subplot in the top row at the center

## Scatter Plot

```{python}
plt.style.use('_mpl-gallery')

# make the data
np.random.seed(3)
x = 4 + np.random.normal(0, 2, 24)
y = 4 + np.random.normal(0, 2, len(x))
# size and color:
sizes = np.random.uniform(15, 80, len(x))
colors = np.random.uniform(15, 80, len(x))

# plot
fig, ax = plt.subplots()

ax.scatter(x, y, s=sizes, c=colors, vmin=0, vmax=100)

ax.set(xlim=(0, 8), xticks=np.arange(1, 8),
       ylim=(0, 8), yticks=np.arange(1, 8))

plt.show()
```

## Bar Plot

```{python}
plt.style.use('_mpl-gallery')

# make data:
np.random.seed(3)
x = 0.5 + np.arange(8)
y = np.random.uniform(2, 7, len(x))

# plot
fig, ax = plt.subplots()

ax.bar(x, y, width=1, edgecolor="white", linewidth=0.7)

ax.set(xlim=(0, 8), xticks=np.arange(1, 8),
       ylim=(0, 8), yticks=np.arange(1, 8))

plt.show()
```

## Box Plot

```{python}

plt.style.use('_mpl-gallery')

# make data:
np.random.seed(10)
D = np.random.normal((3, 5, 4), (1.25, 1.00, 1.25), (100, 3))

# plot
fig, ax = plt.subplots()
VP = ax.boxplot(D, positions=[2, 4, 6], widths=1.5, patch_artist=True,
                showmeans=False, showfliers=False,
                medianprops={"color": "white", "linewidth": 0.5},
                boxprops={"facecolor": "C0", "edgecolor": "white",
                          "linewidth": 0.5},
                whiskerprops={"color": "C0", "linewidth": 1.5},
                capprops={"color": "C0", "linewidth": 1.5})

ax.set(xlim=(0, 8), xticks=np.arange(1, 8),
       ylim=(0, 8), yticks=np.arange(1, 8))

plt.show()
```

[**We can learn more with the matplotlib documentation**](https://matplotlib.org)

| Argument   | Description                                                                                                  |
|--------------------|--------------------------------------------------------|
| nrows      | Number of rows of subplots                                                                                   |
| ncols      | Number of columns of subplots                                                                                |
| sharex     | All subplots should use the same x-axis ticks (adjusting the xlim will affect all subplots)                  |
| sharey     | All subplots should use the same y-axis ticks (adjusting the ylim will affect all subplots)                  |
| subplot_kw | Dictionary of keywords passed to add_subplot call used to create each subplot                                |
| **fig_kw** | Additional keywords to subplots are used when creating the figure, such as plt.subplots (2,2, figsize=(8,6)) |

: Table.1: Matplotlib.pyplot.subplots options

## Adjusting the spacing around subplots

By default, matplotlib leaves a certain amount of padding around the outside of the subplots and in spacing between subplots. This spacing is all specified relative to the height and width of the plot, so that if you resize the plot either programmatically or manually using the GUI window, the plot will dynamically adjust itself. You can change the spacing using the subplots_adjust method on Figure objects:

subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=None)

**wspace** and **hspace** control the percent of the figure width and figure height, respectively, to use as spacing between subplots.

```{python}
fig, axes = plt.subplots(2, 2, sharex=True, sharey=True)
for i in range(2):
    for j in range(2):
        axes[i, j].hist(np.random.standard_normal(500), bins=50,
                        color="black", alpha=0.5)
fig.subplots_adjust(wspace=0, hspace=0)
```

## **Colors, Markers, and Line Styles**

matplotlib's line `plot` function accepts arrays of x and y coordinates and optional color styling options. For example, to plot `x` versus `y` with green dashes, you would execute:

```{python}
ax.plot(x, y, linestyle="--", color="green")
```

```{python}
ax = fig.add_subplot()

ax.plot(np.random.standard_normal(30).cumsum(), color="black",
linestyle="dashed", marker="o")

plt.show()
```

line plots, you will notice that subsequent points are **linearly interpolated** by default. This can be altered with the **drawstyle** option.

```{python}
fig = plt.figure()

ax = fig.add_subplot()

data = np.random.standard_normal(30).cumsum()

ax.plot(data, color="black", linestyle="dashed", label="Default");
ax.plot(data, color="black", linestyle="dashed",
drawstyle="steps-post", label="steps-post");
ax.legend()
```

## Ticks, Labels, and Legends

Most kinds of plot decorations can be accessed through methods on matplotlib axes objects. This includes methods like **xlim**, **xticks**, and **xticklabels**. These control the **plot range**, **tick locations**, and **tick labels**, respectively. They can be used in two ways:

-   Called with no arguments returns the current parameter value (e.g., **ax.xlim()** returns the current **x-axis** plotting range)

-   Called with parameters sets the parameter value (e.g., **ax.xlim**(\[0, 10\]) sets the x-axis range to 0 to 10)

## Setting the title, axis labels, ticks, and tick labels

```{python}
fig, ax = plt.subplots()

ax.plot(np.random.standard_normal(1000).cumsum());

plt.show()
```

To change the **x-axis ticks**, it's easiest to use **set_xticks** and **set_xticklabels**. The former instructs matplotlib where to place the ticks along the data range; by default these locations will also be the labels. But we can set any other values as the labels using **set_xticklabels:**

The rotation option sets the x tick labels at a 30-degree rotation. Lastly, **set_xlabel** gives a name to the x-axis, and set_title is the subplot title.

## Adding legends

Legends are another critical element for identifying plot elements. There are a couple of ways to add one. The easiest is to pass the label argument when adding each piece of the plot:

```{python}
fig, ax = plt.subplots()

ax.plot(np.random.randn(1000).cumsum(), color="black", label="one");
ax.plot(np.random.randn(1000).cumsum(), color="black", linestyle="dashed",
label="two");
ax.plot(np.random.randn(1000).cumsum(), color="black", linestyle="dotted",
label="three");


ax.legend()

```

The legend method has several other choices for the location **loc argument**. See the docstring (with **ax.legend?**) for more information. The **loc legend** option tells matplotlib where to place the plot. The default is **"best"**, which tries to choose a location that is most out of the way. To exclude one or more elements from the legend, pass **no label** or **label="*nolegend*"**.

## Saving Plots to File

You can save the active figure to file using the figure object's savefig instance method. For example, to save an SVG version of a figure, you need only type:

```{python}
fig.savefig("figpath.png", dpi=400)
```

| Argument             | Description                                                                                                                                                |
|-----------------|-------------------------------------------------------|
| fname                | String containing a filepath or a Python file-like object. The figure format is inferred from the file extension (e.g., `.pdf` for PDF or `.png` for PNG). |
| dpi                  | The figure resolution in dots per inch; defaults to 100 in IPython or 72 in Jupyter out of the box but can be configured.                                  |
| facecolor, edgecolor | The color of the figure background outside of the subplots; `"w"` (white), by default.                                                                     |
| format               | The explicit file format to use (`"png"`, `"pdf"`, `"svg"`, `"ps"`, `"eps"`, \...).                                                                        |

: Table 9.2: Some fig.savefig options

### **matplotlib Configuration**

matplotlib comes configured with color schemes and defaults that are geared primarily toward preparing figures for publication. Fortunately, nearly all of the default behavior can be customized via global parameters governing figure size, subplot spacing, colors, font sizes, grid styles, and so on. One way to modify the configuration programmatically from Python is to use the **`rc`** method; for example, to set the global default figure size to be 10 × 10, you could enter:

The first argument to `rc` is the component you wish to customize, such as **`"figure"`**, **`"axes"`**, **`"xtick"`**, **`"ytick"`**, **`"grid"`**, **`"legend"`**, or many others. After that can follow a sequence of keyword arguments indicating the new parameters. A convenient way to write down the options in your program is as a dictionary:

```{python}
plt.rc("font", family="monospace", weight="bold", size=8)
```

```{python}
plt.rc("figure", figsize=(10, 10))
```

```{python}
plt.rc("font", family="monospace", weight="bold", size=8)
```

For more extensive customization and to see a list of all the options, matplotlib comes with a configuration file ***matplotlibrc*** in the ***matplotlib/mpl-data*** **directory**. If you customize this file and place it in your home directory **titled *.matplotlibrc***, it will be loaded each time you use **matplotlib**.

All of the current configuration settings are found in the **plt.rcParams dictionary**, and they can be restored to their default values by calling the **plt.rcdefaults()** function.

The first argument to **rc** is the component you wish to customize, such as **"figure"**, **"axes"**, **"xtick"**, **"ytick"**, **"grid"**, **"legend"**, or many others. After that can follow a sequence of keyword arguments indicating the new parameters. A convenient way to write down the options in your program is as a dictionary:

## Plotting with pandas and seaborn

**matplotlib** can be a fairly low-level tool. You assemble a plot from its base components: the data display (i.e., the type of plot: line, bar, box, scatter, contour, etc.), legend, title, tick labels, and other annotations. In **pandas**, we may have multiple columns of data, along with row and column labels. pandas itself has built-in methods that simplify creating visualizations from DataFrame and Series objects. Another library is **seaborn**, a high-level statistical graphics library built on matplotlib. seaborn simplifies creating many common visualization types.

## Plotting with Pandas

## Line Plots

```{python}
import pandas as pd
import numpy as np

s = pd.Series(np.random.standard_normal(10).cumsum(), index=np.arange(0,
 100, 10))

s.plot()
```

The Series object's index is passed to **matplotlib** for plotting on the x-axis, though you can disable this by passing **use_index=False**. The x-axis ticks and limits can be adjusted with the **xticks** and **xlim** options, and the **y-axis** respectively with **yticks** and **ylim**.

| Argument  | Description                                                                                                          |
|----------------|--------------------------------------------------------|
| label     | Label for plot legend                                                                                                |
| ax        | matplotlib subplot object to plot on; if nothing passed, uses active matplotlib subplot                              |
| style     | Style string, like `"ko--"`, to be passed to matplotlib                                                              |
| alpha     | The plot fill opacity (from 0 to 1)                                                                                  |
| kind      | Can be `"area"`, `"bar"`, `"barh"`, `"density"`, `"hist"`, `"kde"`, `"line"`, or `"pie"`; defaults to `"line"`       |
| figsize   | Size of the figure object to create                                                                                  |
| logx      | Pass `True` for logarithmic scaling on the x axis; pass `"sym"` for symmetric logarithm that permits negative values |
| logy      | Pass `True` for logarithmic scaling on the y axis; pass `"sym"` for symmetric logarithm that permits negative values |
| title     | Title to use for the plot                                                                                            |
| use_index | Use the object index for tick labels                                                                                 |
| rot       | Rotation of tick labels (0 through 360)                                                                              |
| xticks    | Values to use for x-axis ticks                                                                                       |
| yticks    | Values to use for y-axis ticks                                                                                       |
| xlim      | x-axis limits (e.g., `[0, 10]`)                                                                                      |
| ylim      | y-axis limits                                                                                                        |
| grid      | Display axis grid (off by default)                                                                                   |

: Table 9.3: Series.plot method arguments

## Line Graph

```{python}
df = pd.DataFrame(np.random.standard_normal((10, 4)).cumsum(0),
columns=["A", "B", "C", "D"],
index=np.arange(0, 100, 10))

plt.style.use('grayscale')

df.plot()
```

Here I used `plt.style.use('grayscale')` to switch to a color scheme more suitable for black and white publication, since some readers will not be able to see the full color plots. The `plot` attribute contains a "family" of methods for different plot types. For example, `df.plot()` is equivalent to `df.plot.line()`

## Bar Plots

The **plot.bar()** and **plot.barh()** make vertical and horizontal bar plots, respectively. In this case, the Series or DataFrame index will be used as the x **(bar)** or y **(barh)** ticks

```{python}
fig, axes = plt.subplots(2, 1)

data = pd.Series(np.random.uniform(size=16), index=list("abcdefghijklmnop"))

data.plot.bar(ax=axes[0], color="black", alpha=0.7)

data.plot.barh(ax=axes[1], color="black", alpha=0.7)
```

## With a DataFrame, bar plots group the values in each row in bars, side by side, for each value.

```{python}
df = pd.DataFrame(np.random.uniform(size=(6, 4)),
index=["one", "two", "three", "four", "five", "six"],
columns=pd.Index(["A", "B", "C", "D"], name="Genus"))

df
 

df.plot.bar()
```

## We create stacked bar plots from a DataFrame by passing **stacked=True**, resulting in the value in each row being stacked together horizontally

```{python}
df.plot.barh(stacked=True, alpha=0.5)
```

## Visualizing categorical data with Seaborn

Plotting functions in seaborn take a `data` argument, which can be a pandas DataFrame. The other arguments refer to column names.

```{python}
import seaborn as sns
tips = sns.load_dataset("tips")
sns.catplot(data=tips, x="day", y="total_bill")

```

```{python}
sns.catplot(data=tips, x="day", y="total_bill", hue="sex", kind="swarm")
```

## Boxplot

```{python}
sns.catplot(data=tips, x="day", y="total_bill", kind="box")
```

## Barplot

```{python}
titanic = sns.load_dataset("titanic")
sns.catplot(data=titanic, x="sex", y="survived", hue="class", kind="bar")
```

## Scatter Plot

```{python}
import pandas as pd
import seaborn as sns
penguin=pd.read_excel('data/penguins.xlsx')

ax = sns.regplot(x="bill_length_mm", y="flipper_length_mm", data=penguin)

ax

```

In exploratory data analysis, it's helpful to be able to look at all the scatter plots among a group of variables; this is known as a *pairs* plot or *scatter plot matrix.*

```{python}
 sns.pairplot(penguin, diag_kind="kde", plot_kws={"alpha": 0.2})

```

This **plot_kws** enables us to pass down configuration options to the individual plotting calls on the off-diagonal elements.

**Point Plot**

```{python}
sns.catplot(data=titanic, x="sex", y="survived", hue="class", kind="point")
```

## Line Plot

```{python}
import numpy as np
import pandas as pd
import seaborn as sns
sns.set_theme(style="whitegrid")

rs = np.random.RandomState(365)
values = rs.randn(365, 4).cumsum(axis=0)
dates = pd.date_range("1 1 2016", periods=365, freq="D")
data = pd.DataFrame(values, dates, columns=["A", "B", "C", "D"])
data = data.rolling(7).mean()

sns.lineplot(data=data, palette="tab10", linewidth=2.5)
```

## **Facet Grids and Categorical Data**

One way to visualize data with many categorical variables is to use a ***facet grid***, which is a **two-dimensional layout** of plots where the data is split across the plots on each axis based on the distinct values of a certain variable. **seaborn** has a useful built-in function **`catplot`** that simplifies making many kinds of faceted plots split by **categorical variables**

```{python}
sns.catplot(x="species", y="bill_length_mm", hue="sex", col="island",
kind="bar", data=penguin)
```

**`catplot`** supports other plot types that may be useful depending on what you are trying to display. For example, *box plots* (which show the median, quartiles, and outliers) can be an effective visualization type.

```{python}
sns.catplot(x="bill_length_mm", y="island", kind="box",
data=penguin)

```

## **Other Python Visualization Tools**

-   There are many other tools for data visualization in python such as \[[Altair](https://altair-viz.github.io/)\](https://altair-viz.github.io/); \[[Bokeh](http://bokeh.pydata.org/)\](http://bokeh.pydata.org/) and \[[Plotly](https://plotly.com/python)\](https://plotly.com/python)

-   For creating static graphics for print or web, I recommend using matplotlib and libraries that build on matplotlib, like pandas and seaborn, for your needs.