seattle_csv <- open_dataset(sources = "data/seattle-library-checkouts.csv",
- format = "csv"
-)\n\n\n\n
\n\n::: {style=\"font-size: 70%;\"}\n
\n\n::: {style=\"font-size: 70%;\"}\n
\n\n**Which file formats do you use most often?**\n\n
\n\n- 1️⃣ CSV (.csv)\n- 2️⃣ MS Excel (.xls and .xlsx)\n- 3️⃣ Parquet (.parquet)\n- 4️⃣ Something else\n\n\n## Arrow & File Formats\n\n\n\n## Seattle
Checkouts
Big CSV\n\n{.absolute top=\"0\" left=\"300\"}\n\n::: {style=\"font-size: 60%; margin-top: 440px; margin-left: 330px;\"}\n
\n\nLet's extract the schema:\n\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nschema(seattle_csv)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nSchema\nUsageClass: string\nCheckoutType: string\nMaterialType: string\nCheckoutYear: int64\nCheckoutMonth: int64\nCheckouts: int64\nTitle: string\nISBN: null\nCreator: string\nSubjects: string\nPublisher: string\nPublicationYear: string\n```\n\n\n:::\n:::\n\n\n\n\n## Arrow Data Types\n\nArrow has a rich data type system, including direct analogs of many R data types\n\n- `
\n\n
\n\nArrow scans 👀 1MB of data to impute or \"guess\" the data types\n\n::: {style=\"font-size: 80%; margin-top: 200px;\"}\n📚 arrow vs readr blog post:
\n\nThis will take a lot of typing with 12 columns 😢\n\n## Let's Control the Schema\n\nUse the code() method to extract the code from the schema:\n\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nseattle_csv$schema$code() \n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nschema(UsageClass = utf8(), CheckoutType = utf8(), MaterialType = utf8(), \n CheckoutYear = int64(), CheckoutMonth = int64(), Checkouts = int64(), \n Title = utf8(), ISBN = null(), Creator = utf8(), Subjects = utf8(), \n Publisher = utf8(), PublicationYear = utf8())\n```\n\n\n:::\n:::\n\n\n\n\n
\n\n🤩\n\n## Let's Control the Schema\n\nSchema defines column names and types, so we need to skip the first row (skip = 1):\n\n\n\n\n::: {.cell}\n\n```{.r .cell-code code-line-numbers=\"|12\"}\nseattle_csv <- open_dataset(sources = \"data/seattle-library-checkouts.csv\",\n format = \"csv\",\n skip = 1,\n schema = schema(\n UsageClass = utf8(),\n CheckoutType = utf8(),\n MaterialType = utf8(),\n CheckoutYear = int64(),\n CheckoutMonth = int64(),\n Checkouts = int64(),\n Title = utf8(),\n ISBN = string(), #utf8()\n Creator = utf8(),\n Subjects = utf8(),\n Publisher = utf8(),\n PublicationYear = utf8()\n )\n)\nseattle_csv\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nFileSystemDataset with 1 csv file\n12 columns\nUsageClass: string\nCheckoutType: string\nMaterialType: string\nCheckoutYear: int64\nCheckoutMonth: int64\nCheckouts: int64\nTitle: string\nISBN: string\nCreator: string\nSubjects: string\nPublisher: string\nPublicationYear: string\n```\n\n\n:::\n:::\n\n\n\n\n## Let's Control the Schema\n\nSupply column types for a subset of columns by providing a partial schema:\n\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nseattle_csv <- open_dataset(\n sources = \"data/seattle-library-checkouts.csv\",\n format = \"csv\",\n col_types = schema(ISBN = string()) #utf8()\n)\nseattle_csv\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nFileSystemDataset with 1 csv file\n12 columns\nUsageClass: string\nCheckoutType: string\nMaterialType: string\nCheckoutYear: int64\nCheckoutMonth: int64\nCheckouts: int64\nTitle: string\nISBN: string\nCreator: string\nSubjects: string\nPublisher: string\nPublicationYear: string\n```\n\n\n:::\n:::\n\n\n\n\n\n## Your Turn\n\n1. The first few thousand rows of `ISBN` are blank in the Seattle Checkouts CSV file. Read in the Seattle Checkouts CSV file with `open_dataset()` and ensure the correct data type for `ISBN` is `
\n\nParquet about half the size of the CSV file on-disk 💾\n\n## Your Turn\n\n1. Re-run the query counting the number of `Checkouts` by `CheckoutYear` and arranging the result by `CheckoutYear`, this time using the Seattle Checkout data saved to disk as a single, Parquet file. Did you notice a difference in compute time?\n\n➡️ [Data Storage Engineering Exercises Page](3_data_engineering-exercises.html)\n\n## 4.5GB Parquet file + arrow + dplyr\n\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nopen_dataset(sources = seattle_parquet, \n format = \"parquet\") |>\n group_by(CheckoutYear) |>\n summarise(sum(Checkouts)) |>\n arrange(CheckoutYear) |> \n collect() |>\n system.time()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n user system elapsed \n 2.018 0.265 0.595 \n```\n\n\n:::\n:::\n\n\n\n\n42 million rows -- much better! But could be *even* faster....\n\n## File Storage:
Partitioning\n\n
\n\n::: columns\n::: {.column width=\"50%\"}\nDividing data into smaller pieces, making it more easily accessible and manageable\n:::\n\n::: {.column width=\"50%\"}\n{.absolute top=\"0\"}\n:::\n:::\n\n::: notes\nalso called multi-files or sometimes shards\n:::\n\n## Poll: Partitioning?\n\nHave you partitioned your data or used partitioned data before today?\n\n
\n\n- 1️⃣ Yes\n- 2️⃣ No\n- 3️⃣ Not sure, the data engineers sort that out!\n\n## Art & Science of Partitioning\n\n
\n\n- avoid files \\< 20MB and \\> 2GB\n- avoid \\> 10,000 files (🤯)\n- partition on variables used in `filter()`\n\n::: notes\n- guidelines not rules, results vary\n- experiment, especially with cloud\n- arrow suggests avoid files smaller than 20MB and larger than 2GB\n- avoid partitions that produce more than 10,000 files\n- partition by variables that you filter by, allows arrow to only read relevant files\n:::\n\n## Rewriting the Data Again\n\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nseattle_parquet_part <- \"data/seattle-library-checkouts\"\n\nseattle_csv |>\n group_by(CheckoutYear) |>\n write_dataset(path = seattle_parquet_part,\n format = \"parquet\")\n```\n:::\n\n\n\n\n## What Did We \"Engineer\"?\n\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nseattle_parquet_part <- \"data/seattle-library-checkouts\"\n\nsizes <- tibble(\n files = list.files(seattle_parquet_part, recursive = TRUE),\n size_GB = file.size(file.path(seattle_parquet_part, files)) / 10**9\n)\n\nsizes\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 18 × 2\n files size_GB\n
\n\n42 million rows -- not too shabby!\n\n## Your Turn\n\n1. Let's write the Seattle Checkout CSV data to a multi-file dataset just one more time! This time, write the data partitioned by `CheckoutType` as Parquet files.\n\n2. Now compare the compute time between our Parquet data partitioned by `CheckoutYear` and our Parquet data partitioned by `CheckoutType` with a query of the total number of checkouts in September of 2019. Did you find a difference in compute time?\n\n➡️ [Data Storage Engineering Exercises Page](3_data_engineering-exercises.html)\n\n\n## Partitions & NA Values\n\nADD content\n\n## Partition Design\n\n::: columns\n::: {.column width=\"50%\"}\n- Partitioning on variables commonly used in `filter()` often faster\n- Number of partitions also important (Arrow reads the metadata of each file)\n:::\n\n::: {.column width=\"50%\"}\n{.absolute top=\"0\"}\n:::\n:::\n\n## Performance Review: Single CSV\n\nHow long does it take to calculate the number of books checked out in each month of 2021?\n\n
\n\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nopen_dataset(sources = \"data/seattle-library-checkouts.csv\", \n format = \"csv\") |> \n\n filter(CheckoutYear == 2021, MaterialType == \"BOOK\") |>\n group_by(CheckoutMonth) |>\n summarise(TotalCheckouts = sum(Checkouts)) |>\n arrange(desc(CheckoutMonth)) |>\n collect() |>\n system.time()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n user system elapsed \n 13.362 1.763 12.438 \n```\n\n\n:::\n:::\n\n\n\n\n## Performance Review: Partitioned Parquet\n\nHow long does it take to calculate the number of books checked out in each month of 2021?\n\n
\n\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nopen_dataset(sources = \"data/seattle-library-checkouts\",\n format = \"parquet\") |> \n filter(CheckoutYear == 2021, MaterialType == \"BOOK\") |>\n group_by(CheckoutMonth) |>\n summarise(TotalCheckouts = sum(Checkouts)) |>\n arrange(desc(CheckoutMonth)) |>\n collect() |> \n system.time()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n user system elapsed \n 0.330 0.039 0.091 \n```\n\n\n:::\n:::\n\n\n\n\n## Engineering Data Tips for Improved Storage & Performance\n\n
\n\n- consider \"column-oriented\" file formats like Parquet\n- consider partitioning, experiment to get an appropriate partition design 🗂️\n- watch your schemas 👀\n\n## R for Data Science (2e)\n\n::: columns\n::: {.column width=\"50%\"}\n{.absolute top=\"100\" width=\"400\"}\n:::\n\n::: {.column width=\"50%\"}\n
\n\n[Chapter 23: Arrow](https://r4ds.hadley.nz/arrow.html)\n\n
\n\n
\n\n\n\n
\n\n::: {style=\"font-size: 70%;\"}\n
\n\n::: {style=\"font-size: 70%;\"}\n
\n\n**Which file formats do you use most often?**\n\n
\n\n- 1️⃣ CSV (.csv)\n- 2️⃣ MS Excel (.xls and .xlsx)\n- 3️⃣ Parquet (.parquet)\n- 4️⃣ Something else\n\n## Arrow & File Formats\n\n\n\n## Seattle
Checkouts
Big CSV\n\n{.absolute top=\"0\" left=\"300\"}\n\n::: {style=\"font-size: 60%; margin-top: 440px; margin-left: 330px;\"}\n
\n\nLet's extract the schema:\n\n\n::: {.cell}\n\n```{.r .cell-code}\nschema(seattle_csv)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nSchema\nUsageClass: string\nCheckoutType: string\nMaterialType: string\nCheckoutYear: int64\nCheckoutMonth: int64\nCheckouts: int64\nTitle: string\nISBN: null\nCreator: string\nSubjects: string\nPublisher: string\nPublicationYear: string\n```\n\n\n:::\n:::\n\n\n## Arrow Data Types\n\nArrow has a rich data type system, including direct analogs of many R data types\n\n- `
\n\n
\n\nArrow scans 👀 1MB of data to impute or \"guess\" the data types\n\n::: {style=\"font-size: 80%; margin-top: 200px;\"}\n📚 arrow vs readr blog post:
\n\nThis will take a lot of typing with 12 columns 😢\n\n## Let's Control the Schema\n\nUse the code() method to extract the code from the schema:\n\n\n::: {.cell}\n\n```{.r .cell-code}\nseattle_csv$schema$code() \n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nschema(UsageClass = utf8(), CheckoutType = utf8(), MaterialType = utf8(), \n CheckoutYear = int64(), CheckoutMonth = int64(), Checkouts = int64(), \n Title = utf8(), ISBN = null(), Creator = utf8(), Subjects = utf8(), \n Publisher = utf8(), PublicationYear = utf8())\n```\n\n\n:::\n:::\n\n\n
\n\n🤩\n\n## Let's Control the Schema\n\nSchema defines column names and types, so we need to skip the first row (skip = 1):\n\n\n::: {.cell}\n\n```{.r .cell-code code-line-numbers=\"|11|17\"}\nseattle_csv <- open_dataset(sources = \"data/seattle-library-checkouts.csv\",\n format = \"csv\",\n schema = schema(\n UsageClass = utf8(),\n CheckoutType = utf8(),\n MaterialType = utf8(),\n CheckoutYear = int64(),\n CheckoutMonth = int64(),\n Checkouts = int64(),\n Title = utf8(),\n ISBN = string(), #utf8()\n Creator = utf8(),\n Subjects = utf8(),\n Publisher = utf8(),\n PublicationYear = utf8()\n ),\n skip = 1,\n)\nseattle_csv\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nFileSystemDataset with 1 csv file\n12 columns\nUsageClass: string\nCheckoutType: string\nMaterialType: string\nCheckoutYear: int64\nCheckoutMonth: int64\nCheckouts: int64\nTitle: string\nISBN: string\nCreator: string\nSubjects: string\nPublisher: string\nPublicationYear: string\n```\n\n\n:::\n:::\n\n\n## Let's Control the Schema\n\nSupply column types for a subset of columns by providing a partial schema:\n\n\n::: {.cell}\n\n```{.r .cell-code}\nseattle_csv <- open_dataset(\n sources = \"data/seattle-library-checkouts.csv\",\n format = \"csv\",\n col_types = schema(ISBN = string()) #utf8()\n)\nseattle_csv\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nFileSystemDataset with 1 csv file\n12 columns\nUsageClass: string\nCheckoutType: string\nMaterialType: string\nCheckoutYear: int64\nCheckoutMonth: int64\nCheckouts: int64\nTitle: string\nISBN: string\nCreator: string\nSubjects: string\nPublisher: string\nPublicationYear: string\n```\n\n\n:::\n:::\n\n\n\n## Your Turn\n\n1. The first few thousand rows of `ISBN` are blank in the Seattle Checkouts CSV file. Read in the Seattle Checkouts CSV file with `open_dataset()` and ensure the correct data type for `ISBN` is `
\n\nParquet about half the size of the CSV file on-disk 💾\n\n## Your Turn\n\n1. Re-run the query counting the number of `Checkouts` by `CheckoutYear` and arranging the result by `CheckoutYear`, this time using the Seattle Checkout data saved to disk as a single, Parquet file. Did you notice a difference in compute time?\n\n➡️ [Data Storage Engineering Exercises Page](3_data_engineering-exercises.html)\n\n## 4.5GB Parquet file + arrow + dplyr\n\n\n::: {.cell}\n\n```{.r .cell-code}\nopen_dataset(sources = seattle_parquet, \n format = \"parquet\") |>\n group_by(CheckoutYear) |>\n summarise(sum(Checkouts)) |>\n arrange(CheckoutYear) |> \n collect() |>\n system.time()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n user system elapsed \n 1.771 0.431 0.568 \n```\n\n\n:::\n:::\n\n\n42 million rows -- much better! But could be *even* faster....\n\n## File Storage:
Partitioning\n\n
\n\n::: columns\n::: {.column width=\"50%\"}\nDividing data into smaller pieces, making it more easily accessible and manageable\n:::\n\n::: {.column width=\"50%\"}\n{.absolute top=\"0\"}\n:::\n:::\n\n::: notes\nalso called multi-files or sometimes shards\n:::\n\n## Poll: Partitioning?\n\nHave you partitioned your data or used partitioned data before today?\n\n
\n\n- 1️⃣ Yes\n- 2️⃣ No\n- 3️⃣ Not sure, the data engineers sort that out!\n\n## Art & Science of Partitioning\n\n
\n\n- avoid files \\< 20MB and \\> 2GB\n- avoid \\> 10,000 files (🤯)\n- partition on variables used in `filter()`\n\n::: notes\n- guidelines not rules, results vary\n- experiment, especially with cloud\n- arrow suggests avoid files smaller than 20MB and larger than 2GB\n- avoid partitions that produce more than 10,000 files\n- partition by variables that you filter by, allows arrow to only read relevant files\n:::\n\n## Rewriting the Data Again\n\n\n::: {.cell}\n\n```{.r .cell-code}\nseattle_parquet_part <- \"data/seattle-library-checkouts\"\n\nseattle_csv |>\n group_by(CheckoutYear) |>\n write_dataset(path = seattle_parquet_part,\n format = \"parquet\")\n```\n:::\n\n\n## What Did We \"Engineer\"?\n\n\n::: {.cell}\n\n```{.r .cell-code}\nseattle_parquet_part <- \"data/seattle-library-checkouts\"\n\nsizes <- tibble(\n files = list.files(seattle_parquet_part, recursive = TRUE),\n size_GB = file.size(file.path(seattle_parquet_part, files)) / 10**9\n)\n\nsizes\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 18 × 2\n files size_GB\n
\n\n42 million rows -- not too shabby!\n\n## Your Turn\n\n1. Let's write the Seattle Checkout CSV data to a multi-file dataset just one more time! This time, write the data partitioned by `CheckoutType` as Parquet files.\n\n2. Now compare the compute time between our Parquet data partitioned by `CheckoutYear` and our Parquet data partitioned by `CheckoutType` with a query of the total number of checkouts in September of 2019. Did you find a difference in compute time?\n\n➡️ [Data Storage Engineering Exercises Page](3_data_engineering-exercises.html)\n\n## Partition Design\n\n::: columns\n::: {.column width=\"50%\"}\n- Partitioning on variables commonly used in `filter()` often faster\n- Number of partitions also important (Arrow reads the metadata of each file)\n:::\n\n::: {.column width=\"50%\"}\n{.absolute top=\"0\"}\n:::\n:::\n\n## Partitions & NA Values\n\nDefault:\n\n\n::: {.cell}\n\n```{.r .cell-code}\npartition_na_default_path <- \"data/na-partition-default\"\n\nwrite_dataset(starwars,\n partition_na_default_path,\n partitioning = \"hair_color\")\n\nlist.files(partition_na_default_path)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n [1] \"hair_color=__HIVE_DEFAULT_PARTITION__\"\n [2] \"hair_color=auburn\" \n [3] \"hair_color=auburn%2C%20grey\" \n [4] \"hair_color=auburn%2C%20white\" \n [5] \"hair_color=black\" \n [6] \"hair_color=blond\" \n [7] \"hair_color=blonde\" \n [8] \"hair_color=brown\" \n [9] \"hair_color=brown%2C%20grey\" \n[10] \"hair_color=grey\" \n[11] \"hair_color=none\" \n[12] \"hair_color=white\" \n```\n\n\n:::\n:::\n\n\n## Partitions & NA Values\n\nCustom:\n\n\n::: {.cell}\n\n```{.r .cell-code}\npartition_na_custom_path <- \"data/na-partition-custom\"\n\nwrite_dataset(starwars,\n partition_na_custom_path,\n partitioning = hive_partition(hair_color = string(),\n null_fallback = \"no_color\"))\n\nlist.files(partition_na_custom_path)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n [1] \"hair_color=auburn\" \"hair_color=auburn%2C%20grey\" \n [3] \"hair_color=auburn%2C%20white\" \"hair_color=black\" \n [5] \"hair_color=blond\" \"hair_color=blonde\" \n [7] \"hair_color=brown\" \"hair_color=brown%2C%20grey\" \n [9] \"hair_color=grey\" \"hair_color=no_color\" \n[11] \"hair_color=none\" \"hair_color=white\" \n```\n\n\n:::\n:::\n\n\n## Performance Review: Single CSV\n\nHow long does it take to calculate the number of books checked out in each month of 2021?\n\n
\n\n\n::: {.cell}\n\n```{.r .cell-code}\nopen_dataset(sources = \"data/seattle-library-checkouts.csv\", \n format = \"csv\") |> \n\n filter(CheckoutYear == 2021, MaterialType == \"BOOK\") |>\n group_by(CheckoutMonth) |>\n summarise(TotalCheckouts = sum(Checkouts)) |>\n arrange(desc(CheckoutMonth)) |>\n collect() |>\n system.time()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n user system elapsed \n 11.718 1.106 11.250 \n```\n\n\n:::\n:::\n\n\n## Performance Review: Partitioned Parquet\n\nHow long does it take to calculate the number of books checked out in each month of 2021?\n\n
\n\n\n::: {.cell}\n\n```{.r .cell-code}\nopen_dataset(sources = \"data/seattle-library-checkouts\",\n format = \"parquet\") |> \n filter(CheckoutYear == 2021, MaterialType == \"BOOK\") |>\n group_by(CheckoutMonth) |>\n summarise(TotalCheckouts = sum(Checkouts)) |>\n arrange(desc(CheckoutMonth)) |>\n collect() |> \n system.time()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n user system elapsed \n 0.224 0.040 0.068 \n```\n\n\n:::\n:::\n\n\n## Engineering Data Tips for Improved Storage & Performance\n\n
\n\n- consider \"column-oriented\" file formats like Parquet\n- consider partitioning, experiment to get an appropriate partition design 🗂️\n- watch your schemas 👀\n\n## R for Data Science (2e)\n\n::: columns\n::: {.column width=\"50%\"}\n{.absolute top=\"100\" width=\"400\"}\n:::\n\n::: {.column width=\"50%\"}\n
\n\n[Chapter 23: Arrow](https://r4ds.hadley.nz/arrow.html)\n\n
\n\n
\n\n`library(arrow)`\n\n- `read_parquet()`\n- `read_csv_arrow()`\n- `read_feather()`\n- `read_json_arrow()`\n\n**Value**: `tibble` (the default), or an Arrow Table if `as_data_frame = FALSE` --- both *in-memory*\n\n## Your Turn\n\n1. Read in a single NYC Taxi parquet file using `read_parquet()` as an Arrow Table\n\n2. Convert your Arrow Table object to a `data.frame` or a `tibble`\n\n## Read a Parquet File (`tibble`)\n\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(arrow)\n\nparquet_file <- \"data/nyc-taxi/year=2019/month=9/part-0.parquet\"\n\ntaxi_df <- read_parquet(file = parquet_file)\ntaxi_df\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 6,567,396 × 22\n vendor_name pickup_datetime dropoff_datetime passenger_count\n
\n\n- `data.frame` & `tibble` are R objects *in-memory*\n- `Table` is an Arrow object *in-memory*\n\n## Data frames\n\n\n\n## Arrow Tables\n\n\n\n::: notes\nArrow Tables are collections of chunked arrays\n:::\n\n## Table \\| Dataset: A `dplyr` pipeline\n\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nparquet_file |>\n read_parquet(as_data_frame = FALSE) |>\n group_by(vendor_name) |>\n summarise(all_trips = n(),\n shared_trips = sum(passenger_count > 1, na.rm = TRUE)) |>\n mutate(pct_shared = shared_trips / all_trips * 100) |>\n collect()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 3 × 4\n vendor_name all_trips shared_trips pct_shared\n
\n\nFunctions available in Arrow dplyr queries:
\n\n\n\n\n::: {.cell}\n\n```{.r .cell-code code-line-numbers=\"|2,8\"}\nparquet_file |>\n read_parquet() |>\n group_by(vendor_name) |>\n summarise(all_trips = n(),\n shared_trips = sum(passenger_count > 1, na.rm = TRUE)) |>\n mutate(pct_shared = shared_trips / all_trips * 100) |>\n collect() |>\n system.time()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n user system elapsed \n 2.214 0.575 0.814 \n```\n\n\n:::\n:::\n\n\n\n\n## Arrow for Efficient In-Memory Processing\n\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nparquet_file |>\n read_parquet(as_data_frame = FALSE) |>\n nrow()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] 6567396\n```\n\n\n:::\n:::\n\n\n\n\n
\n\n\n\n\n::: {.cell}\n\n```{.r .cell-code code-line-numbers=\"|2,8\"}\nparquet_file |>\n read_parquet(as_data_frame = FALSE) |>\n group_by(vendor_name) |>\n summarise(all_trips = n(),\n shared_trips = sum(passenger_count > 1, na.rm = TRUE)) |>\n mutate(pct_shared = shared_trips / all_trips * 100) |>\n collect() |>\n system.time()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n user system elapsed \n 1.995 0.343 0.366 \n```\n\n\n:::\n:::\n\n\n\n\n## Read a Parquet File Selectively\n\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nparquet_file |>\n read_parquet(\n col_select = c(\"vendor_name\", \"passenger_count\"),\n as_data_frame = FALSE\n )\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nTable\n6567396 rows x 2 columns\n$vendor_name
\n\n`library(arrow)`\n\n- `read_parquet()`\n- `read_csv_arrow()`\n- `read_feather()`\n- `read_json_arrow()`\n\n**Value**: `tibble` (the default), or an Arrow Table if `as_data_frame = FALSE` --- both *in-memory*\n\n## Your Turn\n\n1. Read in a single NYC Taxi parquet file using `read_parquet()` as an Arrow Table\n\n2. Convert your Arrow Table object to a `data.frame` or a `tibble`\n\n## Read a Parquet File (`tibble`)\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(arrow)\n\nparquet_file <- \"data/nyc-taxi/year=2019/month=9/part-0.parquet\"\n\ntaxi_df <- read_parquet(file = parquet_file)\ntaxi_df\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 6,567,396 × 22\n vendor_name pickup_datetime dropoff_datetime passenger_count\n
\n\n- `data.frame` & `tibble` are R objects *in-memory*\n- `Table` is an Arrow object *in-memory*\n\n## Watch Your Schemas 👀\n\n\n:::: {.columns}\n\n::: {.column width=\"50%\"}\n\n::: {.cell}\n\n```{.r .cell-code}\nschema(taxi_df)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nSchema\nvendor_name: string\npickup_datetime: timestamp[us, tz=America/Vancouver]\ndropoff_datetime: timestamp[us, tz=America/Vancouver]\npassenger_count: int32\ntrip_distance: double\npickup_longitude: double\npickup_latitude: double\nrate_code: string\nstore_and_fwd: string\ndropoff_longitude: double\ndropoff_latitude: double\npayment_type: string\nfare_amount: double\nextra: double\nmta_tax: double\ntip_amount: double\ntolls_amount: double\ntotal_amount: double\nimprovement_surcharge: double\ncongestion_surcharge: double\npickup_location_id: int32\ndropoff_location_id: int32\n```\n\n\n:::\n:::\n\n:::\n\n::: {.column width=\"50%\"}\n\n::: {.cell}\n\n```{.r .cell-code}\nschema(taxi_table)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nSchema\nvendor_name: string\npickup_datetime: timestamp[ms]\ndropoff_datetime: timestamp[ms]\npassenger_count: int64\ntrip_distance: double\npickup_longitude: double\npickup_latitude: double\nrate_code: string\nstore_and_fwd: string\ndropoff_longitude: double\ndropoff_latitude: double\npayment_type: string\nfare_amount: double\nextra: double\nmta_tax: double\ntip_amount: double\ntolls_amount: double\ntotal_amount: double\nimprovement_surcharge: double\ncongestion_surcharge: double\npickup_location_id: int64\ndropoff_location_id: int64\n```\n\n\n:::\n:::\n\n:::\n\n::::\n\n## Data frames\n\n\n\n## Arrow Tables\n\n\n\n::: notes\nArrow Tables are collections of chunked arrays\n:::\n\n## Table \\| Dataset: A `dplyr` pipeline\n\n\n::: {.cell}\n\n```{.r .cell-code}\nparquet_file |>\n read_parquet(as_data_frame = FALSE) |>\n group_by(vendor_name) |>\n summarise(all_trips = n(),\n shared_trips = sum(passenger_count > 1, na.rm = TRUE)) |>\n mutate(pct_shared = shared_trips / all_trips * 100) |>\n collect()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n# A tibble: 3 × 4\n vendor_name all_trips shared_trips pct_shared\n
\n\nFunctions available in Arrow dplyr queries:
\n\n\n::: {.cell}\n\n```{.r .cell-code code-line-numbers=\"|2,8\"}\nparquet_file |>\n read_parquet() |>\n group_by(vendor_name) |>\n summarise(all_trips = n(),\n shared_trips = sum(passenger_count > 1, na.rm = TRUE)) |>\n mutate(pct_shared = shared_trips / all_trips * 100) |>\n collect() |>\n system.time()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n user system elapsed \n 1.157 0.261 0.509 \n```\n\n\n:::\n:::\n\n\n## Arrow for Efficient In-Memory Processing\n\n\n::: {.cell}\n\n```{.r .cell-code}\nparquet_file |>\n read_parquet(as_data_frame = FALSE) |>\n nrow()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n[1] 6567396\n```\n\n\n:::\n:::\n\n\n
\n\n\n::: {.cell}\n\n```{.r .cell-code code-line-numbers=\"|2,8\"}\nparquet_file |>\n read_parquet(as_data_frame = FALSE) |>\n group_by(vendor_name) |>\n summarise(all_trips = n(),\n shared_trips = sum(passenger_count > 1, na.rm = TRUE)) |>\n mutate(pct_shared = shared_trips / all_trips * 100) |>\n collect() |>\n system.time()\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\n user system elapsed \n 1.047 0.203 0.220 \n```\n\n\n:::\n:::\n\n\n## Read a Parquet File Selectively\n\n\n::: {.cell}\n\n```{.r .cell-code}\nparquet_file |>\n read_parquet(\n col_select = c(\"vendor_name\", \"passenger_count\"),\n as_data_frame = FALSE\n )\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nTable\n6567396 rows x 2 columns\n$vendor_name
We Assume
Setup
+-
+
- Log onto Workbench at the following URL:
- Create a new session; select “Resource Profile: Large” +
- Run
usethis::use_course("posit-conf-2024/arrow")
+ - Open
data/setup.Rand run the script
+
Schemas
@@ -293,28 +292,28 @@
Schemas
seattle_csv <- open_dataset(sources = "data/seattle-library-checkouts.csv",
format = "csv",
- skip = 1,
- schema(
- UsageClass = utf8(),
- CheckoutType = utf8(),
- MaterialType = utf8(),
- CheckoutYear = int64(),
- CheckoutMonth = int64(),
- Checkouts = int64(),
- Title = utf8(),
- ISBN = string(), #or utf8()
- Creator = utf8(),
- Subjects = utf8(),
- Publisher = utf8(),
- PublicationYear = utf8()
- )
+ schema(
+ UsageClass = utf8(),
+ CheckoutType = utf8(),
+ MaterialType = utf8(),
+ CheckoutYear = int64(),
+ CheckoutMonth = int64(),
+ Checkouts = int64(),
+ Title = utf8(),
+ ISBN = string(), #or utf8()
+ Creator = utf8(),
+ Subjects = utf8(),
+ Publisher = utf8(),
+ PublicationYear = utf8()
+ ),
+ skip = 1,
)or
seattle_csv <- open_dataset(sources = "data/seattle-library-checkouts.csv",
format = "csv",
- col_types = schema(ISBN = string()) #utf8()
+ col_types = schema(ISBN = string()) # or utf8()
)
seattle_csv
@@ -407,7 +406,7 @@
Schemas
system.time() user system elapsed
- 11.474 1.084 11.003 Querying 42 million rows of data stored in a CSV on disk in ~10 seconds, not too bad.
@@ -457,7 +456,7 @@Parquet
system.time() user system elapsed
- 2.076 0.287 0.646 A much faster compute time for the query when the on-disk data is stored in the Parquet format.
@@ -517,7 +516,7 @@Partitioning
system.time() user system elapsed
- 0.965 0.160 0.409 Total number of Checkouts in September of 2019 using partitioned Parquet data by CheckoutYear and CheckoutMonth:
Partitioning
system.time() user system elapsed
- 0.058 0.006 0.052 Faster compute time because the filter() call is based on the partitions.
arrow::open_dataset() with a CSV
seattle_csv <- open_dataset(sources = "data/seattle-library-checkouts.csv", - format = "csv") + format = "csv") seattle_csvFileSystemDataset with 1 csv file
@@ -592,23 +592,23 @@ Let’s Control the Schema
Let’s Control the Schema
Schema defines column names and types, so we need to skip the first row (skip = 1):
-seattle_csv <- open_dataset(sources = "data/seattle-library-checkouts.csv",
+seattle_csv <- open_dataset(sources = "data/seattle-library-checkouts.csv",
format = "csv",
- skip = 1,
- schema = schema(
- UsageClass = utf8(),
- CheckoutType = utf8(),
- MaterialType = utf8(),
- CheckoutYear = int64(),
- CheckoutMonth = int64(),
- Checkouts = int64(),
- Title = utf8(),
- ISBN = string(), #utf8()
- Creator = utf8(),
- Subjects = utf8(),
- Publisher = utf8(),
- PublicationYear = utf8()
- )
+ schema = schema(
+ UsageClass = utf8(),
+ CheckoutType = utf8(),
+ MaterialType = utf8(),
+ CheckoutYear = int64(),
+ CheckoutMonth = int64(),
+ Checkouts = int64(),
+ Title = utf8(),
+ ISBN = string(), #utf8()
+ Creator = utf8(),
+ Subjects = utf8(),
+ Publisher = utf8(),
+ PublicationYear = utf8()
+ ),
+ skip = 1,
)
seattle_csv
@@ -660,7 +660,7 @@ Let’s Control the Schema
user system elapsed
- 11.581 1.136 11.117
+ 10.688 1.099 10.451
42 million rows – not bad, but could be faster….
@@ -769,7 +769,7 @@ Storage: Parquet vs CSV
-[1] 4.423348
+[1] 4.424267
@@ -794,7 +794,7 @@ 4.5GB Parquet file + arrow + dplyr
system.time() user system elapsed
- 2.018 0.265 0.595 42 million rows – much better! But could be even faster….
@@ -887,7 +887,7 @@What Did We “Engineer”?
# A tibble: 18 × 2
files size_GB
<chr> <dbl>
- 1 CheckoutYear=2005/part-0.parquet 0.114
+ 1 CheckoutYear=2005/part-0.parquet 0.115
2 CheckoutYear=2006/part-0.parquet 0.172
3 CheckoutYear=2007/part-0.parquet 0.186
4 CheckoutYear=2008/part-0.parquet 0.204
@@ -901,7 +901,7 @@ What Did We “Engineer”?
12 CheckoutYear=2016/part-0.parquet 0.315
13 CheckoutYear=2017/part-0.parquet 0.319
14 CheckoutYear=2018/part-0.parquet 0.306
-15 CheckoutYear=2019/part-0.parquet 0.302
+15 CheckoutYear=2019/part-0.parquet 0.303
16 CheckoutYear=2020/part-0.parquet 0.158
17 CheckoutYear=2021/part-0.parquet 0.240
18 CheckoutYear=2022/part-0.parquet 0.2524.5GB partitioned Parquet files + arrow + dplyr
system.time() user system elapsed
- 1.640 0.220 0.267 Your Turn
Performance Review: Partitioned Parquet
How long does it take to calculate the number of books checked out in each month of 2021?
-
+
diff --git a/_site/materials/5_arrow_single_file.html b/_site/materials/5_arrow_single_file.html
index 06a9782..bbc5840 100644
--- a/_site/materials/5_arrow_single_file.html
+++ b/_site/materials/5_arrow_single_file.html
@@ -431,16 +431,16 @@ open_dataset(sources = "data/seattle-library-checkouts",
- format = "parquet") |>
- filter(CheckoutYear == 2021, MaterialType == "BOOK") |>
- group_by(CheckoutMonth) |>
- summarise(TotalCheckouts = sum(Checkouts)) |>
- arrange(desc(CheckoutMonth)) |>
- collect() |>
- system.time()open_dataset(sources = "data/seattle-library-checkouts",
+ format = "parquet") |>
+ filter(CheckoutYear == 2021, MaterialType == "BOOK") |>
+ group_by(CheckoutMonth) |>
+ summarise(TotalCheckouts = sum(Checkouts)) |>
+ arrange(desc(CheckoutMonth)) |>
+ collect() |>
+ system.time() user system elapsed
- 0.330 0.039 0.091 Read a Parquet File (tibble)
# A tibble: 6,567,396 × 22
vendor_name pickup_datetime dropoff_datetime passenger_count
<chr> <dttm> <dttm> <int>
- 1 VTS 2019-09-01 06:14:09 2019-09-01 06:31:52 2
- 2 VTS 2019-09-01 06:36:17 2019-09-01 07:12:44 1
- 3 VTS 2019-09-01 06:29:19 2019-09-01 06:54:13 1
- 4 CMT 2019-09-01 06:33:09 2019-09-01 06:52:14 2
- 5 VTS 2019-09-01 06:57:43 2019-09-01 07:26:21 1
- 6 CMT 2019-09-01 06:59:16 2019-09-01 07:28:12 1
- 7 CMT 2019-09-01 06:20:06 2019-09-01 06:52:19 1
- 8 CMT 2019-09-01 06:27:54 2019-09-01 06:32:56 0
- 9 CMT 2019-09-01 06:35:08 2019-09-01 06:55:51 0
-10 CMT 2019-09-01 06:19:37 2019-09-01 06:30:52 1
+ 1 CMT 2019-08-31 18:09:30 2019-08-31 18:15:42 1
+ 2 CMT 2019-08-31 18:26:30 2019-08-31 18:44:31 1
+ 3 CMT 2019-08-31 18:39:35 2019-08-31 19:15:55 2
+ 4 VTS 2019-08-31 18:12:26 2019-08-31 18:15:17 4
+ 5 VTS 2019-08-31 18:43:16 2019-08-31 18:53:50 1
+ 6 VTS 2019-08-31 18:26:13 2019-08-31 18:45:35 1
+ 7 CMT 2019-08-31 18:34:52 2019-08-31 18:42:03 1
+ 8 CMT 2019-08-31 18:50:02 2019-08-31 18:58:16 1
+ 9 CMT 2019-08-31 18:08:02 2019-08-31 18:14:44 0
+10 VTS 2019-08-31 18:11:38 2019-08-31 18:26:47 1
# ℹ 6,567,386 more rows
# ℹ 18 more variables: trip_distance <dbl>, pickup_longitude <dbl>,
# pickup_latitude <dbl>, rate_code <chr>, store_and_fwd <chr>,
@@ -506,6 +506,70 @@ tibble <-> Table <-> data.frame<
Table is an Arrow object in-memory