Handling of single element arrays in datapackage.json

[sannegovaert]

There is a discrepancy in how single-element arrays are written and read back into the datapackage.json file. Specifically, single-element arrays are being converted into strings when the package is read and then rewritten.

Example:

1. Create a DP:

library(frictionless)

temp <- here::here("frictionless")

resource <- data.frame(1:10, 11:20)
dp_v1 <-
  frictionless::create_package() %>% 
  add_resource(resource_name = "numbers", data = resource) %>% 
  frictionless::create_package()

dp_v1$project <- list(
  single_element = list(c("media")),
  double_element = list(c("media", "event"))
)

frictionless::write_package(dp_v1, file.path(temp, "1"))

2. The resulting datapackage.json is as follows (as expected):

{
  "resources": [
    {
      "name": "numbers",
      "path": "numbers.csv",
      "profile": "tabular-data-resource",
      "format": "csv",
      "mediatype": "text/csv",
      "encoding": "utf-8",
      "schema": {
        "fields": [
          {
            "name": "X1.10",
            "type": "integer"
          },
          {
            "name": "X11.20",
            "type": "integer"
          }
        ]
      }
    }
  ],
  "project": {
    "single_element": [
      "media"
    ],
    "double_element": [
      ["media", "event"]
    ]
  }
}

Read datapackage.json and write it back:

dp_v2 <- frictionless::read_package(file.path(temp, "1", "datapackage.json"))

frictionless::write_package(dp_v2, file.path(temp, "2"))

The resulting datapackage.json after being rewritten is not as expected:

{
  "resources": [
    {
      "name": "numbers",
      "path": "numbers.csv",
      "profile": "tabular-data-resource",
      "format": "csv",
      "mediatype": "text/csv",
      "encoding": "utf-8",
      "schema": {
        "fields": [
          {
            "name": "X1.10",
            "type": "integer"
          },
          {
            "name": "X11.20",
            "type": "integer"
          }
        ]
      }
    }
  ],
  "project": {
    "single_element": "media",
    "double_element": [
      ["media", "event"]
    ]
  }
}

[peterdesmet]

Cause

This is caused by simplifyVector = TRUE (the default for fromJSON):

frictionless-r/R/utils.R

Line 76 in 6b62581

jsonlite::fromJSON(x, simplifyDataFrame = FALSE, simplifyVector = TRUE)

Changing this to simplifyVector = FALSE will preserve the single-element array as an array. When writing, it will first write the non-pretty version:

  "project": {
    "single_element": [
      "media"
    ],
    "double_element": [
      ["media", "event"]
    ]
  }

And after a read and write, a pretty:

  "project": {
    "single_element": [
      "media"
    ],
    "double_element": [
      [
        "media",
        "event"
      ]
    ]
  }

It stays like this for additional read and writes, so the following test is met:

frictionless-r/tests/testthat/test-write_package.R

Line 47 in 6b62581

test_that("write_package() writes unaltered datapackage.json as is", {

Issue

However, it changing to simplifyVector = FALSE breaks read_resource() functionality, specifically the reading of missingValues, which are now expressed as:

[[1]]
[1] ""

[[2]]
[1] "NA"

[[3]]
[1] "NaN"

Rather than:

[1] ""    "NA"  "NaN"

Causing this line:

frictionless-r/R/read_from_path.R

Line 67 in 6b62581

na = schema$missingValues %||% "",

To result in:

Caused by error in `enc2utf8()`:
! argument is not a character vector
---
Backtrace:
     ▆
  1. └─frictionless::read_resource(p_json, "deployments")
  2.   └─frictionless:::read_from_path(package, resource_name, col_select) at frictionless-r/R/read_resource.R:66:5
  3.     └─purrr::map_df(...) at frictionless-r/R/read_from_path.R:51:3
  4.       └─purrr::map(.x, .f, ...)
  5.         └─purrr:::map_("list", .x, .f, ..., .progress = .progress)
  6.           ├─purrr:::with_indexed_errors(...)
  7.           │ └─base::withCallingHandlers(...)
  8.           ├─purrr:::call_with_cleanup(...)
  9.           └─frictionless (local) .f(.x[[i]], ...)
 10.             └─readr::read_delim(...) at frictionless-r/R/read_from_path.R:54:7
 11.               └─vroom::vroom(...)

Solution?

So, we probably need to unlist() the missing values (and maybe other properties) at specific places where we want vectors rather than lists.

[peterdesmet]

@sannegovaert if you want to pick this up, I started experimenting in the https://github.com/frictionlessdata/frictionless-r/tree/arrays branch.