Canonical approach to lookup-based replacement?

[MichaelChirico]

I am finding it a bit awkward to refactor code using plyr::mapvalues(), which looks like:

mapvalues(x, from_vector, to_vector)

Items in x found in from_vector are replaced by the corresponding entry in to_vector.

The closest analogue is case_match(), but using it often requires pretty substantial refactoring, e.g. if from_vector/to_vector come from arguments to a function. Elsewhere it is often convenient to keep from_vector/to_vector instead of writing them out manually in different arguments.

Another approach I've tried would basically be to do a join-and-replace lookup with a left_join() to tibble(from_vector, to_vector).

plyr::revalue() is similarly not trivial to migrate.

Am I missing something else?

[DavisVaughan]

Not really. It's a hole in the API that I want to fix in the next minor vctrs release with some kind of new helper that probably lives slightly lower than dplyr itself (because it is typically useful for people writing functions, not people doing interactive analysis).

Out of curiosity, what is the data.table approach here?

(Also, assume that from_vector and to_vector are not character vectors, so assume you can't used a simple named vector to tie the two vectors together)

[DavisVaughan]

See also https://bsky.app/profile/randvegan.bsky.social/post/3lsab7xfb6s2x

[DavisVaughan]

Another approach I've tried would basically be to do a join-and-replace lookup with a left_join() to tibble(from_vector, to_vector).

I think that would totally work, but it feels like it's at the wrong level of abstraction (table level) since all of the required info is at the level of a single column (vector level)

[DavisVaughan]

I think one key feature should be that we don't retain the type of x by default, similar to case_when() and case_match().

x <- c(1, 2, 5)

from <- c(2, 5)
to <- c("a", "b")

dplyr::case_replace(x, from, to)

# should work like case_match() below

# doesn't retain type of `x` (or values of `x`) by default, allowing for arbitrary
# mapping of type A -> type B (here, numeric -> character)
dplyr::case_match(x, 2 ~ "a", 5 ~ "b")

# and like case_match(), you can provide a default as long as it is a type compatible
# with the rhs values
dplyr::case_match(x, 2 ~ "a", 5 ~ "b", .default = as.character(x))

# or you can make it more like `plyr::mapvalues()` like this
# (i.e. replacing values in `x`, using previous values of `x` if not replaced,
# and strictly requiring the ptype of `x` as well, so the types of the input
# and the output can't change)
function(x, from, to) {
  dplyr::case_replace(x, from, to, .default = x, .ptype = x)
}

The ugly implementation but good UI is kind of like this

case_recode <- function(x, from, to, ..., default = NULL, ptype = NULL) {
  rlang::check_dots_empty0(...)
  from <- vctrs::vec_chop(from)
  to <- vctrs::vec_chop(to)
  cases <- purrr::map2(from, to, \(x, y) rlang::new_formula(x, y))
  dplyr::case_match(x, !!!cases, .default = default, .ptype = ptype)
}

x <- c(1, 2, 5)

from <- c(2, 5)
to <- c("a", "b")

case_recode(x, from, to)
#> [1] NA  "a" "b"

case_recode(x, from, to, default = "unknown")
#> [1] "unknown" "a"       "b"

# stricter, no type change allowed. what `plyr::mapvalues()` should be
case_recode(x, from, to, default = x, ptype = x)
#> Error in `dplyr::case_match()`:
#> ! Can't convert `..1 (right)` <character> to <double>.

case_recode(x, from, 99, default = x, ptype = x)
#> [1]  1 99 99

^{Created on 2025-06-23 with reprex v2.1.1}

[MichaelChirico]

Also, assume that from_vector and to_vector are not character vectors

I do think that's the most common case (probably by far), but something more general makes sense. I will say that most often, this is about partially updating a column, where the data type would not be edited.

not people doing interactive analysis

Maybe, maybe not. I would say it comes up interactively a fair amount, e.g. normalizing names:

normalized_names <- data.frame(
  messy = c("Disney", "Disney, Inc.", "TWDCo"),
  clean = "The Walt Disney Company"
)

I guess your point is that interactively, it's easier+cleaner to just write it out, e.g. in case_match(). And the case I cited, where we take the look-up vectors as input, is more programmatic. But interactively, that could apply e.g. when we get the look-up as a data dictionary, e.g. in CSV. So it's a bit of a gray area, but lean towards yes, it's more programmatic.

it feels like it's at the wrong level of abstraction

Totally agree, though I suspect the join & update approach is familiar to those using SQL a lot. A table is a natural data structure to show up because the replacements are paired 1-1, a table disciplines us not to let them get out-of-sync.

Out of curiosity, what is the data.table approach here?

Definitely look-up joins as of today:

DT[lookup_table, on = 'from', from := i.to]

DT = data.table(a=letters[1:8])
idx = seq(1, 26, by=2)
lookup = data.table(a=letters[idx], b=LETTERS[idx])
DT[lookup, on='a', a := i.b][]
#         a
#    <char>
# 1:      A
# 2:      b
# 3:      C
# 4:      d
# 5:      E
# 6:      f
# 7:      G
# 8:      h

[DavisVaughan]

Here's my other API thought. When reading the code 6 months later, no one can ever remember if it's new, old or old, new, so require the arguments to be named

case_recode <- function(x, ..., from, to, default = NULL, ptype = NULL) {
case_recode <- function(x, ..., old, new, default = NULL, ptype = NULL) {

Forcing

lookup <- c(a = 2, b = 3)
case_recode(x, from = names(lookup), to = lookup)

case_recode(x, from = normalized_names$messy, to = normalized_names$clean)

(See also, confusion about recode() and rename() being backwards of each other)

---

It's possible we'd expose a direct replacement to mapvalues() for the ~90% case, but I want to allow the more generic case as well

# i.e., maybe expose a direct wrapper around this
case_recode(x, from, to, default = x, ptype = x)

Maybe case_update(), which nicely implies you are updating a single column and keeping its size and type, and also has nice ties to sql UPDATE

And case_recode() implies a "full recode" of the value, so you don't necessarily keep its type or original values afterwards

---

Implementation using dplyr internals and vec_case_when(). It's possible vec_case_when() should take indices rather than logical vectors, and require a size argument. I think that would avoid the to <- vec_slice(to, loc) and maybe allow for some optimizations (imagine replacing 1 or 2 values in a large vector, you don't want to slice to to full size mostly full of NAs).

case_recode <- function(
  x,
  ...,
  from,
  to,
  default = NULL,
  ptype = NULL
) {
  check_dots_empty0(...)

  obj_check_vector(x)
  obj_check_vector(from)
  obj_check_vector(to)

  from <- vec_cast(from, x)

  # i.e. `case_recode(x, from = c(a, b, c), to = d)`
  to <- vec_recycle(to, size = vec_size(from))

  loc <- vec_match(x, from, na_equal = TRUE)
  to <- vec_slice(to, loc)

  loc <- !vec_detect_missing(loc)

  conditions <- list(
    loc
  )
  values <- list(
    to = to
  )

  size <- vec_size(x)

  vec_case_when(
    conditions = conditions,
    values = values,
    conditions_arg = "",
    values_arg = "",
    default = default,
    default_arg = "default",
    ptype = ptype,
    size = size,
    call = current_env()
  )
}

case_update <- function(
  x,
  ...,
  from,
  to
) {
  ptype <- vec_ptype_finalise(x)
  case_recode(x, from = from, to = to, default = x, ptype = ptype)
}

[DavisVaughan]

To close the loop here - with dev dplyr and recode_values() and replace_values()

library(dplyr)

x <- c(1, 2, 5)

from <- c(2, 5)
to <- c("a", "b")

recode_values(x, from = from, to = to)
#> [1] NA  "a" "b"

# If you think you covered all the cases and want to be sure
recode_values(x, from = from, to = to, unmatched = "error")
#> Error in `recode_values()`:
#> ! Each location must be matched.
#> ✖ Location 1 is unmatched.


x <- c(
  "Disney",
  "TWDCo",
  "Apple",
  "Apple Inc",
  "Samsung",
  "Disney, Inc.",
  "Apple",
  "Walmart"
)

# Some one off normalization
normalized <- tribble(
  ~from, ~to,
  c("Disney", "Disney, Inc.", "TWDCo"), "The Walt Disney Company",
  "Apple Inc", "Apple"
)

replace_values(x, from = normalized$from, to = normalized$to)
#> [1] "The Walt Disney Company" "The Walt Disney Company"
#> [3] "Apple"                   "Apple"                  
#> [5] "Samsung"                 "The Walt Disney Company"
#> [7] "Apple"                   "Walmart"

These are powered by the lower level vctrs::vec_recode_values() and vctrs::vec_replace_values() for the use case of package author who wants something like this but doesn't want to bring in a heavy-ish dependency like dplyr for it.

[MichaelChirico]

QQ: "Location 1 is unmatched." is that referring to the index or the value? (since x[1] is 1)

[DavisVaughan]

It's the index, you have to go back and look at x[1] to see the unmatched value yourself.

I think ideally we'd show the values that are unmatched and tell you where they are but that's arbitrarily hard because x can be as simple as an integer vector or as complicated as a data frame (you can recode using entire data frames if you want for complex cases).