Change destructuring syntax to align with "match"

[allie-jo]

Python 3.10 introduced a canonical destructuring syntax with match. I think we should change the existing clojure like syntax to align with the syntax introduced by Python match which is already used by our existing match implementation.

Here's what I had in mind.

(let+ [{"a key" a-sym  {"a" [{:keys [b c d]} :as nested  e f #* rest]}} some-big-dict]
  [a-sym b c d nested e f rest])

This has key and binding symbol swapped compared to clojure, but Hy isn't Clojure 🤷‍♀️ and I like the consistent syntax with our existing match implementation. the only addition here is the :keys option. Which, looks up by string keys and maintains the idiom of keywords being arguments not identifiers themselves (even though there's nothign stopping you from destructuring keywords here or in match)

[gilch]

What happens if you have dicts/maps

• {:keys [1 2 3]} and
• {:a 1 :b 2 :c 3}?

In Clojure, you can destructure the former as

• {[a b c] :keys}
  and the latter as
• {a :a, b :b, c :c}, or
• {:keys [a b c]} for short.

With your proposal, you can destructure the former as

• {:keys [a b c]}
  and the latter as
• what? {:keys [a b c]}?
  It magically reads your mind to know which case you meant by the same binding form! 🪄 😉

Or, since I don't know how to program telepathy 🪄 🧠,

• we just can't destructure any dicts that happen to have a :keys key. Too bad ☹️ Or,
• we get rid of the :keys abbreviation altogether. Or,
• we destructure the second case as before, but the first case as {[a b c] :keys}? But Python dicts can't have lists as keys. Unlike Clojure's vectors, they're unhashable types. This shouldn't work either, but (unlike Clojure) Hy's dict models are not actually dicts, but the code for generating them, so maybe a clever macro can avoid this issue if it never actually builds dicts with the dict models?

Did I understand your proposal correctly?

[allie-jo]

as far as my idea for this proposal is concerned, you'd quote it. Same as we do with match if you want to return :as which is a reserved keyword in that macro.

https://github.com/hylang/hy/blob/123b82e366ba6909029633eb91be22db55642dac/tests/native_tests/py3_10_only_tests.hy#L39

  (assert (= :as (match 1 1 :if True ':as)))

so here it would be

(let+ [{':keys [a b c]} {:keys [1 2 3]}]
  [a b c])

I have not tried to implement this yet, but that's the idea

[gabriel-francischini]

About implementing it, it shouldn't be too hard (apparently reordering the arguments of dicts is done in hyrule.destructure.dest-dict, possibly in hyrule.destructure.dest-dict.expand-lookup, and the :key and :as should be close to those functions).

This is an amazing idea because

(let+ [{"key" foo}
       {"key" "quux"}]
  (print foo)) ; => "quux"

better reflects the structure we are trying to match than:

(let+ [{foo "key"}
       {"key" "quux"}]
  (print foo)) ; => "quux"

---

Allowing the (discouraged) use of keywords as keys may be useful in some edge cases. But it probably should be discouraged, so :keys should only (or primarily) match string keys. This is one of those cases, although I don't think it will ever happen in practice:

(let+ [{"with" foo}
       {"this" "is" "a" "big" "dictionary" "with" "many" "many" "similar" "things"}]
  (print foo)) ; => None

(let+ [{:with foo}
       {:this "is" :a "different" :big "dictionary" :with "many" :similar "things"}]
  (print foo)) ; => many

[allison-casey]

It's encouraged in Hy to write dict literals with two spaces between key value pairs (clojure uses a comma + space for large maps, but commas are not whitespace in Hy) which makes cases like that easier to deal with. hyrule.hypprint displays maps like this as well. so it would look something like this

(let+ [{"with" foo} {"this" "is"  "a" "big"  "dictionary" "with"  "many" "many"  "similar" "things"}]
  (print foo)) ; => None

Or use the dict kwarg method of constructing the dictionary

(let+ [{"with" foo} (dict :this "is" :a "big" :dictionary "with" :many "many" :similar "things")]
  (print foo)) ; => None

[gabriel-francischini]

Double spaces and (dict ...) work like a charm. They indeed make it a lot easier to deal with.

While we are making the destructuring macros more Hy-like, I would suggest replacing :& with #*:

(let+ [[a b #* rest :as full]
       [0 1 2 3 4]]
  [a b rest full]) ; => [0 1 [2 3 4] [0 1 2 3 4]]

vs

(let+ [[a b :& rest :as full]
       [0 1 2 3 4]]
  [a b rest full]) ; => [0 1 [2 3 4] [0 1 2 3 4]]

Maybe there's potential for #** somewhere here.

I've been looking around defn+ and fn+ because I suspect there are some bugs in them, and apparently the presence of #* and #** and other things inside the destructuring patterns would never clash with the def syntax (because the def operates on the outer list of arguments while the destructuring patterns only operates on the nested ones). After reading a destructuring PR I'm not so sure anymore, but it still may be an interesting idea.

[allison-casey]

I would suggest replacing :& with #*:

Very much agree with this. I'm sure it's possible, but I have some other PR's in the main repo to deal with right now. If you want to take a stab at this, make sure to look over the match destructuring syntax to make sure we stay in alignment. Feel free to ask questions, I'm happy to help as much as I can.

[gabriel-francischini]

as far as my idea for this proposal is concerned, you'd quote it. Same as we do with match if you want to return :as which is a reserved keyword in that macro.

https://github.com/hylang/hy/blob/123b82e366ba6909029633eb91be22db55642dac/tests/native_tests/py3_10_only_tests.hy#L39

  (assert (= :as (match 1 1 :if True ':as)))

so here it would be

(let+ [{':keys [a b c]} {:keys [1 2 3]}]
  [a b c])

I have not tried to implement this yet, but that's the idea

I've been reading the PEP that @allison-casey linked, What's New in Python 3.10, and the match form and apparently the :as already have the same behavior in both the destructuring macros and in the match form, so that's one thing less to implement.

Apparently :as don't need quotes in either of them, so I don't see why we should quote them if they both already work in their unquoted form.

If I understood python match's or-patterns correctly, once the destructuring macros have the | pattern that match currently has, we may be just a few steps shy of being able to do this type of destructuring in Hy (this version is Meander's Clojure code):

(def skynet-widgets
  [{:basic-info {:producer-code "Cyberdyne"}
    :widgets [{:widget-code "Model-101"
               :widget-type-code "t800"}
              {:widget-code "Model-102"
               :widget-type-code "t800"}
              {:widget-code "Model-201"
               :widget-type-code "t1000"}]
    :widget-types [{:widget-type-code "t800"
                    :description "Resistance Infiltrator"}
                   {:widget-type-code "t1000"
                    :description "Mimetic polyalloy"}]}
   {:basic-info {:producer-code "ACME"}
    :widgets [{:widget-code "Dynamite"
               :widget-type-code "c40"}]
    :widget-types [{:widget-type-code "c40"
                    :description "Boom!"}]}])

(require '[meander.match.alpha :as m])
(m/search skynet-widgets
          (scan {:basic-info {:producer-code ?producer-code}
                 :widgets (scan {:widget-code ?widget-code
                                 :widget-type-code ?widget-type-code})
                 :widget-types (scan {:widget-type-code ?widget-type-code
                                      :description ?description})})
          [?producer-code ?widget-code ?description])

;;; Which results in:
(["Cyberdyne" "Model-101" "Resistance Infiltrator"]
 ["Cyberdyne" "Model-102" "Resistance Infiltrator"]
 ["Cyberdyne" "Model-201" "Mimetic polyalloy"]
 ["ACME" "Dynamite" "Boom!"])      

There are some examples using meander/or in here and a few more in this talk.

While I don't like Meander's syntax, it serves to show that a few primitives for destructuring data can go a long way when dealing with complex and cumbersome data. So I think that adding match's | to the destructuring macros is a step in the right direction, because it will let us more easily match against data that we couldn't before.

[allison-casey]

:as needs to be quoted in match if you want to use it as a clause's return value since :as has special behavior. So that something like this (admittedly convoluted example)

(match some-value
  [1 _ a] :as arr  [a arr]
  1                ':as
  anything-else    :not-found)

Doesn't bind the 1 as the fall through var anything-else. Hopefully that makes sense?

I've never seen that kind or in destructuring before, it seems extremely powerful. I do, however, think we might want to limit the scope of a potential first PR to only do the same kind of destructuring that the existing destructuring module already provides so we can ensure feature and test parity. And you know, see if this would actually work 😆

[Kodiologist]

I see the motivation here, but it's not my itch to scratch, and bear in mind that making such changes would require a major-version bump if they're done after Hyrule 1.0.0 is released.