New Miller/Go verbs

[aborruso]

Hi @johnkerl in the great Miller go version there are flatten, unflatten, gap, json-parse and json-stringify new verbs.

Is there some file I can read to understand what these do?

Thank you

[johnkerl]

@aborruso there will be :)

There isn't yet :(

For now I will say that one key thing about the Go port is this:

• In the C implementation all the keys and values in the key-value pairs of each record are strings. They are parsed as int/float/bool/string/map only in the put/filter DSLs and only as-modified. Then back to strings at the end of put and filter. (Also the sort, stats1, and stats2 verbs do type-inference ... step as well.)
• In the Go implementation the keys are all strings still but the values are typed all the way through the verb-chain -- from the moment they're read off disk until the moment they're written back to disk. All the way in between -- for each verb, put/filter or any other -- the values are int/float/bool/string/map/array (arrays are new in Miller 6).
• This has many effects & benefits, as you can see from multiple issues tagged with 'Go port'
• One of the effects/benefits is that JSON data is now richly typed all the way through the verb-chain. This is splendid for JSON-to-JSON processing -- values can be maps or arrays or scalars and that's fine. It's also fine for CSV-to-CSV processing where all the values obtained from input data are scalars.
• This raises the question though of what about CSV-to-JSON or JSON-to-CSV. The flatten is for JSON-to-CSV -- how to take "x":{"a":1,"b":2} (fine in JSON) and flatten it to CSV via x.a=1,x.b=2. (One field becomes two.) The unflatten is for the reverse.
• Another option, instead of flatten/unflatten, is to json_parse / json_stringify fields. E.g. the one field "x":{"a":1,"b":2} becomes the one field "x":"{\"a\":1,\"b\":2}".

That leaves only gap unexplained, and it's simply a way to put blank lines every so often -- when some field changes value, or every 10th record, etc. I find this super-handy for all sorts of stuff in my day job.

[johnkerl]

There's also currently auto-flatten & auto-unflatten (you could just use mlrgo --ojson cat above) which may be the 'right and intuitive thing' for people, or not ...

[aborruso]

THank you @johnkerl .

In the help I have

Usage: mlrgo unflatten [options]
Reverses flatten. Example: field with name 'a:b:c' and value 4
becomes name 'a' and value '{"b": { "c": 4 }}'.

And for this reason I have used a:b:c.

Thank you

[johnkerl]

oh nuts.

thanks, i'll fix that! :)

[johnkerl]

1c322c3

thanks @aborruso !! :D

[johnkerl]

Also 9592a41