Evaluate dumping .lib.json without indentantion

[umarcor]

This is a draft for showing some CI results. NOT TO BE MERGED.

https://twitter.com/wavedrom/status/1383289045812273157
@mithro you would probably shave few Gigs from 42GB #skywaterPDK by storing single-line JSON files instead of nicely indented files

In this PR, a workflow and a Python script are added. The script searches all *.lib.json files in libraries/*/latest and _dump.write_text(dumps(loads(item.read_bytes()))). Results show that the size can be reduced to ~39%:

sky130_fd_sc_hd: 0.427467473782599 GB -> 0.1797396233305335 GB [42.047555511076226%]
sky130_fd_sc_hvl: 0.14845740050077438 GB -> 0.06090841814875603 GB [41.0275391750769%]
sky130_fd_sc_lp: 2.0116734048351645 GB -> 0.7661424418911338 GB [38.084832262019745%]
sky130_fd_io: 0.0 GB -> 0.0 GB 
sky130_fd_sc_hs: 2.260457427240908 GB -> 0.8601267794147134 GB [38.05100547567381%]
sky130_fd_sc_ms: 1.9792483588680625 GB -> 0.7715314263477921 GB [38.98103150576985%]
sky130_fd_sc_ls: 2.31650467030704 GB -> 0.936532448977232 GB [40.42868814303317%]
sky130_fd_sc_hdll: 0.5418667104095221 GB -> 0.21883693151175976 GB [40.38574935640006%]
sky130_fd_pr: 0.0 GB -> 0.0 GB 
Total: 9.68567544594407 GB -> 3.7938180696219206 GB [39.16937017759151%]

/cc @mithro @drom

[drom]

@umarcor thanks for testing the lower bound.
What if we stringify all JSON:

1. with indent 0 ?
2. with normal indent, but arrays of numbers as a single line?

[mithro]

If it is not going to be human readable, might as well just use compressed JSON files which is much higher than 60% (I think like 95%!).

FYI - A good option is to do a blobless clone - https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/

[umarcor]

I think the content should not be "easily" human redable, but it should neither be compressed JSON. If it was compressed JSON, I would suggest using some more efficient binary format instead. The point of having it compact but plain JSON is that any user can quickly reformat it. Just opening it in any online JSON linter will provide a nice indented view (e.g. https://codebeautify.org/jsonviewer). Something such as https://drom.io/inspect might be added to the skywater site for supporting either browsing a list or drag&drop.

On the other hand, git does not like single long lines at all. Therefore, we might want to use multiple lines, indeed, for avoiding each change forcing all the file to be recorded. Otherwise, the repo will grow larger than it would without "optimisation". @drom, were the rules you proposed meant to handle this?

A good option is to do a blobless clone - https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/

I think we are talking about the size of the JSON files in a currently ready-to-use (cloned) dir. That is, we are not considering the size of .git. These optimisations would apply even (specially) if the libraries are downloaded as an archive.

[drom]

I am for preserving "human readable" JSON format.
Printing arrays of numbers in a single line, for example, not only makes JSON files 60% smaller, but make files more readable:

7280397459 -> 2931218556 = 40.26179302033076 %

https://gist.github.com/drom/86facb54fe0875b0e63bb7598df06310

example:

...
 "pin,Y": {
  "direction": "output",
  "function": "(!A1&!B1&!C1&!D1) | (!A2&!B1&!C1&!D1)",
  "internal_power": [
   {
    "fall_power,pwr_template13x7": {
     "index_1": [0.01, 0.01735, 0.02602, 0.03903, 0.05855, 0.08782, 0.13172, 0.19757, 0.29634, 0.44449, 0.6667, 1, 1.5],
     "index_2": [0, 0.00636, 0.01271, 0.02542, 0.05085, 0.10171, 0.20341],
     "values": [
      [0.04431, 0.03219, 0.02014, -0.00392, -0.05218, -0.14876, -0.34199],
      [0.04497, 0.03287, 0.0208, -0.00333, -0.05158, -0.14815, -0.34152],
      [0.04544, 0.03328, 0.02133, -0.00287, -0.05105, -0.14762, -0.34084],
      [0.04606, 0.03386, 0.02186, -0.00226, -0.05057, -0.14719, -0.3404],
      [0.04631, 0.03422, 0.02224, -0.00183, -0.05012, -0.1467, -0.33993],
      [0.0461, 0.03421, 0.02226, -0.00177, -0.0499, -0.14643, -0.33957],
...

[drom]

Something such as https://drom.io/inspect might be added to the skywater site for supporting either browsing a list or drag&drop.

yes, @umarcor I was thinking about Web UI for browsing / searching libraries, and rendering selected cells online.
That would require extra library index information, but can be done.