Zen schemas sourcing and dependency loading

[KGOH]

Currently I have to solve a problem of distributing my zen schemas. It would be good if zen offered tools or guidelines.

Use-case:

• I created a set of zen namespaces, I need to deliver it to my working environments
• I implemented a model defined in zen and now want others to supply configs using my model. My model is a dependency for their configs and must be supplied with their configs for them to be valid. Currently I have to share a copy of the my namespace file and share it to my users
• I developed a library with, let's say json-rpc engine. I want others to use my library and write own json-rpcs using my json-rpc tag. Currently they have to copy my namespace with tag and put into their zrc
• Zen delivery is done at the configuration time, but I have to implement zen schemas delivery in my project. Zen could do that and I could use zen to achieve this

Urgency: medium
Currently I already implemented solutions for all of these problems in one of my projects, but in my upcoming project I will have to solve all of this problems again

[KGOH]

Suggestion:

• We can forbid breaking changes and move in an append-only way.
  If there are breaking changes needed we suggest to create a new package.
• We can check in zen if changes are backwards compatible

zen module = git repo:
history’s always available, we always will be able check breaking changes
We can do shallow clone without history
We could do semantic diffs
dependencies == list of git repos (we also could create package-lock file)
dependency check could work as a git hook

zen jar could provide tools for doing this

Example module:

app/

• zrc/
  • module1/
  • module2/
  • module1.edn
  • module2.edn
• doc/
• README.md
• LICENSE.md

Example usage:

$ zen init module1
$ ls
zrc/
  module1.edn:
 {ns module1
                :zen/deps ["[email protected]:zen-lang/fhir.git"
                           "[email protected]:MyOrg/b.git"]}

$ zen pull-deps
$ ls
zen-package-lock.edn (collects deps zen.edn’s)
zen-modules/
  a.edn
  b.edn
zrc/
  module1/
  module1.edn
  ...

$ git add . && git push # hook ensures that there are no breaking changes

$ zen clone [email protected]:MyOrg/app.git
$ ls
zen-package-lock.edn: (calculated shas and all transitive deps)
zen-modules/
  module1.edn
  ...
  a.edn
  b.edn

zen jar could use git under the hood

zen jar could be compiled with graal/work on bb

zen pull-deps:

1. get all deps
2. pull all deps into tmp
3. if this dep already loaded -- skip
4. go through loaded tmp dirs copying from bottom to top
5. copy loaded deps to zen-modules

zen build:

• same as pull but also put your zrc in that dir
• if some files are not imported -- remove

zen clone :
clones to zen-modules/

zen init:
creates zrc/, adds git hook, provides a program for git hook

git add . && git push

zen hook which was added on zen init

[KGOH]

From aa66252e947a618d02e72910f78dd50fde0e01c6 Mon Sep 17 00:00:00 2001
From: Oleg Veschin <[email protected]>
Date: Wed, 31 Aug 2022 13:38:40 +0300
Subject: [PATCH] add zen-pm POC

Co-authored-by: @islambegkatibov <[email protected]>
---
 src/zen/package.clj | 68 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 68 insertions(+)
 create mode 100644 src/zen/package.clj

diff --git a/src/zen/package.clj b/src/zen/package.clj
new file mode 100644
index 0000000..3017aea
--- /dev/null
+++ b/src/zen/package.clj
@@ -0,0 +1,68 @@
+(ns zen.package
+  (:require
+   clojure.edn
+   [clojure.java.shell :as sh]
+   ;; [clojure.java.io :as io]
+   [clojure.string :as string]
+   [clojure.test :as t]))
+
+(def root "/tmp/zen")
+
+(defn git-init [path] (sh/sh "git" "init" :dir path))
+
+(defn zen-clone [package-file]
+  (->> package-file
+       slurp
+       clojure.edn/read-string
+       (map (comp #(->> (string/replace (first %) #"\." "/")
+                        (str root "/zen_modules/")
+                        (sh/sh "git" "clone" (second %)))
+                  #(update % 0 str)))))
+
+(comment
+
+  (doseq [repo ["/a" "/b" "/c"]
+          :let [dir (str "/tmp" repo)]]
+    (sh/sh "mkdir" "-p" dir)
+    (git-init dir)
+    (spit (str dir "/main.edn") (str {'ns 'main (symbol (str (random-uuid))) {}}))
+    (sh/sh "git" "add" "." :dir dir)
+    (sh/sh "git" "commit" "-m" "\"Initial commit\"" :dir dir))
+
+  (do
+    (def zrc (str root "/zrc"))
+    (sh/sh "rm" "-rf" root)
+    (.mkdir (java.io.File. root))
+    (.mkdir (java.io.File. zrc))
+    (git-init root)
+    (spit (str root "/.gitignore") "/zen_modules")
+
+    (def precommit-hook-file (str root "/.git/hooks/pre-commit"))
+    (spit precommit-hook-file "#!/bin/bash \n\necho \"hello world!\"")
+    (sh/sh "chmod" "+x" precommit-hook-file)
+
+    (spit (str zrc "/module1.edn") (str '{ns module1}))
+
+    (def package-file (str zrc "/../package.edn"))
+    (spit package-file (str '[[a "/tmp/a"]
+                              [b.dir "/tmp/b"]
+                              [c "/tmp/c"]]))
+    (zen-clone package-file))
+  )
+
+(defn get-hash [path]
+  (->> "/.git/refs/heads/master"
+       (str path)
+       slurp
+       string/trim-newline))
+
+(t/deftest zen-pm
+
+    (t/is (= (get-hash "/tmp/a")
+             (get-hash "/tmp/zen/zen_modules/a")))
+
+    (t/is (= (get-hash "/tmp/b")
+             (get-hash "/tmp/zen/zen_modules/b/dir")))
+
+    (t/is (= (get-hash "/tmp/c")
+             (get-hash "/tmp/zen/zen_modules/c"))))
-- 
2.37.2