v0.310

This release is fabricated -- reproducibly! -- by the scripts in the omnibus dir of this repo. All source hashes can be seen pinned there. All build instructions are captured in formulas. There are two implementations of the build pipeline; the first uses bash, the second uses our own reach tool. Both should produce identical results, and those results should also be identical to the attachments to this release on github.

This release adds 💚 🔴 💛 💜 🔵 COLORS 🔵 💜 💛 🔴 💚 to the output of Reach, which is... honestly, nice. It helps. More work on the UI of these tools is always welcome, but the good news is now there's some large dividing lines to make output with lots of steps easier to read.

---

Major changes since the last release:

• reach:
  • Colors colors colors!
  • The 'anonymous' modules system is replaced by 'unnamed' modules, which is... similar, honestly, but we fixed a bunch of bugs and weird special-case UX around them. polydawn/reach#8
  • ... a couple dozen other cool things; I'll edit this later

v0.300

This release is fabricated -- reproducibly! -- by the scripts in the omnibus dir of this repo. All source hashes can be seen pinned there. All build instructions are captured in formulas. There are two implementations of the build pipeline; the first uses bash, the second uses our own reach tool. Both should produce identical results, and those results should also be identical to the attachments to this release on github.

This release is minimal in semantic changes, but contains a couple of significant renamings, so be sure to read the changelog bullet points; scripts may need adjustments. (All changes are at Layer 2; everything Layer 1 and below remains unchanged and should need no updates.)

---

Major changes since the last release:

• reach (nee stellar):
  • Renamed! 🎉 reach seems like a suitable name for the part of our tooling that's all about reaching and building towards new targets. Bonus: it keeps on our theme of 'r'-prefixed project names. (The old working title, 'stellar', was not very thematically consistent, and -- though heaven knows all names will collide with some other project eventually -- that one was particularly unfortunate as it collided with a contemporary and prominent cryptocurrency project name.)
  • catalog.tl files are now known as lineage.tl files. We can now refer to "catalogs" consistently as the whole tree; a "catalog" is an aggregation of "lineages". This change of file name requires rearranging any catalog file trees.
    • If you need to bulk-rename some files, this shell snippet should do the trick: find -name catalog.tl | xargs -I {} bash -c 'mv {} $(echo {} | sed [email protected]@lineage.tl@)'
• go-timeless-api
  • Rename: api.ModuleCatalog -> api.Lineage. As discussed above, this new name is a much better description of what's going on, it's terser, and it should resolve a source of much confusion. We can now refer clearly to "catalogs" as an aggregation of "lineages". 👍

v0.220

This release is fabricated -- reproducibly! -- by the scripts in the omnibus dir of this repo. All source hashes can be seen pinned there. All build instructions are captured in formulas. There are two implementations of the build pipeline; the first uses bash, the second uses our own stellar tool. Both should produce identical results, and those results should also be identical to the attachments to this release on github.

This release has several massive changes; both in small quality-of-life improvements, in new features, and in a few schema changes. Lots of first-time setup operations when using stellar are now done automatically; we have recursive eval across modules (...!!!); the concept of "workspaces" in stellar has been dramatically simplified; and yes, there's one change to the module schema which will require some migrations (but everything Layer 1 and below is still unchanged). So there's a lot to read in the detailed changelog below!

Oh, and the release binaries are static. Because if you're working on bootstrap stuff, you probably don't appreciate having to bring your own libc. Yep, we hear you. Fixed. Sorry we didn't get on it earlier.

---

Major changes since the last release:

• stellar:
  • workspace rework & simplification! This is mostly documented at polydawn/reach#4 . In short: a dir named .timeless is now always (and exclusively) the denotation of a workspace root. And a workspace is always necessary.
    • if you were previously confused about the difference between anonymous module mode and workspace mode, yep, you weren't alone. That was silly, and now the difference simply doesn't exist. Much easier!
  • lots of uninteresting initialization work (making directories, etc) is now automatic. Details at polydawn/reach#6 .
  • module-recursive eval mode!!!! 🎆 🎆 🎆
    • (this is has sometimes been referred to as "commissioning".)
    • stellar emerge -r $moduleName will now load the named module, and if it has any imports which refer to releases of other modules with a version name of "candidate"... it will automatically recurse to those modules, build their latest version, and use that.
    • yes, this works recursively.
    • each such evaluation will produce a "candidate" release. You can see the catalog files for these at $WKSPC/.timeless/candidates/default/*.
      • these "candidate" files are stored in the same format as other catalog releases. They're totally normal. So, they're imported in a fundamentally normal way to the module that references them.
      • the one exception to "candidate" release normalcy is that they're not suitable for publishing publicly. The intended path is to rewrite them into releases with a more permanent name, and then do so and publish that... at your leisure. (We'll introduce tools for automating this in the near future.)
    • this entire "candidate" system also works with regular stellar emerge; every module evaluation will leave behind a candidate release that other modules can import. -r-mode just makes all evals automatic so you don't have to sequence commands manually.
    • so! Now you're really free to start building large systems. Use a single module (with steps within it whenever you need separate containers with unique filesystems) when working on a single project; split things up into several modules whenever you deem it appropriate to do so, and when you want to be able to label and make releases of the products of each module. You're in control, and both forms of structure work easily and recursively.
  • there is a breaking change to the Module format. The Operation.Inputs map flipped key-value ordering: it is now Path:SlotRef ordering.
    • breaking changes are always unfortunate, but on a happy note, the migration is easy and mechanical; and it's only modules that have changed (no Layer 1 components or below are altered).
    • see polydawn/go-timeless-api#1 / polydawn/go-timeless-api@cf6aecf for more details on the reasons for this.
  • internally, some improvements to testing. Since we're doing a lot of stuff that infers things from surrounding context on the filesystem, it's important to make sure where the source repo is and its surroundings can't interfere too much! So, now we have a little more of the needed infrastructure for that: test fixture files are now copied into a tmpdir before test launch.
• repeatr:
  • nothing changed, actually :3
  • we've successfully reached the point where all the interesting new challenges are appearing in how we use it (e.g., stellar)!
• rio:
  • some error messages are improved, and typos fixed.
  • bug fixed in the git transmat: previously, if you asked for a hash you hadn't cached before from a repo which you'd already cloned, and the update fetch found that all remote refs were identical to their previous values, a nonsensical error would be raised. Now, that no-op is just a no-op. As it should be.
  • that's also it! As much as we might like to be adding new packtypes and warehousing transport protocols, all's quiet; and that's okay: things are settling into simply being... stable.

---

Oh, P.S.: You might want to check out the https://github.com/warpfork/smsh project. It's a small shell -- in golang, so its easy to compile, cross-compile, and ship statically -- and it's specifically designed to do the right thing when you pass an array of command strings to it (which can be cleanly appended, unlike bash, where unsafe concatenation is your only option for composition) -- and it keeps bash syntax so there's no new language to learn. It might be useful in your containers!

---

P.P.S.: There's a new project beginning to produce rigorously portable and well-packaged (and composably, path-agnostically packaged) software... and of course, building it with the Timeless Stack, because why wouldn't you... starting over at https://github.com/polydawn/radix . This is early days, and going to take a while to show really exciting results, but watch this space if portable packaging is your jam.

---

Live long and function purely!

v0.210

This release is fabricated -- reproducibly! -- by the scripts in the omnibus dir of this repo. All source hashes can be seen pinned there. All build instructions are captured in formulas. There are two implementations of the build pipeline; the first uses bash, the second uses our own stellar tool. Both should produce identical results, and those results should also be identical to the attachments to this release on github.

The biggest fanfare in this release is probably the new ingest-pack system in Stellar: it's now a LOT easier to scrape some content from your local filesystem into your automated builds processes. Check that out. In practical application, this also gives us a great solution to "my action script is too long" problems, with neither yaml nor quoting&escaping hell: you can just keep more files next to your module, and "ingest:pack" them! 🎉

---

Major changes since the last release:

• stellar:
  • stellar emerge now also accepts a path to the module to build. This can be used instead of frequently changing of your shell working directory.
  • stellar now has a new ingest system: "pack"! The usage of this is roughly as per "ingest:pack:tar:./packme", and you can use this to easy refer to scripts or patches on your local filesystem and get them pipelined into the Timeless Stack without a lot of fuss. (Like other ingest systems, the intent is to make this future-proofed for replay instructions in releases by auto-rewriting the module imports into frozen references that replace the ingest config.)
  • stellar catalog lint has some bugfixes where it could previously be tricked into skipping way too many files.
• rio:
  • The format of filters on pack and unpack has changed. All of these options are now expressed as one long string parameter in the CLI, and resembles the options used in unixy mount configurations.
    • E.x.: "uid=1000,gid=1000,mtime=@1262304000,sticky=keep,setid=keep,dev=keep".
    • As before, you can simply skip specifying any of these for which you accept the default values.
  • The implementation of filters has also changed: we have several new features. Filters now contain separate support for all these attributes:
    • uid, gid, mtime, sticky, setid, dev. (Previously, "sticky" and "setid" were conflated, which was poor because they have very different implications; and "dev" wasn't an option at all, and now is.)
    • the sticky, setid, and dev filters can be set to "ignore", "keep", or (new) "reject". This lets you choose the behavior when an interesting mode is encountered (behave lossily but complete the operation; or halt immediately).
  • There's a new exit code: "rio-filter-rejection". This is reported when filters for e.g. device nodes is set to "reject" and you attempt to pack or unpack a filesystem containing device nodes.
  • Internally: Rio was refactored to use a major jump in the shared go-timeless-api packages, which brings into line with the API packages used by Stellar. These had diverged and getting everything back in one place is a big relief.
  • Reading tar files with type-'g' ("global header") entries is now supported (and ignored). In practice, this means tarballs produced by git-archive (which use this "global header" for an immaterial comment field containing the git hash) can be handled without issue. You will receive a warning message informing you when this happens; you can generally ignore it.
• repeatr:
  • repeatr has experimental but working support for using Gvisor! Just run with --executor=gvisor. You'll need to provide your own binary in the plugins directory and name it "repeatr-plugin-runsc" (nightlies seem to work fine; we're refraining from bundling anything until the upstream project starts cutting non-nightly releases).
  • Bugfix: specifying non-default working directories in the formula works correctly again.
  • The format for output filters in formulas has changed -- slightly. The new options as documented for Rio all exist in repeatr output specs now; existing formulas will continue to work, since all of the existing options still exist with the same names.
  • Internally: Repeatr was refactored to use a major jump in the shared go-timeless-api packages, which brings into line with the API packages used by Stellar. These had diverged and getting everything back in one place is a big relief.
  • repeatr batch has been removed. As was noted in the previously release, this feature was both always marked extremely experimental, and explicitly deprecated in the last release. The "modules" system in stellar now completely supercedes it -- go use that!
  • Some exit codes have been changed. The numeric codes are now in the 40's block, except for job-exit-unsuccessful, which remains at 32. If using the error categories in the json API, nothing has changed.
• go-timeless-api:
  • The API has had a major overhaul. The branch we previously named "redux" and incubated the new features for stellar over the last few months has been finished and has replaced the previous master branch. All the code for modules, ingests, operations, etc -- all is now landed on master.
  • There are exec-based clients for both Repeatr and Rio in the API repo. (Previously, some of these clients resided in the individual projects, but we've decided it's operationally much easier to have them in the shared repo.)

---

Go forth and build!

v0.202

This release is fabricated -- reproducibly! -- by the scripts in the omnibus dir of this repo. All source hashes can be seen pinned there. All build instructions are captured in formulas.

All the changes in this release are are in stellar, the higher-level multi-step build and pipelining tool: it now supports a concept called "workspaces", and also has sprouted a "CI" mode.

Major changes from the last release:

workspaces

Workspaces are denoted by a "workspace.tl" file, and make it possible to share some important configuration between large groups of projects easily. In particular, the releases catalog.

You can now clone a catalog into .timeless/catalogs/upstream (multiple catalogs and overlays will be supported soon(tm), hence the "upstream" hunk of the path) next to your workspace.tl file, and any modules within that will be able to refer to it automatically.

So, a filesystem might for example look like this:

/youism/workplace/workspace.tl
/youism/workplace/.timeless/catalogs/upstream/ # ... release info files go here
/youism/workplace/.timeless/warehouse/ # ... intermediates will be stored here (maybe link it to /tmp)
/youism/workplace/projectFoo/.git # you're not a monster, right?
/youism/workplace/projectFoo/module.tl # this project builds/CIs/releases as a timeless stack module
/youism/workplace/projectBar/module.tl # and so does this *other* project, neato

This makes it easy to maintain (commit, push, pull, etc) the release catalog tree in one place, and refer to it from many projects without a fuss.

Soon(tm), the roadmap is also to replicate the relevant subsets of the release catalog info into a dir next to each module.tl file. Vendoring this info (it's just the releasename->hash mappings plus recommended mirrors) will result in reproducibly buildable projects if you put nothing but the module.tl file and its sibling dir into a git repo. At the same time, the workspace catalog can be your focal point for updates; and we'll build tools to sanity check that the vendored subset is cleanly reconcilable with the more complete catalog stored in your workspace. This should result in the best of both works: easy to manage, and yet also context-free works-anywhere build instructions in each project.

Catalog maintenance

Stellar has a new group of commands -- stellar catalog -- which contains tools for operating on release catalogs.

The first addition to this new command group is stellar catalog lint, which reads in everything in a catalog directory, verifies its sanity, and exits. It also has a --rewrite mode in which it will rewrite every file to have a canonical ordering of all maps and fields, in a standardized pretty-printing, etc.

These tools are intended to be useful in the context of a CI system which checks all alterations to a catalog to make sure everything is in order.

CI mode

"CI mode" is available via a new subcommand: stellar ci.

This "CI mode" is almost exactly like regular stellar emerge, but with a couple of restrictions, and in exchange one killer new feature:

• your module.tl must specify exactly one "ingest" import;
• and it must be a "git" ingest, a la "ingest:git:.:HEAD".
• Given such a module, stellar ci will build it once, then quietly remain running and automatically build it again fresh every time the git ref changes.

Long story short: you can run stellar ci in one of your terminals, and then keep working normally on that project; if stellar ci ever exits, the module failed to execute on the most recent commit. Put your tests in one of your operations inside the module.tl file and you've got a CI system.

(This "CI mode" feature is a PoC and will almost certainly change. The UX leaves many, many things to be desired. But it does demo with flying colors...!)

v0.201

This release is fabricated -- reproducibly! -- by the scripts in the omnibus dir of this repo. All source hashes can be seen pinned there. All build instructions are captured in formulas.

This release is kind of wild because those omnibus build instructions are now available in TWO forms -- firstly, in the build.sh script; and secondly, in a module.tl file which can be evaluated by the new stellar tool. The shell script is familiar from previous releases, and continues to be a self-documenting proof-of-concept of how you can glue formulas together even with basic tools. The new "module" file is... well, the major feature of this release! So, more about that in a second.

Major changes from the last release:

• the stellar command is introduced!
  • stellar is our new tool for working on multi-step build pipelines. Like earlier drafts, the core concept remains that we shall wire a series of containers together by some human-readable names, while translating this into a series of entirely-hashed formulas for execution. New in this tool, we have "modules", which are composed of "steps"... and each step may be either a plan for generating a single formula, or, an entire further sub module.
  • recursive sub-modules have locally scoped names for wiring their steps together; and can import names from and offer to export names to their parent module. This local scoping provides a mechanism for writing reusable sub-modules without having to reconcile naming to align over the entire module at once. Hopefully this will make it easy to grow reusable build pipeline components.
  • stellar handles resolving import names into hashes via release catalog files.
  • stellar also handles loading warehouse addresses from a new format introduced in this release and stored adjacent to release catalog files.
  • these two features together mean you can write stellar modules entirely without referring to hashes or manually copy-pasting any opaque strings... and as long as you track a hash covering your release catalogs as long as your module file together, you still have well-defined reproducible operations from stellar itself when it plans out all your containers in the pipeline.
  • stellar handles the topological sort and dependency-ordered execution of your modules.
  • stellar includes a new "ingest" system, which can be used for importing as an alternative to "catalog" lookups. Ingests look at local conditions and resolve a hash; e.g. "ingest:git:.:HEAD" looks up the checked out commit for a git repo in the current directory and yields that as a WareID. (Ingests obviously won't be suitable to see when we use module data as replay instructions, but we expect to address this in the future by supporting automatic rewriting of ingest imports into more durable references.) If you're thinking about using this to build a localhost CI tool... yes, please do.
  • all in all... stellar basically provides everything necessary to do multi-step and complex pipelines of containers and builds. No more turing-complete/bash glue code in sight on the host, at all. Sweet, sweet freedom.
• repeatr now supports choice of API-friendly output modes (again; this was a feature lost in the v0.15.x->v0.200 rewrite; and now happily we have it back again), via the --format=json flag.
• the heft and hitch commands are dropped from this release. heft was an experiment from our last release, and at this point has served its purpose for research and to be useful again would need a reboot to mesh into the stellar system; hitch is also making more sense as a library than an independent tool at this point -- we still have interfaces in a package called hitch, and stellar has both providers and consumers of these interfaces, and that's sufficient for now.
• while repeatr batch support remains, it also should now also be considered deprecated; the concept formerly known as "basting" is already fully replaced by the stellar module system.
• internally, the go-timeless-api sources have a major new clean rewrite branch, which is used in the stellar tool, but has not completely replaced the extant master branch and been linked in all other tools. This rewrite branch was partially to give us room to develop all of the structures for expressing stellar modules, submodules, steps and refs... and partially so we could take advantage of the new unions support in refmt.

Lots of documentation will need updating in the aftermath of this release. While that's being churned through... for now, enjoy the fine demos.

v0.200.4

This release is fabricated -- reproducibly! -- by the scripts in the omnibus dir of this repo. All source hashes can be seen pinned there. All build instructions are captured in formulas.

Major changes from the last release:

• Overall: Minimal. We've been working on higher-level logistics and systems for commissioning large suites of build plans, and that's in some experimental repos and doesn't affect the current APIs.
• Docs available on http://repeatr.io/ are much improved!
• repeatr now prints much, much prettier logs.
• repeatr now has a batch subcommand. This takes a new kind of json document (working title: "basting") which composes several formulas together, and allows them to refer to outputs of other formulas in the same group for their own inputs. It still generates fully hash-pinned formulas for each step; memoization still composes with batch mode.
  • This feature is experimental and subject to change.
• You can follow the development of further large-scale build planning systems in the heft repo. This is still incubation, however -- expect a few more releases before it matures.

The new colorized logs clearly highlight system progress messages (data fetching, container setup, etc) vs the job's own output (the container's stdout and stderr) vs the final runRecord printout:

Long story short: this should be a painless upgrade from the previous release, and things got prettier. Upgrade. And stay tuned for more exciting news about commissioning larger suites of formulas in the near future.

v0.200.1

Another month, another release.

This release is fabricated -- reproducibly! -- by the scripts in the omnibus dir of this repo. All source hashes can be seen pinned there. All build instructions are captured in formulas.

Major changes:

• Error handling continues to improve iteratively...
• We fixed some nasty bugs with handling some kinds of paths in the tar transmat. (No security notice; previously, some inputs would simply fail. Now they work.)
• git: It's back in. You can now use rio unpack git:<commitHash> to get a checkout of a working tree (and yes, submodules are also supported).
  • this is now implemented via the delightful go-git library -- meaning it's in pure golang, and thus bundled completely in our release and does not depend on your host's git on the $PATH. This is a massive lifestyle improvement over the previous Repeatr v0.15 git implementation, which merely hoped you had a global git installation that looked roughly right.
• mount "inputs" now support files, FIFOs, and even device nodes in addition to directories. (This might be interesting if you wanted to, say, mount your X11 socket, or your video and sound device nodes. Yep, that works.)
• built-in memoization 🎉 🎉 🎉
  • set REPEATR_MEMODIR=./memos/ and repeatr run will store RunRecords there, each in their own file named after the Formula SetupHash. When repeatr run is invoked, it will look for a memoized RunRecord, and immediately return it.
  • this is AWESOME because it means you can trivially string a bunch of steps together, and run the whole script repeatedly, and it'll no-op instantaneously if nothing changed. Change any input, and it cachebusts. Correctly. Every time. Hello this is the future, weclome

So yeah, this release is pretty great. Use it.

v0.200rc1

First release candidate of the "r200" development branch -- this is a massive amount of internals rework and has significant user facing API changes as well. In short, every API change we've identified in the history of the project so far has been saved up, and this is all of that wishlist in one go.

As such, the changelist is so massive it hardly makes sense. There ARE breaking changes, and they are sweeping, and if you don't currently have time to absorb any refactors to your usage of repeatr, please, stay on the v0.15 release.

Among the highlights:

• Massive rewrite of core components has improved error messaging significantly. (We hope. Bugs reports and PRs for further improvements on UX of error messages always welcome!)
• The fileset snapshotting and transport parts have been split out into a new source repo and are built in a standalone binary!
• Fileset snapshotting and transporting runs in separate processes. Sysadmins may rejoice at the more fine-grained ps tree.
• Many components are even more API driven than before. (Do those separated file handling tools have a smooth JSON API? You bet your bippy.)
• The format for identifying wares has changed! They are now a single string. This makes many parts of the API much simpler for third-party tools to work with.
• The way we handle URLs has changed! They are now a fully separate object from formulas, called "context". This should make the semantics of formula hashing much more self-explanatory.
• and many, many more changes which we hope should make for a better user experience across the board.

Furthermore: this new "r200" API has been designed to work with new systems for releasing, managing, publishing, and sharing Wares, and the human-meaningful names and version info you assign to them. This system, called hitch, is another fully separate tool, and is included in this release. We hope you find it useful! If not, you're not tied to it.

Happy hacking! We hope you like the changes.

v0.0.1

This is a snapshot of the omnibus build. You can use it to bootstrap a self-hosting timeless stack toolchain!

It is built by the scripts in the ./omnibus/ dir. You should be able to build this binary reproducibly...!