Ecosystem-wide compatibility and build systems

[Benjmhart]

Broadly speaking, the Cardano Ecosystem relies heavily on Nix.

While Nix can help lead a project to reproducible builds, it can make the process of upgrading dependencies very difficult to manage.

In other Ecosystems (Haskell, Nodejs/npm, etc), a 'constraint-solver' algorithm is used to determine the packages that are known to build together. Haskell's ecosystem has also produced 'Stackage' which creates sets of packages that are known to build together, this makes it easier for library maintainers to see when their library has lost compatibility with a particular package, and for users to know which of their dependencies may not be ready for an upgrade.

unfortunately this infrastructure is not currently available in the Cardano ecosystem. Two solutions are apparent:

• work on integrating the cardano ecosystem with the mainstream Haskell ecosystem in hopes of being included in Stackage nightly images
• create our own version of stackage or a similar toolchain/CI system to validate builds for developers.

[zliu41]

Could you be more specific about how nix makes upgrading dependencies difficult? I'm not sure Stack enables us to do anything that Nix doesn't.

[Benjmhart]

@zliu41 - I'd say Stack & Cabal do a lot IF you have a proper versioning policy

making sure we know which commits of plutus a given version of, say, plutus-apps supports without having to experiment is a huge productivity boost for plutus developers.

Stack automates this process. CHaP is a great solution so far and I'm excited to see what comes of it.

[klarkc]

We're building nix-templates to tackle part of this problem. The idea is to have up-to-date, pre-tested, minimal templates with different languages/tools (currently, it has CTL and Plutus). While having a constraint system in place is likely the best approach, it can be difficult to implement with the diversity of nix. Perhaps a CHaP-like registry for nix, like a central flake that pins and locks compatible dependencies, would help. This would be a kind of nixpkgs for Cardano-related dependencies.

[harryprayiv]

This ecosystem is already hard enough to onboard people to. Eliminating Nix will make it IMPOSSIBLE. The answer isn't getting rid of Nix. The answer is being better at teaching others Nix and developing a canonical configuration that can be shared to get people up and running. In the context of the node, there really isn't enough documentation.

The elimination of Nix in my workflow will introduce MANY more problems than it solves and I will most definitely stop developing DApps for Cardano if Nix is dropped.

[klarkc]

@harryprayiv I agree that nix approach is probably the easier for newcomers, especially from foreign languages. But IMO the lack of static types of nix is really a thing, std helps (cardano-world for example), but at the price of boilerplate and domain specific language. I would suggest taking a radical new approach, like PureScript on top of Nix (pure-nix) to build the Cardano universe with proper static types and documentation. But that's is a huge effort.

[harryprayiv]

lack of static types of nix is really a thing

Please elaborate. From my limited understanding, nix flakes are literally the clearest, most robust way of managing dependencies and builds that exists today. You can specify exactly the derivation hash that you are looking for and it is nearly 100% guaranteed to work. Sounds fairly static to me. Maybe I am thinking about static in the wrong context.

[harryprayiv]

We're building nix-templates to tackle part of this problem.

this looks really helpful.

[nrdxp]

static types of nix is really a thing, std helps (cardano-world for example), but at the price of boilerplate and domain specific language.

Glad you have seen Standard, but just wanted to mention that eliminating boilerplate is one of the key goals of Standard. You can read a bit more specifically in the core repo README.

While there is an API for specifying blocktypes, that's about as much "boiler" code as you'll need, and using a flake template can eliminate even that bootstrapping step.

As for static types in the language, there are a few key areas where typechecking would improve UX, and in those cases we just reach for yants, or simply the module system depending on the specific concern.

You can also import JSON, so I've been experimenting already using nickel for validation and so far, it seems to create a sane barrier between packing and config concerns and its something that I'd like to bring into the mix eventually.

As a more general defense of Nix, I will acknowledge that it is a radical departure from your typical packing workflow, and that inherently requires some steeper level of onboarding than an ecosystem you might already be familiar with. However, I think in the case of the tooling around the Cardano ecosystem (Haskell, et al.) and the overall usecase (currency, contracts, etc) it makes absolute sense to use a build tool that cryptographically traces your dependency tree.

There is a whole world of possibilities that haven't even been fleshed out yet that would only serve to make the delivery pipeline that much more secure.

There is also a benefit to using Nix that isn't immediately obvious to the newcomer; while, in general, it does take more time to write your packaging layer, it becomes trivial to maintain in all but the most breaking of updates, likely saving you the time you invested up front and then some.

That is doubly important for those packages that just so happen to represent your development environment.

[klarkc]

@harryprayiv, from an avarage user perspective, Nix flakes are sufficient for almost everything. However, if you're a library author, package maintainer, or even a registry maintainer, there are some fundamental features that Nix flakes lacks. For instance, versioning (which is currently being discussed in NixOS/rfcs#107) is not yet fully addressed, so there is no way to handle breaking changes or apply security upgrades. Moreover, the Nix ecosystem is well-known for lacking proper documentation for APIs and interfaces, partly due to the lack of features in the language. This is arguably caused by the absence of a static type system. Consequently, building something like Hackage or Pursuit for Nix is not straightforward.

Sorry, @nrdxp, I used the term 'bloated' incorrectly. What I meant to say is that in the absence of a proper type system or language abstractions, it's common to end up with a domain-specific language. The 'std' uses a 'shared domain language' that everyone must agree upon to build upon. I hope this clarifies things a bit.

I think Nix's approach to building a shared foundation is great, but we need better tools to create and showcase Cardano packages that can be built on top of this foundation.

[nrdxp]

I think Nix's approach to building a shared foundation is great, but we need better tools to create and showcase Cardano packages that can be built on top of this foundation.

All I can say is that as someone who is soley responsible for a cardano testnet & a sidechain devnet that runs against it, including a ton of auxilary dependencies that don't really exist anywhere else, I'm not sure I'd be able to effectively perform my duties as efficiently as I have without the help of Nix for package caching and just general runtime sanity.

Sorry, @nrdxp, I used the term 'bloated' incorrectly.

Don't worry about it, just wanted to clarify that it is a project goal, so if there is some undo boilerplate somewhere I'd like to know the specifics so we can address it.

'std' uses a 'shared domain language'

Well sure, but in practice it is more like a bolt on language module system. It should make writing your blocks a bit more like a typical source file where you have includes at the top of the file and you can clearly trace where those values come from without having to manage the whole context yourself. As far as what is actually exported by the blocks though, you can write your code fairly liberally, we just enforce a type boundary at the block level so that you can "act" upon your blocks in a type safe way (i.e. publish them somewhere, run them, build them, etc).

In general I don't think Nix is actually lacking as much as it is different. So in the general case, your problem likely does have a solution, its just probably not the one you expected. So long as the solution is sound though, it doesn't really matter and, if we are talking long term, I can only imagine the benefits to operator sanity that have been unique to Nix (in my own experience at least) will be ever more important as the ecosystem scales up.

That said, it would be definitely useful to meet people where they are at as much as possible. For instance, I have an effort going on my work project to transition to Nix built devcontainers that essentially contain the same environment as a regular nix-shell but that developers can easily consume with docker, et al..

[harryprayiv]

@harryprayiv, from an avarage user perspective, Nix flakes are sufficient for almost everything. However, if you're a library author, package maintainer, or even a registry maintainer, there are some fundamental features that Nix flakes lack. For instance, versioning (which is currently being discussed in NixOS/rfcs#107) is not yet fully addressed, so there is no way to handle breaking changes or apply security upgrades. Moreover, the Nix ecosystem is well-known for lacking proper documentation for APIs and interfaces, partly due to the lack of features in the language. This is arguably caused by the absence of a static type system. Consequently, building something like Hackage or Pursuit for Nix is not straightforward.

Sorry, @nrdxp, I used the term 'bloated' incorrectly. What I meant to say is that in the absence of a proper type system or language abstractions, it's common to end up with a domain-specific language. The 'std' uses a 'shared domain language' that everyone must agree upon to build upon. I hope this clarifies things a bit.

I think Nix's approach to building a shared foundation is great, but we need better tools to create and showcase Cardano packages that can be built on top of this foundation.

Well said. I hope I can help in this regard as that is my current goal for the entire ecosystem. I must admit that I am very amateur and unschooled at all of this (nix and everything else), so forgive any ignorance on my part. Rest assured my ignorance can only be matched by my tenaciousness when it comes to achieving that goal.

[harryprayiv]

Excited to learn about it! That sounds like just the thing I was talking about today.

Well the support was merged already a while back. I have a performance patch I'd like to flesh out eventually but its by no means a blocker. I just use std-action to publish them to github cr from CI and voila your team has a devshell with, or without, Nix.

Also, it seems some of our goals are in alignment with the goals of your working group. I'd be happy to collaborate.

[blaggacao]

As the initiator of Standard, which was forged in the heat of fixing some of the problems discussed herein, I'd be happy to put my experiences at the service of this working group.

[harryprayiv]

Regarding types in Nix, would it be helpful to implement something like Unison in that context? From what I understand, Unison is currently vaporware but it does remind me of Nix. Maybe in a constrained environment like system builds, it can be utilized (or something similar that creates hashes from functions which survive renaming). Or am I just suggesting craziness?

[klarkc]

I didn't know about Unison until now, but it looks pretty cool with its unique approach to dependency resolution. However, the fact that you can't use foreign function interfaces (FFI) with Nix could be a dealbreaker for working with the Nix ecosystem. IMO, it's not the best way to handle the problem. Instead, we should be open to using different tools and not force developers to use something they don't want to.

[harryprayiv]

Instead, we should be open to using different tools and not force developers to use something they don't want to.

Agreed. Maybe a simpler (in-the meantime) solution is content addressed storage.

Tweag's article on it

---

On that on-boarding topic, one of the things talked about in the meeting was the non-existence of a CI across the whole ecosystem (which was roughly attributed to organizational silo-ing).

I suggested that perhaps each project should be required to have a flake with a few docker file outputs that can be run through the organizational hydra build system and be readily available. Maybe that is something that can be added to the Cardano Engineering Handbook as a pull request.

A formal, deterministic set of default build endpoints for each project that gets sent to the hydra system on approval would be helpful in many ways at once:

1. It would automate the process of helping those simply looking for docker builds.
2. It would benefit people like me who are looking for a simple canonical working configuration for each part of the ecosystem.
3. (not that it's needed but) it would help keep teams honest on their delivery metrics since it needs to build to be considered ready for use.

[blaggacao]

Hydra itself has been decomissioned, as it was unsustainable to maintain and scale in our context.

But the core of your idea flies well without choosing any particular build system.

And yes, docker images (in reality: OCI images) would be a good & standards-abiding endpoint.

[klarkc]

One thing that bothers me now is relying on Docker for local development, which adds unnecessary complexity. I would like to explore better alternatives that don't require external container services, such as Docker, but I'm not sure if it's currently feasible.

cardano-world does a great job of exposing the entrypoints of the application, which allows for building custom OCI-images or utilizing the entrypoint in other ways, IMO that's the correct way of doing that.

[blaggacao]

cardano-world does a great job of exposing the entrypoints of the application, which allows for building custom OCI-images or utilizing the entrypoint in other ways, IMO that's the correct way of doing that.

Talks are underway in the borader Standard community to explore ways to integrate overmind into a Contribution Environment while instrumenting entrypoints in such a way so they can choreograph resiliently instead of relying on orchestrated startup order based on building up state around healthiness or readiness predicates.

In order to be able to do that consistently is one of the reasons why Standard has extracted the two concepts of healthiness and readiness and decoupled it from OCI or scheduling and rather attached it to the OCI-agnostic concept of an Operable script. Other reasons.

[klarkc]

@blaggacao that's really awesome, I am eager to see this being applied in Cardano packages. Also overmind is mindblowing 🤯