Convert "plan build" code over to Rust #6258
Description
This is work that's been on our long wish list for a while, but the advent of #6257 has crystallized the fact that it's time to revisit the decision to implement hab-plan-build in Bash (and PowerShell on Windows, though the linked issue pertains specifically to Bash).
The initial decision to implement hab plan build in Bash was a pragmatic one. It's easy to get started, it provides us a plan.sh syntax that's familiar and easy to write (and gives us parsing for free), it has low runtime and binary size overhead, etc. However, it is also an incredibly subtle language full of edge cases, many of which are non-obvious. The excellent shellcheck has helped us greatly in improving our code (honestly, use of that tool should be mandatory for all shell scripts everywhere). However, having excellent linting cannot magically give you safe code. While Bash is easy to get started with, becoming well-versed in the subtleties of its more advanced features (many of which we use) is not easy. Additionally, it does not have many of the most basic features that one needs to build non-trivial applications (the fact that we can't return data from a function, have to rely on global data structures everywhere, and have to resort to esoteric gymnastics to use something approximating a map data structure are just three red flags.)
The irony has not escaped us that the Habitat project is using one of the safest programming languages available (Rust) and one of the most dangerous programming languages (Bash) at the same time, each for key components of the Habitat ecosystem.
Aside from the unsafe nature of Bash, there is also the fact that any change to the plan build logic must necessarily be re-implemented in PowerShell for Windows builds. In addition to being an unfortunate duplication of effort, we also have a far shallower bench of PowerShell expertise than we do for Bash.
Given that we now have considerable expertise in Rust, and the fact that we have actually already implemented a number of things that hab-plan-build does in Rust already (e.g. reading and taking action on package metadata files), it is time to seriously give thought to a rewrite of hab-plan-build entirely in Rust.
This would yield a number of benefits, some of which have already been mentioned, but bear repeating:
- Increased safety
- Better testability
- More familiarity with current team members
- No duplication of effort (single codebase)
We're actually in a good position for a rewrite, and much of it can be transparent to outside users. One way we could proceed would be to wrap library calls in Rust in bespoke binaries to begin replacing the Bash (and PowerShell) functions we currently have. Our existing Bash infrastructure would simply call Rust binaries rather than their shell script counterparts. In this way, we could easily offload the most complicated operations into Rust, giving us quick safety and stability returns. This would also allow us to centralize more and more of our logic, reducing the current Bash and PowerShell duplication.
This would likely be a short-term situation, though we could go quite a long way with this approach.
In the long term, we will also have to contend with the fact that Habitat plans themselves are written in Bash / PowerShell, and can include non-trivial callback functions written by end-users. I think our experience at Chef has shown that having the power of a programming language for such situations is a Good Thing, as opposed to using a data-only approach (e.g., creating a plan.yml file or something similar). Only a sadist would force users to write their plans in Rust, though! 😄 One thought has been to look at something like Lua as a plan language. Lua is commonly used for these kinds of pluggable scripting tasks, is multi-platform, and (importantly for this discussion), has Rust bindings.
- Reddit thread on Rust Lua bindings
- rlua - the safest Rust binding, according to the thread above
- luster - an experimental Lua VM written in Rust by the author of
rlua
As terrifying as it sounds, something like shellfn might also be useful in transitioning from Bash / PowerShell to Rust.
(There may be other viable approaches, as well; using Lua immediately springs to mind, but that shouldn't be the end of the discussion.)
As stated earlier, I think we can greatly improve the safety of plan building code by re-implementing the most complex bits in Rust, and this does not have to take place at the same time as implementing a new plan language. Both things are important to consider in the scope of a rewrite, however, and both will be necessary if there is ever a day when the plan building process is completely implemented in Rust. This is also a change that would greatly impact existing users of Habitat; we would need to devote considerable thought to migration strategies, and would realistically need to straddle both worlds for some period of time. Providing users an easy way to opt into the new plan syntax would be a great help (this could also probably be done automatically; after all, if you've got a plan.lua file in place, that's probably a pretty strong signal that you'd want the new version of the code 😄)
(It should go without saying that as more functionality is implemented in Rust, we should take the time to really strengthen the testing --- unit, integration, and functional --- around this core component of our platform.)
UPDATE: As @baumanj pointed out in chat, another issue not explicitly called out here is the brittleness of the Bash code. Making changes is far more difficult and fraught than it needs to be. Moving to Rust would be a big step in the right direction for reducing the brittleness of the code.