Team and User Layered configuration using chezmoi

[Lockszmith-GH]

chezmoi is really versatile, I guess you already know this, but I thought I'd share our particular use-case.

Who

We are a team of remote workers, almost all of us work on Linux machines (Pop!_OS), we have a few mac OS laptops and one or two Windows machine.
Currently chezmoi covers the Linux realm, but we might expand this in the future.
We all work with GitLab SaaS under a Premium subscription, and all of repos are internal. (They are shared with the team, but not with the world).

What

I've developed a shared environment for the Linux users to use.
This environment does a lot - adds needed repos to apt on the local system, provides 'easy install' scripts, keep external resources up to date, customizes the work environment (fonts, prompts, auto-completion, history, git configuration, etc...), add support scripts to allow IT-Ops to quickly connect remotely and assist.
So, in short: A whole bunch of $h!t

I've been working on a layered approach to chezmoi, one that would allow each team member to manage their own personal chezmoi repo, but this personal repo is tied to a shared team configuration which provides shared functionality and configuration.

Why

By layering, if a team member wants to customize something on their own, they have the ability, but still get streamlined updates that are available to the entire team.
This makes chezmoi not only the safe-keeper of personal state, but a convenient delivery channel of updates to the base.

How

Structure

Each user gets a fork of a template repo when they start. That repo has the following components:

• .chezmoi.toml.tmpl
  • This contains a read-only repo specific authentication token (dotfiles_git_token) that is used to pull the shared repo (aka base)
  • The value is initialized by prompting the user to enter it on first init - so this secret is never stored in the repo.
• .chezmoitexternals.toml
  • Points to the 0base dotfiles repo which contains all of the shared functionality.
  • Since all of our repos are private, including the base repo requires credentials. The URL includes a template string pulling the dotfiles_git_token value.
• .chezmoiscripts directory with:
  • run_once_after_10_init_base_chezmoi
    • executes init on the base chezmoi after it was pulled via externals.
  • run_after_apply_base_chezmoi.sh
    • executed on every apply - applying any changes pulled from the external base.

Usage

The 0base chezmoi-repo contains environment variables, scripts and aliases that provide different shorthand for calling chezmoi:

• cz - the user's chezmoi context - pointing to the default chezmoi dir:

  ~/.local/chezmoi/

• czb - the base chezmoi context - pointing to a base chezmoi dir:

  ~/.local/chezmoi.base/

Context switching

For some 'more complex' calls, the context concept is designated using the CONTEXT environment variable.
For example: cz-init is a script used to call chezmoi init with the proper --script-path parameter, so that values are written back into the same source config file it is called with.
Running cz-init does this on the default (user) chezmoi repo.
Running CONTEXT=czb cz-init however, will run it in the base context.

But wait, there's is more...

There is more complexity the shared repo provides, like separating externals from the base configuration, which allows quicker responses from chezmoi status and makes maintaining and tracking changes faster (less accurate, but faster) from the base configuration, which allows quicker responses from chezmoi status

Code? (sometime?)

I'm almost ready to share the code, but not quite yet, as it's a bit messy (also need to have one of my colleagues validate I will not be exposing any secrets/password), but if you're interested, just 👀 this post, and I'll update below when I'm ready.
I've never got to clean it up and publish - it is still in use. Ask questions and I'll do my best to answer.

Visual

flowchart TD
    h{{home}}
    subgraph user ["~/.local/share/chezmoi"]
        cz[[chezmoi<br>local-state]]
    end

    subgraph base ["~/.local/share/chezmoi.base"]
        czbr([".chezmoiroot"])
        czb[[base-chezmoi<br>local-state]]

        subgraph ext ["~/.local/share/chezmoi.base/_externals"]
            czx[[externals-chezmoi<br>local-state]]
        end        
    end

    subgraph git ["Remote git repos (GitHub)"]
        czg[("dotfiles.< USER NAME >.git")]
        czbg[("dotfiles.0base.git")]
    end
    
    h       ===> |cz commands     | cz
    user   <--> |git             | czg
    h      ---- |czg commands    | czg
    czbr   -.-> |_home.base      | czb
    h      ======> |czb commands    | czb
    cz     -..- |externals       | base
    czb     -.- |externals       | ext
    base <----> |git             | czbg

Loading

Praise

I couldn't have done this without the support of Tom Payne the author, his support for this amazing tool is amazing on it's own level.
Thank you! (I didn't tag Tom as I assume he has enough noise as it is)

What about you?

I'm really interested to learn if others have developed other interesting patterns of usage with chezmoi?
Did you ever face similar needs?
As we will venture into multi-platform coverage, I wonder about best approaches.

[Lockszmith-GH]

Updates and backlinks

[Lockszmith-GH]

• Mentioned in Use Chezmoi in a team #3460 🔗comment

[Lockszmith-GH]

• Mentioned in Install Doctor: Chezmoi-Based Multi-OS Provisioning System #3659 🔗comment

[Lockszmith-GH]

• Mentiond in Best Way to Apply System Config Files #3768 🔗comment

[Lockszmith-GH]

• Mentioned in How do you manage machine-to-machine differences when you have lots of different hosts? #3773

[Lockszmith-GH]

• Updated Code section, from soon to 'sometime' - as life has been OBE and I don't know when I'll get to it.

[Lockszmith-GH]

Placeholder for future 2

[lnyousif]

This is brilliant approach, is there any repo link that I am missing for this ?

[Lockszmith-GH]

I have never got to publish it, as I hadn't really sanitized the internal repo.
I can try and tailor something and share.
If you have questions, I would be happy to answer.

[lnyousif]

if you want to take down all the secrets and put empty files, I can help with prepping an example that you can share publicly later. Just a thought

[Lockszmith-GH]

Been busy with some stuff, I'll try and get to it, but can't commit to anything.

[PeterCardenas]

@Lockszmith-GH what are you setting for the index in the externals file? I'm trying to set an empty path so that all external files are merged with the base one, but getting an empty path error

[""]
type = "git-repo"
url = "<url to repo>"

[Lockszmith-GH]

no, I probably confused everything there even more.
I call each of these chezmoi instances 'CONTEXT' or chezmoi context.

The externals was a separate context of it's own - because I was pulling so many externals, that bogged down and caused slowness - today I do something different, I just wrap the externals file with an {{- if -}} block, and enable it when it's time to check externals.

Anyway, the way I use 2 separate chezmoi contexts, was by having separate command (I describe these are cz and czb)

It looks like this:
cz controls ~/.local/share/chezmoi - the default chezmoi instance.
czb controls ~/.local/share/chezmoi.base - the 'base' chezmoi instance (the shared instance).

cz is just plain chezmoi (an alias or function will do the trick)

czb is something like:

chezmoi --source "${USER_HOME:-$HOME}/.local/share/chezmoi.base" --config ${USER_HOME:-$HOME}/.config/chezmoi/_base/chezmoi.toml "${@}"

I hope this helps understand what's going on up there.

[PeterCardenas]

nice! --source sounds like exactly what i was looking for

[PeterCardenas]

just for my knowledge externals mean every time you apply it fetches the externals?

[LEI]

@PeterCardenas refreshPeriod is set to 0 by default meaning "never re-download", you can find more details following the link to the reference manual here: https://www.chezmoi.io/user-guide/include-files-from-elsewhere/

[Lockszmith-GH]

However, it's not just refreshPeriod, the state of externals is read every time and compared to the active state, regardless of online state.
So while it's not reaching online, it is reading the local cache - if that cache is large, and disk is slow - it takes time.

For more details, see #2375