Updatable templates for template hierarchies and reuse

[sisp]

Is your feature request related to a problem? Please describe.

I'd like to create a Copier base template (e.g. a general Python project template) and Copier child templates (e.g. a Python web app project template using FastAPI, a Python ML library template using PyTorch, etc.). The Copier child templates would mostly extend the Copier base template (e.g. add new files, add content to existing files from the Copier base template) but might also overwrite files/folders in the Copier base template (e.g. modify content in existing files from the Copier base template) or even delete files from the Copier base template (although this might be a rare case). Naively, I could create independent Copier templates, but then maintenance wouldn't scale because I'd have to sync the common parts manually. Instead, I would like to update a Copier child template with the changes of its Copier base template in the same way as I can update a generated project with the changes of its associated Copier template.

Describe the solution you'd like

I know this topic has been discussed a few times already (#79, #402, #416), but I haven't found a satisfactory solution and came up with a (to my knowledge) new approach that I'd like to suggest and discuss as it doesn't seem possible out of the box.

To clarify possibly confusing terminology upfront, I'd like to explicitly distinguish between Copier templates (i.e. project templates) and Jinja templates (i.e. content templates used in text files or file/folder names).

So far, I've seen this topic under the term "template inheritance" which (to me) sounds similar to the idea of, e.g., OverlayFS where multiple filesystem sources are merged, possibly enhanced with Jinja inheritance such that an overwritten file may actually be an extension of the base file rather than a complete replacement. I believe #79 tried to implement such an approach, but it was not merged for apparently good reason.

I've been giving this problem some thought for a while and believe that the word "inheritance" might be misleading. Instead of trying to inherit from a Copier base template, I think a Copier child template should be a (modified) copy of the Copier base template containing a reference to the Copier base template similar to how a generated project has a reference to its corresponding Copier template in .copier-answers.yml. Then, a very similar update mechanism as for updating a generated project could be used to update a Copier child template with the changes of its associated Copier base template:

graph TD
%% nodes ----------------------------------------------------------
base_template_repo("base template repository")
base_template_current("/tmp/base_template<br>(current tag)")
base_template_latest("/tmp/base_template<br>(latest tag)")
child_template_regen("/tmp/child_template<br>(fresh, current version)")
child_template_current("current child template")
child_template_half("half migrated<br>child template")
child_template_updated("updated child template")
child_template_applied("updated child template<br>(diff applied)")
child_template_full("fully updated<br>and migrated child template")
update["update current<br>child template in-place<br>+ run tasks again"]
compare["compare to get diff"]
apply["apply diff"]
diff("diff")
%% edges ----------------------------------------------------------
        base_template_repo    --> |git clone| base_template_current
        base_template_repo    --> |git clone| base_template_latest
        base_template_current --> |copy and run tasks| child_template_regen
       child_template_current --> compare
       child_template_current --> |apply pre-migrations| child_template_half
         child_template_regen --> compare
          child_template_half --> update
         base_template_latest --> update
                       update --> child_template_updated
                      compare --> diff
                         diff --> apply
       child_template_updated --> apply
                        apply --> child_template_applied
       child_template_applied --> |apply post-migrations| child_template_full
%% style ----------------------------------------------------------
classDef blackborder stroke:#000;
class compare,update,apply blackborder;

Loading

This approach would offer significant flexibility for customizing a Copier child template because anything can be changed without requiring the developer of the Copier base template to anticipate extension points (i.e. Jinja blocks).

In contrast to generating projects from a Copier template, generating a Copier child template doesn't involve a questionnaire or Jinja templating. Currently, I think only the following settings (from copier.yml) remain relevant:

• answers_file (But since there are no questions to ask for generating Copier child templates, this setting might be renamed or removed entirely and a hardcoded file name might be used instead.)
• migrations
• min_copier_version
• skip_if_exists
• tasks

So perhaps a new file, e.g. copier-template.yml, should be introduced where the settings for generating Copier child templates are put. The Git rev (_commit: <rev>) of the Copier base template might be written to a file like .copier-extension.yml (or a different file name; subject to discussion).

Let's make this idea more concrete.

The filesystem layout of a Copier base template might look like this (with _subdirectory: template in copier.yml):

.
├── copier.yml
├── copier-template.yml
└── template
    ├── {{_copier_conf.answers_file}}.jinja
    ├── pyproject.toml.jinja
    └── ...

And the filesystem layout of a Copier child template (which itself could serve as a Copier base template again) might look like this:

.
├── .copier-extension.yml
├── copier.yml
├── copier-template.yml
└── template
    ├── {{_copier_conf.answers_file}}.jinja
    ├── Dockerfile.jinja
    ├── pyproject.toml.jinja
    └── ...

In order to speed-up the adoption of this idea, it might be possible to make copier-template.yml optional and assume sane defaults (i.e. no migrations, no tasks, ...) when this file is not present.

Finally, two new CLI subcommands would need to be added to the Copier CLI, one for generating a new Copier child template from a Copier base template and one for updating an existing Copier child template from its associated Copier base template:

copier template <base_template_src> <child_template_path>
copier template update

Describe alternatives you've considered

Both YAML includes of several copier.yml files and appyling multiple templates to the same subproject don't solve this problem.

I also thought about using Git submodules to include a Copier base template repository in a Copier child template repository, but this approach doesn't work either for several reasons:

• Git submodules cannot be pinned to tags (at least not with SSH URLs).
• The approach would follow the idea of WIP - Folder inheritance #79 (if I'm not mistaken) while missing a lot. For instance, I don't see how Copier would fall back to copying and rendering all files and folders from the Copier base template (included via a Git submodule) unless the Copier child template contained files and folders with the same name, thus taking precedence over those in the Copier base template. And in any case, WIP - Folder inheritance #79 seems like the wrong approach.

Additional context

I think it would be a huge win for Copier to support Copier template hierarchies. If others agree with this idea or we refine it to a promising proposal, I'd be happy to work on a draft PR.

[yajo]

Seems interesting indeed. As you know, I always try to keep it as simple as possible here. So I've been thinking about a possible solution for your problem that doesn't require a lot of refactoring.

Quoting from https://copier.readthedocs.io/en/stable/#basic-concepts:

Copier doesn't replace the DRY principle... but sometimes you simply can't be DRY and you need a DRYing machine...

So maybe it's time for Copier to DRY itself!

Have you thought about having a metatemplate? A template that generates templates? This way:

1. Child templates are just normal copier templates that have other templates applied.
2. You get full reproducibility and updatability, just with the well known copier update... but from the child template itself.
3. Still doesn't replace all the well known DRY methods you already mentioned that Git, Copier and Jinja provide you. But it also fills the gap where they're still not enough.

I have made a proof of concept in https://github.com/yajo/copierception. According to my tests, the only problem is that the child template doesn't get the copier.yaml file created as it should. This is because it's explicitly skipped in Copier for obvious reasons. But we can safely no skip it if there's a _subdirectory specified, because in that case anything inside of it is put there on purpose.

Seems like an easy solution. What do you think?

[sisp]

That's a great idea! 👏 Also, thanks for the proof of concept! 🙏 (And great name copierception and meme! 😂)

I think this could work indeed. 🎉 I've created another proof of concept with a chain of meta-templates to check whether/how this could be done:

flowchart TD
    copier-python-meta --> copier-python
    copier-python --> python-project
    copier-python-meta --> copier-python-django-meta
    copier-python-django-meta --> copier-python-django
    copier-python-django --> python-django-project

Loading

For this to work, I think the workflow looks like this:

copier-python-meta

The filesystem layout of a (simplified) generic Python meta-template might look like this:

copier-python-meta
├── copier.yml
└── meta-template
    ├── [[ _copier_conf.answers_file ]].j2
    ├── copier.yml
    └── template
        ├── {{ _copier_conf.answers_file }}.jinja
        ├── {{ project_name }}
        │   └── __init__.py
        └── pyproject.toml.jinja

As you mentioned, copier.yml needs to be removed from the _exclude list in /copier.yml.

copier-python

Using copier-python-meta, a regular Copier template copier-python for generic Python projects can be generated:

$ copier copier-python-meta copier-python
No git tags found in template; using HEAD as ref

Copying from template version 0.0.0.post1.dev0+b21a356
    create  .
    create  template
    create  template/{{ project_name }}
    create  template/{{ project_name }}/__init__.py
    create  template/pyproject.toml.jinja
    create  template/{{ _copier_conf.answers_file }}.jinja
    create  .copier-answers.yml
    create  copier.yml

And the resulting filesystem layout of the copier-python template looks like this:

copier-python
├── .copier-answers.yml
├── copier.yml
└── template
    ├── {{ _copier_conf.answers_file }}.jinja
    ├── {{ project_name }}
    │   └── __init__.py
    └── pyproject.toml.jinja

Using this template, a generic Python project (e.g. python-project) can be generated as usual.

copier-python-django-meta

Now, let's create a more specific meta-template for a (simplified) Python web project that uses Django based on copier-python-meta.

The following filesystem layout needs to be created manually:

copier-python-django-meta
├── copier.yml
└── meta-template
    └── [[ _copier_conf.answers_file ]].j2

Then, the content of the folder meta-template/ is further populated by the generated output of the copier-python-meta template:

$ copier --answers-file .copier-answers-meta.yml copier-python-meta copier-python-django-meta/meta-template
No git tags found in template; using HEAD as ref

Copying from template version 0.0.0.post1.dev0+b21a356
 identical  .
    create  template
    create  template/{{ project_name }}
    create  template/{{ project_name }}/__init__.py
    create  template/pyproject.toml.jinja
    create  template/{{ _copier_conf.answers_file }}.jinja
    create  .copier-answers-meta.yml
    create  copier.yml

Now, just for the sake of this example, we could add the file meta-template/template/manage.py.jinja, that Django uses, to exemplify that the child template is an extension of the copier-python(-meta) template. The resulting filesystem layout looks like this:

copier-python-django-meta
├── copier.yml
└── meta-template
    ├── .copier-answers-meta.yml
    ├── [[ _copier_conf.answers_file ]].j2
    ├── copier.yml
    └── template
        ├── {{ _copier_conf.answers_file }}.jinja
        ├── {{ project_name }}
        │   └── __init__.py
        ├── manage.py.jinja
        └── pyproject.toml.jinja

Note the flag --answers-file .copier-answers-meta.yml which is important to avoid a conflict between the answers file created by copier-python-meta and the answers file [[ _copier_conf.answers_file ]].j2 that is rendered when generating the copier-python-django template (see below). On the other hand, .copier-answers-meta.yml shouldn't be copied into the generated child template copier-python-django, so it needs to be excluded via _exclude in /copier.yml.

When the copier-python-meta meta-template changes and we would like to update this meta-template accordingly, we would run the following command:

copier --answers-file .copier-answers-meta.yml update copier-python-django-meta/meta-template

Again, note the flag --answers-file .copier-answers-meta.yml.

copier-python-django

Using copier-python-django-meta, a regular Copier template copier-python-django for Python web projects using Django can be generated:

$ copier copier-python-django-meta copier-python-django
No git tags found in template; using HEAD as ref

Copying from template version 0.0.0.post1.dev0+04c5d90
    create  .
    create  template
    create  template/{{ project_name }}
    create  template/{{ project_name }}/__init__.py
    create  template/pyproject.toml.jinja
    create  template/manage.py.jinja
    create  template/{{ _copier_conf.answers_file }}.jinja
    create  .copier-answers.yml
    create  copier.yml

The resulting filesystem layout looks like this:

copier-python-django
├── .copier-answers.yml
├── copier.yml
└── template
    ├── {{ _copier_conf.answers_file }}.jinja
    ├── manage.py.jinja
    ├── {{ project_name }}
    │   └── __init__.py
    └── pyproject.toml.jinja

Using this template, a Python web project using Django (e.g. python-django-project) can be generated as usual.

---

WDYT, @yajo? If you agree with this extended proof of concept:

1. Do you think we should automatically allow copying a copier.yml file (i.e. remove it from the _exclude list) in a template when _subdirectory is used and is not "" or "."?
2. Should we add a proper guide in the docs that describes this workflow?

[sisp]

We could even create a template for meta-templates that generates this filesystem layout. 🤣

[yajo]

Note the flag --answers-file .copier-answers-meta.yml which is important to avoid a conflict between the answers file created by copier-python-meta and the answers file [[ _copier_conf.answers_file ]].j2 that is rendered when generating the copier-python-django template (see below). On the other hand, .copier-answers-meta.yml shouldn't be copied into the generated child template copier-python-django, so it needs to be excluded via _exclude in /copier.yml.

IIUC this isn't really a big problem because both the meta-template and the sub-template use _subdirectory. Thus, the sub-template's answers file (which points to the version in the meta-template) would be outside that subdirectory, so it would be ignored when producing a project based on the sub-template.

Do you think we should automatically allow copying a copier.yml file (i.e. remove it from the _exclude list) in a template when _subdirectory is used and is not "" or "."?

Sure. Actually we should completely remove any default value if using a subdirectory.

Should we add a proper guide in the docs that describes this workflow?

Definitely. Template reusability strategies are a common subject in forums, so it makes sense to make them a chapter in docs.

[sisp]

IIUC this isn't really a big problem because both the meta-template and the sub-template use _subdirectory. Thus, the sub-template's answers file (which points to the version in the meta-template) would be outside that subdirectory, so it would be ignored when producing a project based on the sub-template.

The conflict is between the answers file of the parent meta-template copier-python-meta and the (templated) answers file of the child meta-template copier-python-django-meta. If I ran

-copier --answers-file .copier-answers-meta.yml copier-python-meta copier-python-django-meta/meta-template
+copier copier-python-meta copier-python-django-meta/meta-template

then copier-python-django-meta's filesystem layout would be

 copier-python-django-meta
 ├── copier.yml
 └── meta-template
-    ├── .copier-answers-meta.yml
+    ├── .copier-answers.yml
     ├── [[ _copier_conf.answers_file ]].j2
     ├── copier.yml
     └── template
         ├── {{ _copier_conf.answers_file }}.jinja
         ├── {{ project_name }}
         │   └── __init__.py
         ├── manage.py.jinja
         └── pyproject.toml.jinja

and generating python-copier-django using this meta-template would show a conflict for .copier-answers.yml

$ copier copier-python-django-meta copier-python-django
No git tags found in template; using HEAD as ref

Copying from template version 0.0.0.post2.dev0+c422bf7
    create  .
    create  .copier-answers.yml
    create  template
    create  template/{{ project_name }}
    create  template/{{ project_name }}/__init__.py
    create  template/pyproject.toml.jinja
    create  template/manage.py.jinja
    create  template/{{ _copier_conf.answers_file }}.jinja
  conflict  .copier-answers.yml
 overwrite  .copier-answers.yml
    create  copier.yml

because meta-template/.copier-answers.yml (which points to the version of the copier-python-meta meta-template) and meta-template/[[ _copier_conf.answers_file ]].j2 after rendering have identical file names. It looks like meta-template/[[ _copier_conf.answers_file ]].j2 overwrites meta-template/.copier-answers.yml, but this behavior is ambiguous even if its implementation is deterministic. Do you see my point?

I think it would be great if there was a way to avoid a custom answers file name for pointing to the parent meta-template, but I don't see one at the moment. Any idea?

Do you think we should automatically allow copying a copier.yml file (i.e. remove it from the _exclude list) in a template when _subdirectory is used and is not "" or "."?

Sure. Actually we should completely remove any default value if using a subdirectory.

👍 I'd be happy to send a PR next week.

Should we add a proper guide in the docs that describes this workflow?

Definitely. Template reusability strategies are a common subject in forums, so it makes sense to make them a chapter in docs.

👍 I'd be happy to write a draft and send a PR.

[sisp]

Perhaps a slightly better alternative to the --answers-file .copier-answers-meta.yml flag in the command

copier --answers-file .copier-answers-meta.yml copier-python-meta copier-python-django-meta/meta-template

is

copier --answers-file ../.copier-answers.yml copier-python-meta copier-python-django-meta/meta-template

which

1. doesn't introduce a different name for the answers file, and
2. moves the answers file related to the parent meta-template (copier-python-meta) to the root folder of the child meta-template (copier-python-django-meta), which seems better to me.

The resulting filesystem layout of copier-python-django-meta then looks as follows:

copier-python-django-meta
├── .copier-answers.yml
├── copier.yml
└── meta-template
    ├── [[ _copier_conf.answers_file ]].j2
    ├── copier.yml
    └── template
        ├── {{ _copier_conf.answers_file }}.jinja
        ├── {{ project_name }}
        │   └── __init__.py
        └── pyproject.toml.jinja

[yajo]

I'm trying to understand but I really don't get what's the problem of using just the default copier answers file.

According to my tests, after #941 was merged, the meta-template sample works as expected. Check out this shell session:

$ subtpl=$(mktemp -d)
$ dst=$(mktemp -d)

# Create a sub-template from the meta-template
$ copier -r 84e05b9ae447a4e36e9e678270af8cd3e0cad026 copy https://github.com/yajo/copierception $subtpl
🎤 python (bool)
   Yes
🎤 pre_commit (bool)
   Yes

Copying from template version 0.0.0.post2.dev0+84e05b9
 identical  .
    create  template
    create  template/{{ _copier_conf.answers_file }}.jinja
    create  template/{% if pre_commit %}.pre-commit.yaml{% endif %}
    create  template/pyproject.toml.jinja
    create  template/README.md.jinja
    create  copier.yaml
    create  .copier-answers.yml


$ cd $subtpl

$ git init
Inicializado repositorio Git vacío en /tmp/tmp.eC3Bv4Ijmq/.git/

$ git add -A

$ git commit -m 1
[main (commit-raíz) 6499725] 1
 6 files changed, 55 insertions(+)
 create mode 100644 .copier-answers.yml
 create mode 100644 copier.yaml
 create mode 100644 template/README.md.jinja
 create mode 100644 template/pyproject.toml.jinja
 create mode 100644 template/{% if pre_commit %}.pre-commit.yaml{% endif %}
 create mode 100644 template/{{ _copier_conf.answers_file }}.jinja

# Generate a final project from the sub-template
$ copier copy $subtpl $dst
/nix/store/asr4mbhx1va7ha0kn3cvx4n0qlxmg8vk-python3.10-copier-7.1.0a0.dev20230221141200+nix-git-ab72507/lib/python3.10/site-packages/copier/vcs.py:155: ShallowCloneWarning: The repository '/tmp/tmp.eC3Bv4Ijmq' is a shallow clone, this might lead to unexpected failure or unusually high resource consumption.
  warn(
No git tags found in template; using HEAD as ref
🎤 project_name
   my project
🎤 build_system
   poetry

Copying from template version 0.0.0.post1.dev0+6499725
 identical  .
    create  .copier-answers.yml
    create  pyproject.toml
    create  README.md

# Update the sub-template getting last changes from the meta-template
$ copier update
No git tags found in template; using HEAD as ref
Updating to template version 0.0.0.post3.dev0+fea6c75
No git tags found in template; using HEAD as ref
🎤 python (bool)
   Yes
🎤 pre_commit (bool)
   Yes

Copying from template version 0.0.0.post3.dev0+fea6c75
 identical  .
 identical  template
 identical  template/{{ _copier_conf.answers_file }}.jinja
 identical  template/{% if pre_commit %}.pre-commit.yaml{% endif %}
 identical  template/pyproject.toml.jinja
 identical  template/README.md.jinja
 identical  copier.yaml
  conflict  .copier-answers.yml
 overwrite  .copier-answers.yml

# Check that the sub-template was properly updated
$ git diff
diff --git a/.copier-answers.yml b/.copier-answers.yml
index bb3d46a..55ab783 100644
--- a/.copier-answers.yml
+++ b/.copier-answers.yml
@@ -1,5 +1,5 @@
 # Changes here will be overwritten by Copier; NEVER EDIT MANUALLY
-_commit: 84e05b9
+_commit: fea6c75
 _src_path: https://github.com/yajo/copierception
 pre_commit: true
 python: true

[sisp]

Thanks for sticking with me and challenging me! 🙏 I really appreciate your time invest into this discussion! 🙏

In your shell session example, you have only one meta-template (copierception) from which you generate the template in $subtpl from which you generate the project in $dst. In this scenario, there's no problem.

But I'm considering an extended meta-template scenario in which there are two meta-templates: copier-python-meta and copier-python-django-meta where copier-python-django-meta is based on the generated sub-template from copier-python-meta (and possibly edited, which is fine because changes in the meta-template copier-python-meta can be applied by the regular update mechanism). Only this scenario causes a conflict of the answers files.

If you go through the example workflow in #934 (comment) and run

copier copier-python-meta copier-python-django-meta/meta-template

instead of

copier --answers-file .copier-answers-meta.yml copier-python-meta copier-python-django-meta/meta-template

or (as I later improved IMO)

copier --answers-file ../.copier-answers.yml copier-python-meta copier-python-django-meta/meta-template

then you'll see

$ copier copier-python-django-meta copier-python-django
No git tags found in template; using HEAD as ref

Copying from template version 0.0.0.post2.dev0+c422bf7
    ...
  conflict  .copier-answers.yml
 overwrite  .copier-answers.yml
    ...

which is the result of copier-python-django-meta/meta-template/.copier-answers.yml and copier-python-django-meta/meta-template/[[ _copier_conf.answers_file ]].j2 conflicting after rendering the latter.

I know it's a brain twist with the many meta-templates and sub-templates generated into other meta-templates. 🤣

Can you reproduce the problem now? 🤞

[yajo]

Haha I need a meta-brain to follow you! 🤭

Could you prototype something and give me a code snippet to reproduce it please? I guess it'll be easier to understand.

[sisp]

Sure, I'll prepare something next week. 🙂