Optional prefix to filter Git tags relevant to Copier

[emcd]

Actual Situation

I maintain the following in the same repository:

• Copier Template: copier.yaml, template/**
• Common Github Workflows: .github/workflows/xrepo--*.yaml
• Common Documentation: documentation/common/*.rst
• Project Maintenance Package: sources/** (static website generation for index of per-version documentation/coverage reports, code coverage badge/shield generation, release process helpers, etc...)

These exist in the same repository so that they can be updated in lock-step, when necessary.

• E.g., the Copier template generates project-specific Github workflows which rely on the common workflows.
• E.g., the common workflows rely on some of the tooling provided by the project maintenance package published on PyPI.
• E.g., the Copier template generates project-specific documentation stubs which reference the common documentation.

I have the common documentation independently versioned with docs-* Git tags and will be doing the same for the common Github workflows soon (gha-* tags). However, I am in a situation where new releases of the Copier template also trigger new releases of the PyPI package and vice versa, because they currently share the same set of tags, out of necessity. While it is true that I could exclude the Copier template paths from the Github workflow trigger for releases, there are still two issues with that:

• it can lead to non-contiguous package release versions, if Copier template releases occur between Python package releases;
• the repository itself, including a release management Github workflow, is generated from the Copier template that it contains and the Copier template does not (and should not) have special logic for this case.

Desired Situation

To me, the cleanest solution would be for Copier to support a tag filter prefix. E.g., if I gave a copier- prefix during copier copy, then it would look for the latest tag starting with copier-. The prefix would be recorded as a new underscore variable in the answers file.

Proposed solution

(Same as desired situation.)

I looked at copier/_vcs.py and it seems that an additional parameter for the prefix would need to be plumbed into checkout_latest_tag. Then, if a prefix is supplied, a filter would need to be applied before the valid_version filter. Seems like a fairly small change, even including tests and documentation updates.

I can make a PR, but does this idea have maintainer approval? (Happy to discuss my use case in more detail, if it is not clear.)

[sisp]

Thanks for proposing this enchancement, @emcd! 👍 And sorry for the delayed reply! 🫣

Requests for monorepo support have come up several times before. The documentation also addresses this question: https://copier.readthedocs.io/en/latest/configuring/#subdirectory

Can I have multiple templates in a single repo using this option?

The Copier recommendation is: 1 template = 1 Git repository.

[...]

At least #315, #1078, #1279, and #1571 are related. See especially #315 (comment) regarding this topic.

Nevertheless, I'm curious to better understand why factoring out the Copier template into a dedicated repository doesn't seem like a good approach in your case.

• Are the GitHub workflows used independently and also in the Copier template? If yes, could you release them independently as GitHub Actions and reuse them in the Copier template?
• Does the common documentation document the GitHub workflows, and you'd also like to use the same documentation in the Copier template where the same GitHub workflows are used, too? If yes, would releasing the GitHub workflows as GitHub Actions along with the documentation solve the documentation problem, too?

Sorry if I totally misunderstand your scenario and my questions make absolutely no sense. 😄

[emcd]

Thanks for proposing this enchancement, @emcd! 👍 And sorry for the delayed reply! 🫣

No worries, @sisp . It is the nature of open source... I've experienced delays far longer than this. :) Thanks for responding.

Requests for monorepo support have come up several times before. The documentation also addresses this question: https://copier.readthedocs.io/en/latest/configuring/#subdirectory

To clarify, this is not really a monorepo situation. I think maybe my initial description of the problem was too abstract and this is causing confusion. Let me try with something more concrete below. (If it is too big of a wall of text to read, then feel free to skip down to the final part of my response, since that drives to the heart of the problem.)

• My Copier template is here.
• This template has been used to generate a number of projects that I maintain, such as dynadoc, mimeogram, classcore, and others. (With more on their way.) Thanks for Copier, btw! It has worked well.
• As you may note, the template is a subdirectory of the python-project-common repository and the _subdirectory flag is already present in the copier.yaml file in the top-level of the repository.
• The python-project-common repository also contains common cross-repo Github workflows that are invoked by the repo-specific workflows that Copier generates. (All the ones starting with xrepo--; the others are generated by the Copier template when I use it on the python-project-common repo itself.) This allows me to make fixes and improvements without updating all of the dependent projects with Copier.
• The python-project-common repository also contains common documentation that is referenced by the documentation stubs that Copier generates from the template. This allows me to make fixes and improvements without updating all of the dependent projects with Copier.
• The python-project-common repository also contains tooling which is used to validate the Copier template and which is used within some of the Github workflows to perform tasks, such as updating the static site which tracks the documentation and coverage reports for all versions of a project. (This replaces the need for Coverage.io and ReadTheDocs or other such services.)

Nevertheless, I'm curious to better understand why factoring out the Copier template into a dedicated repository doesn't seem like a good approach in your case.

• Why does the same repository contain all of these related facets? So that they can be updated in lock-step, if needed.
  • Example: suppose that I want to change the argument structure of the tooling. This would affect how it is used in the workflows. If I had to release the tooling separately, then I would have to alter the workflows to use Git refs and Github HTTPS URLs to install the tooling to iterate on and test the changes. If the workflows and the tooling package are in the same repo, then integration of changes is simpler. And, I can ensure that there are no mismatched Git refs/versions, which is a danger if I had two separate repos (one for tooling and one for the template or the Github workflows).
  • Example: suppose that I want to restructure the common documentation. If it was in a separate repo, then I would likewise need to alter Git refs to iterate on and test the changes. If the common documentation and the Copier template are in the same repo, the integration of changes is simpler. And, I can ensure that there are no mismatched Git refs/versions, which is a danger if I had two separate repos (one for the common documentation and one for the template).
• Generating python-project-common from its own Copier template is a good way to test changes.

• Are the GitHub workflows used independently and also in the Copier template? If yes, could you release them independently as GitHub Actions and reuse them in the Copier template?

There are two flavors of Github workflows:

• Cross-repo workflows. These are used from the workflows generated by Copier.
• Project-specific workflows generated from the Copier template. These use the cross-repo workflows in python-project-common.

Example: the qa workflow template references the xrepo--tester and xrepo--reporter cross-repo workflows.

But, this is a solved problem. Perhaps I should not have mentioned it at all, since it is adding noise to the conversation.

• Does the common documentation document the GitHub workflows, and you'd also like to use the same documentation in the Copier template where the same GitHub workflows are used, too? If yes, would releasing the GitHub workflows as GitHub Actions along with the documentation solve the documentation problem, too?

While I agree that one could convert Github workflows to Github composite actions, it really does not address the synchronization issue. And, sorry that I wasn't clear, the common documentation relates to development practices, etc... which are shared across all of the projects. Does not really have anything to do with the Github workflows.

Again, I probably should not have mentioned this, since it is a solved problem and is adding a distraction to the conversation.

Sorry if I totally misunderstand your scenario and my questions make absolutely no sense. 😄

No worries at all! Let me try to succinctly state the actual problem that I am trying to solve:

• If you look at the tags for the project, you will see that the same tags are being shared by the tooling and by the Copier template version.
• If you look at the tooling release history on PyPI, you will see that the history is littered with a bunch of alpha versions, which were triggered by version bumps to the Copier template.
• Furthermore, if I were to release another version of the tooling today, the current Copier template version is v1.21. To synchronize with this, I would have to bump the package version to v1.21. This would result in a non-contiguous release history on PyPI.

This is the heart of the problem that I am trying to solve.

If I could have copier-* version tags, then I could use v* version tags solely for the package.

Of course, you're welcome to challenge the premise that the template and the tooling that generated projects use should be in the same repo, but hopefully I've managed to justify why it makes sense. (Namely, synchronized changes.)