templates: support string patterns in template language #6899

lf- · 2025-07-05T02:02:55Z

This is a basic implementation of the same string pattern system as in
the revset language. It's currently only used for string.matches, so
you can now do:

"foo".matches(regex:'[a-f]o+')

In the future this could be added to more string functions (and e.g.
the ability to parse things out of strings could be added).

CC: #6893

Checklist

If applicable:

I have updated CHANGELOG.md
I have updated the documentation (README.md, docs/, demos/)
I have updated the config schema (cli/src/config-schema.json)
I have added/updated tests to cover my changes

lf- · 2025-07-05T02:49:07Z

cc @yuja

cli/src/generic_templater.rs

cli/src/commit_templater.rs

cli/src/template_builder.rs

cli/src/template_parser.rs

PhilipMetzger · 2025-07-05T10:39:06Z

just a reminder to adhere to commit guidelines here: https://github.com/jj-vcs/jj/blob/main/docs/contributing.md#commit-guidelines since. a feat: isn't really a topic. My suggestion would be templater:.

lf- · 2025-07-06T00:03:43Z

The full completion of this feature to be able to pull out the matching text in question probably involves introducing a new type that represents a match result (and match groups in regex), but doing so requires some rework and semantics design of the string pattern system itself.

I think that would probably be .match. Which probably means this boolean one should be called is_match actually as matches might mean "what is the list of matches in this string". Either that or this one should be called match and then we extend the output type to be a more substantive type later and just assume that everyone's using it as truthiness now. Thoughts?

yuja · 2025-07-06T01:09:39Z

The full completion of this feature to be able to pull out the matching text in question probably involves introducing a new type that represents a match result (and match groups in regex), but doing so requires some rework and semantics design of the string pattern system itself.

For this use case, I think we can add .match/es and .replace. The user might have to write the same regex twice, but that seems okay?

# .match(pat) -> String and/or .matches(path) -> List<String>
# .replace_all(pat, subst) -> String
.match(regex:'#\d+\b").replace_all(regex:'^#(.*)', 'issue-$2')

(match/matches might be a bad name, but I'm not sure.)

Implementation-wise, we'll need StringPattern::to_regex() or something, and use string-based substitution provided by Regex. We could add lambda syntax for subst expression, but that would be more complex, and I don't know if we'll need expressivity provided by arbitrary templates.

https://docs.rs/regex/latest/regex/struct.Regex.html#method.replace

lf- · 2025-07-06T07:24:34Z

The issue I have is that it seems nontrivial to make a to_regex translator for globs, and that IME it is most ideal to have the ability to match groups with a regex. But that's probably going to require creating a language type to be able to extract groups and matches; a regex feature that only does full matches is quite a bit harder to use.

So I'm partially trying to figure out how to incrementally deliver something useful without simply . inventing good regex support in my first code PR to the project heh. I think the path forward on that is calling the function introduced in this PR is_match.

yuja · 2025-07-06T08:16:03Z

The issue I have is that it seems nontrivial to make a to_regex translator for globs,

I think we'll need to switch to the globset crate at some point. The current StringPattern API doesn't have a separate .to_matcher() step, so adapting to GlobSet might not be straightforward. Still I think we can leverage globset to implement .to_regex().

https://docs.rs/globset/0.4.16/globset/struct.Glob.html

Since we already have globset in transitive dependencies, this change helps reduce the amount of dependencies. Another reason is that globset provides a function to convert glob to regex. This is nice because we use globs to match against strings or internal repository paths instead of platform-native paths. The GlobPattern wrapper is boxed because globset::Glob type is relatively big.

Unlike string patterns, backslash-escape syntax isn't forcibly enabled. Fileset globs are constructed from platform-native path inputs.

Closes jj-vcs#6919

This makes it clear that it doesn't give a list of matches.

This is a basic implementation of the same string pattern system as in the revset language. It's currently only used for `string.matches`, so you can now do: ``` "foo".matches(regex:'[a-f]o+') ``` In the future this could be added to more string functions (and e.g. the ability to parse things out of strings could be added). CC: jj-vcs#6893

This allows for any matcher type and allows extracting a capture group by number.

lf- · 2025-07-18T00:11:07Z

I've rebased and rewritten this based on #6967. It should be ready to go from my end minus one code question I couldn't figure out and need help with :)

lf- · 2025-07-18T00:12:13Z

cli/src/template_builder.rs

+                                        todo!(
+                                            "how do I fix this? I can't keep the span here due to \
+                                             lifetimes. this error doesn't appear statically"
+                                        ),


I need help with how to write this diagnostic in a way that actually works

I wouldn't add capture_group argument to .match(). I think we can instead add .replace(pattern, subst), which can be used as description.match(regex:'h(ell)o').replace(regex:'h(ell)o', '$1').

yuja · 2025-07-18T02:44:44Z

cli/src/template_builder.rs

+                                        todo!(
+                                            "how do I fix this? I can't keep the span here due to \
+                                             lifetimes. this error doesn't appear statically"
+                                        ),


I wouldn't add capture_group argument to .match(). I think we can instead add .replace(pattern, subst), which can be used as description.match(regex:'h(ell)o').replace(regex:'h(ell)o', '$1').

yuja · 2025-07-18T02:50:13Z

lib/src/str_util.rs

+    /// XXX: this is kind of problematic for byte regexes as would be created
+    /// by globs. I dunno what to do about that.


I'm going to change the underlying regex type to bytes::Regex to fix bytes handling in diff_contains() revset. Suppose this change is accepted, I think there are two options:

in templater, use bytes::Regex and convert the result back to String. If the matched slice wasn't at unicode boundary, .match() returns an error.

make this to_regex() fail if bytes regex cannot be re-interpreted as a string regex.

I think (1) is simpler.

yuja · 2025-07-18T02:56:51Z

cli/src/commit_templater.rs

@@ -12,6 +12,10 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.

+//! Template environment for `jj log`, `jj evolog` and similar.
+//!
+//! See: <https://jj-vcs.github.io/jj/latest/templates/#commit-keywords>.


nit: I would remove the doc link because the commit templater can have various context types than the Commit type.

yuja · 2025-07-18T02:57:20Z

cli/src/operation_templater.rs

@@ -12,6 +12,10 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.

+//! Templates for `jj op log`.


nit: "Template environment" sounds more correct.

yuja · 2025-07-18T02:57:44Z

cli/src/operation_templater.rs

@@ -51,6 +55,7 @@ pub trait OperationTemplateLanguageExtension {
    fn build_cache_extensions(&self, extensions: &mut ExtensionsMap);
 }

+/// Templates for `jj op log`.


nit: Same here. This isn't a "Template", but "Template environment".

yuja · 2025-07-18T02:58:56Z

cli/src/templater.rs

+/// Implements conversion from any error type to support `expr?` in function
+/// binding. This type doesn't implement `std::error::Error` instead.
+/// <https://github.com/dtolnay/anyhow/issues/25#issuecomment-544140480>


nit: This is an implementation comment. Revert the change.

yuja · 2025-07-18T03:03:03Z

cli/src/templater.rs

+/// Template which, upon being forced (`extract()`ed), will evaluate its
+/// condition and select between a template for the true case and a template
+/// for the false case (which will yield nothing if it is [`None`]).


This sounds more like a textual explanation of the code. Can you simplify? I think it'll be something like "Template which selects output based on a boolean condition."

yuja · 2025-07-18T03:12:17Z

cli/src/template.pest

@@ -88,11 +91,18 @@ formal_parameters = {
  identifier ~ (whitespace* ~ "," ~ whitespace* ~ identifier)* ~ (whitespace* ~ ",")?
  | ""
 }
+string_pattern_identifier = @{ (ASCII_ALPHA | "_") ~ (ASCII_ALPHANUMERIC | "_" | "-")* }


Just a comment. It's nice that you noticed that - is allowed as a pattern prefix whereas - can also be infix/prefix operators. The grammar isn't ambiguous yet, but it's a bit odd. I don't have a good idea to address this potential problem.

yuja · 2025-07-18T03:15:10Z

cli/src/template_builder.rs

+    // Don't write default implementations on these (and certainly not ones
+    // using another function on self): wrapper types that contain a sum type
+    // of a CoreTemplatePropertyKind or a custom type will hit the default case
+    // every time rather than the implementation on CoreTemplatePropertyKind if
+    // a custom implementation is omitted.


nit: I wouldn't add this comment because I don't think it is any special here. We shouldn't add default implementation unless that makes sense.

yuja · 2025-07-18T03:17:45Z

cli/src/template_parser.rs

+                other => {
+                    return Err(TemplateParseError::expression(
+                        format!("Unexpected literal in string pattern: {other:?}"),
+                        span,
+                    ))
+                }


nit: should be panic!()?

yuja · 2025-07-18T03:19:26Z

cli/src/template_parser.rs

+            // XXX(jade): this should probably make the StringPattern
+            // itself here, but because StringPattern is not PartialEq (and
+            // cannot be due to regexes), we defer that to later and Deal With
+            // the AST being slightly less clean (and slightly less validated)
+            // than is ideal.
+            ExpressionNode::new(ExpressionKind::StringPattern { kind, value: text }, span)


I think the current implementation is correct. We aren't doing any revset/fileset/string pattern parsing in the parser. And we might add other kind of patterns (e.g. date pattern) that share the same syntax.

yuja · 2025-07-18T03:21:50Z

cli/src/templater.rs

+impl Template for StringPattern {
+    fn format(&self, formatter: &mut TemplateFormatter) -> io::Result<()> {
+        write!(formatter, "{self}")
+    }
+}
+


nit: shouldn't be needed anymore?

lf- requested a review from a team as a code owner July 5, 2025 02:02

lf- force-pushed the jade/push-nlllwnwltntw branch 4 times, most recently from 8ffafc9 to cc7eb2f Compare July 5, 2025 02:20

lf- commented Jul 5, 2025

View reviewed changes

cli/src/generic_templater.rs Outdated Show resolved Hide resolved

lf- force-pushed the jade/push-nlllwnwltntw branch 2 times, most recently from 3ebcc42 to 4cae57d Compare July 5, 2025 04:36

yuja reviewed Jul 5, 2025

View reviewed changes

cli/src/generic_templater.rs Outdated Show resolved Hide resolved

cli/src/commit_templater.rs Outdated Show resolved Hide resolved

cli/src/template_builder.rs Outdated Show resolved Hide resolved

cli/src/template_parser.rs Outdated Show resolved Hide resolved

lf- force-pushed the jade/push-nlllwnwltntw branch 3 times, most recently from 3ea5b1a to 7c68277 Compare July 5, 2025 23:55

lf- changed the title ~~feat: support string patterns in template language~~ templates: support string patterns in template language Jul 6, 2025

lf- mentioned this pull request Jul 7, 2025

Switch to globset #6919

Open

9999years mentioned this pull request Jul 7, 2025

Docs: Explain why jj doesn't have Git-style branches #6926

Open

10 tasks

yuja added 3 commits July 12, 2025 15:39

fileset: switch to globset

1459715

Unlike string patterns, backslash-escape syntax isn't forcibly enabled. Fileset globs are constructed from platform-native path inputs.

cli: remove remaining users of glob crate

44c3ec5

Closes jj-vcs#6919

lf- mentioned this pull request Jul 17, 2025

template: Add the replace method to string #6998

Open

4 tasks

lf- added 3 commits July 17, 2025 15:30

templates: improve rustdocs

974cb82

str_lib: matches -> is_match

e62ea15

This makes it clear that it doesn't give a list of matches.

lf- force-pushed the jade/push-nlllwnwltntw branch 2 times, most recently from 28917ec to 52da308 Compare July 18, 2025 00:10

templates: add string.match function

a6dbfd7

This allows for any matcher type and allows extracting a capture group by number.

lf- force-pushed the jade/push-nlllwnwltntw branch from 52da308 to a6dbfd7 Compare July 18, 2025 00:10

lf- commented Jul 18, 2025

View reviewed changes

yuja reviewed Jul 18, 2025

View reviewed changes

		/// XXX: this is kind of problematic for byte regexes as would be created
		/// by globs. I dunno what to do about that.

templates: support string patterns in template language #6899

Are you sure you want to change the base?

templates: support string patterns in template language #6899

Conversation

lf- commented Jul 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

lf- commented Jul 5, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

PhilipMetzger commented Jul 5, 2025

Uh oh!

lf- commented Jul 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yuja commented Jul 6, 2025

Uh oh!

lf- commented Jul 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yuja commented Jul 6, 2025

Uh oh!

lf- commented Jul 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yuja Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lf- commented Jul 5, 2025 •

edited

Loading

lf- commented Jul 6, 2025 •

edited

Loading

lf- commented Jul 6, 2025 •

edited

Loading

yuja Jul 18, 2025 •

edited

Loading