Recent change in Julia syntax highlighting

[pfitzseb]

We recently got a comment about a regression in Julia syntax highlighting: JuliaEditorSupport/atom-language-julia#218 (comment):

transpose' + tokenized + as + a + string' + instead

As far as I can tell, nothing about the pulled in submodule changed in the most recent linguist release apart from an update for a dev dependency.

Tests for this functionality still pass, but aren't fully relevant for github.com, since I'm using vscode-oniguruma there instead of whatever PCRE flavor seems to be used here.

Anyways, did something change about the actual highlighter further up in the toolchain? And is there a reasonable way for me to test against the GH toolchain as well?

[lildude]

Anyways, did something change about the actual highlighter further up in the toolchain?

No. All we did was pull the latest version of the submodule. Any problems we encounter with grammars are recorded in #3924 and Julia is in there so may be related.

And is there a reasonable way for me to test against the GH toolchain as well?

Unfortunately not. The only thing is our grammar compiler which essentially just does some basic PCRE validation and then converts the grammar into JSON which you can view in the tarball attached to each release (this is the exact file the syntax highlighter uses).

[pfitzseb]

Ah, perfect, didn't see that the compiled grammars are in the release.

Turns out that it's using a template file, not the full grammar. Given that there doesn't seem to be any reference to a specific file anywhere in this repo, do you just pick a (basically random) file that defines the correct scope?

Is there a straightforward way to specify a file as the source of truth here, or should I reorganize the upstream (and if so, how)?

[lildude]

The grammar compiler expects the grammar files to be in the location commonly used by the various editors (code here) and expects the file to contain a scope. If there are multiple files with a scope, it'll use the last (code here). As such there isn't a direct way of specifying a file, but you can effectively do so by ensuring there is only one file with a scope in the location the compiler/editor expects the grammar to be.

[pfitzseb]

Alright, thanks for the help! Cleaned up the repo in JuliaEditorSupport/atom-language-julia#292.

Any chance for a patch release? 😅

[DecimalTurn]

@pfitzseb If you wish to catch those issues earlier in the future, it might be good to have a CI check in place the like one below (inspired from here)(:

# This github action checks if the grammar(s) in this repo will compile with the latest version of linguist.

name: Linguist Compilation Check

env:
  GRAMMAR_REPO: atom-language-julia

on:
  push:
    branches:
      - master
  pull_request:

jobs:
  compile-check:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout linguist
        uses: actions/checkout@v6
        with:
          repository: github-linguist/linguist

      - name: Checkout grammar repo
        uses: actions/checkout@v6
        with:
          path: vendor/grammars/${{ env.GRAMMAR_REPO }}
          clean: false

      - name: Compile linguist grammar
        run: script/grammar-compiler add vendor/grammars/${{ env.GRAMMAR_REPO }}

[pfitzseb]

Hm, yes, with some modification that should work. Thanks for the suggestion!