Crashing on Julia 1.11-rc1

[marcelo-g-simas]

When I try to use TestReports.jl on Julia 1.11.0-rc1 (2024-06-25) I get a crash before any tests run. Here's what happens:

julia> using TestReports; TestReports.test("WesVAR")
[ Info: Testing WesVAR
ERROR: KeyError: key "REPL" not found
Stacktrace:
  [1] getindex
    @ ./dict.jl:477 [inlined]
  [2] Base.Precompilation.ExplicitEnv(envpath::String)
    @ Base.Precompilation ./precompilation.jl:201
  [3] Base.Precompilation.ExplicitEnv()
    @ Base.Precompilation ./precompilation.jl:29
  [4] precompilepkgs(pkgs::Vector{String}; internal_call::Bool, strict::Bool, warn_loaded::Bool, timing::Bool, _from_loading::Bool, configs::Pair{Cmd, Base.CacheFlags}, io::Base.TTY, fancyprint::Bool)
    @ Base.Precompilation ./precompilation.jl:367
  [5] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:2377
  [6] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2216
  [7] #invoke_in_world#3
    @ ./essentials.jl:1077 [inlined]
  [8] invoke_in_world
    @ ./essentials.jl:1074 [inlined]
  [9] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2207
 [10] macro expansion
    @ ./loading.jl:2146 [inlined]
 [11] macro expansion
    @ ./lock.jl:273 [inlined]
 [12] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2103
 [13] #invoke_in_world#3
    @ ./essentials.jl:1077 [inlined]
 [14] invoke_in_world
    @ ./essentials.jl:1074 [inlined]
 [15] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2096
ERROR: TestReports failed to generate the report.
See error log above.

[mharradon]

I'm also running into this issue on julia 1.11.0.

[KristofferC]

TestReports.jl generates an invalid manifest when it creates the test manifest that it uses. The issue is:

[[deps.Pkg]]
deps = ["Artifacts", "Dates", "Downloads", "FileWatching", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "Random", "SHA", "TOML", "Tar", "UUIDs", "p7zip_jll"]
uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"
version = "1.11.0"
weakdeps = ["REPL"]

    [deps.Pkg.extensions]
    REPLExt = "REPL"

If there is no [[deps.REPL]] in the manifest this weak dependency needs to be saved in "expanded form" (which includes the UUID) as this:

[[deps.Pkg]]
deps = ["Artifacts", "Dates", "Downloads", "FileWatching", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "Random", "SHA", "TOML", "Tar", "UUIDs", "p7zip_jll"]
uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"
version = "1.11.0"

    [deps.Pkg.extensions]
    REPLExt = "REPL"

    [deps.Pkg.weakdeps]
    REPL = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"

The Pkg project writer handles this but this package instead chooses to directly print the manifest dictionary which circumvents that.

[mmiller-max]

Hey, please can you see if this is fixed on the latest version (v"1.2.0")? Now switched to using TestEnv which has been updated for 1.11. Thanks

[mharradon]

I'm getting a similar error still:

Precompiling project...
    877.3 ms  ✓ CoverageTools
   1197.4 ms  ✓ Coverage
  2 dependencies successfully precompiled in 2 seconds. 367 already precompiled.
   Resolving package versions...
   Installed EzXML ─────── v1.2.0
   Installed TestEnv ───── v1.102.0
   Installed TestReports ─ v1.2.0
      Compat entries added for TestReports
    Updating `/builds/project/project/Project.toml`
  [dcd651b4] + TestReports v1.2.0
    Updating `/builds/project/project/Manifest.toml`
  [8f5d6c58] + EzXML v1.2.0
  [1e6cf692] + TestEnv v1.102.0
  [dcd651b4] + TestReports v1.2.0
Precompiling project...
   1259.9 ms  ✓ EzXML
   1394.7 ms  ✓ TestEnv
   1867.7 ms  ✓ TestReports
  3 dependencies successfully precompiled in 4 seconds. 369 already precompiled.
[ Info: Testing PACKAGE
ERROR: KeyError: key "REPL" not found
Stacktrace:
  [1] getindex
    @ ./dict.jl:477 [inlined]
  [2] Base.Precompilation.ExplicitEnv(envpath::String)
    @ Base.Precompilation ./precompilation.jl:201
  [3] Base.Precompilation.ExplicitEnv()
    @ Base.Precompilation ./precompilation.jl:29
  [4] precompilepkgs(pkgs::Vector{String}; internal_call::Bool, strict::Bool, warn_loaded::Bool, timing::Bool, _from_loading::Bool, configs::Pair{Cmd, Base.CacheFlags}, io::Base.PipeEndpoint, fancyprint::Bool)
    @ Base.Precompilation ./precompilation.jl:367
  [5] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:2466
  [6] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2300
  [7] #invoke_in_world#3
    @ ./essentials.jl:1088 [inlined]
  [8] invoke_in_world
    @ ./essentials.jl:1085 [inlined]
  [9] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2287
 [10] macro expansion
    @ ./loading.jl:2226 [inlined]
 [11] macro expansion
    @ ./lock.jl:273 [inlined]
 [12] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2183
 [13] #invoke_in_world#3
    @ ./essentials.jl:1088 [inlined]
 [14] invoke_in_world
    @ ./essentials.jl:1085 [inlined]
 [15] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2176
ERROR: TestReports failed to generate the report.
See error log above.

[KristofferC]

The issue is in this function

TestReports.jl/src/runner.jl

Lines 78 to 109 in ef6dd38

	function make_testreports_environment(manifest)
	all_deps = get_deps(manifest, "TestReports")
	push!(all_deps, "TestReports")
	if VERSION >= v"1.7.0"
	new_manifest = Dict{String, Any}()
	new_manifest["deps"] = Dict(pkg => manifest["deps"][pkg] for pkg in all_deps)
	new_manifest["julia_version"] = manifest["julia_version"]
	new_manifest["manifest_format"] = manifest["manifest_format"]
	new_project = Dict(
	"deps" => Dict(
	"Test" => new_manifest["deps"]["Test"][1]["uuid"],
	"TestReports" => new_manifest["deps"]["TestReports"][1]["uuid"]
	)
	)
	else
	new_manifest = Dict(pkg => manifest[pkg] for pkg in all_deps)
	new_project = Dict(
	"deps" => Dict(
	"Test" => new_manifest["Test"][1]["uuid"],
	"TestReports" => new_manifest["TestReports"][1]["uuid"]
	)
	)
	end
	testreportsenv = mktempdir()
	open(joinpath(testreportsenv, "Project.toml"), "w") do io
	Pkg.TOML.print(io, new_project)
	end
	open(joinpath(testreportsenv, "Manifest.toml"), "w") do io
	Pkg.TOML.print(io, new_manifest, sorted=true)
	end
	return testreportsenv
	end

.

It creates TOML dictionaries and prints them but fails to uphold the requirements of a proper manifest (#121 (comment)).

[victoryforphil]

Was there every a workaround or update to this issue? Running into this after the Julia 1.11 update.

[oxinabox]

@omus might have a fix

[pjaap]

We encountered the same problem. However, only if TestReports is in the global environment @v1.11. If we add the package to the to-be-tested package, everything works as expected.

[oxinabox]

This is weird, but we do mess around with internals a lot

[KristofferC]

Not weird, already explained in #121 (comment) and #121 (comment).

[pjaap]

I uploaded a hotfix PR, that implements the missing Manifest entry, as pointed out in #121 (comment)

I also tried to provide a minimal working example in the PR #125 to trigger the issue here.

[oxinabox]

The Pkg project writer handles this but this package instead chooses to directly print the manifest dictionary which circumvents that.

Would be happy to see a PR that changes this to use the Pkg project writer (at least on v1.11+ or wherever that was introduced)