.ipynb files in v1.12 re-JIT-compiling already precompiled code

[mattsignorelli]

This issue does NOT appear with Julia v1.11. First develop the package:

import Pkg; Pkg.develop("SciBmad")

Then, using Julia v1.12, run the following script in both a fresh terminal and in an .ipynb file (I use Jupyter lab) with a fresh kernel:

using SciBmad 

K1 = 0.36;
K2 = 0.1;
qf = Quadrupole(Kn1=DefExpr(() -> +K1), L=0.5);
sf = Sextupole(Kn2=DefExpr(() -> +K2), L=0.2);
d  = Drift(L=0.6);
b  = SBend(L=6.0, angle=pi/132);
qd = Quadrupole(Kn1=DefExpr(() -> -K1), L=0.5);
sd = Sextupole(Kn2=DefExpr(() -> -K2), L=0.2);
fodo_line = [qf, sf, d, b, d, qd, sd, d, b, d];
fodo = Beamline(fodo_line, species_ref=Species("electron"), E_ref=18e9);

@time t = twiss(fodo); # This function has been precompiled using PrecompileTools.jl

In the .ipynb, the output is:

1.936909 seconds (4.12 M allocations: 207.496 MiB, 1.92% gc time, 99.41% compilation time: 68% of which was recompilation)

In the terminal, the output is:

0.378638 seconds (455.82 k allocations: 23.735 MiB, 96.82% compilation time: <1% of which was recompilation)

[JamesWrigley]

My guess is that something in Ijulia or its dependencies is causing invalidations. Do you see the same amount of recompilation if you run import IJulia in the terminal beforehand? If you can find the invalidated methods with SnoopCompile that'd be great: https://juliadebug.github.io/SnoopCompile.jl/dev/tutorials/invalidations/

[mattsignorelli]

Unfortunately I don't have the time at the moment... we are doing a small debut of our software next week at a workshop. I will point our new users to the LTS version of Julia instead where this issue doesn't occur

[JamesWrigley]

Reduced to:

struct Foo <: Base.AbstractPipe end
Base.lock(::Foo) = nothing

using SciBmad 

K1 = 0.36;
K2 = 0.1;
qf = Quadrupole(Kn1=DefExpr(() -> +K1), L=0.5);
sf = Sextupole(Kn2=DefExpr(() -> +K2), L=0.2);
d  = Drift(L=0.6);
b  = SBend(L=6.0, angle=pi/132);
qd = Quadrupole(Kn1=DefExpr(() -> -K1), L=0.5);
sd = Sextupole(Kn2=DefExpr(() -> -K2), L=0.2);
fodo_line = [qf, sf, d, b, d, qd, sd, d, b, d];
fodo = Beamline(fodo_line, species_ref=Species("electron"), E_ref=18e9);

@time t = twiss(fodo);

IJulia defines Base.lock for the IJuliaStdio type, and that causes a bunch of invalidations that I assume hit SciBmad particularly hard:

julia> using SnoopCompileCore

julia> invs, trees = begin
           using SnoopCompileCore
           invs = @snoop_invalidations import IJulia
           using SnoopCompile, AbstractTrees
           trees = invalidation_trees(invs)
           invs, trees
       end;

julia> trees
5-element Vector{SnoopCompile.MethodInvalidations}:
 rebinding Core.Binding(:(Main.Revise), #undef, Core.BindingPartition(#undef, 0x0000000000009727, 0xffffffffffffffff, #undef, 0x0000000000000009), Any[CodeInstance for MethodInstance for IJulia.run_kernel()], 0x00) invalidated:
   mt_backedges: 1: signature Main.Revise triggered MethodInstance for IJulia.run_kernel() (0 children)

 inserting readline(io::IJulia.IJuliaStdio) @ IJulia ~/git/IJulia.jl/src/stdio.jl:227 invalidated:
   backedges: 1: superseding readline(s::IO; keep) @ Base io.jl:619 with MethodInstance for readline(::IO) (1 children)

 inserting unsafe_convert(::Type{Ptr{UInt8}}, zmsg::Base.RefValue{ZMQ.lib.zmq_msg_t}) @ ZMQ ~/.julia/packages/ZMQ/6FLvs/src/_message.jl:32 invalidated:
   mt_backedges: 1: signature Tuple{typeof(Base.unsafe_convert), Type{Ptr{T}} where T, Base.RefValue} triggered MethodInstance for Base._reinterpret(::Type, ::UInt16) (0 children)
                 2: signature Tuple{typeof(Base.unsafe_convert), Type{Ptr{T}} where T, Base.RefValue} triggered MethodInstance for Base._reinterpret_padding(::Type, ::UInt16) (3 children)
   1 mt_cache

 inserting in(key_value::Pair, io::IJulia.IJuliaStdio) @ IJulia ~/git/IJulia.jl/src/stdio.jl:19 invalidated:
   backedges: 1: superseding in(key_value::Pair, io::IO) @ Base show.jl:428 with MethodInstance for in(::Pair{Symbol, TypeVar}, ::IO) (4 children)

 inserting lock(io::IJulia.IJuliaStdio) @ IJulia ~/git/IJulia.jl/src/stdio.jl:17 invalidated:
   backedges: 1: superseding lock(::IO) @ Base io.jl:26 with MethodInstance for lock(::IO) (1370 children)

Not sure why it's so much worse on 1.12 though.