[BUG] sudden explosion in memory consumption

[baggepinnen]

I have experience a couple of times the behavior shown in the screenshot below

It's the julia process suddenly eating up all available ram and swap, and it happens even if I'm not running any julia code for the moment. When this instance of the problem occured, I had julia in Juno running (cycle mode not active) and I was editing code without running anything (I might have been resizing the editor, but it sometimes happens when I'm definitely not resizing anything). When I noticed the mouse pointer getting stuck, I saw the juno striped progress bar indicating that juno was up to something. When the swap is full, the julia process is automatically terminated and I regain control of the computer. I know it's the julia process eating the ram as verified by the system monitor.

This has happened 10+ times now, and in different scripts doing completely different things. Debuginfo below
...
actually, when I asked juno for debuginfo, the julia process restarted and the bug immediately occurred again, crashing the entire Ubuntu login session (thankfully the text I had written in this issue was saved somehow)

# Atom:
Version: 1.44.0
Dev Mode: false
Official Release: true
{
  "http_parser": "2.8.0",
  "node": "10.11.0",
  "v8": "6.9.427.31-electron.0",
  "uv": "1.23.0",
  "zlib": "1.2.11",
  "ares": "1.14.0",
  "modules": "69",
  "nghttp2": "1.33.0",
  "napi": "3",
  "openssl": "1.1.0",
  "electron": "4.2.7",
  "chrome": "69.0.3497.128",
  "icu": "62.2",
  "unicode": "11.0",
  "cldr": "33.1",
  "tz": "2019a"
}
# julia-client:
Version: 0.12.3
Config:
{
  "firstBoot": false,
  "juliaOptions": {
    "bootMode": "Basic",
    "workingDir": "/home/fredrikb/"
  },
  "juliaPath": "/home/fredrikb/julia/julia",
  "juliaSyntaxScopes": [
    "source.julia",
    "source.weave.md",
    "source.weave.latex",
    "source.gfm",
    "source.embedded.julia",
    "meta.function.environment.general.latex",
    "meta.embedded.block.source",
    "text.tex.latex"
  ],
  "remoteOptions": {
    "remoteJulia": "export JULIA_NUM_THREADS=6;/home/fredrikb/julia/julia"
  },
  "uiOptions": {
    "cellDelimiter": [
      "##",
      "#---",
      "#%%",
      "# %%",
      "# #"
    ],
    "docsDisplayMode": "inline",
    "enableMenu": true,
    "errorNotifications": false,
    "highlightCells": false,
    "layouts": {
      "console": {
        "defaultLocation": "center",
        "split": "right"
      },
      "defaultPanes": {
        "documentation": false,
        "plotPane": false,
        "workspace": false
      },
      "plotPane": {
        "split": "up"
      },
      "terminal": {
        "defaultLocation": "center",
        "split": "right"
      },
      "workspace": {
        "defaultLocation": "right",
        "split": "no split"
      }
    }
  }
}


# ink:
Version: 0.12.3
Config:
undefined


# uber-juno:
Version: 0.3.0
Config:
{
  "disable": true
}


# language-julia:
Version: 0.19.2
Config:
undefined


# language-weave:
Version: 0.6.7
Config:
undefined


# indent-detective:
Version: 0.4.0
Config:
undefined


# latex-completions:
Version: 0.3.6
Config:
{
  "disableForSelector": ".nothing"
}


# versioninfo():
Julia Version 1.4.0-rc2.0
Commit b99ed72c95 (2020-02-24 16:51 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Core(TM) i7-4550U CPU @ 1.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, haswell)
Environment:
  JULIA_NUM_THREADS = 2
  JULIA_EDITOR = atom  -a

Status `~/.julia/environments/v1.4/Project.toml`
  [14f7f29c] AMD v0.3.1
  [c52e3926] Atom v0.12.9
  [dbf13d8f] AudioClustering v0.1.0 [`~/.julia/dev/AudioClustering`]
  [70b36510] AutomaticDocstrings v0.1.1 [`~/.julia/dev/AutomaticDocstrings`]
  [aae01518] BandedMatrices v0.14.3
  [6e4b80f9] BenchmarkTools v0.5.0
  [336ed68f] CSV v0.5.26
  [aaaa29a8] Clustering v0.13.5
  [3abffc1c] ControlSystemIdentification v0.2.0 [`../../dev/ControlSystemIdentification`]
  [a6e380b2] ControlSystems v0.5.6 [`~/.julia/dev/ControlSystems`]
  [717857b8] DSP v0.6.3
  [a93c6f00] DataFrames v0.20.2
  [bb1859e0] DefaultArrays v1.0.1
  [0c46a032] DifferentialEquations v6.11.0
  [b4f34e82] Distances v0.8.2
  [31c24e10] Distributions v0.22.5
  [ffbed154] DocStringExtensions v0.8.1
  [f6369f11] ForwardDiff v0.10.9
  [994df76e] ForwardDiff2 v0.2.2
  [28b8d3ca] GR v0.47.0
  [891a1506] GaussianProcesses v0.11.2 [`~/.julia/dev/GaussianProcesses`]
  [c27321d9] Glob v1.2.0
  [775960c6] Grep v0.2.0
  [dbf13d8f] HDBSCAN v0.1.0 [`~/.julia/dev/HDBSCAN`]
  [e91730f6] Hungarian v0.6.0
  [7073ff75] IJulia v1.20.0
  [916415d5] Images v0.22.0
  [e5e0dc1b] Juno v0.8.1
  [26dcc766] LPVSpectral v0.1.5
  [3abffc1c] LTVModels v0.1.0 [`~/.julia/dev/LTVModels`]
  [5078a376] LazyArrays v0.15.0
  [82ba68a1] LazyWAVFiles v1.0.0 [`~/.julia/dev/LazyWAVFiles`]
  [6e84ca90] LengthChannels v1.0.0 [`~/.julia/dev/LengthChannels`]
  [093fc24a] LightGraphs v1.3.1
  [d3d80556] LineSearches v7.0.1
  [d9d29d28] LowLevelParticleFilters v0.4.1 [`~/.julia/dev/LowLevelParticleFilters`]
  [f0e99cf1] MLBase v0.8.0
  [eff96d63] Measurements v2.2.0
  [0987c9cc] MonteCarloMeasurements v0.8.3
  [6f286f6a] MultivariateStats v0.7.0
  [872c559c] NNlib v0.6.6
  [b8a86587] NearestNeighbors v0.4.4
  [429524aa] Optim v0.20.1
  [1dea7af3] OrdinaryDiffEq v5.29.0
  [9b87118b] PackageCompiler v1.1.0
  [2dcacdae] ParallelDataTransfer v0.5.0
  [18e31ff7] Peaks v0.1.0
  [58dd65bb] Plotly v0.3.0
  [91a5bcdd] Plots v0.27.1
  [a725b495] ProximalOperators v0.10.3
  [438e738f] PyCall v1.91.4
  [d330b81b] PyPlot v2.8.2
  [1a8c2f83] Query v0.12.2
  [77f5812f] SingularSpectrumAnalysis v0.3.1 [`~/.julia/dev/SingularSpectrumAnalysis`]
  [8ce77f84] Soss v0.7.0
  [2b0dec9d] SpectralDistances v0.1.0 [`../../dev/SpectralDistances`]
  [90137ffa] StaticArrays v0.12.1
  [2913bbd2] StatsBase v0.32.2
  [f3b207a7] StatsPlots v0.12.0
  [a2db99b7] TextAnalysis v0.7.0 [`~/.julia/dev/TextAnalysis`]
  [dbf13d8f] ThreadTools v0.1.1 [`~/.julia/dev/ThreadTools`]
  [028f657a] TotalLeastSquares v1.4.1 [`~/.julia/dev/TotalLeastSquares`]
  [c4f8c510] UMAP v0.1.5
  [c4a57d5a] UnsafeArrays v0.4.0
  [8149f6b0] WAV v1.0.3

Some more info:

• It has happened while editing both scripts in module main, and in code files that are part of a loaded package.
• When it happens, some times it's enough to restart atom. On a few occasions, this has not been enough and the bug reappeared immediately. Restarting the computer makes it run again for a couple of hours or something.

[baggepinnen]

This is the file I was editing while it happened
https://gist.github.com/baggepinnen/52e27ec6001b758d5e8ac7c54ee432c1
nothing special in it as far as I can see. I was editing the function in the bottom

[pfitzseb]

I still didn't have any luck reproducing this. My best bet is that CSTParser and/or CodeTools run into infinite recursion and crash everthing, but not sure what we can do about that...

[baggepinnen]

Can I disable one of them and see if the problem goes away? It has messed up 4 sessions today alone, so I'm eager to help getting it resolved.

[pfitzseb]

pkg> dev Atom and apply

diff --git a/src/completions.jl b/src/completions.jl
index 397a457..2830fa5 100644
--- a/src/completions.jl
+++ b/src/completions.jl
@@ -72,7 +72,7 @@ function basecompletionadapter(

   # initialize suggestions with local completions so that they show up first
   prefix = line[replace]
-  comps = if force || !isempty(prefix)
+  comps = if false # force || !isempty(prefix)
     filter!(let p = prefix
       c -> startswith(c[:text], p)
     end, localcompletions(context, row, column, prefix))
diff --git a/src/outline.jl b/src/outline.jl
index 496d6e1..a695a68 100644
--- a/src/outline.jl
+++ b/src/outline.jl
@@ -6,11 +6,11 @@ handle("updateeditor") do data
         updateSymbols || true
     ] = data

-    return try
-        todict.(updateeditor(text, mod, path, updateSymbols))
-    catch err
+    # return try
+    #     todict.(updateeditor(text, mod, path, updateSymbols))
+    # catch err
         []
-    end
+    # end
 end

 # NOTE: update outline and symbols cache all in one go

to disable local completions and the outline. You should be able to use something like

diff --git a/src/modules.jl b/src/modules.jl
index 166c9c3..1dbaa55 100644
--- a/src/modules.jl
+++ b/src/modules.jl
@@ -285,34 +285,40 @@ end
 const MAIN_MODULE_LOCATION = Ref{Tuple{String, Int}}(moduledefinition(Main))

 handle("module") do data
-  main, sub = modulenames(data, cursor(data))
-
-  mod = CodeTools.getmodule(main)
-  smod = CodeTools.getmodule(mod, sub)
-
-  if main == "Main" && sub == ""
-    MAIN_MODULE_LOCATION[] = get!(data, "path", ""), data["row"]
-  end
-
-  loaded_mods = copy(Base.loaded_modules_array())
-  if main == "Main"
-    filter!(m -> string(m) == sub, loaded_mods)
-    if !isempty(loaded_mods)
-      return Dict(
-               :main => string(loaded_mods[1]),
-               :sub  => "",
-               :inactive => false,
-               :subInactive => false
-             )
-    end
-  end
-
+  # main, sub = modulenames(data, cursor(data))
+  #
+  # mod = CodeTools.getmodule(main)
+  # smod = CodeTools.getmodule(mod, sub)
+  #
+  # if main == "Main" && sub == ""
+  #   MAIN_MODULE_LOCATION[] = get!(data, "path", ""), data["row"]
+  # end
+  #
+  # loaded_mods = copy(Base.loaded_modules_array())
+  # if main == "Main"
+  #   filter!(m -> string(m) == sub, loaded_mods)
+  #   if !isempty(loaded_mods)
+  #     return Dict(
+  #              :main => string(loaded_mods[1]),
+  #              :sub  => "",
+  #              :inactive => false,
+  #              :subInactive => false
+  #            )
+  #   end
+  # end
+  #
+  # return Dict(
+  #          :main => main,
+  #          :sub  => sub,
+  #          :inactive => (mod==nothing),
+  #          :subInactive => smod==nothing
+  #        )
   return Dict(
-           :sub  => sub,
-           :main => main,
-           :inactive => (mod==nothing),
-           :subInactive => smod==nothing
-           :sub  => sub,
-         )
+    :main => "Main",
+    :sub  => "",
+    :inactive => false,
+    :subInactive => false
-           :inactive => (mod==nothing),
-           :subInactive => smod==nothing
+  )
-         )
 end
+    :main => "Main",


+    :sub  => "",
+    :inactive => false,
@@ -321,9 +327,10 @@ find all modules
 =#
+    :subInactive => false
+  )

 handle("allmodules") do
 end
-  sort!([string(m) for m in CodeTools.allchildren(Main)])

+  # sort!([string(m) for m in CodeTools.allchildren(Main)])
+  []
 end

 handle("ismodule") do mod

-  return CodeTools.getthing(mod) isa Module || mod in string.(Base.loaded_modules_array())
+  return false # CodeTools.getthing(mod) isa Module || mod in string.(Base.loaded_modules_array())
 end
@@ -321,9 +327,10 @@ find all modules
 =#

 handle("allmodules") do
-  sort!([string(m) for m in CodeTools.allchildren(Main)])
+  # sort!([string(m) for m in CodeTools.allchildren(Main)])
+  []
 end

 handle("ismodule") do mod
-  return CodeTools.getthing(mod) isa Module || mod in string.(Base.loaded_modules_array())
+  return false # CodeTools.getthing(mod) isa Module || mod in string.(Base.loaded_modules_array())
 end

to disable most CodeTools features.

Try the first patch first though, the other stuff was in Juno for a long time.

[baggepinnen]

Done! I'll report back in a while when I know if it works or not. (And if I don't, it might mean that the problem has been solved and I've forgotten about it ^^)

[baggepinnen]

I can confirm that it does not solve the problem of the REPL suddenly freezes and stops responding while evaluation in the editor keeps working.

[pfitzseb]

Ok, but the memory leak issue didn't happen again?

[baggepinnen]

So far it has not appeared which is a good sign. I would give it a few more days though to be sure.

[aviatesk]

well, I am VERY MUCH interested in for what script you had encontered the memory leak.
Every file that was being open at that time can be to blame, so I would appreciate if you report them when you see the symptom again.

[pfitzseb]

Any file that's open and any file that's reachable from any of those, yes...

[aviatesk]

😓

[baggepinnen]

I have used Juno extensively a couple of days now without seeing the explosion of memory use, it seems likely that the first patch @pfitzseb provided above resolved the issue. I did not apply the second patch.