Use `enso.dev.insight` property to turn Insight on #11385

JaroslavTulach · 2024-10-23T07:15:46Z

Pull Request Description

This PR provides development only (e.g. no long term) API to enable (simple) usage of GraalVM Insight in the IDE and also in CLI. The insight script is sent to the internal EPB language responsible for interop with JavaScript, Python and other languages. Such integration revealed an inconsistency as insight object in the Insight script wasn't visible under the right name. Fixed by f50329a - which also changes behavior of foreign js - self is available (next to already available this) in instance foreign methods.

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

Insight & Enso Manual has been provided as part of this PR
All code follows the
Scala,
Java,
Unit tests have been written where possible.

…M Insight in the IDE as well as CLI

…cified

JaroslavTulach · 2024-10-23T07:25:01Z

Insight & Enso Manual

Create Insight Script

Simple script to start with is for example methods.js:

print("Initializing Insight: " + insight);
insight.on("enter", function(ctx) {
  print("Calling " + ctx.name);
}, {
  roots: true
});

insight variable is only available because of f50329a fix.

Learn more about writing Insight Scripts:

GraalVM Insight Manual
GraalVM Insight API docs

Usage in CLI

Execute either a script or use REPL:

sbt:enso> runEngineDistribution --vm.D=enso.dev.insight=methods.js --repl
[info] Executing enso/built-distribution/enso-engine-*/enso-*/bin/enso --vm.D=enso.dev.insight=methods.js --repl
Initializing Insight: [object Object]
Calling Internal_Repl_Module___::Internal_Repl_Module___::internal_repl_entry_point___
> "Ahoj".reverse
Calling Standard.Base.Data.Text.Extensions::Standard.Base.Data.Text.Text::reverse
Calling Text.reverse.iterate
Calling Text.reverse.iterate
Calling Text.reverse.iterate
Calling Text.reverse.iterate
Calling Text.reverse.iterate
>>> johA

Usage in IDE

Create an Insight script like the methods.js above and launch project-manager with additional options:

enso$ ENSO_JVM_OPTS=-Denso.dev.insight=`pwd`/methods.js ./project-manager

use absolute path to reference the Insight script file. In another terminal launch the IDE:

enso$ corepack pnpm run dev:gui

open a project. Output of your Insight script will be displayed in the project-manager console whenever IDE requests the engine to execute something. Alternatively one can use sbt to execute the project manager:

enso$ ENSO_JVM_OPTS=-Denso.dev.insight=`pwd`/methods.js sbt --java-home /graalvm/ runProjectManagerDistribution

with the above project one gets following output:

Initializing Insight: [object Object]
Calling local.NewProject4.Main::local.NewProject4.Main::main
Calling Standard.Base.Data.Vector::Standard.Base.Data.Vector.Vector::length
Calling Standard.Base.Data.Text.Encoding::Standard.Base.Data.Text.Encoding.Encoding.type::utf_32_le
Calling Standard.Base.Data.Text.Encoding::Standard.Base.Data.Text.Encoding.Encoding::Value
Calling <inline_source>.Standard.Visualization.Helpers
Calling Standard.Visualization.Helpers::Standard.Base.Any.Any::default_visualization
Calling Standard.Visualization.Id::Standard.Visualization.Id.Id.type::table
Calling Standard.Visualization.Id::Standard.Visualization.Id.Id::Builtin
Calling Standard.Visualization.Id::Standard.Visualization.Id.Id::to_js_object
...

modification of the code causes re-execution of the code and produces more messages. E.g. GraalVM Insight is active.

Developer's Workflow

When you make a change to the Insight script, there is no need to restart everything, just:

save changes to the same Insight script
close the IDE tab with opened project
open the project again

The new script will be used. Enough for now or do you want more?

JaroslavTulach · 2024-10-23T07:53:46Z

Quirks

The biggest problem right now is multi-threaded execution. CLI is fine (as it is single threaded), but IDE tends to sometimes execute the program in parallel and Graal.js doesn't like it. Graal.js insists on enforcing single threaded ness. That causes issue that we need to work around:

Allow control of throwDeniedThreadAccess via TruffleContext.threadAccessDeniedHandler oracle/graal#8266 might help - but it is not in an official release yet (as of Oct 2024)
write your insight script in Python as demonstrate here
fix the language server to avoid multi threaded ness - we shouldn't be using multiple thread anyway, our code isn't really ready for that (yet)

None of the above shall stop us from integrating this -Denso.dev.insight support. Quite opposite: by having it in we can start experimenting with insights written in JavaScript and discover quirks that prevent us to put it into production.

Less Damage than Expected

I was experiencing multi threaded execution problems with the simple Insight script that instruments invocations of all methods in any file, including methods in visualizations. That seems to cause issues.

However the primary use-case we have is to instrument statements in main function of the Main.enso file. E.g. something done by Detect Assign Statements script - I haven't seen any problems while doing that. E.g. restricting the application of Insights to the user file seems to work in the IDE even right now.

JaroslavTulach · 2024-10-23T08:40:28Z

engine/common/src/main/java/org/enso/common/ContextInsightSetup.java

+    var instrument = ctx.getEngine().getInstruments().get("insight");
+    @SuppressWarnings("unchecked")
+    var insight = (Function<Source, AutoCloseable>) instrument.lookup(Function.class);
+    insightHandle = insight.apply(insightSrc);


We could detect changes to the insightFile via filesystem watcher and re-initialize the Insight script automatically. Just:

insightHandle.close() to disable the currently active insights

load the new content

insight.apply(theNewSrc)

that way the speed of development would be closer to JavaScript development experience.

JaroslavTulach · 2024-10-23T08:49:06Z

Useful Scripts: Detect Assign Statements

Following script attaches to main function statements and prints out values of all local variables that are initialized together with location in the Enso code and information about just assigned local variable:

print("Initializing Insight: " + insight);

insight.on("return", function(ctx, frame) {
  let equals = ctx.characters.indexOf('\n') == -1 ? ctx.characters.indexOf('=') : -1;
  let text = equals == -1 ? "" : `variable ${ctx.characters.substring(0, equals)} has just assigned`;
  print(`At ${ctx.name}:${ctx.line} ${text}`);
  for (let p in frame) {
    let v = frame[p]
    if (v != null) {
      print(` property ${p} has value ${v}`);
    }
  }
}, {
  statements: true,
  rootNameFilter: ".*main"
});

a sample output may look like:

At local.NewProject4.Main::local.NewProject4.Main::main:12 variable vector1  has just assigned
 property vector1 has value 1,2,32,13
At local.NewProject4.Main::local.NewProject4.Main::main:13 variable integer1  has just assigned
 property integer1 has value 4
 property vector1 has value 1,2,32,13
At local.NewProject4.Main::local.NewProject4.Main::main:14 variable node1  has just assigned
 property node1 has value 11,12,42,23
 property integer1 has value 4
 property vector1 has value 1,2,32,13
At local.NewProject4.Main::local.NewProject4.Main::main:15 variable node2  has just assigned
 property node1 has value 11,12,42,23
 property integer1 has value 4
 property node2 has value 22,24,84,46
 property vector1 has value 1,2,32,13

…efined in GraalVM Insight scripts using JavaScript. That also means self argument is available next to this in foreign js functions.

jdunkerley

approving the libs bit

enso-bot · 2024-10-24T02:06:12Z

Jaroslav Tulach reports a new STANDUP for yesterday (2024-10-23):

Progress: .

enso.dev.insight property PR: Use enso.dev.insight property to turn Insight on #11385
fighting with instabilities while debugging: More robust enso4igv when "find in projects" invoked #11384
value with multiple types: https://github.com/orgs/enso-org/discussions/11366#discussioncomment-11028066
meetings, standup, AMA:
@Tail_Case & Array_Like_Helpers
review safepoints: Don't cancel aborted jobs immediately #11375 (review)
avoid IR when generating bytecode: Investigate usage of bytecode interpreter #11328
reporting bug for GraalVM native-image: [Native Image] Missing support for ServiceLoader services registered via module-info oracle/graal#9952 It should be finished by 2024-10-25.

GitHub
Better UX for "type classes" in the IDE · enso-org · Discussion #11366
Using Enso Type Classes in the IDE There is a blueprint describing how to write Type Classes in Enso. Can such type classes be used in the IDE? there is nothing special on the actual type class its...

…sightDev11159

4e6 · 2024-10-24T08:54:54Z

Let's merge it!

Akirathan · 2024-10-23T11:12:54Z

engine/common/src/main/java/org/enso/common/ContextInsightSetup.java

+    }
+    var insightSrc =
+        Source.newBuilder("epb", insightFile.toFile())
+            .content(language + ":0#" + insightCode)


Where is this cryptic :0# string constant comming from? What does it mean?

epb is just an internal language. As such the documentation is provided on the level of tests https://github.com/enso-org/enso/pull/11385/files#diff-208ccc44a38434f59f03b9e2f77d891939bcd11de260a8a9c00c2e652fa3df5dR25 and sibling source code only.

The support is primarily for foreign functions. Language is language. The number is the line number where the script is located in the original file. The text behind # is the code of the foreign function.

We are re-using epb as it has special support for invoking JavaScript in its own inner context to workaround the single threaded limitation.

JaroslavTulach added 2 commits October 23, 2024 09:11

Specify -Denso.dev.insight=insightScript.js property to enable GraalV…

1a02b7c

…M Insight in the IDE as well as CLI

Sent JavaScript print to stdout when enso.dev.insight property is spe…

1cd842b

…cified

JaroslavTulach added the CI: No changelog needed Do not require a changelog entry for this PR. label Oct 23, 2024

JaroslavTulach requested a review from Frizi October 23, 2024 07:15

JaroslavTulach self-assigned this Oct 23, 2024

JaroslavTulach requested review from 4e6, hubertp and Akirathan as code owners October 23, 2024 07:15

JaroslavTulach mentioned this pull request Oct 23, 2024

Deliver expression updates via Ydoc AST elements #11159

Open

JaroslavTulach requested a review from radeusgd October 23, 2024 07:22

JaroslavTulach changed the title ~~Use enso.dev.insight=insightScript.js property to play with GraalVM Insight~~ Use enso.dev.insight=insightScript.js property to turn Insight on Oct 23, 2024

JaroslavTulach changed the title ~~Use enso.dev.insight=insightScript.js property to turn Insight on~~ Use enso.dev.insight=in.js property to turn Insight on Oct 23, 2024

JaroslavTulach changed the title ~~Use enso.dev.insight=in.js property to turn Insight on~~ Use enso.dev.insight property to turn Insight on Oct 23, 2024

JaroslavTulach commented Oct 23, 2024

View reviewed changes

JaroslavTulach added 2 commits October 23, 2024 11:39

Make the error messages more verbose

b3259ff

Don't throw away the first argument. That means insight variable is d…

f50329a

…efined in GraalVM Insight scripts using JavaScript. That also means self argument is available next to this in foreign js functions.

JaroslavTulach requested review from jdunkerley, GregoryTravis, AdRiley and marthasharkey as code owners October 23, 2024 14:10

No need for this prelude. insight variable is always defined.

954e628

jdunkerley approved these changes Oct 23, 2024

View reviewed changes

JaroslavTulach requested a review from farmaazon October 24, 2024 02:21

Merge remote-tracking branch 'origin/develop' into wip/jtulach/EnsoIn…

a3938bf

…sightDev11159

JaroslavTulach added the CI: Clean build required CI runners will be cleaned before and after this PR is built. label Oct 24, 2024

4e6 approved these changes Oct 24, 2024

View reviewed changes

Akirathan reviewed Oct 24, 2024

View reviewed changes

JaroslavTulach merged commit fe45da9 into develop Oct 24, 2024
42 checks passed

JaroslavTulach deleted the wip/jtulach/EnsoInsightDev11159 branch October 24, 2024 11:56

This was referenced Nov 4, 2024

Reload insight script on file change #11415

Merged

Allow to execute Y.js and Insight together #11477

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use `enso.dev.insight` property to turn Insight on #11385

Use `enso.dev.insight` property to turn Insight on #11385

JaroslavTulach commented Oct 23, 2024 •

edited

Loading

JaroslavTulach commented Oct 23, 2024 •

edited

Loading

JaroslavTulach commented Oct 23, 2024 •

edited

Loading

JaroslavTulach Oct 23, 2024

JaroslavTulach commented Oct 23, 2024 •

edited

Loading

jdunkerley left a comment

enso-bot bot commented Oct 24, 2024 •

edited by unfurl-links bot

Loading

4e6 commented Oct 24, 2024

Akirathan Oct 23, 2024

JaroslavTulach Oct 24, 2024

JaroslavTulach Oct 24, 2024

Use enso.dev.insight property to turn Insight on #11385

Use enso.dev.insight property to turn Insight on #11385

Conversation

JaroslavTulach commented Oct 23, 2024 • edited Loading

Pull Request Description

Checklist

JaroslavTulach commented Oct 23, 2024 • edited Loading

Insight & Enso Manual

Create Insight Script

Usage in CLI

Usage in IDE

Developer's Workflow

JaroslavTulach commented Oct 23, 2024 • edited Loading

Quirks

Less Damage than Expected

JaroslavTulach Oct 23, 2024

Choose a reason for hiding this comment

JaroslavTulach commented Oct 23, 2024 • edited Loading

Useful Scripts: Detect Assign Statements

jdunkerley left a comment

Choose a reason for hiding this comment

enso-bot bot commented Oct 24, 2024 • edited by unfurl-links bot Loading

4e6 commented Oct 24, 2024

Akirathan Oct 23, 2024

Choose a reason for hiding this comment

JaroslavTulach Oct 24, 2024

Choose a reason for hiding this comment

JaroslavTulach Oct 24, 2024

Choose a reason for hiding this comment

Use `enso.dev.insight` property to turn Insight on #11385

Use `enso.dev.insight` property to turn Insight on #11385

JaroslavTulach commented Oct 23, 2024 •

edited

Loading

JaroslavTulach commented Oct 23, 2024 •

edited

Loading

JaroslavTulach commented Oct 23, 2024 •

edited

Loading

JaroslavTulach commented Oct 23, 2024 •

edited

Loading

enso-bot bot commented Oct 24, 2024 •

edited by unfurl-links bot

Loading