Turning Sieve benchmarks into Enso benchmarks #8475

JaroslavTulach · 2023-12-06T12:46:43Z

Pull Request Description

Having the benchmarks in Enso will make it easier to rewrite certain parts of the algorithm to other languages.

Important Notes

Execute as

sbt:enso> runEngineDistribution --run test/Benchmarks Sieve

or as

sbt:enso> std-benchmarks/benchOnly Sieve

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

All code follows the
Java, Enso
style guides.
All code has been tested:
- Benchmarks can be executed
- New benchmarks provide similar results as old ones

JaroslavTulach · 2023-12-06T12:47:44Z

Current results on my computer:

[info] Benchmark            Mode  Cnt    Score   Error  Units
[info] Sieve.Ascribed       avgt       291.542          ms/op
[info] Sieve.Original       avgt       289.745          ms/op
[info] Sieve.Without_Types  avgt       270.837          ms/op

JaroslavTulach · 2023-12-06T14:00:21Z

test/Benchmarks/src/Sieve/Sieve_Java_Script.enso

+    }
+    return new Gen()
+
+compute_nth_prime_natural_in_js = Sieve_Without_Types.compute_nth_prime gen=natural_in_js


These JavaScript benchmarks work with:

sbt:enso> runEngineDistribution --run test/Benchmarks Sieve

but not with std-benchmarks/benchOnly Sieve. Please help, @Akirathan!

JaroslavTulach · 2023-12-06T15:32:02Z

a2acce6 adds JavaScript benchmarks. They however seem to show the price we pay for JavaScript interop is high!

[info] Sieve.Ascribed             avgt    3   268.487 ±   60.973  ms/op
[info] Sieve.Java_Script_All      avgt    3   888.633 ±  276.752  ms/op
[info] Sieve.Java_Script_Filter   avgt    3  1831.328 ± 4293.687  ms/op
[info] Sieve.Java_Script_Natural  avgt    3  2152.961 ±  804.918  ms/op
[info] Sieve.Original             avgt    3   278.269 ±  104.952  ms/op
[info] Sieve.Without_Types        avgt    3   274.963 ±   96.071  ms/op

Java_Script_Filter and Java_Script_Natural pay a price for interop - clearly not done at the fastest possible manner. But why Java_Script_All is slow remains a mystery - standalone benchmarks show JavaScript is faster than Enso.

Running official benchmarks and once again.

The picture confirms (not surprisingly) that the "Enso Sieve Benchmarks" run at the same speed as the "JMH Sieve Benchmarks". E.g. we can integrate and use/watch the new benchmarks.

JaroslavTulach · 2023-12-06T18:13:15Z

lib/scala/bench-processor/src/main/java/org/enso/benchmarks/processor/BenchProcessor.java

@@ -231,7 +231,7 @@ private void generateClassForGroup(
        "      throw new IllegalStateException(\"Project root directory does not exist or cannot be read: \" + Objects.toString(projectRootDir));");
    out.println("    }");
    out.println("    File languageHomeOverride = Utils.findLanguageHomeOverride();");
-    out.println("    var ctx = Context.newBuilder(LanguageInfo.ID)");
+    out.println("    var ctx = Context.newBuilder()");


I thought it was going to be a classpath issue, not "restricted to Enso language" issue! Thank you for your help, @Akirathan!

radeusgd · 2023-12-06T21:19:18Z

But why Java_Script_All is slow remains a mystery - standalone benchmarks show JavaScript is faster than Enso.

Hmm this is very curious indeed 🤔

hubertp

Does it make sense to keep JMH benchmark as well - the more the merrier and gives us some reference?

JaroslavTulach · 2023-12-07T18:13:20Z

Official results are here. Java_Script_File takes 70ms - that's the goal we'd like to reach one day... Original, Ascribed and Without_Types take 400+ms - they are purely written in Enso. E.g. Enso is 6 times slower than JavaScript. The other Java_Script_... benchmark somehow mix Enso and JavaScript - they are pretty slow - e.g. certainly there is a cost of crossing the polyglot boundary between Enso & JavaScript.

Sieve.Java_Script_File                                                avgt    3    70.976 ±      6.440  ms/op

Sieve.Ascribed                                                        avgt    3   440.056 ±     54.563  ms/op
Sieve.Original                                                        avgt    3   412.924 ±     30.009  ms/op
Sieve.Without_Types                                                   avgt    3   405.650 ±      6.148  ms/op

Sieve.Java_Script_All                                                 avgt    3  1123.099 ±    391.940  ms/op
Sieve.Java_Script_Filter                                              avgt    3  2609.217 ±   3943.999  ms/op
Sieve.Java_Script_Natural                                             avgt    3  3182.236 ±   7700.466  ms/op

enso-bot · 2023-12-08T05:03:17Z

Jaroslav Tulach reports a new STANDUP for yesterday (2023-12-07):

Progress: - "removing JMH sieve": c849ff9

AoC: https://discord.com/channels/401396655599124480/1182230447141437531/1182263147550220298
Reporting bug: if then else and chained block syntax #8489
investigating why JavaScript interop is slow It should be finished by 2023-12-11.

Next Day: Bugfix something

Discord
Discord - A New Way to Chat with Friends & Communities
Discord is the easiest way to communicate over voice, video, and text. Chat, hang out, and stay close with your friends and communities.

JaroslavTulach · 2023-12-08T09:27:33Z

Does it make sense to keep JMH benchmark as well - the more the merrier and gives us some reference?

Probably not. The results are the same with the Enso benchmarking infrastructure 400-420ms.

enso-bot · 2023-12-11T08:30:27Z

Jaroslav Tulach reports a new STANDUP for the last Friday (2023-12-08):

Progress: - AoC: https://discord.com/channels/401396655599124480/1182572489025531955/1182572641333280860

discussion about warnings: Support on_problems=Problem_Behavior.Report_Warning and Map_Error wrapping in Vector.map #8371 (comment)
benchmarks merged: Turning Sieve benchmarks into Enso benchmarks #8475
pair debugging classloaders with Pavel
prefer fat JAR: Add org.jline as module to component directory #8496 (comment) It should be finished by 2023-12-11.

Next Day: Bugfix something

Discord
Discord - A New Way to Chat with Friends & Communities
Discord is the easiest way to communicate over voice, video, and text. Chat, hang out, and stay close with your friends and communities.

Turning Sieve benchmarks into Enso benchmarks

089de78

JaroslavTulach requested review from jdunkerley, radeusgd, GregoryTravis, 4e6, hubertp and Akirathan as code owners December 6, 2023 12:46

JaroslavTulach self-assigned this Dec 6, 2023

JaroslavTulach added the CI: No changelog needed Do not require a changelog entry for this PR. label Dec 6, 2023

Adding comparative benchmarks in JavaScript

4d13f13

JaroslavTulach commented Dec 6, 2023

View reviewed changes

Akirathan and others added 2 commits December 6, 2023 15:19

std-benchmarks can run with all installed Truffle languages

32177ba

JavaScript benchmarks compute the right values

a2acce6

JavaScript benchmark from a file is eight times faster

13e569a

JaroslavTulach commented Dec 6, 2023

View reviewed changes

radeusgd approved these changes Dec 6, 2023

View reviewed changes

enso-bot bot mentioned this pull request Dec 7, 2023

Inconsistent type reported by Meta for [] #8255

Closed

JaroslavTulach added the CI: Clean build required CI runners will be cleaned before and after this PR is built. label Dec 7, 2023

Removing the JMH Sieve benchmarks in favor of Enso ones

c849ff9

hubertp approved these changes Dec 7, 2023

View reviewed changes

JaroslavTulach merged commit e4b2b56 into develop Dec 8, 2023

JaroslavTulach deleted the wip/jtulach/SieveBenchmarksInEnso branch December 8, 2023 09:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Turning Sieve benchmarks into Enso benchmarks #8475

Turning Sieve benchmarks into Enso benchmarks #8475

JaroslavTulach commented Dec 6, 2023 •

edited

Loading

JaroslavTulach commented Dec 6, 2023

JaroslavTulach Dec 6, 2023

JaroslavTulach commented Dec 6, 2023 •

edited

Loading

JaroslavTulach Dec 6, 2023 •

edited

Loading

radeusgd commented Dec 6, 2023

hubertp left a comment •

edited

Loading

JaroslavTulach commented Dec 7, 2023

enso-bot bot commented Dec 8, 2023 •

edited by unfurl-links bot

Loading

JaroslavTulach commented Dec 8, 2023

enso-bot bot commented Dec 11, 2023 •

edited by unfurl-links bot

Loading

Turning Sieve benchmarks into Enso benchmarks #8475

Turning Sieve benchmarks into Enso benchmarks #8475

Conversation

JaroslavTulach commented Dec 6, 2023 • edited Loading

Pull Request Description

Important Notes

Checklist

JaroslavTulach commented Dec 6, 2023

JaroslavTulach Dec 6, 2023

Choose a reason for hiding this comment

JaroslavTulach commented Dec 6, 2023 • edited Loading

JaroslavTulach Dec 6, 2023 • edited Loading

Choose a reason for hiding this comment

radeusgd commented Dec 6, 2023

hubertp left a comment • edited Loading

Choose a reason for hiding this comment

JaroslavTulach commented Dec 7, 2023

enso-bot bot commented Dec 8, 2023 • edited by unfurl-links bot Loading

JaroslavTulach commented Dec 8, 2023

enso-bot bot commented Dec 11, 2023 • edited by unfurl-links bot Loading

JaroslavTulach commented Dec 6, 2023 •

edited

Loading

JaroslavTulach commented Dec 6, 2023 •

edited

Loading

JaroslavTulach Dec 6, 2023 •

edited

Loading

hubertp left a comment •

edited

Loading

enso-bot bot commented Dec 8, 2023 •

edited by unfurl-links bot

Loading

enso-bot bot commented Dec 11, 2023 •

edited by unfurl-links bot

Loading