Add a performance measuring top-level user guide page

[malteneuss]

Another step of the user guide improvement initiative #9214:

Add a simple profiling user-guide page based on https://discourse.haskell.org/t/ghc-profiling-a-cabal-project-with-an-interactive-application/10465/2?u=malteneuss, which the author allowed us to use.

Feel free to modify yourself if that is faster.

• Patches conform to the coding conventions.
• Is this a PR that fixes CI? If so, it will need to be backported to older cabal release branches (ask maintainers for directions).

[Kleidukos]

Thanks a bunch for starting this. Several things:

• I would recommend that you make use of the ghc-prof-options key, which is used like this:

ghc-prof-options: -fprof-auto -fno-prof-count-entries -fprof-auto-calls

• You mention Speedscope and the -pj format. Is the JSON produced by -pj natively understood by Speedscope? Or do you still need something like https://github.com/mpickering/hs-speedscope ?

[malteneuss]

Thanks for the quick review @Kleidukos and @geekosaur. I've added all your review remarks. Yes, the json output can be loaded into speedscope directly. This https://github.com/mpickering/hs-speedscope only deals with eventlog files, which irrc is another profiling report format to analyze memory consumption (some topic for a future section in this guide), although it also measures cpu performance.

[tchoutri]

@malteneuss Much better, only two more changes to make and we should be good to go! :)

[jasagredo]

My understanding is that eventlog profiling is far superior than just -pj, and it is (for recent GHCs) just a matter of passing -l -p instead (or if a smaller eventlog is wanted -l-agu -p). I have personally never used the json profiling report.

[malteneuss]

@jasagredo Do you have a link to a post or blog that discusses the differences? I have never used it. Would you like to add a second section showing how to use it in another MR? I think we should add another section for profiling memory, where eventlog should be discussed after all.

[ulysses4ever]

Thank you for taking this on! It's amazing to see efforts in expanding the guide section! I have certain critique below but, please, realize that documentation is a matter where reasonable people can disagree. I can survive the current version, especially given that you already got the mandatory two approvals.

IMO the text uses a cabal workflow that is suboptimal. In particular:

1. It advocates for explicit cabal build, while you never need an explicit build in simple scenarios (including profiling). You should use cabal run where possible and rely on cabal to manage (re)compilation.

2. (In part, because of (1)) it has to use $(cabal list-bin my-app) in an earlier, supposedly lighter subsection. This is long (i.e. annoying), quite platform-dependent (i.e. it assumes bash; I normally don't use bash even under Linux, for instance), and perfectly avoidable (see below).

3. In an earlier, supposedly lighter subsection, it mentions adding profiling-related GHC options in a cabal file. Even though it's done as a note, this idea comes too early (advanced options should come later) and also gives a bad advice (you shouldn'g use cabal file for it -- a project file is the right place).

I propose solving most (all) of these with the following reformatting.

• Start with cabal configure and explain that it creates cabal.project.local with local configuration.
• Now, simply cabal run works.
• Now you can mention the advanced GHC options. You are able to put them in the local project file at this point. Optionally, you could have a little subsection at the very end instead of here. You could also say two words why these options are good in there.
• Now comes the section Profiling your dependencies too.

[ulysses4ever]

I temporarily block it to make sure you have enough time to respond.

[jasagredo]

Re-reading the document again I am unsure this describes Cabal's job on profiling. I think the section should be more about what options does cabal need to enable profiling than how to produce a profiling report, therefore I think the title is misleading and we are stepping into GHC's User Guide territory.

My suggestion would be to leave most of the description and RTS options to the GHC User guide and describe only the following in the text:

• A brief explanation on why building for profiling is different (one has to link to the profiling RTS and insert cost-centres)
• How to enable profiling (either via flag or via cabal.project)
• The different options to enable profiling (library-vanilla for example)
• The different customizations cabal provides (like profiling-detail which uses different names than GHC) and how they map to GHC options
• How to pass other GHC profiling options.
• How to enable profiling for specific dependencies or for all dependencies.
• The usual caveats (for example I build with --enable-profiling, then using only run will re-build without profiling because the flags used during building are not persistent)

This way the choice of p vs pj, or what files are produced or where to load them is deferred to the User Guide, as it is GHC-specific business.

This way we don't mix "How to profile" and "How to configure for profiling". The latter is Cabal's job, but the former is GHC or general Haskell information which should not live in Cabal's docs I think. It will always be too specific or too general what you can say there and it might depend on GHC versions and so on.

[Mikolaj]

OTOH, for some users a full example session of profiling would be immensely valuable and precisely what they are looking for, in despair reaching even for the cabal documentation. So maybe at least link to the original discourse post, saying it contains practical examples, including how to tweak GHC to profile best, which is out of scope of the cabal guide?

[jasagredo]

I think this deserves a place in something like https://haskell.foundation/hs-opt-handbook.github.io, not in cabal's documentation. Cabal docs should talk about how to configure cabal for the different profiling options, but not about how to profile or interpret the results.

[tchoutri]

@jasagredo I disagree part of what you said. The cabal manual should describe how to operate cabal. Interpretation of the results certainly deserves to be centralised in the optimisation handbook but producing a profile is certainly well within what one could expect of the cabal manual.

[jasagredo]

I think there is a hope that Cabal's docs could tell you how to do everything, but IMHO that is not how it should be.

For example I think Rust usually has had a much more comprehensible documentation than Haskell, and they usually made very correct choices in my opinion.

The equivalent to what we are discussing here is this page on "The Cargo Book" https://doc.rust-lang.org/cargo/reference/profiles.html which describes the options that exist to customize optimization or debug information.

Then it defers most of the information to "The rustc book" which describes how each option works. How to do profiling is covered in "The Rust Performance book" https://nnethercote.github.io/perf-book/profiling.html which links to different tools to produce and analyze profiles.

Why is this different than in Haskell? Because Rust performance can be analyzed by standard tools, whereas in Haskell it is the GHC RTS the one that produces its own reports, and third party tools the ones that interpret those reports in different ways.

GHC's User Guide already speaks about how to produce reports and gives a brief outline on how to analyze them, but it is mostly via 3rd party tools that those reports are consumed (profiteur, eventlog2html, hp2pretty/hp2ps, speedscope, ...). It therefore is reasonable that each tool explains their business, and having a central place that outlines the overall complete process, which I think is the Haskell Optimization Handbook.

In any case, both producing and analyzing documentation live outside of "The Cargo Book" in the Rust ecosystem, which I think is the right choice, and as such I would argue we could do the same and leave this outside of the cabal documentation.

[jasagredo]

Maybe I should re-phrase my suggestion after my wall of text above (sorry for that):

• cabal documentation should explain just how to configure your project for profiling
• It could mention explicitly that RTS options will have to be passed to produce a report, but for what flags to pass that is GHC's work.
• Distinction between memory and CPU profiling is GHC's work.
• Interpreting reports is Haskell Optimization Handbook's work.

[malteneuss]

Here's the second proposal after having it streamlined similar to @ulysses4ever suggestions. I don't mention cabal.project.local as i find it too unintuitive why it exists in the first place (cabal.project is also "local"; maybe something for another how-to-configure things guide).

I'm with @Mikolaj and @tchoutri on having a full example session (if i need such a guide, i don't want to scramble snippets together from different places) and that the main interface where we configure things is Cabal here. However, i emphasize now that Cabal does only a configuring part, and GHC the actual work; and i mention the optimization handbook.

[geekosaur]

I don't mention cabal.project.local as i find it too unintuitive why it exists in the first place (cabal.project is also "local"; maybe something for another how-to-configure things guide).

cabal.project is shared between all developers of a particular project; cabal.project.local is for individual developers to customize or adapt to their local machine/development environment.

[ulysses4ever]

cabal.project is shared between all developers of a particular project; cabal.project.local is for individual developers to customize or adapt to their local machine/development environment.

correct. Even simpler rule of thumb is: cabal.project holds options that are useful to store in Git and cabal.project.local is for local experiments (like profiling) that, in general, shouldn't be committed to Git.

[jasagredo]

As this is going forward anyways, let me add some suggestions to improve it. I won't "Request changes" so feel free to disregard my comments.

[jasagredo]

I would rather first mention the cabal configure path, then in a note mention that you can do this in one go but it is not persistent.

[ulysses4ever]

Yes, please!

[jasagredo]

I would mention that there are three outputs, in particular -p, -pj and -l -p and that you will, just for educational purposes, show the -pj in particular.

[jasagredo]

From my comment below, I would actually mention the three if we are going to mention any at all.

[ulysses4ever]

Yes, please! The eventlog property that Javier is getting at is called time series, I believe? https://en.wikipedia.org/wiki/Time_series

[jasagredo]

I would mention something like:

• -pj produces JSON output, can be visualized in speedscope.
• -p produces GHC's own .prof format, can be visualized in profiteur or ghcprofview.
• -l -p produces an eventlog that can be visualized in speedscope by first converting it via hs-speedscope.

[jasagredo]

Also I'm not 100% sure of that I'm about to say, but my understanding is that -p and -pj shows total time spent, and even in speedscope you see totals. One can see how much time a function took, but not when.

I think the eventlog path shows totals but also shows when this happened, i.e. information will be interleaved with GCs or switching contexts, and split by capabilities. IMHO this is more useful and the thing I usually use.

[jasagredo]

As said, I would mention this first and then the compressed command.