Add the ability to execute notebooks via the CLI

[bt-maps]

Add options to cli to inject input parameters at runtime and allow parameterized livebooks to programmatically execute a workflow/broadway multi-stage data pipeline.

Basically, allow livebooks with inputs (or exported scripts) to be used as-is in multi-stage broadway pipelines and pass values downstream.

• similar to Papermill for Jupyter notebooks

Brent

[josevalim]

You can use inputs to parameterized Livebook and fill those in the UI. Because our inputs use the IO API, they also work when the notebooks run in the CLI (i.e. we will request the user to fill info in as it happens). So this functionality is already in place but in a way.

The biggest issue is that we don't have an option to execute notebooks via the CLI.

[jonatanklosko]

Depends on livebook-dev/kino#54.

[houllette]

I was hoping to bump this issue as I'm currently working on a pretty novel idea using Livebooks in an educational manner and would like a way to execute notebooks via the CLI in order to automate grading code output. Any help would be appreciated! 😄

[jonatanklosko]

Hey @houllette, this is not a priority for us right now, but I put together a script for basic evaluation here. It needs to be run within the Livebook mix project:

git clone https://github.com/livebook-dev/livebook.git
cd livebook
mix dev.setup

# Download the script and use as:
mix run livebook_run.exs ./path/to/notebook.livemd

Hopefully this helps or gives you a starting point :)

[houllette]

Hey @jonatanklosko - Sorry to bother you again after so much time as elapsed since you wrote the aforementioned script. I was just starting to work with it more and am running into issues - I tried debugging it myself as best as possible but it appears I'm running into a wall.

I'm getting an error when the code enters the Livebook.Session.Data.apply_operation/2 function during your script execution on Line 38. The error I'm specifically seeing is 1st argument: not an atom.

At first I thought it was the .livemd file I was running the script on, but I've come to realize that it won't work even on just a super basic .livemd file that only has one code block returning an atom.

[jonatanklosko]

Hey @houllette! There was an internal change that broke it, I've just updated the script :)

[sonic182]

In jupyter there is nbconvert for notebooks to other formats (python scripts included of course) https://nbconvert.readthedocs.io/en/latest/index.html

It may be nice to have something like nbconvert in elixir for livebooks

[dderooy]

Bumping this idea b/c I think its great. These features would open up lots of potential new app ideas.

> livebook run ./playbook.livemd -x someval -y anotherval

Also a companion hex package would be great too.

output_livemd = livebook.run(file, params) 

An app user story:

1. user creates a livebook to automate events from a stream (to do ETL, data enrichments etc).
2. user saves livebook and goes to do other things in life.
3. an orchestrator process consumes the stream and runs the livebook for each event. A runtime log is created and saved.
4. User later logs into app and can view a runtime for some past event. When the livebook is loaded for the past event, any front end rendered things (visualizations etc) are created from saved output of code cells.

Applications: ETL, Security SOAR, if-this-then-that low code apps, any event driven system handling.

Livebook is great b/c it can bridge the gap between non-coders and developers. In the security industry as example, a developer could make an advanced livebook to run and enrich a security alert stream. Later a SOC analyst could investigate an enriched alert with an already-ran livebook and see pretty readable output with visualizations etc.

[joshuataylor]

Sorry for the bump, for those coming from Jupyter Notebooks (plus to help SEO), this would be equivalent to the following command:

jupyter nbconvert --inplace --to notebook --execute example.ipynb

docs for reference

Anything more complicated than that requires papermill etc.

My use case is I could edit this outside the browser and rerun the workbook as I need.

However, IMHO, if you need to do this as a script, please write a mix task or consider something else :).