update section of page(component) based on reactive property change

[AbhimanyuAryan]

reported by @zygmuntszpak

I have created a timeline, and I would like to update the timeline entries in response to a user selection. How do I ensure that the UI gets updated? My UI references a reactive component, but after updating the reactive component, the UI doesn't automatically update. Please see the following MWE:

using Stipple
using StippleUI

@reactive mutable struct Model <: ReactiveModel
    button_pressed::R{Bool} = false
    timeline_entries::R{Vector{ParsedHTMLString}} = initialise_timeline_entries()
end

function initialise_timeline_entries()
    timeline_entries =  [  timelineentry("Timeline", heading=true),  
                           timelineentry(;title = "Title A"),
                           timelineentry(;title = "Title B")
                        ]
    return Reactive(timeline_entries)
end

function ui(model)

    Stipple.onbutton(model.button_pressed) do
        @show "Button Pressed"
        entries =  [ timelineentry(;title = "Title C"),  timelineentry(;title = "Title D") ]
        model.timeline_entries[] = entries
    end

    page(
        model,
        class = "container",
        [
            btn("Update Timeline", @click("button_pressed = true"))
            timeline("",color = "primary", model.timeline_entries[])   
        ],
    )
end

model = Stipple.init(Model)

route("/") do
  html(ui(model), context = @__MODULE__)
end

up(async = true)

[hhaensel]

Your thinking the wrong way 😉
If the content needs to be reactive, the loop needs to be on the client side. Vue.js has the v-for syntax for such loops. In Stipple we translated this to @recur.

using Stipple
using StippleUI

@vars TimeSeries begin
    button_pressed = false
    titles = ["Title A", "Title B"]

    private = "private", PRIVATE
    readonly = "readonly", READONLY
end

function ui(model)
    Stipple.onbutton(model.button_pressed) do
        @show "Button Pressed"
        model.titles[] = ["Title C", "Title D"]
    end

    page(
        model,
        class = "container",
        [
            btn("Update Timeline", @click("button_pressed = true"))
            
            timeline("",color = "primary", [
                timelineentry("Timeline", heading=true)

                timelineentry("", @recur("t in titles"), title = :t)
            ])  
        ],
    )
end

model = Stipple.init(TimeSeries)

route("/") do
  html(ui(model), context = @__MODULE__)
end

Note the new definition syntax for reactive models, which we introduced tonight with Stipple v0.25.14.
You can now easily redefine models, i.e. add, delete or modify parameters and just rerun the model definition.
Fields are automatically reactive. If you want them private or readonly, just append , PRIVATE or , READONLYto the definition line (see above).
Documentation still needs to be done. - Happy coding!

[AbhimanyuAryan]

Oh yes thanks @hhaensel I didn't pay attention to that line

timeline("",color = "primary", model.timeline_entries[])

[AbhimanyuAryan]

I need to find some free time to document stipple code ☺️

[AbhimanyuAryan]

@hhaensel I really like this PRIVATE but I don't understand READONLY(is it like one in typescript)?

[hhaensel]

READONLY is there rigjt from the beginning. It means that fields from the model are pushed to the client but the client can't update the fields back to the server. PRIVATE fields don't appear on the client side.

[AbhimanyuAryan]

never used it. I can vaguely recall I think Adrian contributed it few months back. Regardless I need to start documenting Stipple 📝😔

[zygmuntszpak]

Thank you very much for that. I'm still trying to build a proper mental model of what is going on and trying to find the similarity with GenieFramework/Stipple.jl#147

I noticed that you said it was important in that other issue to have something like:

	Stipple.on(model.isready) do ready
		ready || return
		push!(model)
	end

which is missing here. From the previous post, my impression is that one must also have something like

Stipple.notify(model.titles) 

inside the button press handler after updating the titles (but I don't see that as part of your solution here either).

Must the data structure I loop over with @recur be a plain array of strings, or could it, for instance, be an array of DataFrameRow? In my case, the title, subtitle etc. of the timeline_entries will be extracted from a DataFrame, where there are already columns for title, subtitle etc. I would like to convert the DataFrame to an array of DataFrameRow and then reference the title, subtitle fields as I populate the timeline entries.
For instance, I define

model.titles::R{Vector{DataFrameRow}} = copy(eachrow(DataFrame(Message = ["Title A", "Title B", "Title C"])))

and then try to use

timelineentry("", title = Symbol("item.Message"),  @recur("(item, index) in titles"))

but this does not work. I end up with a timeline of three entries but no title text for the entries.

[hhaensel]

Part I - Updating values:

There are 3 main possibilities of sending updated values to the front end (or client):

• standard: set the value of a reactive model field to a new value, e.g. model.titles[] = ["a", "b", "c"]
  This first sets the value of the field and then pushes the value to the client (via notify)
• notify: the method notify() signals to a reactive field that the value has changed and that all connected listeners should be triggered. (If you have defined handlers with on(model.fieldname) do ... these will also be called. But the first handler is the one that pushes the value of the field to the client. This handler has been invisibly defined be the init() routine.)
• update client: push!(model, fieldname => fieldvalue) will update the model field on the client side only. This way the server value and the client value can get out of sync. This is typically used when you want to define a helper field on the client which does not need to exist in the model. Recently we added push!(model, fieldname) which sends the current value of the field to the client, which can be of advantage if you want to keep the client updated but don't want to trigger the handlers.
  The short answer is, if you want to loop over something with @recur you loop over a javascript object.

Part II - isready handler

• the isready handler is necessary if your initialisation routine, e.g. route("/") do ..., changes values of the model. The reason is that init(model) intialises the model with the default values defined in @reactive (or now @vars). Any changes to these values need to be sent to the client at some point. When the model on the client side is ready to receive values it sends isready. So the handler sends all model data to the client. If you only change one field you could also decide to only do push!(model, updatedfieldname). If you always start with the default values, no isready handler is necessary.

Part III - @recur

You need to be aware that @recur always loops over javascript objects. Looping over arrays is the easiest solution as it is very similar to Julian syntax. Note that indexing is 0-based. If you want to loop over dataframes it is a bit more complicated. The reason for this is how JSON3 handles dataframes. Have a look at JSON3.write(df) to see how it is coded.
There are two ways out:

• transform the DataFrame into a Dict with dict = Dict(zip(names(df), eachcol(df))) and refer to the columns as you did above.
• refer to the columns with Symbol(df.columns[df.colindex.lookup.Message - 1])
  your model would then look like

@vars TimeSeries begin
    button_pressed = false
    df = DataFrame(Message => ["Title A", "Title B"])

    private = "private", PRIVATE
    readonly = "readonly", READONLY
end

[hhaensel]

@essenciary
We might consider defining
render(df::DataFrame) = Dict(zip(names(df), eachcol(df)))
I just verified that JSON and JSON3 render DataFrames differently 😢

[hhaensel]

Also spotted another no-go: The handler should not go into the ui().
If you do it as is, everytime the ui is called, another layer of handlers is put on the model.
If you define your model locally in the rout() this doesn't matter, but if you define it globally, it does.

My final version would look like this:

using Stipple
using StippleUI
using DataFrames

@vars TimeSeries begin
    button_pressed = false
    df = DataFrame(:Message => ["Title A", "Title B"]), READONLY
    private = "private", PRIVATE
    nonreactive = "nr", Stipple.NON_REACTIVE
end

function ui(model)
    page(
        model,
        class = "container",
        row(cell(class = "st-module", [
            btn("Update Timeline", color = "primary", @click("button_pressed = true"))
            timeline("", color = "primary", [
                timelineentry("Timeline", heading=true),
                timelineentry("", @recur("t in df.columns[df.colindex.lookup.Message - 1]"), title = :t)
            ])   
        ])),
    )
end

function handlers(model)
    Stipple.onbutton(model.button_pressed) do
        @show "Button Pressed"
        model.df[] = DataFrame(:Message => ["Title C", "Title D", "Title E"])
    end

    model
end

model = Stipple.init(TimeSeries) |> handlers

route("/") do
  html(ui(model), context = @__MODULE__)
end

up()

[hhaensel]

@essenciary We'd rather build a stippleparse() for DataFrames ...

[hhaensel]

Let's reopen this, because the current situation of rendering dataframes is unsatisfying.

[essenciary]

I've been looking over this and I don't understand the value of an abstract renderer for DataFrames. Surely, the JSON output needs to be structured in a way that matches whatever the specific UI component expects.

[hhaensel]

@essenciary

I propose to add the following lines to Stipple.__init__()

@require DataFrames  = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0" function render(df::DataFrames.DataFrame, fieldname::Union{Nothing, Symbol} = nothing)
    Dict(zip(names(df), eachcol(df)))
end

This will render dataframes as a js dictionary (as would JSON.json(df) BTW)

Withh this modification

timelineentry("", @recur("t in df.columns[df.colindex.lookup.Message - 1]"), title = :t)

would become

timelineentry("", @recur("t in df.Message"), title = :t)

[hhaensel]

So the benefit is that you can manage your data as a dataframe and can use @recur to display the content of its columns.

[essenciary]

Again, I don't see the value of this as part of the core. It can be easily implemented by hand. That's why we have Stipple.render. Also, it can conflict with other packages that implement render for DataFrames. IMO, if it should not be in core, it should be a plugin or not at all.

Also, I wanted to discuss the use of @require as a separate topic as I'd like to remove it because it breaks the best practices for dependencies management in Julia. We should design extensible APIs that can be enhanced with plugins.

[essenciary]

See for example the case of StippleUI.Tables

StippleUI.jl/src/Tables.jl

Line 110 in 6a4ec59

Base.@kwdef mutable struct DataTable{T<:DataFrames.DataFrame}

StippleUI.jl/src/Tables.jl

Line 233 in 6a4ec59

function Stipple.render(t::T, fieldname::Union{Symbol,Nothing} = nothing) where {T<:DataTable}

[essenciary]

Finally, we should probably not care about DataFrames but talk about tabular data iteration in general via https://github.com/JuliaData/Tables.jl in the form of a StippleTables.jl plugin.

[essenciary]

Adding to this, Stipple itself, as the core library that it is, is not concerned with rendering of specific data structures. It only provides low-level communication primitives for server-client 2-way data exchanges. Any specific renderers are meant to go into plugin libraries that leverage the Stipple.render API (like StippleUI or StipplePlotly).

We can see that by looking at the Stipple.render methods:

[hhaensel]

Also, I wanted to discuss the use of @require as a separate topic as I'd like to remove it because it breaks the best practices for dependencies management in Julia. We should design extensible APIs that can be enhanced with plugins.

Agree 100%

[hhaensel]

Finally, we should probably not care about DataFrames but talk about tabular data iteration in general via https://github.com/JuliaData/Tables.jl in the form of a StippleTables.jl plugin.

We can easily write a renderer for the Tables.jl interface:

Stipple.render(table::T, fieldname) where T = OrderedDict(zip(Tables.columnnames(df), Tables.columns(df)))

But I wonder, how we'd define the types T that support the Tables.jl interface.

[hhaensel]

function Stipple.render(t::T, fieldname::Union{Symbol,Nothing} = nothing) where {T<:DataTable}

I've not really understood why you decided to define a fieldname-dependent render function

function Stipple.render(t::T, fieldname::Union{Symbol,Nothing} = nothing) where {T<:DataTable}
  data(t, fieldname)
end

function data(t::T, fieldname::Symbol; datakey = "data_$fieldname", columnskey = "columns_$fieldname")::Dict{String,Any} where {T<:DataTable}
  Dict(
    columnskey  => columns(t),
    datakey     => rows(t)
  )
end

so that

julia> df = DataFrame(:a => 1:3, :b => 2:4)
3×2 DataFrame
 Row │ a      b     
     │ Int64  Int64 
─────┼──────────────
   1 │     1      2
   2 │     2      3
   3 │     3      4

julia> render(DataTable(df), :test)
Dict{String, Any} with 2 entries:
  "columns_test" => Column[Column("a", false, "a", :left, "a", true), Column("b", false, "b", :left, "b", true)]
  "data_test"    => Dict{String, Any}[Dict("__id"=>1, "b"=>2, "a"=>1), Dict("__id"=>2, "b"=>3, "a"=>2), Dict("__id"=>3, "b"=>4, "a"=>3)]

I think that's the only place where we have fieldname-dependent rendering, right?

[hhaensel]

@zygmuntszpak
So it seems the best way for you to go forward is to define

Stipple.render(df::DataFrame, fieldname::Union{Nothing, Symbol} = nothing) = Dict(zip(names(df), eachcol(df)))

yourself and go with

timelineentry("", @recur("t in df.Message"), title = :t)

in th ui() until we have implemented the plugin.

[zygmuntszpak]

Thank you very much for all the explanations and suggestions. In the meantime, I had gone with something like this (I hadn't made the switch to @vars yet) :

@reactive mutable struct TimeSeries begin
main_table::R{DataTable} = DataTable(load_dataframe(), table_options)
timeline_entries::R{Dict{String, Vector{String}}} = initialise_timeline_entries(main_table)
end

function initialise_timeline_entries(table)
  df = DataFrames.transform(table.data, :Message, :Date, :X => (ByRow(x-> x > 0 ? "left" : "right") )=> :Side, :X => (ByRow(x->abs(x))) => :Score)
  sort!(df, :Score, rev = true)
  df₂ = DataFrames.select(df, :Message, :Date, :Side)
  dict = Dict(zip(names(df₂), eachcol(df₂)))
return Reactive(dict)

timelineentry("", title = Symbol("timeline_entries.Message[index]"), subtitle = Symbol("timeline_entries.Date[index]"), side = Symbol("timeline_entries.Side[index]"),  @recur("(message, index) in timeline_entries.Message"))

If I understood correctly, then what you are suggesting is that I can define timeline_entries::R{DataTable} instead and then use

Stipple.render(df::DataFrames.DataFrame, fieldname::Union{Nothing, Symbol} = nothing) = Dict(zip(names(df), eachcol(df)))

to take care of converting to the dictionary which yields a convenient syntax to iterate over, i.e.

@recur("(message, index) in timeline_entries.Message"))

[hhaensel]

That's correct except that df should be of type DataFrame not DataTable

[essenciary]

function Stipple.render(t::T, fieldname::Union{Symbol,Nothing} = nothing) where {T<:DataTable}

I've not really understood why you decided to define a fieldname-dependent render function

I think that's the only place where we have fieldname-dependent rendering, right?

I'm not sure the fieldname part is relevant. My points were:
1/ we support multiple libraries/components/stipple plugins that expect their data, on the frontend, to have a certain structure (as JSON)
2/ on the server side, these components leverage common Julia libraries and data types
3/ the Stipple API for rendering components data as JSON is via Stipple.render
4/ defining Stipple.render for DataFrame is not allowed because it's type piracy (defining a method that dispatches on a type that is not defined by the package that defines the method)
5/ also, if multiple components library would do type piracy on Stipple.render that would crash the app when using multiple such libraries in the same app.

So users should implement their own type that renders a DataFrame without type piracy.

[zygmuntszpak]

Regarding the plug-in system, is the suggestion then to develop a StippleDataFrames package which implements Stipple.render for DataFrame?

In that case wouldn't your definition of type piracy be violated in that package as well (defining a method that dispatches on a type that is not defined by the package that defines the method)?

I came across the following definition of type piracy:

Right, type piracy is defining function f(x::T) where you “own” neither f nor T. If f is your own function, or T is a type you defined, there’s no issue at all. But even if neither is true, it still might be OK. It’s just a smell that something bad might be happening, such as:

The author/designer of f really did not want it to support type T, so you’re misunderstanding what the function is supposed to mean.
The code is in the wrong place, and should be moved to where f or T is defined.

If I understood that definition, part of the issue is who owns f (Stipple.render) and not just T.

[hhaensel]

If you are the autor of the core package, it is ok to do this, see https://discourse.julialang.org/t/how-bad-is-type-piracy-actually/37913/9

[hhaensel]

In the case of dataframes it is even clear how a default json rendering should look like. JSON.jl implements it exactly in that way so that df.colname works on both, server side and client side. This is what I would definitely expect. The different rendering by JSON3.jl comes from the fact that for unknown types JSON3 renders the fields of the type and not the properties.
So defining a module that cares about expected dataframe rendering should be part of the Stipple plugins.

[hhaensel]

@essenciary
How do we proceed with this? Shall we build an extension to incorporate Stipple.render for DataFrames?
If we aim to be backwards compatible, we would need to implement a fallback with Require.

[essenciary]

@hhaensel Extension makes sense.
I would not worry about backwards compatibility.

[hhaensel]

I mean julia < v1.9

[essenciary]

Ah bummer... Honestly I'd prefer bumping the Julia compat to 1.9 -- but this will be too steep of a jump I think.

In this case we have to support Julia 1.6/1.8 (technically 1.8 is now the LTS) and remove all the Require code when we reach Julia 1.10 (as 1.9 will become LTS so we can bump to 1.9 Julia compat then).

With this approach it would make sense to have a clean implementation. Ex putting the 1.9 extension implementation in a file and the 1.8 Require implementation in a different file. And have some sort of static include to leverage precompilation (if possible). Then at 1.10 we just remove the "dead code".

[hhaensel]

Is this issue solved now? With the latest PR we have moved DataFrames to an extension and any type that supports the Tables API is now rendered as OrderedDict, e.g.

julia> df = DataFrame(:a => [1, 2, 3], :b => ["a", "b", "c"])
3×2 DataFrame
 Row │ a      b
     │ Int64  String
─────┼───────────────
   1 │     1  a
   2 │     2  b
   3 │     3  c

julia> render(df)
OrderedDict{String, AbstractVector} with 2 entries:
  "a" => [1, 2, 3]
  "b" => ["a", "b", "c"]

[hhaensel]

Just to update this issue:

• we now support rendering table-like datatypes as ordered dicts
• we had to excempt some table-api compatible types from table-rendering, e.g. certain Dicts or vectors, as they are already expected default-rendering
• we have removed the necessity of specifying "fieldname" for render-functions

@essenciary , @AbhimanyuAryan , @zygmuntszpak Actually, I think this issue can now be closed, as all the questions and proposals in this slightly diverging issue have been addressed.
Please specify what needs to be done in order to close it, if you disagree.

[AbhimanyuAryan]

@hhaensel this can be closed as the original question was about looping and dynamic rendering