DataTables breaking with Ajax errors when running a containerized Shiny application - multiple containers at same URL

[Mkranj]

We have a Shiny app running inside a docker container at a specific URL.
After going to the app and retrieving new data multiple times, which refreshes the DataTable with new data, suddenly we get an Ajax error and the table is empty, even though R reports no errors.

So, the app works several times, and then any further actions results in a broken table. All other components relying on new data (i.e. plots) still work, only the DT breaks!

Looking at the Network tab in Devtools, we get 404 errors:
Failed to load resource: the server responded with a status of 404 (Not Found)

<myURL>/session/fc012ccedd40a438092f8403d45c96a9/dataobj/<myDataTableID>?w=&nonce=79f70b402396a814

Each time we get a new table OR go to the next table page, a new request is sent and results in 404.

No other aspects of the app send http/XHR requests.

Potential cause

I believe the issue is that we have several containers running in parallel, serving multiple users at once. All containers are routed to the same URL.

From my understanding DataTable creates mini-endpoints that fetch new data when e.g. you fetch the next page, and those endpoints live at myURL/dataobj/table-id . After some time in the app, when the request is sent to the url listed above, it gets routed to a different container - and in that container, the endpoint is not created so the Ajax error occurs.
The user itself is not redirected as long as he stays in the same tab, but DataTable alone communicates in a different way than the rest of the Shiny session. So other components still works.

Does this explanation make sense? How can it be avoided while still serving multiple containers?

[philibe]

I have a big R Shiny app with many DT tables. I've done modules which create the sub DT only at their beginning.

Nowadays I made also Flask VueJs with Datatable.net with Ajax: when I reload a datable.net table with changes I destroy it and recreate it.

In other terms are you sure that the cause is the implementations in Docker ? Have you a reprex ?

PS: I'm a simple user, not from DT team.

[Mkranj]

I mean, I can reprex a minimal app with a datatable with randomly changing data:

library(shiny)
library(DT)

ui <- fluidPage(
  actionButton("generate", "Generate"),
  DTOutput("table")
)

server <- function(input, output, session) {
  
  data_rand <- eventReactive(input$generate, {
    # Simulate pulling data from database.
    # Depending on specific report, the number of columns differ

    n_cols <- sample(3:7, 1)
    
    n_rows <- round(runif(1, 2, 20))
    
    df <- data.frame("col1" = rnorm(n_rows))
    
    for (i in 1:n_cols) {
      df[[paste0("col", i)]] <- rnorm(n_rows)
    }
    
    df
  })
  
  output$table <- renderDT({
    req(data_rand())
    datatable(data_rand())
  })
}

shinyApp(ui, server)

But the point is that this should be run in an environment with multiple containers running this app, and all being routed to the same URL.
After some point running the app, when the balancer changes the container I get routed to, the datatable will give an Ajax error every time the Generate button is pressed, and from then on it won't work.

Understandably I can't reprex the network environment.

[philibe]

I don't know if those urls can help you:

• Ajax Error with Load Balancer on Chrome #642 "Ajax Error with Load Balancer on Chrome"
• R Shiny DataTables giving Ajax error using Docker swarm #849 "R Shiny DataTables giving Ajax error using Docker swarm"
• from R Shiny DataTables giving Ajax error using Docker swarm #849 : https://github.com/shrektan/DT-load-balancer
• https://dev.to/fabiancdng/scaling-nodejs-web-apps-with-docker-mep
• https://github.com/josephDev123/loadbalancer-nginx-docker-nodejs

When I run a shiny app over behind reverse proxy (but not in docker) , I have in F12:

• http://192.x.x.x:3838/apps/test11/session/c556ceb55265.....80a8c2a2fc7/dataobj/tbl?w=&nonce=9fb...c63c62b

In other terms the "mini-endpoints" go throught 3838 port.

My config with
/etc/apache2/sites-available/my-shiny-conf.conf

ProxyPassMatch ^/shiny/p/([0-9]+)/(websocket|.*/websocket)/$ ws://localhost:3838//p/$1/$2/
ProxyPass /shiny/ http://localhost:3838/
ProxyPassReverse /shiny/ http://localhost:3838/

/etc/shiny-server/shiny-server.conf

run_as shiny;
  server {
    listen 3838;
    location / {
      site_dir /srv/shiny-server;
      ...
  }
}

sudo ln -fs /opt/shiny-server/myapps/ /srv/shiny-server/apps.

And I call my apps by http://192.x.x.x/shiny/apps/test11/ (test11 is a folder where there is app.R).

When I run a shiny app over behind reverse proxy (but not in docker) , I have in F12:

• http://192.x.x.x/shiny/apps/test11/session/482429b8f88......37d9e5ff0b/dataobj/tbl?w=&nonce=d59b.....848a6

Therefore you could try adapt it to this:
https://github.com/josephDev123/loadbalancer-nginx-docker-nodejs/blob/master/docker-compose.yml

I am not a network specialist.

[philibe]

And a last url if you want to edit the DT session name:

• https://r-packages.io/packages/DT/dataTableAjax DT::dataTableAjax()

[Mkranj]

Thank you very much for the links! Sticky sessions fixed the issue.
We use haproxy, so these settings were the solution:

stick-table type ip size 5000 expire 1h
stick on src

So each user is guaranteed to be connected to the same container for an hour, based on their IP.