Using MBrace in distributed processes on-premise / local network

[smoothdeveloper]

I'd like to have a look at MBrace which looks amazing, but my use case implies having distributed processes on-premise / internal network.

I tried to figure out in the documentation if this was supported use-case, I found there is Thespian which is "not for production" for "test only" but it is not clear how I could at least investigate it's usage on the network.

I tried this:

on cluster (supervising process)

#r "packages/MBrace.Core/lib/net45/MBrace.Core.dll"
#r "packages/MBrace.Runtime/lib/net45/MBrace.Runtime.dll"
#r "packages/MBrace.Thespian/lib/net45/MBrace.Thespian.dll"

#I "packages/Vagabond/lib/net45/"
#I "packages/FsPickler/lib/net45/"
#I "packages/Thespian/lib/net45/"
#I "packages/Mono.Cecil/lib/net45/"
#I "packages/Argu/lib/net40/"
open System.IO
open MBrace.Library
open MBrace.Thespian
MBrace.Thespian.ThespianWorker.LocalExecutable <- Path.Combine(__SOURCE_DIRECTORY__, __SOURCE_FILE__)
let cluster = MBrace.Thespian.ThespianCluster.InitOnCurrentMachine(0, false)
let workerref1 = ThespianWorker.Connect("mbrace://machine1:2468")
let workerref2 = ThespianWorker.Connect("mbrace://machine2:2469")
cluster.AttachWorker(workerref1)
cluster.AttachWorker(workerref2)

on workers:

#r "packages/MBrace.Core/lib/net45/MBrace.Core.dll"
#r "packages/MBrace.Runtime/lib/net45/MBrace.Runtime.dll"
#r "packages/MBrace.Thespian/lib/net45/MBrace.Thespian.dll"
#r "packages/Vagabond/lib/net45/Vagabond.dll"
#I "packages/Vagabond/lib/net45/"
#I "packages/FsPickler/lib/net45/"
#I "packages/Thespian/lib/net45/"
#I "packages/Mono.Cecil/lib/net45/"
#I "packages/Argu/lib/net40/"
open System.IO
open MBrace.Library
open MBrace.Thespian

let getWorkDir i =
  let dir = Path.Combine(__SOURCE_DIRECTORY__, sprintf "thespianworkdir%i" i)
  Directory.CreateDirectory dir |> ignore
  dir

ThespianWorker.LocalExecutable <- Path.Combine(__SOURCE_DIRECTORY__, __SOURCE_FILE__)
let workDir1 = getWorkDir 1
let worker1 = ThespianWorker.SpawnAsync("machine1", 2468, workDir1, 4)
Async.RunSynchronously worker1

this seems to get stuck on either side.

Can anyone provide guidance?

Is there any plan for production support of on-premise distributed setup?

[palladin]

Unfortunately we don't have support for on-premise deployment. Currently our focus is to cover the two main cloud providers (Azure and Aws). I don't know if Azure Stack https://azure.microsoft.com/en-us/overview/azure-stack/ is a possible future solution for you.

[smoothdeveloper]

@palladin thanks for your feedback.

No I don't have plans at this stage for cloud deployed app, I'm far from that for my current use case.

Do you have an idea of what needs to happen in Thespian at least to have "fiddling with mbrace on locally distributed setup" story enabled?

Having subscription to Azure or AWS means $$$ and subscription, which is not easily enabled for people wanting to look into mbrace. I think (despite in process test setup with Thespian) this is a hindrance for adoption of this framework which looks awesome.

[palladin]

If you want to test locally with thespian the best place is to start with the https://github.com/mbraceproject/MBrace.StarterKit

[eiriktsarpalis]

Hi Gauthier,

In order to function properly, an mbrace cluster depends on a few things: a
file storage service, a table storage service and a queueing system.
Thespian uses the local filesystem for the former and in-memory emulation
for the latter two. This expedites deployment for testing purposes, but
makes it unsuitable for persistable, fault tolerant applications.

So the question of building an on-premises deployment really depends on
what type of storage services your data center is offering. Obviously using
HDFS/SQL server/0MQ would require a different mbrace implementation than a
data center that uses, say, NFS/MongoDB/RabbitMQ. The MBrace.Runtime
library offers a set of interfaces for quickly and easily setting up new
MBrace cluster implementations, depending on your requirements. We are
always keen on providing help with such custom solutions.

On Wed, 13 Apr 2016 at 22:50, Nick Palladinos notifications@github.com
wrote:

If you want to test locally with thespian the best place is to start with
the https://github.com/mbraceproject/MBrace.StarterKit

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#165 (comment)

[smoothdeveloper]

@eiriktsarpalis thanks for additional feedback.

In my current use case, I'm not looking for "persistable" and the "fault tolerant" aspect implies the process stays up but not looking for further guaranties.

I'd like to sketch something to replace a codebase using .NET Remoting where remoting was introduced late (code was in process initially) and doesn't provide any guarantees (perstence, fault tolerance) whatsoever.

On the persistence backend, I can eventually consider using sqlserver, or anything on local drive worker and cluster machines), I intend to support Windows Server.

It seems that mbrace is a great tool for distributed processing even without those aspects, and I'll see myself using it if I can have basic distributed processing.

[isaacabraham]

@smoothdeveloper have you had a look at Azure Stack, which is an on-premise version of Azure. It's still in beta but as far as I know comes with the ability to host service bus and storage accounts, which are all you need really - the compute nodes (and @eiriktsarpalis correct me if i'm wrong here) could easily just be run as executables on any old machine that has access to the service bus and storage?

[eiriktsarpalis]

@isaacabraham correct
On Thu, Apr 14, 2016 at 01:44 Isaac Abraham notifications@github.com
wrote:

@smoothdeveloper https://github.com/smoothdeveloper have you had a look
at Azure Stack, which is an on-premise version of Azure. It's still in beta
but as far as I know comes with the ability to host service bus and storage
accounts, which are all you need really - the compute nodes (and
@eiriktsarpalis https://github.com/eiriktsarpalis correct me if i'm
wrong here) could easily just be run as executables on any old machine that
has access to the service bus and storage?

—
You are receiving this because you were mentioned.

Reply to this email directly or view it on GitHub
#165 (comment)

[smoothdeveloper]

@isaacabraham thanks, I'll check what Azure Stack is about (I remember seeing anouncement back then). I hope it is lightweight enough, ultimately this would need to ship in our setup to customers (but this is way way down the road from where I sit).

It seems from there https://azure.microsoft.com/en-us/documentation/articles/azure-stack-deploy/ that it would be difficult to try out in my regular dev environment though.

I think a Thespian that works on-premise (baring persistence / fault tolerant resilience guarantees) would be great, mostly as a lean extension / abstraction to .NET Remoting to support mbrace workflow.

Thanks all!

[0x53A]

hi @smoothdeveloper , I'm currently working on something similar (a local multi-machine cluster), and got it working with the following code:

A few notes:

• Workers are sub-processes, so you need to set ThespianWorker.LocalExecutable to a valid exe. This exe needs to call Worker.main argv
• All nodes in the cluster need access to one fileshare.

You can call this with prg.exe master 7890 \\machine\share to start a cluster.
It will print the url to the console, then run this on all machines to attach workers: prg.exe slave <url> 1

open MBrace.Thespian
open System
open System.Threading
open System.Diagnostics
open MBrace.Core
open MBrace.Runtime


type RunMode = Host of port:int * workingDir:string | SlaveMaster of master:string * nodeCount:int | Worker of argv:string array

[<EntryPoint>]
let main argv = 
    let mode =
        if argv.Length > 0 then
            match argv.[0] with
            | "master" ->
                let port = Int32.Parse argv.[1]
                Host (port, argv.[2])
            | "slave" ->
                let master = argv.[1]
                let nodeCount = int32 argv.[2]
                SlaveMaster(master, nodeCount)
            | _ -> 
                Worker argv
        else
            failwith "No parameters specified"


    match mode with
    | Host (port, wd) ->    
        ThespianWorker.LocalExecutable <- (Process.GetCurrentProcess().MainModule).FileName
        let worker = ThespianWorker.Spawn(Environment.MachineName, port)
        let cluster = ThespianCluster.InitOnWorker(worker, FileSystemStore.Create(wd))
        printfn "Created new cluster on %s" cluster.Uri
        0
    | SlaveMaster (master, nodeCount) ->        
        ThespianWorker.LocalExecutable <- (Process.GetCurrentProcess().MainModule).FileName
        let cluster = ThespianCluster.Connect(master)
        cluster.AttachNewLocalWorkers(nodeCount)
        printfn "Attached %i workers to %s" nodeCount cluster.Uri
        0
    | Worker xx ->
        MBrace.Thespian.Runtime.Worker.main argv

[mpodkolzin]

In order to function properly, an mbrace cluster depends on a few things: a
file storage service, a table storage service and a queueing system.
Thespian uses the local filesystem for the former and in-memory emulation
for the latter two. This expedites deployment for testing purposes, but
makes it unsuitable for persistable, fault tolerant applications.
So the question of building an on-premises deployment really depends on
what type of storage services your data center is offering. Obviously using
HDFS/SQL server/0MQ would require a different mbrace implementation than a
data center that uses, say, NFS/MongoDB/RabbitMQ. The MBrace.Runtime
library offers a set of interfaces for quickly and easily setting up new
MBrace cluster implementations, depending on your requirements. We are
always keen on providing help with such custom solutions.

Do you have any examples documented somewhere? That would be superusefull.
Thanks, Max