Distributed execution

[abylon-io]

Here is what I did on the same workstation (W11)

1. Started the server via: dagu server --host 192.168.47.1 --port 8080
2. Started the scheduler via: dagu scheduler
3. Started the coordinator via: dagu coordinator --coordinator.host 192.168.47.1 --coordinator.port 50055

On a Linux VM (Manjaro) at 192.168.47.3
With the environment variables defined as follows:

export DAGU_COORDINATOR_HOST=192.168.47.1
export DAGU_COORDINATOR_PORT=50055

I launch the worker via dagu worker --worker.labels role=postgres
and I always get the same error: ERROR msg="Failed to send heartbeat" worker_id=manjaro-virtualbox@36147 err="no coordinators available"
The worker does not seem to instantiate a network connection to my workstation (W11)

Can workers register with the registry via IP?
Are workers based only on a file registry system?

Many thanks

[yottahmd]

Hi @abylon-io,
Not certain this will fix the issue, but could you try setting --coordinator.advertise=192.168.47.1 for the dagu coodinator?

[abylon-io]

Hi @abylon-io, Not certain this will fix the issue, but could you try setting --coordinator.advertise=192.168.47.1 for the dagu coodinator?

Hi @yottahmd,
Thank you for your suggestion, but it doesn't seem possible; the option is not recognized in 1.23.2.

> dagu coordinator --coordinator.host 192.168.47.1 --coordinator.port 50055 --coordinator.advertise=192.168.47.1
Error: unknown flag: --coordinator.advertise
Usage:
  dagu coordinator [flags]

Flags:
  -c, --config string                Path to the configuration file (default: $HOME/.config/dagu/config.yaml)
  -H, --coordinator.host string      Coordinator gRPC server host (default: 127.0.0.1) (default "127.0.0.1")
  -P, --coordinator.port string      Coordinator gRPC server port (default: 50055) (default "50055")
      --cpu-profile                  Enable CPU profiling (saves to cpu.pprof)
      --dagu-home string             Override DAGU_HOME for this command
  -h, --help                         help for coordinator
      --peer.cert-file string        Path to TLS certificate file for mutual TLS
      --peer.client-ca-file string   Path to CA certificate file for server verification
      --peer.insecure                Use insecure connection (h2c) instead of TLS
      --peer.key-file string         Path to TLS key file for mutual TLS
      --peer.skip-tls-verify         Skip TLS certificate verification (insecure)
  -q, --quiet                        Suppress output during dag-run

Thanks for your help.

[yottahmd]

@abylon-io Sorry for confusion, we have added the option in v1.23.4 which also includes other fixes on distributed execution. Could you please give it a try?

[abylon-io]

@yottahmd I just tried it, but unfortunately got the same result.
However, I don't seem to have any network issues. When I run curl -v --http2-prior-knowledge http://192.168.47.1:50055 from my Linux VM, the response seems correct for a gRPC server like coordinator:

*   Trying 192.168.47.1:50055...
* Established connection to 192.168.47.1 (192.168.47.1 port 50055) from 192.168.47.3 port 42052 
* using HTTP/2
* [HTTP/2] [1] OPENED stream for http://192.168.47.1:50055/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: http]
* [HTTP/2] [1] [:authority: 192.168.47.1:50055]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.16.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: 192.168.47.1:50055
> User-Agent: curl/8.16.0
> Accept: */*
> 
< HTTP/2 415 
< content-type: application/grpc
< grpc-status: 3
< grpc-message: invalid gRPC request content-type “”
< 
* abort upload
* Connection #0 to host 192.168.47.1:50055 left intact

I think I'm going to go back to a configuration where all components are on the same host (and therefore not distributed).
Thanks.

[yottahmd]

@abylon-io thanks for checking that. I think I understand the issue. As a current limitation of distributed execution of Dagu, you’ll need to share the same volume of Dagu data across worker, server, and coordinator server. We’re planning to add sync feature to worker soon to eliminate the limitation.

[abylon-io]

@yottahmd that's exactly the kind of feature I'm looking for, it would be really great.
Since it was possible to configure the coordinator address, I thought it was a feature that had already been implemented. I misread the documentation ;-)
Thank you, and I'm available to test it if you'd like.

[yottahmd]

Thanks a lot! I'll ask you for testing for sure.