ListDevices returns output of command executed using RunCommand with --reply argument

[kubamaruszczak]

Describe the bug
I have production EC2 instance running Meshcentral server with DocumentDB as its database. There are over 2500 linux agents connected to the server. I've created a bash script which allows me to execute command on each of the agents synchronously and to get the output back. It lists all of the devices using ListDevices, iterates over all of them and executes command on each connected agent using RunCommand --reply. See the screenshot below:

While the script is executing commands on the devices and I try to list device using ListDevices command, instead of the device I get output of the command executed on one of the devices like on the screenshot below:

I can see is that this RunCommand --reply feature was added here -> #5932

To Reproduce
It's harder to reproduce in lower latency environment. However, I managed to do so with the below steps.

Steps to reproduce the behavior:

1. In one terminal window run infinite loop which will execute echo hello command on one device using RunCommand with --reply option
2. In a separate terminal window run infinite loop which will try to list that device using ListDevices --filter <agent-name> command
3. Observe the output of ListDevices command while RunCommand is being constantly executed. For some of the calls it will return 'hello' instead of the actual device info.

Expected behavior
I would like ListDevices command to return device instead of output from RunCommand --reply command.

[kubamaruszczak]

I run my server with trace=1 parameter and observe Console tab in my browser. What I see is that when I execute RunCommand using meshctrl.js I can see websocket response from RunCommand execution there as shown below. However, when I run ListDevices response doesn't appear in the Console.

My assumption is that the command response being returned from meshcore.js to the server creates some kind of conflict. I've just started learning how communication between Meshcentral components works so I might be wrong but it might be good catch at the same time.

Or maybe it might be related to responseid being the same for all requests sent using meshctrl.js... idk yet

[kubamaruszczak]

I changed responseid for runcommands action to meshctrlcommand and added below console log at the begging of ws.on('message', function incoming(rawdata) {...}); callback in meshctrl.js to see what is received.

console.log(`action: ${data.action}, cmd: ${settings.cmd}, responseid: ${data.responseid}`);

When I just run ListDevices meshctrl command, action nodes is received with responseid meshctrl and then requested node is returned as expected.

However, when I first run the ListDevices command in meshctrl and then execute the RunCommand, it seems like the first command gets interrupted. After the msg action with the responseid meshctrlcommand, the result of the second command execution is returned.

[kubamaruszczak]

I found it! Websocket response sent from Meshagent is being broadcasted to all open websocket sessions and causes them to log result and close. That's why I get RunCommand result in ListDevices execution.

MeshCentral/agents/meshcore.js

Line 1586 in f2681de

mesh.SendCommand({ action: 'msg', type: 'runcommands', result: replydata, sessionid: data.sessionid, responseid: data.responseid });

It wouldn't be an issue if it didn't have defined responseid. Since it's action type of msg it lands in this case and disrupts all ongoing processes.
https://github.com/Ylianst/MeshCentral/blob/master/meshctrl.js#L2218-L2253