Subscribe to generic / any type of message in a "wildcard" fashion

[muety]

This might be a bit unusual, but for a very simple use case, where I effectively only want to track the rate of incoming messages on a given topic, regardless of its type, I'm looking for a way to subscribe to any type of message - even such that might not necessarily be available as structs. Of course, the message couldn't be parsed then, but in this particular case, I don't even care about the message's content, but only the fact that something had been received. Essentially, I'd like to be able to do something like this:

// ...
sub, err := goroslib.NewSubscriber(goroslib.SubscriberConf{
	Node:     node,
	Topic:    "/foo",
	Callback: dispatch,
})

// ...

func dispatch(msg *interface{}) {
    println("got new message")
}

// ...

[muety]

Any ideas on this?

[aler9]

Hello @muety, the problem is that subscribers, when they are connecting to publishers, need to include the message checksum (computed from definition) in the handshake, and most publisher implementations (including the one in this library) reject subscribers with the wrong message checksum:

goroslib/publisher.go

Lines 237 to 240 in 349d9a6

	if header.Md5sum != p.msgMd5 {
	return fmt.Errorf("wrong message checksum, expected '%s', got '%s'",
	p.msgMd5, header.Md5sum)
	}

Therefore the problem lies in publishers, not in subscribers. You need to know the message checksum in advance. To know it, you need to know the message structure in advance.

Nonetheless this doesn't prevent you from building code that is able to subscribe to any topic, with the condition that you have its definition stored somewhere.

Download the list of topics from the master:

topics, err := n.MasterGetTopics()
if err != nil {
	panic(err)
}

Then find the type of your topic:

myTopicType := topics[myTopicName].Type

Then store a map which contains structs corresponding to every possible topic type:

myTypes := make(map[string]interface{})
myTypes["geometry_msgs/Accel"] = &geometry_msgs.Accel{}
myTypes["stereo_msgs/DisparityImage"] = &stereo_msgs.DisparityImage{}
...

Then pick, from this map, the struct corresponding to the current topic type:

myStruct := myTypes[myTopicType]

Then use reflect to make a callback which has this struct as argument:

cb := reflect.MakeFunc(
	reflect.FuncOf([]reflect.Type{reflect.TypeOf(myStruct)}, []reflect.Type{}, false),
	func (in []reflect.Value) []reflect.Value {
		fmt.Println("got new message")
		return nil
	}
).Interface()

Then finally use this callback to create a subscriber:

sub, err := goroslib.NewSubscriber(goroslib.SubscriberConf{
	Node:     node,
	Topic:    myTopicType,
	Callback: cb,
})

And you will get what you asked for.

There's an example of this inside the action client implementation, since it needs to subscribe to dynamic topics too:

https://github.com/bluenviron/goroslib/blob/main/action_client.go

[muety]

Thanks for this super detailed reply, @aler9! I even had a go at implementing this earlier today - pretty much the same way that you suggested. I got to the point where I had a generic callback that simply accepts an interface{} argument. However, as you already explained, I'd still need sort of a "registry" mapping message type names to their respective Go structs, which is where I got stuck then.

My (super weird and specific) use case requires to have entirely dynamic subscriptions in a sense that I can, at runtime, subscribe to a topic of any arbitrary type. This, in turn, would require to find and instantiate a Go type only given it's name. Something like this:

func getMsgType(name string) interface{} {
  // iterate through all message types
  // find the one matching the name
  // return an instance or a reflect.Type of it
}

In Java, for example, you could use the class loader to go through all your message definitions' .java files and dynamically load the one that is required according to the topic's type (to then use it in deserialization). However, I don't think Go has a way to accomplish this, except for code generation. The dynamic requirements of my problem sort of contradict the nature of a statically-typed language.

However, it would already be super useful to me if there was a way to simply know that a message had just been published on a topic, without even deserializing it. Something like:

sub, err := goroslib.NewSubscriber(goroslib.SubscriberConf{
	Node:     n,
	Topic:    "test_topic",
	Callback: nil,
	CallbackRaw: onMessage,
})

func onMessage(raw []byte) {
	fmt.Printf("got new message of size %d\n", len(raw))
}

Do you think that's something that you'd consider implementing?

Apart from all of that: thanks a lot for this amazing library! 🙏

[aler9]

In Java, for example, you could use the class loader to go through all your message definitions' .java files and dynamically load the one that is required according to the topic's type (to then use it in deserialization). However, I don't think Go has a way to accomplish this, except for code generation

If message list had some kind of usefulness for any other use case, i would have it automatically generated together with messages, as i'm doing in one of my other libraries, gomavlib:

https://github.com/bluenviron/gomavlib/blob/f0dd0a45286c8220e78d7dc72ebc799bda3ab593/pkg/dialects/ardupilotmega/dialect.go#L15

In case of ROS, it has no particular use, therefore i skipped this part - you need to either code generation or to parse source files during runtime. Go provides utilities to list all structs in a source file, see:

https://gist.github.com/LukaGiorgadze/570a89a5c3c6d006120da8c29f6684ee

You can also use plain bash:

for f in $(find ./pkg/msgs -name 'msg_*'); do echo $(basename $(dirname $f)).$(cat $f | grep -E ^type | sed 's/^type \(.\+\) struct {/\1/'); done

Do you think that's something that you'd consider implementing?

I'd rather not add this logic into the library since it's way too specific - it requires the library to know message types in advance (because knowing message types and checksum is still required) and it works only when a topic is already being published (since the message type is needed), while subscribers can normally be created before publishers.

The logic you want can be easily implemented in a script that makes use of the untouched library and the library source code (for listing messages).