Scatter-gather support for Query API

[jeqo]

Work to be done in order to support scaling storage.

Currently only one instance is supported to get complete response from data stored. If scaled, responses will get partitioned.

Using one instance is limiting storage usage, as when load is high, it starts falling back as one instance cannot keep pace with the ingestion load.

Initial design

In order to query all instances, we need to have a channel where stores can communicate:

------client requests----> | zipkin server 0  |<--+ store requests
                                                  |
------client requests----> | zipkin server 1  |<--+ store requests
                                                  |
------client requests----> | zipkin server 2  |<--+ store requests

Then:

• when receiving client requests: storage will make store requests to other instances.
• when receiving store requests: storage will make requests to local state.

We can expose store endpoint in another port (e.g. 9412), including metadata and metrics that allow better debugging.

Fix #36

[jeqo]

@adriancole this might be a long one :)

[jeqo]

Currently working when testing one instance. Requires fixing tests and validate approach on scaling instances.

[jeqo]

thanks @adriancole and @anuraaga for your feedback! Hope it looks much better now :)

about reusing zipkin-server: is it something we would be able to do right now or requires changes on armeria first?

[anuraaga]

I think until armeria supports Consumer, we can go ahead and add a provided dependency on armeria-spring and use the custom type so this PR works with the current artifacts.

[jeqo]

@anuraaga I'd like to keep the current impl with port 9412, and fix integration and reuse zipkin server on a new PR. I've tried to use armeria-spring but haven't manage to mount the http service.

[codefromthecrypt]

@anuraaga <https://github.com/anuraaga> I'd like to keep the current impl with port 9412, and fix integration and reuse zipkin server on a new PR. I've tried to use armeria-spring but haven't manage to mount the http service.

sgtm TODO and do later is good!

[anuraaga]

Sure no problem 👍

[jeqo]

great! just created #40 to follow up the discussion.

is there any other feedback before merging this?

[codefromthecrypt]

anywhere setting hostname explicitly, comment why important. If important we should probably consider changing upstream docker-compose also..

[codefromthecrypt]

Please do merge this. If you don't mind, I'd like to do a follow-up commit to use routine build infra.