Live data feed query results in pipeline runner

[scottsdevelopment]

What is the proposed architecture for a long running live session of market data?

Currently the PipelineRunner will expect a Source to return a Iterator[Candle]. The AsyncQueryResult is being returned by the AsyncMarketProvider.request_symbol_history and our QuerySubscription attached to the results will wait for the done_event before returning. Once the market provider returns the symbol history and our subscriptions complete, the pipeline ends.

Ideally, the architecture should be able listen for incoming candles from a source and continuously process new incoming data through the pipeline.

Any thoughts on the implementation of this would be greatly appreciated.

This might be related to #2 section 2.

[idanya]

AsyncQueryResult, QuerySubscription, and AsyncMarketProvider, were written entirely to enable history loading from IB (IBMarketProvider). As you can see, the single function there is request_symbol_history.

Ideally, in order to implement streaming market data (running throughout the trading session), you'd want to implement an adaptor against the chosen broken API that will expose the generic Source iterator.

The pipeline itself cares only for the Source class. Meaning that it will probably be up to the adaptor implementation to keep the 3rd party connection open and receiving data, transform it, and pass it to a Source implementation.

We could wrap it in some kind of "adaptor runner" that will bridge the broker to the Source object. But that means creating an interface that will bind the different adaptor's implementations. Because there is no implementation for a streaming provider at the moment, I didn't see a reason to create it yet. After the first implementation, we'll get a sense of how the interface should look and maybe go in that direction.

[scottsdevelopment]

I think because of the current state of my additional providers, the best way to get this feature to the main project would be to create a dummy streaming provider which returns some fixed bars over time and completes the interface for technical reference.

[idanya]

I'm not familiar with many brokers streaming APIs to know how the interface should be designed. There are APIs that are symbol based and you have to request symbols one by one. There are fire hoses that send you everything. There are pulling ones...

I'd really like to have several providers before committing to a unified binding interface.

It is possible to go and create this dummy adaptor that will provide a long-running Source just to close the circle and have a running example end-to-end while keeping in mind the possibility to have a type of adaptor manager that will bridge the two via a unified interface in a later stage when we have a better sense of how the interface should look like (with real brokers).

[scottsdevelopment]

The current provider that I am working on implementing is TradingView Pro unofficial websockets feed.