UBWA - subscribe_to_stream bug

[benniebendiksen]

Solution to Issue cannot be found in the documentation or other Issues and also occurs in the latest version of this library.

• I checked the documentation and other Issues. I am using the latest version of this library.

Version of this library.

2.4.0

Hardware?

VPS or other cloud hosting

Operating System?

Linux

Python version?

Python3.12

Installed wheel files

unicorn_binance_rest_api-2.2.1.dist-info/INSTALLER
  unicorn_binance_rest_api-2.2.1.dist-info/LICENSE
  unicorn_binance_rest_api-2.2.1.dist-info/METADATA
  unicorn_binance_rest_api-2.2.1.dist-info/RECORD
  unicorn_binance_rest_api-2.2.1.dist-info/WHEEL
  unicorn_binance_rest_api-2.2.1.dist-info/top_level.txt
  unicorn_binance_rest_api/__init__.cpython-312-x86_64-linux-gnu.so
  unicorn_binance_rest_api/enums.cpython-312-x86_64-linux-gnu.so
  unicorn_binance_rest_api/exceptions.cpython-312-x86_64-linux-gnu.so
  unicorn_binance_rest_api/helpers.cpython-312-x86_64-linux-gnu.so
  unicorn_binance_rest_api/manager.cpython-312-x86_64-linux-gnu.so

Installed packages

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
aiohttp                   3.9.3                    pypi_0    pypi
aiosignal                 1.3.1                    pypi_0    pypi
aniso8601                 9.0.1                    pypi_0    pypi
appdirs                   1.4.4              pyhd3eb1b0_0  
attrs                     23.1.0          py312h06a4308_0  
automat                   20.2.0                     py_0  
bcrypt                    3.2.0           py312h5eee18b_1  
blas                      1.0                         mkl  
blinker                   1.7.0                    pypi_0    pypi
bottleneck                1.3.7           py312ha883a20_0  
brotli-python             1.0.9           py312h6a678d5_7  
bzip2                     1.0.8                h5eee18b_5  
ca-certificates           2024.3.11            h06a4308_0  
certifi                   2024.2.2        py312h06a4308_0  
cffi                      1.16.0          py312h5eee18b_0  
charset-normalizer        2.0.4              pyhd3eb1b0_0  
cheroot                   10.0.0                   pypi_0    pypi
click                     8.1.7                    pypi_0    pypi
colorama                  0.4.6                    pypi_0    pypi
constantly                23.10.4         py312h06a4308_0  
cryptography              42.0.5          py312hdda0065_0  
cython                    3.0.9                    pypi_0    pypi
dateparser                1.2.0                    pypi_0    pypi
expat                     2.5.0                h6a678d5_0  
flask                     3.0.2                    pypi_0    pypi
flask-restful             0.3.10                   pypi_0    pypi
frozenlist                1.4.1                    pypi_0    pypi
hyperlink                 21.0.0             pyhd3eb1b0_0  
idna                      3.4             py312h06a4308_0  
incremental               22.10.0            pyhd3eb1b0_0  
intel-openmp              2023.1.0         hdb19cb5_46306  
itsdangerous              2.1.2                    pypi_0    pypi
jaraco-functools          4.0.0                    pypi_0    pypi
jinja2                    3.1.3                    pypi_0    pypi
ld_impl_linux-64          2.38                 h1181459_1  
libffi                    3.4.4                h6a678d5_0  
libgcc-ng                 11.2.0               h1234567_1  
libgomp                   11.2.0               h1234567_1  
libstdcxx-ng              11.2.0               h1234567_1  
libuuid                   1.41.5               h5eee18b_0  
lucit-licensing-python    1.8.2                    pypi_0    pypi
markupsafe                2.1.5                    pypi_0    pypi
mkl                       2023.1.0         h213fc3f_46344  
mkl-service               2.4.0           py312h5eee18b_1  
mkl_fft                   1.3.8           py312h5eee18b_0  
mkl_random                1.2.4           py312hdb19cb5_0  
more-itertools            10.2.0                   pypi_0    pypi
multidict                 6.0.5                    pypi_0    pypi
ncurses                   6.4                  h6a678d5_0  
numexpr                   2.8.7           py312hf827012_0  
numpy                     1.26.4          py312hc5e2394_0  
numpy-base                1.26.4          py312h0da6c21_0  
openssl                   3.0.13               h7f8727e_0  
pandas                    2.1.1           py312h526ad5a_0    anaconda
pip                       23.3.1          py312h06a4308_0  
psutil                    5.9.8                    pypi_0    pypi
pyasn1                    0.4.8              pyhd3eb1b0_0  
pyasn1-modules            0.2.8                      py_0  
pycparser                 2.21               pyhd3eb1b0_0  
pycryptodome              3.20.0                   pypi_0    pypi
pyopenssl                 24.0.0          py312h06a4308_0  
pysocks                   1.7.1           py312h06a4308_0  
python                    3.12.2               h996f2a0_0  
python-binance            1.0.19                   pypi_0    pypi
python-dateutil           2.8.2              pyhd3eb1b0_0  
python-tzdata             2023.3             pyhd3eb1b0_0  
pytz                      2023.3.post1    py312h06a4308_0  
readline                  8.2                  h5eee18b_0  
regex                     2023.12.25               pypi_0    pypi
requests                  2.31.0          py312h06a4308_0    anaconda
service_identity          18.1.0             pyhd3eb1b0_1  
setuptools                68.2.2          py312h06a4308_0  
shared-memory-dict        0.7.2                    pypi_0    pypi
simplejson                3.19.2                   pypi_0    pypi
six                       1.16.0             pyhd3eb1b0_1  
sqlite                    3.41.2               h5eee18b_0  
tbb                       2021.8.0             hdb19cb5_0  
tk                        8.6.12               h1ccaba5_0  
twisted                   23.10.0         py312h06a4308_0  
typing_extensions         4.9.0           py312h06a4308_1  
tzdata                    2024a                h04d1e81_0  
tzlocal                   5.2                      pypi_0    pypi
ujson                     5.9.0                    pypi_0    pypi
unicorn-binance-rest-api  2.2.1                    pypi_0    pypi
unicorn-binance-websocket-api 2.4.0                    pypi_0    pypi
unicorn-fy                1.14.1                   pypi_0    pypi
urllib3                   1.26.18         py312h06a4308_0  
websocket-client          1.7.0                    pypi_0    pypi
websockets                11.0.3                   pypi_0    pypi
werkzeug                  3.0.1                    pypi_0    pypi
wheel                     0.41.2          py312h06a4308_0  
xz                        5.4.6                h5eee18b_0  
yarl                      1.9.4                    pypi_0    pypi
zlib                      1.2.13               h5eee18b_0  
zope                      1.0             py312h06a4308_1  
zope.interface            5.4.0           py312h5eee18b_0

Logging output

No response

Processing method?

process_stream_data

Used endpoint?

binance.com

Issue

ubwa_stream_subscribe_issue.py.log.zip
ubwa_stream_subscribe_issue.py.zip

periodic calls into subscribe_to_stream() returns a payload value of False, and the corresponding market does not undergo the subscription. This error happens periodically with no clear pattern as to when.

Print statements at the time of the first occurrence was as follows:

UNSUBSCRIBED btttrx, market count: 702
2024-04-13 23:30:16: FAILED TO SUBSCRIBE TO wintrx STREAM!!! Adding to self.unicorn_failed_market_subscriptions ...
Len of self.unicorn_failed_market_subscriptions: 1
UNSUBSCRIBED chzbnb, market count: 701

Log file simply shows that the coin in question, wintrx, was listed in the list of subscription markets, and subsequently, that the payload value was False.

[oliver-zehentleitner]

Hi Pablo!

I know the code part that is responsible for this and I want to rework it because of the Binance Websocket API. I want to send directly via loop.create_task() to reduce the latency, probably I can fix your issue as well. I will report here as soon as you can try it again.

[benniebendiksen]

Hi Oliver,

Please forgive my delayed response. It is the busiest time of the academic semester for me.

The issue still exists as of today (version 2.4.0). From inspecting the ubwa manager class code statically, I think the issue of the False payload value may be localized here:

https://github.com/LUCIT-Systems-and-Development/unicorn-binance-websocket-api/blob/d212b810828bfd598cadd24d9a7dce44ccedc088/unicorn_binance_websocket_api/manager.py#L4018

If this is correct then I am curious as to why calls into this split_payload() method from the "unsubscribe" method do not result in the returned value of False (I have only observed the issue with subscribe attempts).

Can you show me how calling into subscribe_to_stream() results in sending API requests via the established websocket connection? I cannot find the link between appending to self.stream_list[stream_id]['payload']—which is what I understand subscribe_to_stream() and unsubscribe_from_stream() to be doing (in the absence of this issue)—and the addition of a task to an event loop or some other mechanism to send an API request.

In any case, if self.stream_list[stream_id]['payload'] does not get appended to from within subscribe_to_stream() whenever the issue is encountered, then doesn't that mean that no request gets sent to the Binance Websocket API?

I can try to dynamically trace the issue when I have more time, if it would help, which would be after mid-May.

[oliver-zehentleitner]

Good approach from you, I will test why the False occurs at all.

I first have to integrate the option of accessing the event loop directly and sending it via websocket, which is not yet available. Currently the payload is stored on a stack and another process picks it up from the stack in the loop and then sends it (not optimal ^^) In the future I would like to have this done directly via a separate asyncio task.

[oliver-zehentleitner]

A legitimate reason for a False would be if max_subscriptions_per_stream is exceeded! Have you checked this? You should then find a "The limit of X has been exceeded!" in the logs and you can retrieve the current number of subscriptions:
https://unicorn-binance-websocket-api.docs.lucit.tech/unicorn_binance_websocket_api.html#unicorn_binance_websocket_api.manager.BinanceWebSocketApiManager.get_limit_of_subscriptions_per_stream
https://unicorn-binance-websocket-api.docs.lucit.tech/unicorn_binance_websocket_api.html#unicorn_binance_websocket_api.manager.BinanceWebSocketApiManager.get_number_of_subscriptions

Another reason would be the TypeError that is caught, here you should also find a log note!

I have now equipped all the functions involved with logging at the crucial points, if you activate debug level "ERROR", you will have all the relevant information in the log before the False is returned. So we can better investigate the reason.

[oliver-zehentleitner]

Please test with version 2.6.0 and keep an eye on the max subscriptions!

[benniebendiksen]

A legitimate reason for a False would be if max_subscriptions_per_stream is exceeded! Have you checked this? You should then find a "The limit of X has been exceeded!" in the logs and you can retrieve the current number of subscriptions: https://unicorn-binance-websocket-api.docs.lucit.tech/unicorn_binance_websocket_api.html#unicorn_binance_websocket_api.manager.BinanceWebSocketApiManager.get_limit_of_subscriptions_per_stream https://unicorn-binance-websocket-api.docs.lucit.tech/unicorn_binance_websocket_api.html#unicorn_binance_websocket_api.manager.BinanceWebSocketApiManager.get_number_of_subscriptions

Another reason would be the TypeError that is caught, here you should also find a log note!

I have now equipped all the functions involved with logging at the crucial points, if you activate debug level "ERROR", you will have all the relevant information in the log before the False is returned. So we can better investigate the reason.

Hi Oliver,

Thank you for looking into this. I keep an eye on the stream subscription count by printing the market count associated with the stream. Also, in the example script for this issue—ubwa_stream_subscribe_issue.py, we initialize markets to not exceed the subscription limit (I believe the value is 1024 markets), i.e., markets = markets[:subscribe_limit]. The subscribe_limit we grab from the method call indicated in your first provided web link. We then associate boolean values to these markets by sampling a uniform distribution and create the stream by subscribing to the markets associated with a value of True. We then iteratively unsubscribe from subscribed markets and subscribe to unsubscribed markets. So on average there will be 50% of the subscription number limit of markets subscribed to the stream, and never will there be more than the limit. Nevertheless, as an added measure, I will go ahead and use the get_number_of_subscriptions() method you reference with your second provided web-link. But my belief is that the problem isn't there.

The TypeError, to my understanding, used to result when the program would attempt to iterate the payload variable, thereby assuming it to be of an iterable type. Yet, as we know, that payload variable is actually a Bool at times when the bug shows up. In version 2.4.0 the TypeError stopped getting thrown because of the try-catch wrap you placed before the program attempts to iterate the payload.

As for a log note in either case, I will go ahead and run the example script until the bug appears, terminate the program, and inspect all logs in the generated log file.

Thank you for extending the logger writes around more points surrounding this problem. I'll go ahead and activate the "ERROR" level of the logger when undergoing my testing.

[benniebendiksen]

Good approach from you, I will test why the False occurs at all.

I first have to integrate the option of accessing the event loop directly and sending it via websocket, which is not yet available. Currently the payload is stored on a stack and another process picks it up from the stack in the loop and then sends it (not optimal ^^) In the future I would like to have this done directly via a separate asyncio task.

Ah, thank you for the explanation. I agree that an immediate addition of a task to a running event loop, as a result of a subscription or unsubscription method call, would be optimal.

[benniebendiksen]

Please test with version 2.6.0 and keep an eye on the max subscriptions!

Yes, I will definitely do so. Please allow me to do so either Tuesday or Wednesday of the upcoming week.

[oliver-zehentleitner]

The limits are different "binance.com" supports 1024 while "binance.com-futures" only supports 200: https://github.com/LUCIT-Systems-and-Development/unicorn-binance-websocket-api/wiki/Binance-websocket-endpoint-configuration-overview

UBWA knows these limits and you can use get_limit_of_subscriptions_per_stream() to query the limits for the endpoint used and get_number_of_subscriptions() to calculate the free slots. Or you can make it even easier for yourself and use get_number_of_free_subscription_slots().

[oliver-zehentleitner]

It would actually be useful to throw an exception as soon as too many subscriptions are made.

[benniebendiksen]

It would actually be useful to throw an exception as soon as too many subscriptions are made.

Agreed. Good idea

[benniebendiksen]

get_limit_of_subscriptions_per_stream() to query the limits for the endpoint used

I apologize, I should've specified. I was referring to "binance.com"; I've not tested for the issue in the futures markets. I'll make sure to keep calling get_number_of_free_subscription_slots() to verify that free slots remain available at times when the payload bug occurs.