Intermittent Dataset Retrieval Errors When Making Parallel API Requests

[orlandolaycos]

Bug description

Describe the bug
This issue occurs specifically with the /dataset method. It does not consistently return the correct dataset when called concurrently with Rison encoded filter looking for the "table_name".
The problem manifests about 5% of the time, indicating it may be a race condition or a resource contention issue.

To Reproduce

1. Make multiple parallel API requests to the /dataset endpoint, each querying for different "table_name" values.
2. Compare the returned datasets with the expected results based on the initial queries.

What I've tried:

• When the error occurs, a subsequent call with the same query returns the correct dataset, suggesting that the issue may not be with the dataset itself but rather with how the API handles concurrent requests.
• Serial calls (one after the other) do not produce errors, indicating that the problem is likely related to concurrency.
• Using 'mitmweb' to monitor traffic revealed that the error originates from the Superset response.

Screenshots/recordings

N.A.

Superset version

v5.0.0rc3
v4.1.2

Python version

3.9

Node version

22

Browser

Chrome

Additional context

I'm developing an application that generates dashboards "on the fly" where each chart is basically fed by a dataset (in a 1:1 relationship). Since the dataset's table_name is a unique ID, I'm using it to retrieve the dataset through the API, giving it a fixed UUID based on the user/dashboard.

I am making approximately 12 concurrent requests to retrieve around 70 datasets, which are updated frequently.

Checklist

• I have searched Superset docs and Slack and didn't find a solution to my problem.
• I have searched the GitHub issue tracker and didn't find a similar bug report.
• I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

[orlandolaycos]

I've created a small example in Python, separate from my main project code. Forgive the informality, there are several additional tests to verify that we are properly "connected" to the API. The case we're interested in is the "extress_dataset_api" function.

pruebas.zip

[LisaHusband]

I fixed the bug in the sample code you provided and published it to this repository

[orlandolaycos]

I fixed the bug in the sample code you provided and published it to this repository

For a moment, seeing that you had corrected my code, I thought I had made a childish mistake 😀

[LisaHusband]

For superset, the bug t is true. but for your code , the bug is true also.
obviously，

[LisaHusband]

I fixed the bug in the sample code you provided and published it to this repository

For a moment, seeing that you had corrected my code, I thought I had made a childish mistake 😀

I'm sorry that my expression is not accurate enough. In your code, I added a correctness statistic, and fixed the incomplete collection of uppercase letters in your regular expression, as well as the bug of undefined variables and request parameters.

I also modified the superset code and rewrote the filter method to make it able to handle high concurrent requests correctly,, and published it in this repository. You can merge your local code and test to see if the problem has been solved. I have already pulled the PR request, but for reasons I can't explain, I haven't requested a review.

[LisaHusband]

@orlandolaycos So is your problem solved?

[rusackas]

Would love to see the linked PR made ready for review/merging if it resolves the problem. Thanks for the help @LisaHusband !

[orlandolaycos]

@orlandolaycos So is your problem solved?

@LisaHusband Yes, the issue has been resolved with the changes in the pull request. Thank you for your support!