Limit the total number of queries a user can run concurrently in the system

[ivakegg]

The idea is that a group of webservers needs to coordinate how many queries a user can run concurrently. The webserver zookeeper would be the place to coordinate the number of currently running queries for a user. The system needs to be able to set limit on a user overall, and set a separate limit on the number of queries for a user per query logic. The use case is that lookup UUID queries may have a higher threshold but all other queries would fall under a lower concurrent limit per user.

[jstewart-shield]

I want to add some more details as it's a little more nuanced than this. We limit both systems (SYSTEM_FROM) and users. Sometimes the queries against a system also count against a user and sometimes they don't and we need to be able to set up/manage the limits this way.

There needs to be a user default concurrency limit. Systems that do not have a custom number set should count against the user limit.

For example, lets say I have 4 systems, 5 users, 2 query logics:

User default is 10 concurrent queries.

SYSTEM1 - limit of 100 queries concurrently, does not count against user
SYSTEM2 - limit of 200 queries concurrently, counts against user
SYSTEM3 - not configured
SYSTEM4 - limit of 100 queries concurrently, does not count against user

USER1 - limit of 100 queries concurrently
USER2 - default
USER3 - default
USER4 - default
USER5 - default

Assuming no queries finish during my scenario:
USER1 submits 50 queries via SYSTEM1

• User count = 0, SYSTEM1 count = 50
  USER2 submits 5 queries via SYSTEM2
• user count = 5, SYSTEM2 count = 5
  USER3 submits 10 queries via SYSTEM3
• user count = 10, nothing is counted as a system count
  USER4 submits 100 queries via SYSTEM1
• user count = 0, SYSTEM1 accepts 50 queries and errors the other 50 as hitting the system concurrency limit
  USER5 submits 20 queries via SYSTEM2
• 10 queries accepted, 10 error due to hitting the user concurrency limit, user count = 10, SYSTEM2 = 10
  USER5 submits 20 queries via SYSTEM4
• queries are accepted and user count remains at 10 from previous submission, SYSTEM4 count = 20

To add more complexity, we need to be able to group query logics together. For example, EventQuery and TLDEvent query would count in the same group of queries. Limiting by query pool would also work to solve this part of the problem if that's easier/better.

Hope this makes sense. Reach out with any questions.

[lbschanno]

@jstewart-shield thank you for the additional details, it did clarify some questions I had. For the query logic groups, would you want to impose limits for them per user, per system, or both?

[jstewart-shield]

Both please Julie

…

________________________________ From: Laura Schanno ***@***.***> Sent: Friday, August 22, 2025 11:48:01 AM To: NationalSecurityAgency/datawave ***@***.***> Cc: jstewart-shield ***@***.***>; Mention ***@***.***> Subject: Re: [NationalSecurityAgency/datawave] Limit the total number of queries a user can run concurrently in the system (Issue #3100) [https://avatars.githubusercontent.com/u/52420215?s=20&v=4]lbschanno left a comment (NationalSecurityAgency/datawave#3100)<#3100 (comment)> @jstewart-shield<https://github.com/jstewart-shield> thank you for the additional details, it did clarify some questions I had. For the query logic groups, would you want to impose limits for them per user, per system, or both? — Reply to this email directly, view it on GitHub<#3100 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BLZ573SW57S4CFEBGNV6EFT3O43TDAVCNFSM6AAAAACDXIFALCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEMJUHA2DCNZYHE>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[lbschanno]

Here are the requirements as I understand them. Please let me know if I got anything wrong.

From a configuration basis:

On a system-wide basis we need to be able to configure:

• The default concurrent system query limit that will be applied to each system unless overridden at the system level.
• The default concurrent user query limit that will be applied to each system unless overridden at the system level.
• The default concurrent system query limit for all query logic groups to be applied to each system unless overridden at the system level.
• The default concurrent user query limit for all query logic groups to be applied to each system, unless overriden at the system level. (Possibly not needed, see questions at the end)
• The default of whether the system query limit applies to the user query limit. True unless overridden at the system level.

On a per-system level, we need to be able to configure:

• A concurrent system query limit if different from the default. Overrides the system-wide configuration.
• A concurrent user query limit if different from the default. Overrides the system-wide configuration.
• Whether the system query limits apply to the user query limit if different from the default. Overrides the system-wide configuration.
• For specific query logic groups on a system:
  • The max concurrent query limit for a specified query logic group on the system. Overrides the system-wide configuration.
• For individual users on a system:
  • Their max concurrent query limit on the system. Overrides the system-wide configuration, and overrides the system-level user query limit.
  • Their max concurrent query limit for a specified query logic group on the system. Overrides the system-wide configuration, and overrides the system-level query logic group query limit.

On a per-query logic group level, we need to be able to configure:

• The name of the query logic group
• The names of the query logics included in the group

Some questions:

1. Should we allow query logics to be added to multiple query logic groups?
2. Should query logic group limits be enabled by default across all systems, or only where specified in the system-level configuration. For example, if we have a query logic group 'uuidLogicGroup', should limits for this group be enforced across all existing systems, or should individual systems be required to specify a limit for 'uuidLogicGroup' for its query limit to be enabled and enforced?
3. What limits are given priority over others? For instance, are user limits given priority over query logic group limits, or is it a matter of which limit is lower?

[jstewart-shield]

Ivan - I’m not sure if Laura gets my replies by email so, if not, can you please make sure she sees this. I don’t have my login info with me. Laura - comments in line below. I apologize if it’s hard to read, I’m on vacation so responding from my phone. Julie

…

________________________________ From: Laura Schanno ***@***.***> Sent: Friday, August 22, 2025 7:09 PM To: NationalSecurityAgency/datawave ***@***.***> Cc: jstewart-shield ***@***.***>; Mention ***@***.***> Subject: Re: [NationalSecurityAgency/datawave] Limit the total number of queries a user can run concurrently in the system (Issue #3100) [https://avatars.githubusercontent.com/u/52420215?s=20&v=4]lbschanno left a comment (NationalSecurityAgency/datawave#3100)<#3100 (comment)> Here are the requirements as I understand them. Please let me know if I got anything wrong. From a configuration basis: On a system-wide basis we need to be able to configure: * The default concurrent system query limit that will be applied to each system unless overridden at the system level. Julie - The original thought was that there is no system default, just a user default. If a system is not configured then all queries are based only on the user queries. But also, the system query configuration should allow regex (or at least a generic wildcard as we have some systems that append things at the end of their system name). Something like SYSTEM* gets 50 but SYSTEM125 gets 100. The more specific system should be honored and the wildcarded one matches only if a specific system doesn’t match. * The default concurrent user query limit that will be applied to each system unless overridden at the system level. Julie - The default concurrent user limit is independent of the system limit. If a user is allowed 10 queries they may have all 10 from one system or one from 10 different systems…or some other combination * The default concurrent system query limit for all query logic groups to be applied to each system unless overridden at the system level. Julie - Per above, there is no default system limit - unless that makes this more difficult, or you generally think it’s a good idea. As I think about it it probably does make sense to have one as we could always set it to something pretty high but still have it to make sore no one system is overrunning us. * The default concurrent user query limit for all query logic groups to be applied to each system, unless overriden at the system level. (Possibly not needed, see questions at the end) Julie - Sort of. The query limit is overall for the user, not by system. The thing that is overridden at the system level is just that the queries to that system don’t count against the user. * The default of whether the system query limit applies to the user query limit. True unless overridden at the system level. Julie - Correct On a per-system level, we need to be able to configure: * A concurrent system query limit if different from the default. Overrides the system-wide configuration. Julie - Correct * A concurrent user query limit if different from the default. Overrides the system-wide configuration. Julie - Correct * Whether the system query limits apply to the user query limit if different from the default. Overrides the system-wide configuration. Julie - Correct For specific query logic groups on a system: * The max concurrent query limit for a specified query logic group on the system. Overrides the system-wide configuration. Julie - correct For individual users on a system: * Their max concurrent query limit on the system. Overrides the system-wide configuration, and overrides the system-level user query limit. Julie - correct except there is no system user limit. Just a system limit and a user limit and a configuration on whether or not the query applies just to the system or both the system and the user * Their max concurrent query limit for a specified query logic group on the system. Overrides the system-wide configuration, and overrides the system-level query logic group query limit. Julie - No, if a query hits either the user limit or the system limit it is rejected. However, it would be nice to have a different message returned so it’s clear why the query was rejected - system query limit reached or user query limit reached. On a per-query logic group level, we need to be able to configure: * The name of the query logic group Julie - correct * The names of the query logics included in the group Julie - correct Some questions: 1. Should we allow query logics to be added to multiple query logic groups? Julie - Yes 2. Should query logic group limits be enabled by default across all systems, or only where specified in the system-level configuration. For example, if we have a query logic group 'uuidLogicGroup', should limits for this group be enforced across all existing systems, or should individual systems be required to specify a limit for 'uuidLogicGroup' for its query limit to be enabled and enforced? Julie - Great question. And perfect example because those are faster queries to be serviced so that is a group that we would generally allow more of. I’m thinking we will want to be able to set defaults for users in a couple groups we set (basically a generic lookup one and a catch all one with regular event type queries) but not necessarily group all system queries in those groups. For example, some systems we only allow them to have certain query logics so we might have some groups created that are 0. And one system we have submits all different query logics but they manage what they send when so for them we will just want an overall limit without specific query logic groups. 3. What limits are given priority over others? For instance, are user limits given priority over query logic group limits, or is it a matter of which limit is lower? Julie - whichever is the lower limit — Reply to this email directly, view it on GitHub<#3100 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BLZ573WW4SO25M4CQKD3EJ33O6PKZAVCNFSM6AAAAACDXIFALCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEMJVHEYTOOBZGI>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[lbschanno]

@jstewart-shield I got your reply by email and was notified of it. Thank you for further clarifying and responding to my questions. After reading over your comments, this is now my understanding of the requirements:

On a system-wide basis, we need to be able to configure:

• The default concurrent user query limit. This applies to the total number of queries a user may run across all systems. May be overridden per user.
• The default concurrent system query limit. Primarily to avoid a system getting overloaded. May be overriden per system.
• The default of whether queries submitted to the system are counted against the user's query limit. Default should be true.

On a per-system basis, we need to be able to configure:

• The system names the configuration targets. Must support regex matching to match multiple systems.
• The concurrent system query limit. Overrides the system-wide configuration.
• Whether queries submitted to the system are counted against the user's query limit. Overrides the system-wide configuration.
• The concurrent system query limit for a specific query logic group. Should support regex matching to match multiple groups.

On a per-user basis, we need to be able to configure:

• The user id.
• The user's concurrent query limit. Overrides the system-wide configuration.
• The user's concurrent query limit for a specific query logic group. Should support regex matching to match multiple groups.

On a per-query-logic-group basis we need to be able to configure:

• The name of the query logic group.
• The names of the query logics included in the group. Should support regex matching.
• The default concurrent user query limit. This applies to the total concurrent queries a user may run that originate from a query logic in the group across all systems.

An additional question about query logic limits:

3. What limits are given priority over others? For instance, are user limits given priority over query logic group limits, or is it a matter of which limit is lower?
   Julie - whichever is the lower limit

Given the following scenario:

Default user query limit: 10

Logic group 'UUID_GROUPS'

• Logic names: *UUID*
• User query limit: 20

Based off the your statement above, we would expect all users to by default to run at most 10 queries, even if they fall into the group UUID_GROUP. Is there ever a case when we would want to allow the higher limit to take priority?

[jstewart-shield]

Sorry, I misunderstood the last question. I was thinking system vs user. In the example scenario the user would be allowed to submit 20 queries to any logic that contains UUID then 10 additional queries to any other query logic. So each query logic group is its own count and anything that isn’t specifically set is grouped together as the remainder. Everything else is correct. Julie

…

________________________________ From: Laura Schanno ***@***.***> Sent: Monday, August 25, 2025 2:37:42 PM To: NationalSecurityAgency/datawave ***@***.***> Cc: jstewart-shield ***@***.***>; Mention ***@***.***> Subject: Re: [NationalSecurityAgency/datawave] Limit the total number of queries a user can run concurrently in the system (Issue #3100) [https://avatars.githubusercontent.com/u/52420215?s=20&v=4]lbschanno left a comment (NationalSecurityAgency/datawave#3100)<#3100 (comment)> @jstewart-shield<https://github.com/jstewart-shield> I got your reply by email and was notified of it. Thank you for further clarifying and responding to my questions. After reading over your comments, this is now my understanding of the requirements: On a system-wide basis, we need to be able to configure: * The default concurrent user query limit. This applies to the total number of queries a user may run across all systems. May be overridden per user. * The default concurrent system query limit. Primarily to avoid a system getting overloaded. May be overriden per system. * The default of whether queries submitted to the system are counted against the user's query limit. Default should be true. On a per-system basis, we need to be able to configure: * The system names the configuration targets. Must support regex matching to match multiple systems. * The concurrent system query limit. Overrides the system-wide configuration. * Whether queries submitted to the system are counted against the user's query limit. Overrides the system-wide configuration. * The concurrent system query limit for a specific query logic group. Should support regex matching to match multiple groups. On a per-user basis, we need to be able to configure: * The user id. * The user's concurrent query limit. Overrides the system-wide configuration. * The user's concurrent query limit for a specific query logic group. Should support regex matching to match multiple groups. On a per-query-logic-group basis we need to be able to configure: * The name of the query logic group. * The names of the query logics included in the group. Should support regex matching. * The default concurrent user query limit. This applies to the total concurrent queries a user may run that originate from a query logic in the group across all systems. An additional question about query logic limits: 1. What limits are given priority over others? For instance, are user limits given priority over query logic group limits, or is it a matter of which limit is lower? Julie - whichever is the lower limit Given the following scenario: Default user query limit: 10 Logic group 'UUID_GROUPS' * Logic names: *UUID* * User query limit: 20 Based off the your statement above, we would expect all users to by default to run at most 10 queries, even if they fall into the group UUID_GROUP. Is there ever a case when we would want to allow the higher limit to take priority? — Reply to this email directly, view it on GitHub<#3100 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BLZ573QR6523GYA27C5JQ733PNJXNAVCNFSM6AAAAACDXIFALCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEMRRGMZTQMZYGI>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[lbschanno]

Understood, thanks for verifying.

[jstewart-shield]

One more thought. If a query logic does end up in more than one group that is not the default the it would be the lower of the two. Julie

…

________________________________ From: Laura Schanno ***@***.***> Sent: Monday, August 25, 2025 11:36:43 PM To: NationalSecurityAgency/datawave ***@***.***> Cc: jstewart-shield ***@***.***>; Mention ***@***.***> Subject: Re: [NationalSecurityAgency/datawave] Limit the total number of queries a user can run concurrently in the system (Issue #3100) [https://avatars.githubusercontent.com/u/52420215?s=20&v=4]lbschanno left a comment (NationalSecurityAgency/datawave#3100)<#3100 (comment)> Understood, thanks for verifying. — Reply to this email directly, view it on GitHub<#3100 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BLZ573WCAGRHEONSVYIOPIT3PPI4XAVCNFSM6AAAAACDXIFALCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEMRSGQ3DGOBRHE>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[lbschanno]

What if we end up with something along the following scenario:

Group 1:
Name: ALL_QUERIES
Logic names: *
User query limit: 10

Group 2:
Name: UUID_QUERIES
Logic names: *UUID*
User query limit: 20

Group 3:
Name: LUCENE_UUID_QUERIES
Logic names: LuceneUUID* (More specific regex than group 2)
User query limit: 30

Group 4
Name: LUCENE_UUID_EVENT_QUERIES_ONLY
Logic name: LuceneUUIDEventQuery (direct match only, no regex, more specific than group 3)
User query limit: 40

A user submits queries from the following query logics:

• 1 query from query logic EventQuery - best match is group 1
• 1 query from query logic KeywordUUIDQuery - best match is group 2
• 1 query from query logic LuceneUUIDTokenizedQuery - best match is group 3
• 1 query from query logic LuceneUUIDEventQuery - direct match to group 4

Do we need to try to make a determination of the best-matching regex pattern when no direct match exists, or should we effectively say that a query could fall into one of the following query logic group buckets (in best-match priority):

1. Direct match - find lowest limit among any groups with direct match to query logic name
2. Non-wildcard-only-regex: find lowest limit among any groups with a matching regex that isn't only a wildcard.
3. Wildcard-only regex: Matches all query logics. Find lowest limit among these.

[jstewart-shield]

Yes, I think this logic makes the most sense. Also, you would have this scenario for both user queries and system queries. Also, this I good logic to use with system names as well since we are allowing wildcarding there too (though I think we may have already said that?). Julie

…

________________________________ From: Laura Schanno ***@***.***> Sent: Tuesday, August 26, 2025 12:08:29 AM To: NationalSecurityAgency/datawave ***@***.***> Cc: jstewart-shield ***@***.***>; Mention ***@***.***> Subject: Re: [NationalSecurityAgency/datawave] Limit the total number of queries a user can run concurrently in the system (Issue #3100) [https://avatars.githubusercontent.com/u/52420215?s=20&v=4]lbschanno left a comment (NationalSecurityAgency/datawave#3100)<#3100 (comment)> What if we end up with something along the following scenario: Group 1: Name: ALL_QUERIES Logic names: * User query limit: 10 Group 2: Name: UUID_QUERIES Logic names: *UUID* User query limit: 20 Group 3: Name: LUCENE_UUID_QUERIES Logic names: LuceneUUID* (More specific regex than group 2) User query limit: 30 Group 4 Name: LUCENE_UUID_EVENT_QUERIES_ONLY Logic name: LuceneUUIDEventQuery (direct match only, no regex, more specific than group 3) User query limit: 40 A user submits queries from the following query logics: * 1 query from query logic EventQuery - best match is group 1 * 1 query from query logic KeywordUUIDQuery - best match is group 2 * 1 query from query logic LuceneUUIDTokenizedQuery - best match is group 3 * 1 query from query logic LuceneUUIDEventQuery - direct match to group 4 Do we need to try to make a determination of the best-matching regex pattern when no direct match exists, or should we effectively say that a query could fall into one of the following query logic group buckets: 1. Direct match - find lowest limit among any groups with direct match to query logic name 2. Non-wildcard-only-regex: find lowest limit among any groups with a matching regex that isn't only a wildcard. 3. Wildcard-only regex: Matches all query logics. Find lowest limit among these. — Reply to this email directly, view it on GitHub<#3100 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BLZ573VOFUOEFP76ZMZP7S33PPMT3AVCNFSM6AAAAACDXIFALCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEMRSGU2DGMBYGI>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[lbschanno]

Understood, I'll do a similar bucketing for system configurations based on their names.