-
Notifications
You must be signed in to change notification settings - Fork 271
Add ability to enforce concurrent query limits #3257
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
lbschanno
wants to merge
4
commits into
integration
Choose a base branch
from
task/queryLimit
base: integration
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add the ability to enforce concurrent query limits across a group of webservers. Zookeeper is used to track active queries and the following data: - The query ID - The user who submitted the query - The system the query was submitted on - The query logic the query originated from When the `ActiveQueryTracker` is instructed to track a query, the following nodes will be created in Zookeeper under the 'ActiveQueries' namespace: ``` /users/<userDn>/<queryId> /systems/<systemName>/<queryId> /queryLogics/<queryLogic>/<queryId> /queries/<queryId> /queries/<queryId>/user [data = byte[] value of userDn] /queries/<queryId>/system [data = byte[] value of systemName] /queries/<queryId>/queryLogic [data = byte[] value of queryLogic] /queries/<queryId>/heartbeats ``` This is done through the use of the `ActiveQueryTracker` class. In addition to managing the nodes that record information about the query, the `ActiveQueryTracker` class is also responsible for providing instances of the `QueryHeartbeat` class. A `QueryHeartbeat` is a wrapper around an ephemeral PersistentNode, provided by the Apache Curator library. As long as this node is present in Zookeeper for a particular query, the query will be considered to be active. Should the webservers fail over and the Zookeeper connection drop, these heartbeat nodes will automatically be deleted by Zookeeper. The `ActiveQueryTracker` is also responsible for providing instances of the `ActiveQuerySnapshot` class, which represent a snapshot of total active queries at a point in time that are associated with a particular user, system, or query logic. Query limit enforcement is done through the `QueryLimiter` class. Given a user, system, and query logic, it can determine if any of the following limits have been exceeded: - The max allowed concurrent queries for the user. - The max allowed concurrent queries of the query logic for the user. - The max allowed concurrent queries for the system. - The max allowed concurrent queries of the query logic for the system. Limits may be defined and customized on a per-user and per-system basis. They may also be defined for groups of query logics. The classes `UserLimitProvider`, `SystemLimitProvider`, and `QueryLogicGroupLimitProvider` are respectively responsible for identifying the best limits to enforce for a user, system, and query logic. They will be initialized in the `QueryLimiter` after providing a `QueryLimitConfiguration` instance. The following can be configured: On a system-wide basis: - The default concurrent user query limit. This applies to the total number of queries a user may run across all systems. May be overridden per user. - The default concurrent system query limit. Primarily to avoid a system getting overloaded. May be overridden per system. - The default of whether queries submitted to a system are counted towards the user's concurrent query total. This is always true. On a per-system basis: - The system name/ids the configuration targets. Regex matching is supported. - The concurrent system query limit. Overrides the system-wide value. - Whether queries submitted to the system count towards a user's concurrent query total. Overrides the system-wide value. - The concurrent system query limit for different query logic groups. Regex matching against group names is supported. on a per-user basis: - The user DN. - The user's concurrent query limit. Overrides the system-wide configuration. - The user's concurrent query limit for different query logic groups. Regex matching against group names is supported. On a per-query-logic-group basis: - The group name. - The query logics included in the group. Regex matching is supported. - The default concurrent user query limit. This applies to the total concurrent queries a user may run that originate from a query logic in the group across all systems. Given the possibilities for exact matches, partial regex matches, and wildcard regex matches, the determination of the best limit to use for any particular system or query logic is done by sorting matches into the following 'matching buckets' (in best-match priority): 1. Exact match 2. Partial regex (non-wildcard-only) 3. Wildcard-only regex and then selecting the lowest limit from the best bucket where we first found a match. Currently the `QueryLimiter` is used in `QueryExecutorBean`, along with a `QueryHeartbeatCache` instance to cache heartbeats and keep them alive when a running query is cached for retrieval later. For the purposes of this feature, a query is considered to start when an Accumulo connection is retrieved from the connection factory, and is considered to end when the connection is returned to the factory. The following error codes have been added: 412-20 - Concurrent query limit exceeded 500-164 - Error checking concurrent query limits Closes #3100
156ed51 to
9c59e98
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
NOTE: All tests pass during a build, but I have had issues getting queries to run correctly in quickstart, so further integration testing and fixes are needed.
Add the ability to enforce concurrent query limits across a group of webservers. Zookeeper is used to track active queries and the following data:
When the
ActiveQueryTrackeris instructed to track a query, the following nodes will be created in Zookeeper under the 'ActiveQueries' namespace:This is done through the use of the
ActiveQueryTrackerclass. In addition to managing the nodes that record information about the query, theActiveQueryTrackerclass is also responsible for providing instances of theQueryHeartbeatclass.A
QueryHeartbeatis a wrapper around an ephemeral PersistentNode, provided by the Apache Curator library. As long as this node is present in Zookeeper for a particular query, the query will be considered to be active. Should the webservers fail over and the Zookeeper connection drop, these heartbeat nodes will automatically be deleted by Zookeeper.The
ActiveQueryTrackeris also responsible for providing instances of theActiveQuerySnapshotclass, which represent a snapshot of total active queries at a point in time that are associated with a particular user, system, or query logic.Query limit enforcement is done through the
QueryLimiterclass. Given a user, system, and query logic, it can determine if any of the following limits have been exceeded:Limits may be defined and customized on a per-user and per-system basis. They may also be defined for groups of query logics. The classes
UserLimitProvider,SystemLimitProvider, andQueryLogicGroupLimitProviderare respectively responsible for identifying the best limits to enforce for a user, system, and query logic. They will be initialized in theQueryLimiterafter providing aQueryLimitConfigurationinstance. The following can be configured:On a system-wide basis:
On a per-system basis:
on a per-user basis:
On a per-query-logic-group basis:
Given the possibilities for exact matches, partial regex matches, and wildcard regex matches, the determination of the best limit to use for any particular system or query logic is done by sorting matches into the following 'matching buckets' (in best-match priority):
and then selecting the lowest limit from the best bucket where we first found a match.
Currently the
QueryLimiteris used inQueryExecutorBean, along with aQueryHeartbeatCacheinstance to cache heartbeats and keep them alive when a running query is cached for retrieval later. For the purposes of this feature, a query is considered to start when an Accumulo connection is retrieved from the connection factory, and is considered to end when the connection is returned to the factory.The following error codes have been added:
412-20 - Concurrent query limit exceeded
500-164 - Error checking concurrent query limits
Closes #3100