Add ability to enforce concurrent query limits

Add the ability to enforce concurrent query limits across a group of
webservers. Zookeeper is used to track active queries and the following
data:
- The query ID
- The user who submitted the query
- The system the query was submitted on
- The query logic the query originated from

When the `ActiveQueryTracker` is instructed to track a query, the
following nodes will be created in Zookeeper under the 'ActiveQueries'
namespace:

```
/users/<userDn>/<queryId>
/systems/<systemName>/<queryId>
/queryLogics/<queryLogic>/<queryId>
/queries/<queryId>
/queries/<queryId>/user           [data = byte[] value of userDn]
/queries/<queryId>/system         [data = byte[] value of systemName]
/queries/<queryId>/queryLogic     [data = byte[] value of queryLogic]
/queries/<queryId>/heartbeats
```

This is done through the use of the `ActiveQueryTracker` class. In
addition to managing the nodes that record information about the query,
the `ActiveQueryTracker` class is also responsible for providing
instances of the `QueryHeartbeat` class.

A `QueryHeartbeat` is a wrapper around an ephemeral PersistentNode,
provided by the Apache Curator library. As long as this node is present
in Zookeeper for a particular query, the query will be considered to be
active. Should the webservers fail over and the Zookeeper connection
drop, these heartbeat nodes will automatically be deleted by Zookeeper.

The `ActiveQueryTracker` is also responsible for providing instances of
the `ActiveQuerySnapshot` class, which represent a snapshot of total
active queries at a point in time that are associated with a particular
user, system, or query logic.

Query limit enforcement is done through the `QueryLimiter` class. Given
a user, system, and query logic, it can determine if any of the
following limits have been exceeded:

- The max allowed concurrent queries for the user.
- The max allowed concurrent queries of the query logic for the user.
- The max allowed concurrent queries for the system.
- The max allowed concurrent queries of the query logic for the system.

Limits may be defined and customized on a per-user and per-system basis.
They may also be defined for groups of query logics. The classes
`UserLimitProvider`, `SystemLimitProvider`, and
`QueryLogicGroupLimitProvider` are respectively responsible for
identifying the best limits to enforce for a user, system, and query
logic. They will be initialized in the `QueryLimiter` after providing a
`QueryLimitConfiguration` instance. The following can be configured:

On a system-wide basis:
- The default concurrent user query limit. This applies to the total
  number of queries a user may run across all systems. May be overridden
  per user.
- The default concurrent system query limit. Primarily to avoid a system
  getting overloaded. May be overridden per system.
- The default of whether queries submitted to a system are counted
  towards the user's concurrent query total. This is always true.

On a per-system basis:
- The system name/ids the configuration targets. Regex matching is
  supported.
- The concurrent system query limit. Overrides the system-wide value.
- Whether queries submitted to the system count towards a user's
  concurrent query total. Overrides the system-wide value.
- The concurrent system query limit for different query logic groups.
  Regex matching against group names is supported.

on a per-user basis:
- The user DN.
- The user's concurrent query limit. Overrides the system-wide
  configuration.
- The user's concurrent query limit for different query logic groups.
  Regex matching against group names is supported.

On a per-query-logic-group basis:
- The group name.
- The query logics included in the group. Regex matching is supported.
- The default concurrent user query limit. This applies to the total
  concurrent queries a user may run that originate from a query logic in
  the group across all systems.

Given the possibilities for exact matches, partial regex matches, and
wildcard regex matches, the determination of the best limit to use for
any particular system or query logic is done by sorting matches into the
following 'matching buckets' (in best-match priority):

1. Exact match
2. Partial regex (non-wildcard-only)
3. Wildcard-only regex

and then selecting the lowest limit from the best bucket where we first
found a match.

Currently the `QueryLimiter` is used in `QueryExecutorBean`, along with a
`QueryHeartbeatCache` instance to cache heartbeats and keep them alive
when a running query is cached for retrieval later. For the purposes of
this feature, a query is considered to start when an Accumulo connection
is retrieved from the connection factory, and is considered to end when
the connection is returned to the factory.

The following error codes have been added:
412-20  - Concurrent query limit exceeded
500-164 - Error checking concurrent query limits

Closes #3100

Author: lbschanno

core/base-rest-responses/src/main/java/datawave/webservice/query/exception/DatawaveErrorCode.java

@@ -169,6 +169,7 @@ public enum DatawaveErrorCode {
     QUERY_PLAN_ERROR(500, 161, "Error retrieving plan for query."),
     QUERY_PREDICTIONS_ERROR(500, 162, "Error retrieving predictions for query."),
     QUERY_VALIDATION_ERROR(500, 163, "Error validating query."),
+    CONCURRENT_QUERY_LIMIT_ERROR(500, 164, "Error checking concurrent query limits."),
     // 204 No Content
     NO_QUERIES_FOUND(204, 1, "No queries found for user."),
     RESULTS_NOT_SENT(204, 2, "Results not sent."),
@@ -275,7 +276,8 @@ public enum DatawaveErrorCode {
     CURRENT_AND_PREVIOUS_EVENT_ORDER_INVALID(412, 16, "Current event and previous event are not in chronological order"),
     CURRENT_AND_NEXT_EVENT_ORDER_INVALID(412, 17, "Current event and next event are not in chronological order"),
     FIELD_PHRASE_QUERY_NOT_INDEXED(412, 18, "Field cannot be queried as a phrase since it was not indexed as such."),
-    NO_QUERY_VALIDATION_RULES_CONFIGURED(412, 19, "No query validation rules configured for the query logic.");
+    NO_QUERY_VALIDATION_RULES_CONFIGURED(412, 19, "No query validation rules configured for the query logic."),
+    CONCURRENT_QUERY_LIMIT_EXCEEDED(412, 20, "Concurrent query limit exceeded.");
 
     private String message;
     private int httpCode;

core/utils/type-utils/src/main/java/datawave/query/parser/JavaRegexAnalyzer.java

@@ -23,7 +23,7 @@ public class JavaRegexAnalyzer {
 
     // Types as applied to portions of the regex. We are interested in portions that
     // are literals and those that contain regex constructs.
-    private enum RegexType {
+    public enum RegexType {
         LITERAL(true), // a literal value
         ESCAPED_LITERAL(true), // an escaped literal (e.g. \[ or \.)
         REGEX(false), // a regex
@@ -42,7 +42,7 @@ public boolean isLiteral() {
         }
     }
 
-    private static class RegexPart {
+    public static class RegexPart {
         // the regex is not-final to allow applyRegexCaseSensitivity
         public String regex;
         public RegexType type;
@@ -58,6 +58,18 @@ public RegexPart(String reg, RegexType typ, int nonCapt) {
             this(reg, typ, (nonCapt > 0));
         }
 
+        public String getRegex() {
+            return regex;
+        }
+
+        public RegexType getType() {
+            return type;
+        }
+
+        public boolean isNonCapturing() {
+            return nonCapturing;
+        }
+
         @Override
         public boolean equals(Object o) {
             if (!(o instanceof RegexPart)) {
@@ -153,6 +165,10 @@ public String getRegex() {
         return getRegex(regexParts);
     }
 
+    public RegexPart[] getRegexParts() {
+        return regexParts;
+    }
+
     public static String getRegex(RegexPart[] regexParts) {
         StringBuilder regex = new StringBuilder();
         for (RegexPart part : regexParts) {

pom.xml

@@ -72,8 +72,8 @@
         <version.commons-pool>1.6</version.commons-pool>
         <version.commons-pool2>2.6.1</version.commons-pool2>
         <version.commons-text>1.11.0</version.commons-text>
-        <version.curator>5.7.1</version.curator>
-        <version.curator.test>5.7.1</version.curator.test>
+        <version.curator>5.9.0</version.curator>
+        <version.curator.test>5.9.0</version.curator.test>
         <version.datawave.accumulo-api>4.0.0</version.datawave.accumulo-api>
         <version.datawave.audit-api>4.0.0</version.datawave.audit-api>
         <version.datawave.authorization-api>4.0.0</version.datawave.authorization-api>

web-services/deploy/configuration/src/main/resources/beanRefContext.xml

@@ -18,6 +18,7 @@
                 <value>classpath*:datawave/query/QueryExpiration.xml</value>
                 <value>classpath*:datawave/query/QueryMetricsWriter.xml</value>
                 <value>classpath*:datawave/query/*QueryLogicFactory.xml</value>
+                <value>classpath*:datawave/query/QueryLimiterFactory.xml</value>
                 <value>classpath*:datawave/query/CachedResults*.xml</value>
                 <value>classpath*:datawave/mapreduce/MapReduceJobs.xml</value>
 				<value>classpath*:datawave/modification/ModificationServices.xml</value>

web-services/deploy/configuration/src/main/resources/datawave/query/QueryLimiterFactory.xml

@@ -0,0 +1,24 @@
+<?xml version="1.0" encoding="UTF-8"?>
+
+<beans xmlns="http://www.springframework.org/schema/beans"
+       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+       xmlns:util="http://www.springframework.org/schema/util"
+       xmlns:context="http://www.springframework.org/schema/context"
+       xsi:schemaLocation="http://www.springframework.org/schema/beans
+    http://www.springframework.org/schema/beans/spring-beans-4.0.xsd
+    http://www.springframework.org/schema/context
+    http://www.springframework.org/schema/context/spring-context-4.0.xsd
+    http://www.springframework.org/schema/util
+    http://www.springframework.org/schema/util/spring-util-4.0.xsd">
+
+    <!-- Configuration for the QueryLimitProvider. -->
+    <bean id="queryLimitProviderConfig" class="datawave.webservice.query.limit.QueryLimitConfiguration">
+        <property name="defaultSystemQueryLimit" value="20000"/>
+        <property name="defaultUserQueryLimit" value="100"/>
+    </bean>
+
+    <!-- Configuration for the QueryLimiter. -->
+    <bean id="queryLimiter" class="datawave.webservice.query.limit.QueryLimiter" >
+        <property name="zookeeperConfig" value="${ivarator.zookeeper.hosts}"/>
+    </bean>
+</beans>

web-services/deploy/spring-framework-integration/src/test/resources/springFrameworkBeanRefContext.xml

@@ -15,6 +15,7 @@
                 <value>classpath*:datawave/security/PrincipalFactory.xml</value>
                 <value>classpath*:datawave/query/QueryExpiration.xml</value>
                 <value>classpath*:datawave/query/*QueryLogicFactory.xml</value>
+                <value>classpath*:datawave/query/QueryLimiterFactory.xml</value>
                 <value>classpath*:datawave/query/CachedResults*.xml</value>
                 <value>classpath*:datawave/mapreduce/MapReduceJobs.xml</value>
             </list>

web-services/query/pom.xml

@@ -250,11 +250,26 @@
             <artifactId>datawave-in-memory-accumulo</artifactId>
             <scope>test</scope>
         </dependency>
+        <dependency>
+            <groupId>org.apache.curator</groupId>
+            <artifactId>curator-test</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.assertj</groupId>
+            <artifactId>assertj-core</artifactId>
+            <scope>test</scope>
+        </dependency>
         <dependency>
             <groupId>org.javassist</groupId>
             <artifactId>javassist</artifactId>
             <scope>test</scope>
         </dependency>
+        <dependency>
+            <groupId>org.springframework</groupId>
+            <artifactId>spring-test</artifactId>
+            <scope>test</scope>
+        </dependency>
     </dependencies>
     <build>
         <finalName>${project.artifactId}</finalName>

web-services/query/src/main/java/datawave/webservice/query/cache/QueryHeartbeatCache.java

@@ -0,0 +1,52 @@
+package datawave.webservice.query.cache;
+
+import java.io.IOException;
+
+import javax.annotation.PostConstruct;
+import javax.inject.Singleton;
+
+import org.apache.log4j.Logger;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+
+import datawave.webservice.query.limit.QueryHeartbeat;
+
+/**
+ * A cache for storing query heartbeats of running queries.
+ */
+@Singleton
+public class QueryHeartbeatCache {
+
+    private static final Logger log = Logger.getLogger(QueryHeartbeatCache.class);
+
+    private Cache<String,QueryHeartbeat> cache;
+
+    @PostConstruct
+    public void init() {
+        cache = Caffeine.newBuilder().build();
+    }
+
+    public void put(String queryId, QueryHeartbeat heartbeat) {
+        cache.put(queryId, heartbeat);
+    }
+
+    public QueryHeartbeat get(String queryId) {
+        return cache.getIfPresent(queryId);
+    }
+
+    public void stopAndRemoveHeartbeat(String queryId) {
+        QueryHeartbeat heartbeat = cache.asMap().remove(queryId);
+        if (heartbeat != null) {
+            try {
+                heartbeat.stop();
+            } catch (IOException e) {
+                log.error("Error stopping query heartbeat", e);
+            }
+        }
+    }
+
+    public void clear() {
+        cache.asMap().clear();
+    }
+}

web-services/query/src/main/java/datawave/webservice/query/limit/ActiveQueryException.java

@@ -0,0 +1,27 @@
+package datawave.webservice.query.limit;
+
+/**
+ * Represents an exception that occurred when interacting with the {@link ActiveQueryTracker}.
+ */
+public class ActiveQueryException extends RuntimeException {
+
+    public ActiveQueryException() {
+        super();
+    }
+
+    public ActiveQueryException(String message) {
+        super(message);
+    }
+
+    public ActiveQueryException(String message, Throwable cause) {
+        super(message, cause);
+    }
+
+    public ActiveQueryException(Throwable cause) {
+        super(cause);
+    }
+
+    protected ActiveQueryException(String message, Throwable cause, boolean enableSuppression, boolean writableStackTrace) {
+        super(message, cause, enableSuppression, writableStackTrace);
+    }
+}