3.1.4

Release date: November 2, 2023

New Features

• Supports sort keys for Primary Key tables created in shared-data StarRocks clusters.
• Supports using the str2date function to specify partition expressions for asynchronous materialized views. This helps facilitate incremental updates and query rewrites of asynchronous materialized views created on tables that reside in external catalogs and use the STRING-type data as their partitioning expressions. #29923 #31964
• Added a new session variable enable_query_tablet_affinity, which controls whether to direct multiple queries against the same tablet to a fixed replica. This session variable is set to false by default. #33049
• Added the utility function is_role_in_session, which is used to check whether the specified roles are activated in the current session. It supports checking nested roles granted to a user. #32984
• Supports setting resource group-level query queue, which is controlled by the global variable enable_group_lelvel_query_queue (default value: false). When the global-level or resource group-level resource consumption reaches a predefined threshold, new queries are placed in queue, and will be run when both the global-level resource consumption and the resource group-level resource consumption fall below their thresholds.
  • Users can set concurrency_limit for each resource group to limit the maximum number of concurrent queries allowed per BE.
  • Users can set max_cpu_cores for each resource group to limit the maximum CPU consumption allowed per BE.
• Added two parameters, plan_cpu_cost_range and plan_mem_cost_range, for resource group classifiers.
  • plan_cpu_cost_range: the CPU consumption range estimated by the system. The default value NULL indicates no limit is imposed.
  • plan_mem_cost_range: the memory consumption range estimated by the system. The default value NULL indicates no limit is imposed.

Improvements

• Window functions COVAR_SAMP, COVAR_POP, CORR, VARIANCE, VAR_SAMP, STD, and STDDEV_SAMP now support the ORDER BY clause and Window clause. #30786
• An error instead of NULL is returned if a decimal overflow occurs during queries on the DECIMAL type data. #30419
• The number of concurrent queries allowed in a query queue is now managed by the leader FE. Each follower FE notifies of the leader FE when a query starts and finishes. If the number of concurrent queries reaches the global-level or resource group-level concurrency_limit, new queries are rejected or placed in queue.

Bug Fixes

Fixed the following issues:

• Spark or Flink may report data read errors due to inaccurate memory usage statistics. #30702 #30751
• Memory usage statistics for Metadata Cache are inaccurate. #31978
  BEs crash when libcurl is invoked. #31667
• When StarRocks materialized views created on Hive views are refreshed, an error "java.lang.ClassCastException: com.starrocks.catalog. HiveView cannot be cast to com.starrocks.catalog. HiveMetaStoreTable" is returned. #31004
• If the ORDER BY clause contains aggregate functions, an error "java.lang.IllegalStateException: null" is returned. #30108
• In shared-data StarRocks clusters, the information of table keys is not recorded in information_schema.COLUMNS. As a result, DELETE operations cannot be performed when data is loaded by using Flink Connector. #31458
• When data is loaded by using Flink Connector, the load job is suspended unexpectedly if there are highly concurrent load jobs and both the number of HTTP threads and the number of Scan threads have reached their upper limits. #32251
• When a field of only a few bytes is added, executing SELECT COUNT(*) before the data change finishes returns an error that reads "error: invalid field name". #33243
• Query results are incorrect after the query cache is enabled. #32781
• Queries fail during hash joins, causing BEs to crash. #32219
• DATA_TYPE and COLUMN_TYPE for BINARY or VARBINARY data types are displayed as unknown in the information_schema.columns view. #32678

Behavior Change

• From v3.1.4 onwards, persistent indexing is enabled by default for Primary Key tables created in new StarRocks clusters (this does not apply to existing StarRocks clusters whose versions are upgraded to v3.1.4 from an earlier version). #33374
• A new FE parameter enable_sync_publish which is set to true by default is added. When this parameter is set to true, the Publish phase of a data load into a Primary Key table returns the execution result only after the Apply task finishes. As such, the data loaded can be queried immediately after the load job returns a success message. However, setting this parameter to true may cause data loads into Primary Key tables to take a longer time. (Before this parameter is added, the Apply task is asynchronous with the Publish phase.) #27055

2.5.13

Release date: September 28, 2023

Improvements

• Window functions COVAR_SAMP, COVAR_POP, CORR, VARIANCE, VAR_SAMP, STD, and STDDEV_SAMP now support the ORDER BY clause and Window clause. #30786
• An error instead of NULL is returned if a decimal overflow occurs during queries on the DECIMAL type data. #30419
• Executing SQL commands with invalid comments now returns results consistent with MySQL. #30210
• Rowsets corresponding to tablets that have been deleted are cleaned up, reducing the memory usage during BE startup. #30625

Bug Fixes

Fixed the following issues:

• An error "Set cancelled by MemoryScratchSinkOperator" occurs when users read data from StarRocks using the Spark Connector or Flink Connector. #30702 #30751
• An error "java.lang.IllegalStateException: null" occurs during queries with an ORDER BY clause that includes aggregate functions. #30108
• FEs fail to restart when there are inactive materialized views. #30015
• Performing INSERT OVERWRITE operations on duplicate partitions corrupts the metadata, leading to FE restart failures. #27545
• An error "java.lang.NullPointerException: null" occurs when users modify columns that do not exist in a Primary Key table. #30366
• An error "get TableMeta failed from TNetworkAddress" occurs when users load data into a partitioned StarRocks external table. #30124
• If users use CloudCanal to load data into table columns that are set to NOT NULL but have no default value specified, an error "Unsupported dataFormat value is : \N" is thrown. #30799
• An error "current running txns on db xxx is 200, larger than limit 200" occurs when users load data via the Flink Connector or perform DELETE and INSERT operations. #18393
• Asynchronous materialized views which use HAVING clauses that include aggregate functions cannot rewrite queries properly. #29976

3.1.3

Release date: September 25, 2023

New Features

• The aggregate function group_concat supports the DISTINCT keyword and the ORDER BY clause. #28778
• Stream Load, Broker Load, Kafka Connector, Flink Connector, and Spark Connector support partial updates in column mode on a Primary Key table. #28288
• Data in partitions can be automatically cooled down over time. (This feature is not supported for list partitioning.) #29335 #29393

Improvements

• Executing SQL commands with invalid comments now returns results consistent with MySQL. #30210

Bug Fixes

Fixed the following issues:

• If the BITMAP or HLL data type is specified in the WHERE clause of a DELETE statement to be executed, the statement cannot be properly executed. #28592
• After a follower FE is restarted, CpuCores statistics are not up-to-date, resulting in query performance degradation. #28472 #30434
• The execution cost of the to_bitmap() function is incorrectly calculated. As a result, an inappropriate execution plan is selected for the function after materialized views are rewritten. #29961
• In certain use cases of the shared-data architecture, after a follower FE is restarted, queries submitted to the follower FE return an error that reads "Backend node not found. Check if any backend node is down". #28615
• If data is continuously loaded into a table that is being altered by using the ALTER TABLE statement, an error "Tablet is in error state" may be thrown. #29364
• Modifying the FE dynamic parameter max_broker_load_job_concurrency using the ADMIN SET FRONTEND CONFIG command does not take effect. #29964 #29720
• BEs crash if the time unit in the date_diff() function is a constant but the dates are not constants. #29937
• In the shared-data architecture, automatic partitioning does not take effect after asynchronous load is enabled. #29986
• If users create a Primary Key table by using the CREATE TABLE LIKE statement, an error "Unexpected exception: Unknown properties: {persistent_index_type=LOCAL}" is thrown. #30255
• Restoring Primary Key tables causes metadata inconsistency after BEs are restarted. #30135
• If users load data into a Primary Key table on which truncate operations and queries are concurrently performed, an error "java.lang.NullPointerException" is thrown in certain cases. #30573
• If predicate expressions are specified in materialized view creation statements, the refresh results of those materialized views are incorrect. #29904
• After users upgrade their StarRocks cluster to v3.1.2, the storage volume properties of the tables created before the upgrade are reset to null. #30647
• If checkpointing and restoration are concurrently performed on tablet metadata, some tablet replicas will be lost and cannot be retrieved. #30603
• If users use CloudCanal to load data into table columns that are set to NOT NULL but have no default value specified, an error "Unsupported dataFormat value is : \N" is thrown. #30799

2.5.12

Release date: September 4, 2023

Improvements

• Comments in an SQL are retained in the Audit Log. #29747
• Added CPU and memory statistics of INSERT INTO SELECT to the Audit Log. #29901

Bug Fixes

Fixed the following issues:

• When Broker Load is used to load data, the NOT NULL attribute of some fields may cause BEs to crash or cause the "msg:mismatched row count" error. #29832
• Queries against ORC-formatted files fail because the bugfix ORC-1304 (apache/orc#1299) from Apache ORC is not merged. #29804
• Restoring Primary Key tables causes metadata inconsistency after BEs are restarted. #30135

3.1.2

Release date: August 25, 2023

Bug Fixes

Fixed the following issues:

• If a user specifies which database is to be connected by default and the user only has permissions on tables in the database but does not have permissions on the database, an error stating that the user does not have permissions on the database is thrown. #29767
• The values returned by the RESTful API action show_data for cloud-native tables are incorrect. #29473
• BEs crash if queries are canceled while the array_agg() function is being run. #29400
• The Default field values returned by the SHOW FULL COLUMNS statement for columns of the BITMAP or HLL data type are incorrect. #29510
• If the array_map() function in queries involves multiple tables, the queries fail due to pushdown strategy issues. #29504
  Queries against ORC-formatted files fail because the bugfix ORC-1304 (apache/orc#1299) from Apache ORC is not merged. #29804

Behavior Change

• For a newly deployed StarRocks v3.1 cluster, you must have the USAGE privilege on the destination external catalog if you want to run SET CATALOG to switch to that catalog. You can use GRANT to grant the required privileges.

• For a v3.1 cluster upgraded from an earlier version, you can run SET CATALOG with inherited privilege.

3.1.1

Release date: August 18, 2023

New Features

• Supports Azure Blob Storage for shared-data clusters.
• Supports aggregate functions COVAR_SAMP, COVAR_POP, and CORR.
• Supports the following window functions: COVAR_SAMP, COVAR_POP, CORR, VARIANCE, VAR_SAMP, STD, and STDDEV_SAMP.

Improvements

• Supports implicit conversions for all compound predicates and for all expressions in the WHERE clause. You can enable or disable implicit conversions by using the session variable ENABLE_STRICT_TYPE. The default value of this session variable is false.

Bug Fixes

Fixed the following issues:

• When data is loaded into tables that have multiple replicas, a large number of invalid log records are written if some partitions of the tables are empty. #28824
• Inaccurate estimation of average row size causes partial updates in column mode on Primary Key tables to occupy excessively large memory. #27485
• If clone operations are triggered on tablets in an ERROR state, disk usage increases. #28488
• Compaction causes cold data to be written to the local cache. #28831

3.0.5

Release date: August 16, 2023

New Features

• Supports aggregate functions COVAR_SAMP, COVAR_POP, and CORR.
• Supports the following window functions: COVAR_SAMP, COVAR_POP, CORR, VARIANCE, VAR_SAMP, STD, and STDDEV_SAMP.

Improvements

• Added more prompts in the error message xxx too many versions xxx. #28397
• Dynamic partitioning further supports the partitioning unit to be year. #28386
• The partitioning field is case-insensitive when expression partitioning is used at table creation and INSERT OVERWRITE is used to overwrite data in a specific partition. #28309

Bug Fixes

Fixed the following issues:

• Incorrect table-level scan statistics in FE cause inaccurate metrics for table queries and loading. #27779
• The query result is not stable if the sort key is modified for a partitioned table. #27850
• The version number for a tablet is inconsistent between the BE and FE after data is restored. #26518
• If the bucket number is not specified when users create a Colocation table, the number will be inferred as 0, which causes failures in adding new partitions. #27086
• When the SELECT result set of INSERT INTO SELECT is empty, the load job status returned by SHOW LOAD is CANCELED. #26913
• BEs may crash when the input values of the sub_bitmap function are not of the BITMAP type. #27982
• BEs may crash when the AUTO_INCREMENT column is being updated. #27199
• Outer join and Anti join rewrite errors for materialized views. #28028
• Inaccurate estimation of average row size causes Primary Key partial updates to occupy excessively large memory. #27485
• Activating an inactive materialized view may cause a FE to crash. #27959
• Queries can not be rewritten to materialized views created based on external tables in a Hudi catalog. #28023
• The data of a Hive table can still be queried even after the table is dropped and the metadata cache is manually updated. #28223
• Manually refreshing an asynchronous materialized view via a synchronous call results in multiple INSERT OVERWRITE records in the information_schema.task_runs table. #28060
• FE memory leak caused by blocked LabelCleaner threads. #28311

3.1.0

Release date: August 7, 2023

New Features

Shared-data cluster

• Added support for Primary Key tables, on which persistent indexes cannot be enabled.
• Supports the AUTO_INCREMENT column attribute, which enables a globally unique ID for each data row and thus simplifies data management.
• Supports automatically creating partitions during loading and using partitioning expressions to define partitioning rules, thereby making partition creation easier to use and more flexible.

Data Lake analytics

• Supports accessing views created on tables within Hive catalogs.
• Supports accessing Parquet-formatted Iceberg v2 tables.
• [Preview] Supports sinking data to Parquet-formatted Iceberg tables.
• [Preview] Supports accessing data stored in Elasticsearch by using Elasticsearch catalogs. This simplifies the creation of Elasticsearch external tables.
• [Preview] Supports performing analytics on streaming data stored in Apache Paimon by using Paimon catalogs.

Storage engine, data ingestion, and query

• Upgraded automatic partitioning to expression partitioning. Users only need to use a simple partition expression (either a time function expression or a column expression) to specify a partitioning method at table creation, and StarRocks will automatically create partitions based on the data characteristics and the rule defined in the partition expression during data loading. This method of partition creation is suitable for most scenarios and is more flexible and user-friendly.
• Supports list partitioning. Data is partitioned based on a list of values predefined for a particular column, which can accelerate queries and manage clearly categorized data more efficiently.
• Added a new table named loads to the Information_schema database. Users can query the results of Broker Load and Insert jobs from the loads table.
• Supports logging the unqualified data rows that are filtered out by Stream Load, Broker Load, and Spark Load jobs. Users can use the log_rejected_record_num parameter in their load job to specify the maximum number of data rows that can be logged.
• Supports random bucketing. With this feature, users do not need to configure bucketing columns at table creation, and StarRocks will randomly distribute the data loaded into it to buckets. Using this feature together with the capability of automatically setting the number of buckets (BUCKETS) that StarRocks has provided since v2.5.7, users no longer need to consider bucket configurations, and table creation statements are greatly simplified. In big data and high performance-demanding scenarios, however, we recommend that users continue using hash bucketing, because this way they can use bucket pruning to accelerate queries.
• Supports using the table function FILES() in INSERT INTO to directly load the data of Parquet- or ORC-formatted data files stored in AWS S3. The FILES() function can automatically infer the table schema, which relieves the need to create external catalogs or file external tables before data loading and therefore greatly simplifies the data loading process.
• Supports generated columns. With the generated column feature, StarRocks can automatically generate and store the values of column expressions and automatically rewrite queries to improve query performance.
• Supports loading data from Spark to StarRocks by using Spark connector. Compared to Spark Load, the Spark connector provides more comprehensive capabilities. Users can define a Spark job to perform ETL operations on the data, and the Spark connector serves as the sink in the Spark job.
• Supports loading data into columns of the MAP and STRUCT data types, and supports nesting Fast Decimal values in ARRAY, MAP, and STRUCT.

SQL reference

• Added the following storage volume-related statements: CREATE STORAGE VOLUME, ALTER STORAGE VOLUME, DROP STORAGE VOLUME, SET DEFAULT STORAGE VOLUME, DESC STORAGE VOLUME, SHOW STORAGE VOLUMES.

• Supports altering table comments using ALTER TABLE. #21035

• Added the following functions:

  • Struct functions: struct (row), named_struct
  • Map functions: str_to_map, map_concat, map_from_arrays, element_at, distinct_map_keys, cardinality
  • Higher-order Map functions: map_filter, map_apply, transform_keys, transform_values
  • Array functions: array_agg supports ORDER BY, array_generate, element_at, cardinality
  • Higher-order Array functions: all_match, any_match
  • Aggregate functions: min_by, percentile_disc
  • Table functions: generate_series, FILES
  • Date functions: next_day, previous_day, last_day, makedate, date_diff
  • Bitmap functions：bitmap_subset_limit, bitmap_subset_in_range

Privileges and security

Added [privilege items](https://docs.starrocks.io/en-us/3.1/administration/privilege_item#...

2.5.10

Release date: August 7, 2023

New features

• Supports aggregate functions COVAR_SAMP, COVAR_POP, and CORR.
• Supports the following window functions: COVAR_SAMP, COVAR_POP, CORR, VARIANCE, VAR_SAMP, STD, and STDDEV_SAMP.

Improvements

• Optimized the scheduling logic of TabletChecker to prevent the checker from repeatedly scheduling tablets that are not repaired. #27648
• When Schema Change and Routine Load occur simultaneously, Routine Load jobs may fail if Schema Change completes first. The error message reported in this situation is optimized. #28425
• Users are prohibited from defining NOT NULL columns when they create external tables (If NOT NULL columns are defined, errors will occur after an upgrade and the table must be created again). External catalogs are recommended starting from v2.3.0 to replace external tables. #25485
• Added an error message when Broker Load retries encounter an error. This facilitates troubleshooting and debugging during data loading. #21982
• Supports large-scale data writes when a load job involves both UPSERT and DELETE operations. #17264
• Optimized query rewrite using materialized views. #27934 #25542 #22300 #27557 #22300 #26957 #27728 #27900

Bug Fixes

Fixed the following issues:

• When CAST is used to convert a string into an array, the result may be incorrect if the input includes constants. #19793
• SHOW TABLET returns incorrect results if it contains ORDER BY and LIMIT. #23375
• Outer join and Anti join rewrite errors for materialized views. #28028
• Incorrect table-level scan statistics in FE cause inaccurate metrics for table queries and loading. #27779
• "An exception occurred when using the current long link to access metastore. msg: Failed to get next notification based on last event id: 707602" is reported in FE logs if event listener is configured on the HMS to incrementally update Hive metadata. #21056
• The query result is not stable if the sort key is modified for a partitioned table. #27850
• Data loaded using Spark Load may be distributed to the wrong buckets if the bucketing column is a DATE, DATETIME, or DECIMAL column. #27005
• The regex_replace function may cause BEs to crash in some scenarios. #27117
• BE crashes if the input of the sub_bitmap function is not a BITMAP value. #27982
• "Unknown error" is returned for a query when Join Reorder is enabled. #27472
• Inaccurate estimation of average row size causes Primary Key partial updates to occupy excessively large memory. #27485
• Some INSERT jobs return "[42000][1064] Dict Decode failed, Dict can't take cover all key :0" if low-cardinality optimization is enabled. #26463
• If users specify "hadoop.security.authentication" = "simple" in their Broker Load jobs created to load data from HDFS, the job fails. #27774
• Modifying the refresh mode of materialized views causes inconsistent metadata between the leader FE and follower FE. #28082 #28097
• Passwords are not hidden when SHOW CREATE CATALOG and SHOW RESOURCES are used to query specific information. #28059
• FE memory leak caused by blocked LabelCleaner threads. #28311

3.0.4

Release date: July 18, 2023

New Feature

• Queries can be rewritten even when the queries contain a different type of join than the materialized view. #25099

Improvements

• If the queried fields are not included in the output columns of a materialized view but are included in the predicate of the materialized view, the query can still be rewritten to benefit from the materialized view. #23028
• When the SQL dialect (sql_dialect) is set to trino, table aliases are not case-sensitive, and the json_array function is supported in queries. #26094 #25282
• Added a new field table_id to the table Information_schema.tables_config. You can join the table tables_config with the table be_tablets on the column table_id in the database Information_schema to query the names of the database and table to which a tablet belongs. #24061

Bug Fixes

Fixed the following issues:

• If a query that contains the sum aggregate function is rewritten to directly obtain query results from a single-table materialized view, the values in sum() field may be incorrect due to type inference issues. #25512
• An error occurs when SHOW PROC is used to view information about tablets in a StarRocks shared-data cluster.
• The INSERT operation hangs when the length of CHAR data in a STRUCT to be inserted exceeds the maximum length. #25942
• Some data rows queried fail to be returned for INSERT INTO SELECT with FULL JOIN. #26603
• An error ERROR xxx: Unknown table property xxx occurs when the ALTER TABLE statement is used to modify the table's property default.storage_medium. #25870
• An error occurs when Broker Load is used to load empty files. #26212
• Decommissioning a BE sometimes hangs. #26509