Skip to content

Merge 2024-05-30 internal #252

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 23 commits into from
Feb 17, 2025
Merged

Merge 2024-05-30 internal #252

merged 23 commits into from
Feb 17, 2025

Conversation

lgqss
Copy link
Collaborator

@lgqss lgqss commented Feb 13, 2025

将发布成v3.0.0版本
由于大部分代码库都不支持bazel,后续编译将只支持cmake

New Features:

  • 增加了arrow acero执行引擎,能够大幅提升大数据聚合/join的性能,db需要打开开关-use_arrow_vector=true
  • ddl支持store粒度的限制并发
  • 支持merge into语法(类似replace into语法,只是相同主键会合并而不是覆盖),类似于olap系统的sum聚合模型
  • merge into的表支持rollup索引,可以按照维度进一步聚合
  • 向量索引支持带前缀索引(必须和之间前缀一致),可以保存私域向量,向量搜索只在固定前缀里搜索(类似自动的不限制个数的partition功能)例如:primary(prefix, id), vector index(prefix, vec);使用的时候where prefix = xx and match(vector_field) against ('0.1,0.2,0.3...' in vector mode)
  • alter table modify column 无default时会清理默认值
  • truncate need ddl permission
  • 拒绝修改Schema Conf不为空的表的resource tag,除非指定force强制执行
  • 支持视图,支持with语句
  • 支持dblink功能(link后不同meta的表可以互相访问),db需要打开开关enable_dblink=true。例子:CREATE TABLE tbl1 (
    ) ENGINE=DBLink COMMENT='{"dblink_info": {"type":"LT_BAIKALDB","meta_name":"meta_addr","namespace_name":"ns","database_name":"db1","table_name":"tbl1"}}'
  • 支持multi distinct;例如 select count(distinct a), count(distinct b) from tb group by xx
  • 建表列名检查,不支持空格等特殊字符
  • 定期同步user权限
  • 日期支持CURDATE() - INTERVAL 7 DAY
  • mysql兼容性,建表不指定not null,默认是null

Bug Fixes:

  • 向量索引内存泄露fix
  • 倒排报错修复
  • fix row expr in (null)
  • fix check_and_update_incremental
  • union子查询不支持fullexport
  • fix drop region 导致 meta ddl core
  • join reorder fix
  • range分区预分裂问题fix
  • group_concat core fix
  • 修复like prefix
  • 表达式分区赋值fix
  • 修复union问题
  • 修复子查询类型推导
  • derived_table独立ctx,和外层名称不冲突
  • 子查询类型推导bugfix && 子查询全局索引下推bugfix
  • fix heart beat core
  • 限制ttl表不支持add fulltext
  • 全局索引partition_regions fix
  • 修复prepare默认值问题
  • 修复bugs:(a,b)>(0,0) and b in (1,2,3)
  • fix insert select core

Performance Improvements:

  • 增加了arrow acero执行引擎,能够大幅提升大数据聚合/join的性能,db需要打开开关-use_arrow_vector=true
  • 采用双buf优化meta心跳性能,单meta可以支持更多db和store
  • add_partition时init regions异步执行
  • 支持union和子查询的谓词下推

@lgqss lgqss changed the title Fix conflict 0530 Merge 2024-05-30 internal Feb 13, 2025
@lgqss lgqss merged commit c04e446 into baidu:master Feb 17, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant