Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: feat(merge): merge main to feat/424_taskframework #1310

Closed
wants to merge 113 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
113 commits
Select commit Hold shift + click to select a range
3efbdf9
add ODCMonitor
ungreat Nov 27, 2023
0f3a319
add ODCMonitor
ungreat Nov 27, 2023
3efaf48
rename
ungreat Nov 28, 2023
b0e59c1
Merge remote-tracking branch 'origin/yiang_add_odc_monitor' into yian…
ungreat Nov 28, 2023
7d70d0f
fix(connection):add back connection cluster name (#942)
ungreat Nov 28, 2023
e205022
Merge remote-tracking branch 'origin/dev/4.2.x' into yiang_add_odc_mo…
ungreat Nov 30, 2023
2120a18
rename
ungreat Nov 30, 2023
a90d0dc
add log message
ungreat Nov 30, 2023
ac0d9b6
add log message
ungreat Dec 1, 2023
10104b7
fix(data-security): test masking algorithm may lead to security issue…
smallsheeeep Dec 4, 2023
60d7d38
Merge branch 'dev/4.2.x' into yiang_add_odc_monitor
yhilmare Dec 4, 2023
710938a
feat(osc): lock user is not required when create osc task on ob (#970)
krihy Dec 4, 2023
5dbbd31
Merge branch 'dev/4.2.x' into yiang_add_odc_monitor
yhilmare Dec 4, 2023
0f065ad
feat(data-transfer): support transfer mysql data by DataX (#871)
LuckyPickleZZ Dec 4, 2023
920de2e
fix(variables): variable updating may lead to sql injection (#1008)
yhilmare Dec 5, 2023
00712de
fix(bastion): inactive datasources are not cleared (#997)
smallsheeeep Dec 5, 2023
8f73554
fix(db-browser): DB session list show 0 in execute time for ob mysql …
PeachThinking Dec 5, 2023
0e6b909
feat(sql-execute): unable to obtain locale info in subthread (#994)
LuckyPickleZZ Dec 6, 2023
80f3632
fix(apply-project): project role names are not internationalized (#1000)
smallsheeeep Dec 6, 2023
55a9d32
feat(sql-execute): supports locating specific issue locations in mult…
MarkPotato777 Dec 6, 2023
95b390c
fix(ticket): horizontal unauthorized when query approver related role…
smallsheeeep Dec 6, 2023
651e7a5
fix(data-security): create sensitive columns failed due to scanning d…
smallsheeeep Dec 6, 2023
67d2642
fix(permission): user can create datasource without any project and r…
smallsheeeep Dec 6, 2023
2c488af
fix(integration): cannot deal with array when parsing json or xml res…
smallsheeeep Dec 7, 2023
3658888
fix(database): block built-in databases when auto-sync databases to p…
smallsheeeep Dec 7, 2023
df479e7
fix(sql-rules): cannot add/update any sql rule default values (#1014)
MarkPotato777 Dec 7, 2023
030b793
review
ungreat Dec 7, 2023
e09137f
fix(sql-check): failed to recognize several drop statements (#1026)
yhilmare Dec 7, 2023
85c589e
rename
ungreat Dec 7, 2023
dd40713
fix(partition-plan): create partition plan task failed in obmysql 147…
MarkPotato777 Dec 7, 2023
305c036
fix(dlm):task cannot be executed due to insufficient connections (#1052)
guowl3 Dec 7, 2023
0419805
add log
ungreat Dec 7, 2023
3d5a174
fix(datasource): built-in database still belong to previous project w…
smallsheeeep Dec 7, 2023
d64bf69
enhancement(sql-check): check auto-increment column's datatype and ch…
yhilmare Dec 7, 2023
bce473b
rename package
ungreat Dec 8, 2023
25576b9
feat(data-transfer): support log throughput of datatransfer (#1056)
LuckyPickleZZ Dec 8, 2023
325754e
feat(alarm): add AlarmUtils, use log output as default channel
yizhouxw Dec 8, 2023
2629b84
fix(parse-sid): optimize parse sid failed error message (#1062)
PeachThinking Dec 8, 2023
aa27ad8
fix(pl): no sys_refcursor shown in return type select panel when crea…
yhilmare Dec 8, 2023
932319c
fix(sql-rule): the sql type 'desc' does not work in the allow-sql-typ…
MarkPotato777 Dec 8, 2023
f1b6a95
fix(result-export): there is no log printed for result export task (#…
LuckyPickleZZ Dec 8, 2023
89ea94c
fix(sql-rule): cannot execute sqls with dblink in team space's sql co…
MarkPotato777 Dec 8, 2023
e9137ec
fix(project): project participants can create database and add them i…
MarkPotato777 Dec 11, 2023
1e4f496
fix(full-link-trace): no tags and references in downloaded json file …
LuckyPickleZZ Dec 11, 2023
7e3adce
fix(obclient): do not create os user when it already exists (#1096)
LuckyPickleZZ Dec 11, 2023
d271f47
feat(data-masking): prohibit data-masking for native MySQL datasource…
smallsheeeep Dec 11, 2023
c3c3f59
enhancement(pl): add async result acquisition api for pl execution (#…
yhilmare Dec 11, 2023
1301ebf
fix(project): the project owner can remove all project dbas from the …
MarkPotato777 Dec 11, 2023
a49c4da
fix(ticket): list all tickets returns empty in individual space (#1089)
MarkPotato777 Dec 11, 2023
813b63e
fix(ticket): tickets not filtered by projects (#1111)
MarkPotato777 Dec 12, 2023
246ab2f
fix(connect-plugin): failed to connect to native percona mysql dataso…
PeachThinking Dec 12, 2023
f60bfdc
fix(ticket): approvers viewing shadow table sync ticket fails after t…
MarkPotato777 Dec 12, 2023
09f9762
fix(result-set-export): there is no data in exported xlsx file (#1139)
LuckyPickleZZ Dec 12, 2023
4ecc47c
fix(jdbc): full link trace leads to OOM exception (#1145)
LuckyPickleZZ Dec 12, 2023
097e558
docs: refine developer guide (#1141)
yizhouxw Dec 12, 2023
bc95844
fix(project): could delete users who are currently joining projects (…
MarkPotato777 Dec 12, 2023
2bdde03
fix(database-object) :Provide relevant prompts to users for high-risk…
PeachThinking Dec 13, 2023
5b062f9
fix(dlm): displays incomplete information after editing (#1073)
guowl3 Dec 13, 2023
5d3963e
fix(session): session creation will fail when the oracle schema name …
MarkPotato777 Dec 13, 2023
0713fba
fix(project): transaction timeout when transfer too many databases or…
MarkPotato777 Dec 13, 2023
3ee0738
fix(sql-check): can not give violations related comments normally whe…
yhilmare Dec 13, 2023
3f80fa5
fix(data-transfer): only inject sys tenant config when it's configure…
LuckyPickleZZ Dec 14, 2023
bf0f943
fix(sql-rule): several sql interception bugs (#1165)
MarkPotato777 Dec 14, 2023
9816be0
enhance(monitor): add druid monitor (#1088)
ungreat Dec 14, 2023
164ec8d
fix(iam): users need re-login to access the individual space after th…
MarkPotato777 Dec 14, 2023
c200c4b
fix(osc): osc log is not show totally and flow task is done unnormal…
krihy Dec 15, 2023
02cef2c
fix(concurrent): remove servlet configuration (#1188)
LuckyPickleZZ Dec 15, 2023
40e87b8
enhancement: use git-lfs to storage datax.zip (#1186)
LuckyPickleZZ Dec 15, 2023
7b495c8
feat(partition-plan): support setting scheduling strategy (#1136)
guowl3 Dec 15, 2023
9c0010c
fix(monitor): druid stats use mysql parser (#1208)
ungreat Dec 15, 2023
e83549a
fix(database): optimize error message of synchronizing databases fail…
MarkPotato777 Dec 15, 2023
a6fefcd
fix(sql-rule): disabling the rule 'allow-execute-sql-types' does not…
MarkPotato777 Dec 15, 2023
8ff9138
fix(monitor): fix druid stats parser error (#1213)
ungreat Dec 18, 2023
e154df4
fix(database-change): timeout or oom when upload a large sql files (#…
smallsheeeep Dec 18, 2023
7391d5c
remove lfs files
yizhouxw Dec 18, 2023
e8b8a4e
add build-resource submodule
yizhouxw Dec 18, 2023
4cc1675
refer datax.zip from submodule build-resource
yizhouxw Dec 18, 2023
2ed6287
gitmodules miss branch name
yizhouxw Dec 18, 2023
ffb0b46
remove obclient tar.gz, refer from submodule also
yizhouxw Dec 18, 2023
92ce8c9
build: fix lfs over github quota
yizhouxw Dec 18, 2023
43823f7
fix(database-object):modify the prompt that prompts users about the r…
PeachThinking Dec 19, 2023
7ffc406
fix(audit): several operating records issues after ODC V4.2.0 (#1222)
MarkPotato777 Dec 19, 2023
e799c58
fix(osc): lock ob mysql user failed when host with ip limited (#1072)
krihy Dec 19, 2023
a49321c
feat(datasource): show datasource's connect status in team space's SQ…
MarkPotato777 Dec 19, 2023
a1667ed
feat(datatransfer): support masking data for mysql datatransfer (#1198)
LuckyPickleZZ Dec 19, 2023
1febbcb
feat(monitor):add api alarm (#1212)
ungreat Dec 19, 2023
0dac873
fix(flow): creating flow costs too much time (#1183)
yhilmare Dec 19, 2023
131347e
fix(apply-project): failed to set mdc value (#1237)
smallsheeeep Dec 20, 2023
eeb6f7a
fix(web-framework): invalid csrf token result into Invalid session er…
yizhouxw Dec 20, 2023
2169f3e
fix(rollback-plan): NPE when user input sql content is empty (#1242)
smallsheeeep Dec 20, 2023
f2fcebc
fix(datasource): it occurs 'duplicate data source name' error when cr…
MarkPotato777 Dec 20, 2023
ad3999d
fix(partition plan):failed to disable table partition plan (#1247)
guowl3 Dec 20, 2023
66b45de
fix(schema-plugin): show partition name with identifiers (#1249)
PeachThinking Dec 20, 2023
d14ddea
fix(monitor): format druid log (#1251)
ungreat Dec 20, 2023
3e10bf6
fix(ticket): project "pending approval" tickets shows other project's…
MarkPotato777 Dec 21, 2023
7055e86
fix(datasource): convert the type of ob-mysql-sharding data source t…
PeachThinking Dec 21, 2023
b8f9fd5
fix(flow): close prepared stmt and resultset when batch creating end …
ungreat Dec 21, 2023
fb4795f
fix(mock-data): upgrade mock-data module's version to fix several bug…
yhilmare Dec 21, 2023
27e620e
feat(dlm): supports viewing task logs (#1017)
guowl3 Dec 21, 2023
f8c41d7
fix(mock-data): failed to recognize the charset key of 'UTF8' (#1272)
yhilmare Dec 21, 2023
cac37e4
fix(monitor): format alarm error stack to inline (#1273)
ungreat Dec 21, 2023
a1b5864
docs: add changelog for 4.2.3 (#1108)
yhilmare Dec 21, 2023
4bd0aa6
fxi(web-framework): MissCsrfToken error while logout from personal sp…
yizhouxw Dec 21, 2023
6e6105f
fix(result-export): failed to export mysql data (#1275)
LuckyPickleZZ Dec 22, 2023
608a866
fix(datasource): make ODP_SHARDING_OB_MYSQL not be converted to OB_MY…
PeachThinking Dec 22, 2023
51788c2
fix(result-set-export): failed to rewrite sql for mysql (#1288)
LuckyPickleZZ Dec 22, 2023
3c0fd8b
fix(sql-rule): adjust several sql-console rules' default values (#1281)
MarkPotato777 Dec 22, 2023
c2d53cd
feat(dlm): upgrade dlm's version to 1.0.8 (#1299)
guowl3 Dec 25, 2023
d85b83a
feat(mock-data): increase the max number of the mock data to 100 mill…
yhilmare Dec 25, 2023
5ef3a9b
fix(sql-rule): the rule 'allow-sql-types' of dev environment is disab…
MarkPotato777 Dec 25, 2023
c543368
builds: update submodule (#1304)
yhilmare Dec 25, 2023
09e4ec8
build: fix copy obclient logic (#1305)
yizhouxw Dec 26, 2023
a310761
merge dev/4.2.x into main #1308
yhilmare Dec 26, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion .gitchangelog.rc
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ subject_process = (strip |
## ``tag_filter_regexp`` is a regexp
## Tags that will be used for the changelog must match this regexp.
# Match ODC release tag style
tag_filter_regexp = r'^v([0-9]+\.[0-9]+\.[0-9]+)$'
tag_filter_regexp = r'^v([0-9]+\.[0-9]+\.[0-9]+)(_[a-z0-9]+)?$'

## ``revs`` is a list of callable or a list of string
##
Expand Down
4 changes: 1 addition & 3 deletions .github/workflows/build_artifact.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -160,9 +160,7 @@ jobs:
echo "Start prepare oceanbase-client"
pushd import
echo "Current dir is "`pwd`
rm -rf obclient_aarch.tar.gz
rm -rf obclient_win.tar.gz
mv obclient_x86.tar.gz obclient.tar.gz
cp ../build-resource/obclient/2.2.4/linux_x86/obclient.tar.gz obclient.tar.gz
popd
echo "Prepare oceanbase-client success"
echo "Start build rpm package"
Expand Down
4 changes: 1 addition & 3 deletions .github/workflows/build_daily.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -182,9 +182,7 @@ jobs:
echo "Start prepare oceanbase-client"
pushd import
echo "Current dir is "`pwd`
rm -rf obclient_aarch.tar.gz
rm -rf obclient_win.tar.gz
mv obclient_x86.tar.gz obclient.tar.gz
cp ../build-resource/obclient/2.2.4/linux_x86/obclient.tar.gz obclient.tar.gz
popd
echo "Prepare oceanbase-client success"
echo "Start build rpm package"
Expand Down
4 changes: 1 addition & 3 deletions .github/workflows/build_dev.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -156,9 +156,7 @@ jobs:
echo "Start prepare oceanbase-client"
pushd import
echo "Current dir is "`pwd`
rm -rf obclient_aarch.tar.gz
rm -rf obclient_win.tar.gz
mv obclient_x86.tar.gz obclient.tar.gz
cp ../build-resource/obclient/2.2.4/linux_x86/obclient.tar.gz obclient.tar.gz
popd
echo "Prepare oceanbase-client success"
echo "Start build rpm package"
Expand Down
6 changes: 2 additions & 4 deletions .github/workflows/build_release.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -149,10 +149,8 @@ jobs:
run: |
echo "Start prepare oceanbase-client"
pushd import
echo "Current dir is "`pwd`
rm -rf obclient_aarch.tar.gz
rm -rf obclient_win.tar.gz
mv obclient_x86.tar.gz obclient.tar.gz
echo "Current dir is "`pwd`
cp ../build-resource/obclient/2.2.4/linux_x86/obclient.tar.gz obclient.tar.gz
popd
echo "Prepare oceanbase-client success"
echo "Start build rpm package"
Expand Down
1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
# ignore specific directories
.idea
.repository/
build-resource/
target
_build/
conf/
Expand Down
4 changes: 4 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,7 @@
path = client
url = https://github.com/oceanbase/odc-client.git
branch = dev-4.2.3
[submodule "build-resource"]
path = build-resource
url = https://github.com/oceanbase/odc-build-resource.git
branch = main
185 changes: 185 additions & 0 deletions CHANGELOG-zh-CN.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,190 @@
# OceanBase Developer Center (ODC) CHANGELOG

## 4.2.3 (2023-12-22)

### 功能变化

数据源

- 允许数据源绑定到项目
- 支持 OceanBase Sharding MySQL 数据源
- 支持克隆数据源
- 团队空间中支持在对象树中展示数据源状态

导入导出

- 支持原生 MySQL 数据源的导入导出
- OceanBase 数据源的导入导出任务配置页面中不再提供 SYS 账户配置

数据库对象管理

- OceanBase MySQL 以及原生 MySQL 模式下支持 GIS 数据类型
- 白屏创建或删除索引时给出高风险操作提示

项目

- 增加 2 个内置项目角色:安全管理员,参与者;安全管理员被允许管理项目的敏感列和参与审批,参与者被允许参与审批
- 允许用户申请项目权限
- 禁止删除以 DBA 或 项目 OWNER 角色归属到任意项目中的用户

SQL 开发规范

- 优化了 SQL 拦截交互
- 增加了问题定位功能,支持快速定位原始 SQL 中的具体问题

数据库连接会话

- 增加了自动重连机制,避免长时间不使用场景下会话销毁导致的报错及易用性问题

分区计划

- 支持定时调度

SQL 执行

- SQL 片段最大支持 65535 大小的内容
- 支持 Ctrl+Enter 快捷键执行当前语句

堡垒机集成

- 支持 SQL Check

DLM

- 支持日志查看
- 新增三个任务参数配置:查询超时时间、分片大小、分片策略
- 优化了 MYSQL 5.6 的性能表现
- 优化了 OceanBase 数据清理的性能表现

全链路诊断

- 支持导出 Jaeger 兼容的 JSON 文件
- 视觉效果优化
- 增加了结果的列表视图,支持搜索和排序

工单

- 项目管理员能查看该项目下的所有工单,其它角色能查看自己审批过的工单

### 缺陷修复

数据源

- 用户离开项目且不属于任何项目和角色时依然可以创建数据源
- OceanBase MySQL 以及原生 MySQL 模式下数据库会话中"执行时间"栏目为 0
- OceanBase Oracle 模式下通过会话变量管理功能进行时间输出格式的修改在 SQL 执行窗口不生效
- OceanBase Oracle 模式下无法连接小写的 schema
- 无法连接 percona 分支构建的 MySQL 数据源

SQL 执行

- SQL 执行过程中抛出错误没有国际化
- 无法在团队空间下执行带有 dblink 的 SQL
- 安全规则允许的前提下,无法在团队空间中执行 desc 语句
- OceanBase Oracle 模式下执行 SELECT ... ORDER BY 1 样式的语句时 ORDER BY 会失效
- 禁用"SQL 窗口允许执行的 SQL 类型"规则不生效

数据库对象管理

- OceanBase MySQL 模式下左侧对象树上展示的表的分区等对象的名字带有反引号包围符

结果集导出

- 任务没有日志打印
- 以 excel 格式导出后没有数据

PL 对象

- 交互式创建函数过程中,无法通过下拉菜单定义 sys_refcursor 类型的返回值
- OceanBase MySQL 模式下 PL 参数值没有对单引号转义

DLM

- 数据库连接池过小导致任务执行失败

分区计划

- 在 OceanBase 1.4.79 版本的 MySQL 模式下创建任务失败
- 不设置分区策略的表依然会执行分区计划变更

SQL 开发规范

- 无法识别`alter table xxx drop index`语句为 DROP INDEX 语句

外部审批集成

- 无法识别索引集合中的数据的表达式
- 外部系统返回的 xml 形式的数据在反序列化时会丢失原始 xml 的根 tag

数据脱敏

- 当扫描到重复列时会导致敏感列添加失败

项目

- 用户被赋予"个人空间"权限后必须重新登陆才能生效
- 同步大量数据库或 schema 到项目时发生事务超时
- 无法以项目维度过滤工单
- 项目 OWNER 可以将项目中所有 DBA 角色的用户全部移除

堡垒机集成

- 不活跃连接没有被清理

回收站

- 无法删除回收站中的特定对象

模拟数据

- 任务占用过多内存
- 不支持 ZHSGB232 编码
- OceanBase MySQL 以及原生 MySQL 模式下无法对宽度在 8 以下的 bit 类型生成任务
- 无法跳过自增的主键列
- OceanBase MySQL 以及原生 MySQL 模式下 bit 类型宽度显示错误

数据库变更任务

- 上传大文件场景下出现内存溢出错误

全链路诊断

- 将驱动加入全链路诊断后导致的内存溢出问题

影子表同步

- 工单被同意或拒绝后审批者无法查看任务详情

obclient 集成

- 重复创建同名操作系统用户导致报错

工单

- 创建工单耗时过久
- 一个项目的"待审批"工单列表中存在另一个项目的"待审批"工单

操作记录

- 操作记录中"数据源"栏目为空
- SQL 执行事件没有被记录
- 打开 SQL 窗口事件没有被记录

### 改进

- 提升 SQL 执行性能,减少不必要的耗时操作
- 允许用户配置登录失败场景下的最大重试次数以及账户锁定时间
- 只允许用户白屏修改带有主键约束,唯一键约束以及 rowid 的表数据
- 优化同步数据库出错时的报错文案

### 依赖库升级

- 升级 obclient 版本到 2.2.4
- 升级 spring security 版本到 5.7.10
- 升级 hutool 版本到 5.8.23
- 升级 pf4j 版本到 3.10.0
- 升级 netty 版本到 4.1.94.FINAL

## 4.2.2 bp (2023-11-23)

### 缺陷修复
Expand Down
Loading
Loading