1.1.0

Highlights

• Upgrade Apache Kafka from version 3.4.0 to 3.7.0

What's Changed

• feat(auto_balancer): support dynamic configs for AutoBalancer by @SCNieh in #1076
• fix(issues1078): fix RequestChannel shutdown blocking by @superhx in #1080
• feat(auto_balancer): optimzed distribution goal action score calculation by @SCNieh in #1083
• docs(config): add config usage suggestion by @superhx in #1084
• fix(auto_balancer): avoid broker load affected by excluded topics by @SCNieh in #1086
• fix(maven): correct the maven build system for s3stream by @daniel-y in #1089
• fix(auto_balancer): add missing variable initialization by @SCNieh in #1090
• feat(auto_balancer): record network metrics by throttle priority by @SCNieh in #1093
• feat(MM2): allow customizing MM2 offset-syncs/checkpoints/heartbeats topic easily by @aaron-ai in #1095
• build(issues1033): switch log appender from DailyRollingFileAppender to RollingFileAppender by @warr99 in #1092
• feat(issues1097): increase api call timeout after timeout by @superhx in #1098
• fix(gradle): specify the classifier for netty-tcnative by @Chillax-0v0 in #1103
• fix(s3stream): fail fast if openssl not available by @Chillax-0v0 in #1105
• feat(issues1087): add data block cache by @superhx in #1107
• fix(docker): use specific localstack version; update aws-cli version by @mooc9988 in #1108
• feat(auto_balancer): split and merge swap actions by @SCNieh in #1109
• chore(util): print commit id when calling scripts wich "--version" as it used to be by @Chillax-0v0 in #1110
• fix(bin): support to run in Windows by @Chillax-0v0 in #1112
• feat(issues1087): add stream reader by @superhx in #1114
• feat(auto_balancer): support broker rack in cluster model by @SCNieh in #1115
• fix(s3): ControllerKVClient putKV should return result by @yangl in #1113
• feat(issues1087): add stream readers by @superhx in #1116
• fix(issues1087): event loop reject task after a task fail by @superhx in #1117
• fix(issues1087): fix long running issues by @superhx in #1118
• fix(issues1087): fix object not exist handle by @superhx in #1119
• fix(core): fix race issue in transaction verification by @ShadowySpirits in #1121
• feat(issues1087): add cold down time to reset readahead by @superhx in #1123
• feat(issue1120): support slow broker detect in controller by @SCNieh in #1122
• docs(config): refine metrics related config documentations by @SCNieh in #1125
• feat(issue1120): support reporting broker metrics in AutoBalancerMetricsReporter by @SCNieh in #1124
• feat(issues1087): allow readahead when cache is free by @superhx in #1126
• fix(metrics): fix method name by @SCNieh in #1129
• feat(issues1087): set s3.object.block.size default from 1MB to 512KB by @superhx in #1130
• chore: update README.md by @KaimingWan in #1131
• feat(metrics): use delta value on reporting percentile metrics by @SCNieh in #1132
• feat(issues1087): increase readahead size when cache miss by @superhx in #1133
• fix(issues1127): fix 1.0 to 1.1-rc0 launch fail by @superhx in #1128
• fix(metrics): remove redundant error log by @SCNieh in #1135
• chore(grafana): fix broken panels by @SCNieh in #1136
• fix(issues1142): retry get index when object not exist by @superhx in #1143
• feat(auto_balancer): support client ACL configuration by @SCNieh in #1146
• fix(core): compose future result when writing stream object by @SCNieh in #1147
• feat(issue1144): start next compaction in advance when needed by @SCNieh in #1148
• fix(issues1097): specify cleanup/minor/major stream object compaction… by @superhx in #1149
• fix(issues1140): fix S3ObjectControlManager NPE by @superhx in #1151
• fix(auto_balancer): suppress error log for OutOfOrderSequenceException by @SCNieh in #1154
• Revert "fix(issues1140): fix S3ObjectControlManager NPE" by @superhx in #1157
• feat(s3stream): change the checksum algorithm to crc32c by @ShadowySpirits in #1155
• chore: update readme and add comparision by @KaimingWan in #1141
• fix(issues1140): fix S3ObjectControlManager NPE by @superhx in #1158
• chore: adjust table and cloud supported badge by @KaimingWan in #1162
• feat(s3stream): reduce stream object compaction write amplification by @superhx in #1159
• feat(throttle): subtract extra token from total available token by @SCNieh in #1163
• feat(throttle): add constructor for compatibility by @SCNieh in #1164
• Revert "feat(s3stream): change the checksum algorithm to crc32c (#1155)" by @ShadowySpirits in #1166
• fix(issues1072): fix MultiPartWriterTest unstable test by @warr99 in #1165
• fix(s3stream): precise record get objects metrics by @superhx in #1170
• chore: support one line quick start by @KaimingWan in #1168
• feat(issues1171): improve objects deletion performance by @superhx in #1172
• feat(s3stream): speed up block cache get objects by @superhx in #1167
• feat(auto_balancer): reduce massive small objects copy during scheduling by @SCNieh in #1176
• fix(auto_balancer): add client ACL to load retriever consumer by @SCNieh in #1182
• feat(issue1179): accelerate partition reassignments on load balancing by @SCNieh in #1181
• chore: update readme by @KaimingWan in #1183
• chore: Update README.md for more accurate descriptions by @daniel-y in #1185
• feat(issues1169): optimize partition reassignment logic to avoid NotLeaderOrFollowerException by @superhx in #1180
• feat(s3stream): active evict expired data by @superhx in #1186
• feat(auto_balancer): change log level to warning when failed on sendi… by @SCNieh in #1188
• style(auto_balancer): remove unused import by @SCNieh in #1189
• feat(issue668): disable swap action temporarily by @SCNieh in #1190
• feat(auto_balancer): add missing action logging by @SCNieh in #1194
• chore: Update README.md for explanations for Kafka compatibility (#1191) by @amos201600 in #1193
• chore(readme): remove controller from the comparison table, and mark proxy of pulsar as optional by @daniel-y in #1197
• feat(issue1195): introduce goal group on score calculation for goals on different scale by @SCNieh in #1196
• fix(issue1198): use timeline structure to avoid incorrect s3 object size by @SCNieh in #1199
• feat(s3stream): add tags to stream by @superhx in #1200
• chore(controller): more logs by @Chillax-0v0 in #1201
• feat(test): add e2e tests for s3stream by @ShadowySpirits in #1204
• feat(auto_balancer): optimize resource type for better scalability by @SCNieh in #1206
• chore(cicd): removing site-docs file in GitHub release workflow by @Gezi-lzq in #1207
• feat(issues1169): prevent re-open partition when the broker is shutting down by @superhx in #1210
• feat: update AWS S3 bucket and destination directory in GitHub releas… by @Gezi-lzq in #1211
• feat(#1212): update readme，add alibaba cloud by @amos201600 in #1214
• fix(s3stream): fix stream reader busy loop when metadata replay delay by @superhx in #1215
• feat(issues1174): cleanup deleted topic by controller by @superhx in #1213
• chore(log): rename stream_id to streamId by @superhx in #1216
• feat(metrics): support record slow read throughput by @SCNieh in #1217
• refactor: rename dockerhub repo and docker path by @Gezi-lzq in #1218
• feat(cicd): add Determine Image Tags step to Docker release workflow by @Gezi-lzq in #1208
• feat(auto_balancer): refactor cluster model for better scalability by @SCNieh in #1220
• chore(log): convert stream fenced log to warn level by @superhx in #1219
• fix(issues1222): fix #openStream resp nextOffset by @superhx in #1224
• feat(s3stream): add logs on completion of timeout read request by @SCNieh in #1227
• chore: polish the lots of badges of readme by @daniel-y in #1228
• fix(s3stream/wal): check record's cycle during recovery by @Chillax-0v0 in #1229
• chore: polish the readme with new architecture gif of AutoMQ by @daniel-y in #1233
• feat(issue1231): support historical metrics measurement in AutoBalancer by @SCNieh in #1232
• fix(log): fix open stream checker by @superhx in #1235
• feat(s3stream/wal): recover a little more records after reaching the window by @Chillax-0v0 in #1234
• feat(s3stream/wal): print more info in recover tool by @Chillax-0v0 in #1236
• fix(s3stream): avoid starting wal twice during failover by @Chillax-0v0 in #1237
• chore(log): isolate kafka.log log from stdout by @superhx in #1239
• feat(metadata): add partition stream relation by @superhx in #1241
• feat(issues1153): partition status self-check by @superhx in #1238
• chore: indicate how to pick up issue in contribute guide by @KaimingWan in #1242
• chore: add help info in how to contribute by @KaimingWan in #1243
• feat(issues1225): enhance s3 api testing by @warr99 in #1240
• feat(issues1153): add partition status statistics metrics by @superhx in #1245
• feat(metadata): fix data race when close and reopen coordinator partition by @ShadowySpirits in #1246
• fix(issues1248): fix partition reassignment deadlock by @superhx in #1249
• feat(cli): initialize the cli module by @ShadowySpirits in #1252
• fix(issues1253): fix duplicated Tracker#close fail by @superhx in #1254
• fix(metadata): fix potential data race by @ShadowySpirits in #1257
• feat(cli): introduce the stream command by @ShadowySpirits in #1256
• fix(metadata): merge cli module and kshell-sdk module by @ShadowySpirits in #1258
• feat(s3stream): enhance delta histogram by @superhx in #1261
• fix(metadata): ensure that the consumer offset is deleted along with the deletion of the topic by @ShadowySpirits in #1262
• fix(metadata): clean up the consumption offset for the deleted partitions by @ShadowySpirits in #1264
• feat(auto_balancer): make AbstractTimeWindowSamples extendable by @SCNieh in #1266
• fix(s3stream): add missing metrics record by @SCNieh in #1268
• chore: optimize doc by @KaimingWan in #1267
• feat(s3stream): optimize network rate limiter for better precision by @SCNieh in #1269
• feat(tools): a tool to run performance tests by @Chillax-0v0 in #1270
• fix(issues1276): group the batch request by nodeId and apiKey by @superhx in #1277
• fix(auto_balancer): reset partition model on reassignment by @SCNieh in #1279
• fix(auto_balancer): prevent reporting metrics for closed partition by @SCNieh in #1281
• feat(tools/perf): benchmark on catch-up read by @Chillax-0v0 in #1283
• feat(issues1273): Optimize the readme to include more deployment options by @amos201600 in #1275
• fix(core): clean potential leaked partition-level metrics periodically by @SCNieh in #1284
• fix(s3stream): fix stream missing tags by @superhx in #1285
• feat(cli): add force close stream command by @ShadowySpirits in #1260
• feat(readme): add contact #1286 by @amos201600 in #1287
• feat(auto_balancer): introduce metric version for backward compatibility by @SCNieh in #1288
• feat(core): add partition level produce/fetch request count metrics by @SCNieh in #1291
• feat(tools/perf): optimizing user experience by @Chillax-0v0 in #1292
• feat(shell): add s3 log appender by @ShadowySpirits in #1293
• chore: delete misleading contributing file by @KaimingWan in #1295
• fix(log): rollback the log config by @superhx in #1297
• fix(issues1296): fix potential stackoverflow by @superhx in #1298
• fix(issues1300): fix stream reader empty blocks by @superhx in #1301
• fix(auto_balancer): fix race condition by @SCNieh in #1299
• fix(issues1302): block bytebuf leak by @superhx in #1303
• fix(log): readahead error by @superhx in #1304
• feat(shell): add s3 metrics exporter by @ShadowySpirits in #1294
• fix(issues1222): fix stream advance check by @superhx in #1306
• fix(auto_balancer): avoid action score calculation for unsupported goal by @SCNieh in #1305
• feat: close valid streams first when encountering an invalid stream during recovery by @Chillax-0v0 in #1308
• fix(issues1302): fix StreamReader buf LEAK by @superhx in #1311
• chore(log): change mute unmute log level by @superhx in #1312
• feat: support for more customed SASL handlers by @mooc9988 in #1313
• feat(core): use load aware partition selector on broker fence by @SCNieh in #1314
• chore: enable join slack by @KaimingWan in #1309
• feat(auto_balancer): support specify broker listener name for AutoBalancer client by @SCNieh in #1316
• docs(telemetry): remove unused configs by @SCNieh in #1318
• feat(shell): add s3 log uploader by @ShadowySpirits in #1317
• chore(log): suppress open fail error caused by stream fence by @superhx in #1319
• feat(shell): add LogRecorder interface to adapt other logging framework by @ShadowySpirits in #1320
• chore(s3stream): log the phased duration of uploading WAL by @Chillax-0v0 in #1321
• fix(s3stream): release buffer on exception when compacting stream objects by @SCNieh in #1322
• feat(shell): add jitter for log and metrics report by @ShadowySpirits in #1323
• refactor(s3operator): refactor: make objectTagging a Map<String, String> rather than a boolean by @Chillax-0v0 in #1325
• feat(server): make kafka more scalable by @Chillax-0v0 in #1329
• feat(shell): add recreate command by @ShadowySpirits in #1327
• feat(shell): optimize shutdown hook by @ShadowySpirits in #1330
• fix(metrics): avoid using uninitialized counter metrics by @SCNieh in #1331
• feat(shell): cleanup some codes by @ShadowySpirits in #1332
• fix(perf): increase request timeout of the admin client by @Chillax-0v0 in #1337
• feat(core): forward stream apis by @ShadowySpirits in #1341
• feat(core): add command config by @ShadowySpirits in #1343
• fix(auto_balancer): fix broker status change by @SCNieh in #1345
• feat(metadata): add missing tags for the old stream by @superhx in #1347
• fix(s3stream): refine network rate limiter for better precision (#1344) by @SCNieh in #1348
• feat(shell): remove automq cli by @ShadowySpirits in #1356
• fix(s3stream): temporarily disable percentile metrics for performance issue (#1353) by @SCNieh in #1362
• perf(s3stream/allocator): increase trunk size of the netty allocator by @Chillax-0v0 in #1363
• fix(issues1357): fix missing deletes when load from snapshot by @superhx in #1359
• chore(log): fix log by @superhx in #1366
• feat(shell): add basic command for automq cli by @ShadowySpirits in #1371
• fix(issues1367): stream reader prevent adding invalid blocks by @superhx in #1373
• fix(core): add tolerance delay when append with non-zero seq number with empty producer snapshot by @SCNieh in #1377
• chore(s3stream): add toString methods to fix logs by @Chillax-0v0 in #1380
• fix(cache): fix readhead fail when user read fast than readahead by @superhx in #1383
• refactor(s3stream): remove DefaultS3Client's dependency on KafkaConfig by @Chillax-0v0 in #1386
• fix(issues604): fix response size not match by @superhx in #1393
• feat(tools/perf): adjust send rate during warmup and catchup by @Chillax-0v0 in #1394
• chore(gradle): change kafka version to 3.7.0 by @superhx in #1397
• fix(action): fix release action by @superhx in #1398
• fix(TimeIndex): wait for the last append to finish before fetching the stream by @Chillax-0v0 in #1408
• fix(cache): fix cache reuse for re-created topic by @superhx in #1418
• chore(build): update version to 1.1.0 by @superhx in #1422

New Contributors

• @yangl made their first contribution in #1113
• @Gezi-lzq made their first contribution in #1207

Full Changelog: 1.1.0-rc0...1.1.0