v1.22.0

[romange]

Dragonfly v1.22.0

Some prominent changes include:

• Lots of stability fixes around JSON and Search APIS:
  fix: JSON.MSET command #3459, feat(json_family): Add support of the JSON legacy mode #3284, fix(search_family): Add error on creating indexes from non-zero databases #3537, fix(search_family): Fix query parsing for the integer tags in OR expression #3544,fix(json_family): Fix json get crash due to an invalid json path #3580, fix(json_family): Fix JSON.GET crash for the multiple legacy mode paths #3582, feat: add escaping symbols for tag search #3578, refactor: remove double conversion from str to number to str in search #3591, fix: JSON.STRAPPEND #3597, fix(server_family): Remove search indexes during the FLUSHALL command #3539
• Support HELLO AUTH handshake: fix: HELLO AUTH with non default user #3486
• Implement FT.TAGVALS: feat: implement FT.TAGVALS #3493
• Introduce Dragonfly specific lua pragmas: chore: Change Lua embedded flags syntax #3517
• Memcache now replicates flags correctly: fix: MC flags size and serialization #3134 #3538
• Memcache API now supports absolute timestamps as well: fix: fix expiration processing for set command #3607
• PEXPIRETIME and EXPIRETIME are implemented by @Eunoia1729 - thank you!
• Last but not least, a significant contribution by @cunla: a massive test suite, likely the largest in Dragonfly's history, with 12K lines of code. This contribution has sparked further efforts to enhance Dragonfly's compatibility and stability. The initial results are included in this release.

What's Changed

• fix: missing logs on ci timeout by @kostasrim in fix: missing logs on ci timeout #3452
• fix: JSON.MSET command by @BorysTheDev in fix: JSON.MSET command #3459
• chore: integrate quicklist changes from valkey by @romange in chore: integrate quicklist changes from valkey #3440
• feat(json_family): Add support of the JSON legacy mode by @BagritsevichStepan in feat(json_family): Add support of the JSON legacy mode #3284
• fix: pytest teardown exception in os.remove(LAST_LOGS) by @kostasrim in fix: pytest teardown exception in os.remove(LAST_LOGS) #3463
• chore: change how we track memory_budget during evictions by @romange in chore: change how we track memory_budget during evictions #3457
• chore: add timeout to replication sockets by @romange in chore: add timeout to replication sockets #3434
• chore(server): Remove old blocking debug by @dranikpg in chore(server): Remove old blocking debug #3460
• chore: SinkReplyBuilder2 with vec batching by @dranikpg in chore: SinkReplyBuilder2 with vec batching #3454
• feat: Allow pre-declaring Lua SHAs to run with undeclared keys by @chakaz in feat: Allow pre-declaring Lua SHAs to run with undeclared keys #3465
• chore(server): Use parser more in list_family by @dranikpg in chore(server): Use parser more in list_family #3461
• fix: Make replica's REPLCONF IP-ADDRESS optional by @chakaz in fix: Make replica's REPLCONF IP-ADDRESS optional #3473
• chore: make tiered_test more reliable by @romange in chore: make tiered_test more reliable #3477
• chore(facade): Update SinkReplyBuilder2 by @dranikpg in chore(facade): Update SinkReplyBuilder2 #3474
• refactor: set info_replication_valkey_compatible=true by @BorysTheDev in refactor: set info_replication_valkey_compatible=true #3467
• chore: relax the version check in test_replicate_old_master by @romange in chore: relax the version check in test_replicate_old_master #3478
• chore: Download Dragonfly versions only when needed by @chakaz in chore: Download Dragonfly versions only when needed #3479
• fix: division by zero bug by @romange in fix: division by zero bug #3482
• test: fix test_scripts by @BorysTheDev in test: fix test_scripts #3485
• chore(facade): RedisReplyBuilder2 base by @dranikpg in chore(facade): RedisReplyBuilder2 base #3475
• chore(facade): MCReplyBuilder2 by @dranikpg in chore(facade): MCReplyBuilder2 #3480
• chore(facade): RedisReplyBuilder2 (extensions) by @dranikpg in chore(facade): RedisReplyBuilder2 (extensions) #3481
• fix: big value serialization corner cases by @kostasrim in fix: big value serialization corner cases #3430
• fix: HELLO AUTH with non default user by @kostasrim in fix: HELLO AUTH with non default user #3486
• fix: flake in teardown of test_unknown_dfly_env by @kostasrim in fix: flake in teardown of test_unknown_dfly_env #3487
• fix: test_replicate_old_master missing serialization_max_chunk_size by @kostasrim in fix: test_replicate_old_master missing serialization_max_chunk_size #3498
• chore: add logs to debug tiered memory failures by @romange in chore: add logs to debug tiered memory failures #3499
• feat: implement FT.TAGVALS by @romange in feat: implement FT.TAGVALS #3493
• test: fix test_acl_deluser by @BorysTheDev in test: fix test_acl_deluser #3488
• fix: sanitizers post failure on google chat by @kostasrim in fix: sanitizers post failure on google chat #3496
• chore(generic_family): Fix bad data format error in the RESTORE command by @BagritsevichStepan in chore(generic_family): Fix bad data format error in the RESTORE command #3501
• fix: sanitizers false positives by @kostasrim in fix: sanitizers false positives #3495
• feat: Expose replica_reconnect_count for Prometheus metrics by @BagritsevichStepan in feat: Expose replica_reconnect_count for Prometheus metrics #3370
• fix: pytest warnings by @kostasrim in fix: pytest warnings #3497
• fix: missing virtual destructors in ReplyBuilder2 by @kostasrim in fix: missing virtual destructors in ReplyBuilder2 #3506
• chore: split engine_shard file from engine_shard_set by @romange in chore: split engine_shard file from engine_shard_set #3507
• chore: introduce a secondary TaskQueue for shards by @romange in chore: introduce a secondary TaskQueue for shards #3508
• fix: clang warnings by @romange in fix: clang warnings #3509
• chore: reduce pipelining latency by reusing existing shard fibers by @romange in chore: reduce pipelining latency by reusing existing shard fibers #3494
• fix: bugs around the growth of a tiered file by @romange in fix: bugs around the growth of a tiered file #3516
• chore: Change Lua embedded flags syntax by @chakaz in chore: Change Lua embedded flags syntax #3517
• feat(cluster): Allow appending RDB to existing store by @chakaz in feat(cluster): Allow appending RDB to existing store #3505
• fix: disable ThreadLocalMutex when big value ser is off by @kostasrim in fix: disable ThreadLocalMutex when big value ser is off #3521
• chore(io): Introduce (carefully) new io with use_new_io flag by @dranikpg in chore(io): Introduce (carefully) new io with use_new_io flag #3513
• chore(io): Optimize repeated ReservePiece calls by @dranikpg in chore(io): Optimize repeated ReservePiece calls #3525
• fix: disable sanitizers false positive by @kostasrim in fix: disable sanitizers false positive #3522
• chore(config): make pipeline_squash configurable by @dranikpg in chore(config): make pipeline_squash configurable #3529
• chore(traffic loger): use pipelining and print/analyze commands by @dranikpg in chore(traffic loger): use pipelining and print/analyze commands #3527
• fix: skip empty objects on load and replication by @kostasrim in fix: skip empty objects on load and replication #3514
• fix: deadlock in Heartbeat() by @kostasrim in fix: deadlock in Heartbeat() #3530
• fix(search_family): Add error on creating indexes from non-zero databases by @BagritsevichStepan in fix(search_family): Add error on creating indexes from non-zero databases #3537
• fix: macos build by @kostasrim in fix: macos build #3536
• fix: MC flags size and serialization Memcached flags loses during load from snapshot #3134 by @BorysTheDev in fix: MC flags size and serialization #3134 #3538
• fix(transaction): Don't set continuation for blocking by @dranikpg in fix(transaction): Don't set continuation for blocking #3540
• fix: named volume permissions in docker by @romange in fix: named volume permissions in docker #3518
• fix(jsonpath): Add JsonPath grammar for the child identifier in brackets by @BagritsevichStepan in fix(jsonpath): Add JsonPath grammar for the child identifier in brackets #3533
• fix: build errors in sanitizers daily workflow by @kostasrim in fix: build errors in sanitizers daily workflow #3542
• test: tests using fakeredis to check compatibility by @cunla in test: tests using fakeredis to check compatibility #3534
• chore: fix search replication by @dranikpg in chore: fix search replication #3547
• chore: add deallocation logs in the allocation tracker by @romange in chore: add deallocation logs in the allocation tracker #3549
• fix: fakeredis tests support python 3.10 by @cunla in fix: fakeredis tests support python 3.10 #3550
• feat(server_family): Add backup/restore Prometheus metrics by @BagritsevichStepan in feat(server_family): Add backup/restore Prometheus metrics #3520
• docs: Fakeredis tests README by @cunla in docs: Fakeredis tests README #3552
• fix: zinterstore correctly finds weights by @romange in fix: zinterstore correctly finds weights #3554
• fix(search_family): Fix query parsing for the integer tags in OR expression by @BagritsevichStepan in fix(search_family): Fix query parsing for the integer tags in OR expression #3544
• chore(server): Fix watch by @dranikpg in chore(server): Fix watch #3557
• chore(search): fix numeric index query in rev order by @dranikpg in chore(search): fix numeric index query in rev order #3555
• chore: allow limiting pipelining queue by length by @romange in chore: allow limiting pipelining queue by length #3551
• fix: add unsupported_server_types to fakeredis tests by @cunla in fix: add unsupported_server_types to fakeredis tests #3560
• fix: string compatibility issues by @romange in fix: string compatibility issues #3564
• fix(server): Fix ZRANGEBYLEX limit params by @dranikpg in fix(server): Fix ZRANGEBYLEX limit params #3562
• fix: compatibility around list and string commands by @romange in fix: compatibility around list and string commands #3565
• chore(server): Unify zset arg parsing by @dranikpg in chore(server): Unify zset arg parsing #3563
• fix: compatibility around list,string and sort commands by @romange in fix: compatibility around list,string and sort commands #3568
• chore(server): Sort correctly in ZINTER by @dranikpg in chore(server): Sort correctly in ZINTER #3566
• chore(server): Unify zset boolean operations into single function by @dranikpg in chore(server): Unify zset boolean operations into single function #3567
• chore(transaction): More blocking tests by @dranikpg in chore(transaction): More blocking tests #3546
• fix: delete empty dense sets in HGetGeneric by @kostasrim in fix: delete empty dense sets in HGetGeneric #3543
• chore(search): Silence query parser error by @dranikpg in chore(search): Silence query parser error #3570
• chore: improve compatibility of set and ping commands by @romange in chore: improve compatibility of set and ping commands #3569
• fix: disable failing test_cron_snapshot_failed_saving by @kostasrim in fix: disable failing test_cron_snapshot_failed_saving #3571
• fix: disable test_cluster_flushall_during_migration by @kostasrim in fix: disable test_cluster_flushall_during_migration #3573
• chore: add -Wthread-analysis and annotate (part 1/2) by @kostasrim in chore: add -Wthread-analysis and annotate (part 1/2) #3502
• feat(generic_family): Implement EXPIRETIME and PEXPIRETIME commands by @Eunoia1729 in feat(generic_family): Implement EXPIRETIME and PEXPIRETIME commands #3524
• chore: allow running dragonfly pytests in repeat by @romange in chore: allow running dragonfly pytests in repeat #3577
• fix(json_family): Fix json get crash due to an invalid json path by @BagritsevichStepan in fix(json_family): Fix json get crash due to an invalid json path #3580
• fix(json_family): Fix JSON.GET crash for the multiple legacy mode paths by @BagritsevichStepan in fix(json_family): Fix JSON.GET crash for the multiple legacy mode paths #3582
• fix: return an error when invalid number of arguments is passed. by @romange in fix: return an error when invalid number of arguments is passed. #3584
• feat: add oom_deny_ratio to mutable config by @romange in feat: add oom_deny_ratio to mutable config #3585
• fix: warning as error on sanitizers build by @kostasrim in fix: warning as error on sanitizers build #3587
• feat: add escaping symbols for tag search by @BorysTheDev in feat: add escaping symbols for tag search #3578
• fix: skip test_replica_reconnections_after_network_disconnect by @kostasrim in fix: skip test_replica_reconnections_after_network_disconnect #3589
• docs(readme): Fix data mistake by @DerickIT in docs(readme): Fix data mistake #3590
• chore: remove DflyVersion::VER0 by @romange in chore: remove DflyVersion::VER0 #3593
• refactor: remove double conversion from str to number to str in search by @BorysTheDev in refactor: remove double conversion from str to number to str in search #3591
• chore(deps): bump aiohttp from 3.8.4 to 3.10.2 in /tests/dragonfly by @dependabot in chore(deps): bump aiohttp from 3.8.4 to 3.10.2 in /tests/dragonfly #3600
• refactor: remove toUpper() from cmd_arg_parser by @BorysTheDev in refactor: remove toUpper() from cmd_arg_parser #3599
• fix: JSON.STRAPPEND by @romange in fix: JSON.STRAPPEND #3597
• chore(deps): bump golang.org/x/net from 0.7.0 to 0.23.0 in /contrib/charts/dragonfly by @dependabot in chore(deps): bump golang.org/x/net from 0.7.0 to 0.23.0 in /contrib/charts/dragonfly #3601
• chore(deps): bump gopkg.in/yaml.v3 from 3.0.0-20210107192922-496545a6307b to 3.0.0 in /tools/replay by @dependabot in chore(deps): bump gopkg.in/yaml.v3 from 3.0.0-20210107192922-496545a6307b to 3.0.0 in /tools/replay #3603
• chore(deps): bump google.golang.org/protobuf from 1.26.0 to 1.33.0 in /contrib/charts/dragonfly by @dependabot in chore(deps): bump google.golang.org/protobuf from 1.26.0 to 1.33.0 in /contrib/charts/dragonfly #3604
• fix: xreadgroup replies as a map for RESP3 by @romange in fix: xreadgroup replies as a map for RESP3 #3576
• chore: enable experimental_new_io by default. by @romange in chore: enable experimental_new_io by default. #3605
• fix(server_family): Remove search indexes during the FLUSHALL command by @BagritsevichStepan in fix(server_family): Remove search indexes during the FLUSHALL command #3539
• feat(acl): add pub/sub by @kostasrim in feat(acl): add pub/sub #3574
• chore: enable -Werror=thread-safety and add missing annotations (part 2/2) by @kostasrim in chore: enable -Werror=thread-safety and add missing annotations (part 2/2) #3595
• feat: add slave_repl_offset to the replication section. by @romange in feat: add slave_repl_offset to the replication section. #3596
• fix(json_family): Fix JsonFromString method by @BagritsevichStepan in fix(json_family): Fix JsonFromString method #3602
• fix(grafana): Fix grafana dragonfly dashboard datasource by @applike-ss in fix(grafana): Fix grafana dragonfly dashboard datasource #3608
• fix: debug crash inside parsing of ZRANGE by @romange in fix: debug crash inside parsing of ZRANGE #3611
• chore: run fakeredis flow with the debug build by @romange in chore: run fakeredis flow with the debug build #3612
• chore: Split --cluster_announce_ip and --replica_announce_ip by @chakaz in chore: Split --cluster_announce_ip and --replica_announce_ip #3615
• fix: fix expiration processing for set command by @BorysTheDev in fix: fix expiration processing for set command #3607
• fix pytest: use generic random dbfilename in tests by @adiholden in fix pytest: use generic random dbfilename in tests #3617
• fix(bug): zinter command should run on replica by @adiholden in fix(bug): zinter command should run on replica #3620
• fix(test_auth_resp3_bug): release build failing by @kostasrim in fix(test_auth_resp3_bug): release build failing #3621
• chore: improve debug logs in dragonfly_connection by @romange in chore: improve debug logs in dragonfly_connection #3624
• fix: edge cases around mismatched path in json code by @romange in fix: edge cases around mismatched path in json code #3609
• test: fix search tests by @BorysTheDev in test: fix search tests #3625
• refactor: remove extra code from CmdArgParser by @BorysTheDev in refactor: remove extra code from CmdArgParser #3619
• fix: limit parsing in zrange commands by @romange in fix: limit parsing in zrange commands #3626
• chore: disable big value ser on reg tests by @kostasrim in chore: disable big value ser on reg tests #3629
• fix: crash when passing empty arguments by @romange in fix: crash when passing empty arguments #3627
• fix: 'renamenx foo foo' should return 0 if foo exists by @romange in fix: 'renamenx foo foo' should return 0 if foo exists #3630
• fix: debug help printed layout by @kostasrim in fix: debug help printed layout #3635
• fix: crash during SORT DESC call by @romange in fix: crash during SORT DESC call #3637

New Contributors

• @cunla made their first contribution in test: tests using fakeredis to check compatibility #3534
• @Eunoia1729 made their first contribution in feat(generic_family): Implement EXPIRETIME and PEXPIRETIME commands #3524
• @DerickIT made their first contribution in docs(readme): Fix data mistake #3590
• @applike-ss made their first contribution in fix(grafana): Fix grafana dragonfly dashboard datasource #3608

Full Changelog: v1.21.0...v1.22.0

---

This discussion was created from the release v1.22.0.