v1.18.0

[romange]

Dragonfly v1.18.0

Some prominent changes include:

• ACL improvements: chore(acl): allow multiple users in acl deluser #2945 chore(acl): adjust some ACL command responses #2943 fix: remove acl-check and cancel instead when REPLCONF ACK fails to validate #2920 fix: allow non hashed passwords when loading users from acl file #2982 chore(acl): add vlog and check on deluser flow #2995
• Implementation of json.merge feat: implement json.merge #2960
• Replication - memory improvements
• Very much alpha support for data tiering. Try it out with --tiered_prefix=/pathto/ssd/base and see how your memory usage goes down (STRING type only). Do not use it in prod! 😸

What's Changed

• feat: retry ACK if the configs are different don't finish migration until the target node contains config without migration #2833 by @BorysTheDev in feat: retry ACK if the configs are different #2833 #2906
• chore(tiering): Update Get, Set, Del by @dranikpg in chore(tiering): Update Get, Set, Del #2897
• chore: preparation step for lock fingerprints by @romange in chore: preparation step for lock fingerprints #2899
• fix(transaction): Use FinishHop in schedule by @dranikpg in fix(transaction): Use FinishHop in schedule #2911
• chore(tiering): Fix MacOs build by @dranikpg in chore(tiering): Fix MacOs build #2913
• feat(cluster): Migration cancellation support by @chakaz in feat(cluster): Migration cancellation support #2869
• feat: process migration data after FIN opcode Proceed "STABLE-SYNC" state if the timeout for ACK is happened #2864 by @BorysTheDev in feat: process migration data after FIN opcode #2864 #2918
• chore(string_family): Refactor SetCmd by @dranikpg in chore(string_family): Refactor SetCmd #2919
• fix: Improve reply latency of HELLO by @romange in fix: Improve reply latency of HELLO #2925
• chore: improve reply latency of SendScoredArray by @romange in chore: improve reply latency of SendScoredArray #2929
• Namespace support in prometheus rule by @Pothulapati in Namespace support in prometheus rule  #2931
• fix: socket closed when RegisterOnErrorCb is called in HandleRequests by @kostasrim in fix: socket closed when RegisterOnErrorCb is called in HandleRequests #2932
• chore: bring more clarity when replayer fails by @romange in chore: bring more clarity when replayer fails #2933
• Slot migration cancel crash fix by @BorysTheDev in Slot migration cancel crash fix #2934
• feat: add ability reaply config with migration reapply slots into new config from the finished migrations #2924 by @BorysTheDev in feat: add ability reaply config with migration #2924 #2926
• fix(test): Unflake fuzzy cluster migration test by @chakaz in fix(test): Unflake fuzzy cluster migration test #2927
• chore: Remove Schedule() call by @romange in chore: Remove Schedule() call #2938
• chore: get rid of lock keys by @romange in chore: get rid of lock keys #2894
• fix: introduce info_replication_valkey_compatible flag by @romange in fix: introduce info_replication_valkey_compatible flag #2936
• feat(metrics): adding max_clients to metrics and info output (FR: Add maxclients in INFO and /metrics #2912) by @racamirko in feat(metrics): adding max_clients to metrics and info output (#2912) #2940
• chore: adjust transaction code to keystep/3 commands by @romange in chore: adjust transaction code to keystep/3 commands #2941
• feat(tiering): Get, GetSet, Set test by @dranikpg in feat(tiering): Get, GetSet, Set test #2921
• chore(acl): adjust some ACL command responses by @Niennienzz in chore(acl): adjust some ACL command responses #2943
• chore: Pull helio with new future by @dranikpg in chore: Pull helio with new future #2944
• refactor: add cluster namespace by @BorysTheDev in refactor: add cluster namespace #2948
• chore: Introduce ShardArgs as a distinct type by @romange in chore: Introduce ShardArgs as a distinct type #2952
• chore: Log db_index in traffic logger by @dranikpg in chore: Log db_index in traffic logger #2951
• fixes for v1.18.0 by @adiholden in fixes for v1.18.0 #2956
• feat(tiering): Support append (and modifications in general) by @dranikpg in feat(tiering): Support append (and modifications in general) #2949
• feat: extended bracket index in jsonpath by @romange in feat: extended bracket index in jsonpath #2954
• chore: Remove TieringV1 by @dranikpg in chore: Remove TieringV1 #2962
• fix(pytests): replace proc.wait() with proc.communicate() to avoid deadlocks by @kostasrim in fix(pytests): replace proc.wait() with proc.communicate() to avoid deadlocks #2964
• feat(tiering): Registered buffers by @dranikpg in feat(tiering): Registered buffers #2967
• feat: add slot migration error processing by @BorysTheDev in feat: add slot migration error processing #2957
• chore(acl): allow multiple users in acl deluser by @kostasrim in chore(acl): allow multiple users in acl deluser #2945
• feat: implement json.merge by @romange in feat: implement json.merge #2960
• fix: fix deadlock and slot flush for migration cancel deadlock if slot migration is canceled after beggining #2968 by @BorysTheDev in fix: fix deadlock and slot flush for migration cancel #2968 #2972
• chore(tiering): Lots of metrics by @dranikpg in chore(tiering): Lots of metrics #2977
• fix: crash during migration when connection is closing by @kostasrim in fix: crash during migration when connection is closing #2973
• fix: remove acl-check and cancel instead when REPLCONF ACK fails to validate by @kostasrim in fix: remove acl-check and cancel instead when REPLCONF ACK fails to validate #2920
• fix: check return code of process after communicate by @kostasrim in fix: check return code of process after communicate #2976
• fix: allow non hashed passwords when loading users from acl file by @kostasrim in fix: allow non hashed passwords when loading users from acl file #2982
• chore: update our container distributions versions by @romange in chore: update our container distributions versions #2983
• chore: remove version checks when running our regtests by @romange in chore: remove version checks when running our regtests #2988
• chore(acl): add vlog and check on deluser flow by @kostasrim in chore(acl): add vlog and check on deluser flow #2995
• fix(memcached): Register memcached listener to handle --maxclients by @chakaz in fix(memcached): Register memcached listener to handle --maxclients #2985
• chore: another preparation commit to get rid of kv_args in transaction by @romange in chore: another preparation commit to get rid of kv_args in transaction #2996
• chore: improve performance of Scan operation by @romange in chore: improve performance of Scan operation #2990
• fix(server): small string allocations only under 256 bytes str by @adiholden in fix(server): small string allocations only under 256 bytes str #2991
• fix(cluster-migration): Support cancelling migration right after starting it by @chakaz in fix(cluster-migration): Support cancelling migration right after starting it #2992
• chore: fix double header issue by @romange in chore: fix double header issue #3002
• chore: small tiering fixes by @dranikpg in chore: small tiering fixes #2966
• feat(benchmark-tests): run in K8s by @zacharya19 in feat(benchmark-tests): run in K8s #2965
• Benchmark fixes by @zacharya19 in Benchmark fixes #3005
• fix(tiering): rename v2 + max_file_size by @dranikpg in fix(tiering): rename v2 + max_file_size #3004
• chore: fix tiering macos stub by @dranikpg in chore: fix tiering macos stub #3006
• chore: export listener stats by @romange in chore: export listener stats #3007
• chore: pull latest helio by @romange in chore: pull latest helio #3009
• fix(server): lag is 0 when server not in stable state by @adiholden in fix(server): lag is 0 when server not in stable state #3010
• chore: get rid of kv_args and replace it with slices to full_args by @romange in chore: get rid of kv_args and replace it with slices to full_args #2942
• fix(server): non auto journal write after callback finish by @adiholden in fix(server): non auto journal write after callback finish #3016
• fix(server): shrink replication steaming buf by @adiholden in fix(server): shrink replication steaming buf #3012
• fix(zset): fix random in ZRANDMEMBER command by @BagritsevichStepan in fix(zset): fix random in ZRANDMEMBER command #2994
• Fix benchmark by @adiholden in Fix benchmark #3017
• chore: Remove tiering test skip by @dranikpg in chore: Remove tiering test skip #3011
• feat(tiering): simple offload loop by @dranikpg in feat(tiering): simple offload loop #2987
• feat(tiering): MGET support by @dranikpg in feat(tiering): MGET support #3013
• Revert "chore: get rid of kv_args and replace it with slices to full_… by @romange in Revert "chore: get rid of kv_args and replace it with slices to full_… #3024
• chore(dash): Replace comparator with predicate by @dranikpg in chore(dash): Replace comparator with predicate #3025
• feat: add defragment command by @BorysTheDev in feat: add defragment command #3003

Huge thanks to all the contributors! ❤️

New Contributors

• @racamirko made their first contribution in feat(metrics): adding max_clients to metrics and info output (#2912) #2940
• @BagritsevichStepan made their first contribution in fix(zset): fix random in ZRANDMEMBER command #2994

Full Changelog: v1.17.0...v1.18.0

---

This discussion was created from the release v1.18.0.

[snird]

data tiering initial support is exciting!

Not only does Dragonfly offer superior performance, but also data tiering that RedisLabs managed services used to charge top dollar for.
Thank you, guys. You are the next big leap in the space.

[romange]

Thanks, Also I believe Dragonfly tiering will be more efficient in terms of latency, throughput because it uses a totally differrent approach from RoF.