v1.18.0

Dragonfly v1.18.0

Some prominent changes include:

• ACL improvements: #2945 #2943 #2920 #2982 #2995
• Implementation of json.merge #2960
• Replication - memory improvements
• Very much alpha support for data tiering. Try it out with --tiered_prefix=/pathto/ssd/base and see how your memory usage goes down (STRING type only). Do not use it in prod! 😸

What's Changed

• feat: retry ACK if the configs are different #2833 by @BorysTheDev in #2906
• chore(tiering): Update Get, Set, Del by @dranikpg in #2897
• chore: preparation step for lock fingerprints by @romange in #2899
• fix(transaction): Use FinishHop in schedule by @dranikpg in #2911
• chore(tiering): Fix MacOs build by @dranikpg in #2913
• feat(cluster): Migration cancellation support by @chakaz in #2869
• feat: process migration data after FIN opcode #2864 by @BorysTheDev in #2918
• chore(string_family): Refactor SetCmd by @dranikpg in #2919
• fix: Improve reply latency of HELLO by @romange in #2925
• chore: improve reply latency of SendScoredArray by @romange in #2929
• Namespace support in prometheus rule by @Pothulapati in #2931
• fix: socket closed when RegisterOnErrorCb is called in HandleRequests by @kostasrim in #2932
• chore: bring more clarity when replayer fails by @romange in #2933
• Slot migration cancel crash fix by @BorysTheDev in #2934
• feat: add ability reaply config with migration #2924 by @BorysTheDev in #2926
• fix(test): Unflake fuzzy cluster migration test by @chakaz in #2927
• chore: Remove Schedule() call by @romange in #2938
• chore: get rid of lock keys by @romange in #2894
• fix: introduce info_replication_valkey_compatible flag by @romange in #2936
• feat(metrics): adding max_clients to metrics and info output (#2912) by @racamirko in #2940
• chore: adjust transaction code to keystep/3 commands by @romange in #2941
• feat(tiering): Get, GetSet, Set test by @dranikpg in #2921
• chore(acl): adjust some ACL command responses by @Niennienzz in #2943
• chore: Pull helio with new future by @dranikpg in #2944
• refactor: add cluster namespace by @BorysTheDev in #2948
• chore: Introduce ShardArgs as a distinct type by @romange in #2952
• chore: Log db_index in traffic logger by @dranikpg in #2951
• fixes for v1.18.0 by @adiholden in #2956
• feat(tiering): Support append (and modifications in general) by @dranikpg in #2949
• feat: extended bracket index in jsonpath by @romange in #2954
• chore: Remove TieringV1 by @dranikpg in #2962
• fix(pytests): replace proc.wait() with proc.communicate() to avoid deadlocks by @kostasrim in #2964
• feat(tiering): Registered buffers by @dranikpg in #2967
• feat: add slot migration error processing by @BorysTheDev in #2957
• chore(acl): allow multiple users in acl deluser by @kostasrim in #2945
• feat: implement json.merge by @romange in #2960
• fix: fix deadlock and slot flush for migration cancel #2968 by @BorysTheDev in #2972
• chore(tiering): Lots of metrics by @dranikpg in #2977
• fix: crash during migration when connection is closing by @kostasrim in #2973
• fix: remove acl-check and cancel instead when REPLCONF ACK fails to validate by @kostasrim in #2920
• fix: check return code of process after communicate by @kostasrim in #2976
• fix: allow non hashed passwords when loading users from acl file by @kostasrim in #2982
• chore: update our container distributions versions by @romange in #2983
• chore: remove version checks when running our regtests by @romange in #2988
• chore(acl): add vlog and check on deluser flow by @kostasrim in #2995
• fix(memcached): Register memcached listener to handle --maxclients by @chakaz in #2985
• chore: another preparation commit to get rid of kv_args in transaction by @romange in #2996
• chore: improve performance of Scan operation by @romange in #2990
• fix(server): small string allocations only under 256 bytes str by @adiholden in #2991
• fix(cluster-migration): Support cancelling migration right after starting it by @chakaz in #2992
• chore: fix double header issue by @romange in #3002
• chore: small tiering fixes by @dranikpg in #2966
• feat(benchmark-tests): run in K8s by @zacharya19 in #2965
• Benchmark fixes by @zacharya19 in #3005
• fix(tiering): rename v2 + max_file_size by @dranikpg in #3004
• chore: fix tiering macos stub by @dranikpg in #3006
• chore: export listener stats by @romange in #3007
• chore: pull latest helio by @romange in #3009
• fix(server): lag is 0 when server not in stable state by @adiholden in #3010
• chore: get rid of kv_args and replace it with slices to full_args by @romange in #2942
• fix(server): non auto journal write after callback finish by @adiholden in #3016
• fix(server): shrink replication steaming buf by @adiholden in #3012
• fix(zset): fix random in ZRANDMEMBER command by @BagritsevichStepan in #2994
• Fix benchmark by @adiholden in #3017
• chore: Remove tiering test skip by @dranikpg in #3011
• feat(tiering): simple offload loop by @dranikpg in #2987
• feat(tiering): MGET support by @dranikpg in #3013
• Revert "chore: get rid of kv_args and replace it with slices to full_… by @romange in #3024
• chore(dash): Replace comparator with predicate by @dranikpg in #3025
• feat: add defragment command by @BorysTheDev in #3003

Huge thanks to all the contributors! ❤️

New Contributors

• @racamirko made their first contribution in #2940
• @BagritsevichStepan made their first contribution in #2994

Full Changelog: v1.17.0...v1.18.0

v1.17.1

Dragonfly v1.17.1

This is a patch release.

What's Changed

fix: potential data race in the transactional code (#2911)

v1.17.0

Dragonfly v1.17.0

Some prominent changes include:

• Improved performance for MGET operations (#2453)
• Fix argument parsing in json.objkeys (#2872)
• Fix ipv6 support for replication (#2889)
• Support serialisation of bloom filters - saving to and loading from snapshots (#2846)
• Support of HLL PFADD (#2761)
• Support bullmq workloads that do not have {} hashtags in their queue names (#2890)

What's Changed

• fix: #2745 don't start migration process again after apply the same the same config is applied by @BorysTheDev in #2822
• feat(transaction): Idempotent callbacks (immediate runs) by @dranikpg in #2453
• refactor(cluster): replace sync_id with node_id for slot migration #2835 by @BorysTheDev in #2838
• feat(tiering): Simple OpManager by @dranikpg in #2781
• chore: implement path mutation for JsonFlat by @romange in #2805
• feat(cluster): add migration removing by config #2835 by @BorysTheDev in #2844
• chore: expose direct API on Bloom objects by @romange in #2845
• chore: generalize CompactObject::AllocateMR by @kostasrim in #2847
• feat(tiering): Simplest small bins by @dranikpg in #2810
• refactor: clean cluster slot migration code by @BorysTheDev in #2848
• fix(tests): Fix numsub test by @dranikpg in #2852
• fix: healthcheck for docker containers by @romange in #2853
• fix: possible crash in tls code by @romange in #2854
• fix(server): Do not block admin-port commands by @chakaz in #2842
• fix(pytest): make pytests fail if server crash on shutdown by @adiholden in #2827
• feat(server): add prints on takeover timeout by @adiholden in #2856
• fix(pytest): dont check process return code on kill by @adiholden in #2862
• fix: authorize the http connection to call commands by @romange in #2863
• feat(cluster): Send number of keys for incoming and outgoing migrations. by @chakaz in #2858
• feat(tiering): TieredStorageV2 by @dranikpg in #2849
• bug(server): set connection flags block/pause flag on all blocking commands by @adiholden in #2816
• chore: serialize SBF by @romange in #2846
• fix: test_replicaof_reject_on_load crash on stop by @kostasrim in #2818
• feat(dbslice): Add self-laundering iterator in DbSlice by @chakaz in #2815
• chore: License update by @romange in #2767
• fix(acl): incompatibilities with acl load by @kostasrim in #2867
• fix(json): make path optional in json.objkeys by @kostasrim in #2872
• fix: return wrong type errors for SET...GET command by @romange in #2874
• fix(redis replication): remove partial sync flow ,not supported yet by @adiholden in #2865
• chore: limit traffic logger only to the main interface by @romange in #2877
• chore: relax repltakeover constraints to only exclude write commands by @kostasrim in #2873
• chore(replayer): Roll back to go1.18 by @dranikpg in #2881
• fix: brpoplpush single shard to wake up blocked transactions by @kostasrim in #2875
• chore: LockTable tracks fingerprints of keys by @romange in #2839
• chore: reject TLS handshake when our listener is plain TCP by @romange in #2882
• Add support for Sparse HLL PFADD by @azuredream in #2761
• feat server: bring visibility to script errors by @adiholden in #2879
• chore: clean up REPLTAKEOVER flow by @romange in #2887
• chore(tiering): Move files and move kb literal to common by @dranikpg in #2868
• chore(interpreter): Support object replies by @dranikpg in #2885
• fix(ci/helm): Stick to v0.73.0 version of prom operator by @Pothulapati in #2893
• fix(acl): authentication with UDS socket by @kostasrim in #2895
• feat(cluster): add repeated ACK if an error is happened by @BorysTheDev in #2892
• chore(blocking): Remove faulty DCHECK by @dranikpg in #2898
• chore: add a clear link on how to build dragonfly from source by @romange in #2884
• feat(server): Allow configuration of hashtag extraction by @chakaz in #2890
• fix: fix build under macos by @BorysTheDev in #2901
• fix(cluster_replication): replicate redis cluster node bug fix by @adiholden in #2876
• fix(acl): skip http and add check on connection traversals by @kostasrim in #2883
• fix(zset): Better memory consumption calculation by @chakaz in #2900
• fix: fix ld for num converting by @BorysTheDev in #2902
• chore: add help string for memory_fiberstack_vms_bytes by @romange in #2903
• fix(sanitizers): false positive fail on multi_test::Eval by @kostasrim in #2896
• chore: pull helio and add ipv6 replication test by @dranikpg in #2889
• chore: add ipv6 support for native linux release by @romange in #2908

Huge thanks to all the contributors! ❤️

Full Changelog: v1.16.0...v1.17.0

v1.16.1

Dragonfly v1.16.1

This is a patch release.

What's Changed

• fix: authorize the http connection to call DF commands (#2863)
• possible crash in tls code (#2854 )
• healthcheck for docker containers (#2853)

Full Changelog: v1.16.0...v1.16.1

v1.16.0

Dragonfly v1.16.0

Our spring release. We are getting closer to 2.0 with some very exciting features ahead. Stay tuned!

Some prominent changes include:

• Improved memory accounting of client connections (#2710 #2755 and #2692 )
• FT.AGGREGATE call (#2413)
• Properly handle and replicate Memcache flags (#2787 #2807)
• Intoduce BF.AGGREGATE BD.(M)ADD and BF.(M)EXISTS methods (#2801). Note, that it does not work with snapshots and replication yet.
• Dragonfly builds natively on MacOS. We would love some help with extending the release pipeline to create a proper macos binary.
• Following the requests from the Edge developers community, we added a basic HTTP API support! Try running Dragonfly with:
  --expose_http_api flag and then call curl -X POST -d '["ping"]' localhost:6379/api. We will follow up with more extensive docs later this month.
• Lots of stability fixes, especially around Sidekiq and BullMQ workloads.

What's Changed

• chore: make usan asan optional and enable them on CI by @kostasrim in #2631
• fix: missing manual trigger for daily sanitizers by @kostasrim in #2682
• bug(tiering): fix overflow in page offset calculation and wrong hash offset calculation by @theyueli in #2683
• Chore: Fixed Docker Health Check by @manojks1999 in #2659
• chore: Increase disk space in the coverage runs by @romange in #2684
• fix(flushall): Decommit memory after releasing tables. by @chakaz in #2691
• feat(server): Account for serializer's temporary buffer size by @chakaz in #2689
• refactor: remove FULL-SYNC-CUT cmd #2687 by @BorysTheDev in #2688
• chore: add malloc-based stats and decommit by @romange in #2692
• feat(cluster): automatic slot migration finalization #2697 by @BorysTheDev in #2698
• Basic FT.AGGREGATE by @dranikpg in #2413
• chore: little transaction cleanup by @dranikpg in #2608
• fix(channel store): add acquire/release pair in fast update path by @dranikpg in #2704
• chore: add ubuntu22 devcontainer by @romange in #2700
• feat(cluster): Add --cluster_id flag by @chakaz in #2695
• feat(server): Use mimalloc in SSL calls by @chakaz in #2710
• chore: Pull helio with new BlockingCounter by @dranikpg in #2711
• chore(transaction): Simplify PollExecution by @dranikpg in #2712
• chore(transaction): Don't call GetLocalMask from blocking controller by @dranikpg in #2715
• chore: improve compatibility of EXPIRE functions with Redis by @romange in #2696
• chore: disable flaky fuzzy migration test by @kostasrim in #2716
• chore: update sanitizers workflow by @kostasrim in #2686
• chore: Use c-ares for resolving hosts in ProtocolClient by @chakaz in #2719
• Remove check-fail in ExpireIfNeeded and introduce DFLY LOAD by @romange in #2699
• chore: Record cmd stat from invoke by @dranikpg in #2720
• fix(transaction): nullptr access on non transactional commands by @kostasrim in #2724
• fix(BgSave): async from sync by @kostasrim in #2702
• chore: remove core/fibers by @romange in #2723
• fix(transaction): Replace with armed sync point by @dranikpg in #2708
• feat(json): Deserialize ReJSON format by @chakaz in #2725
• feat: add flag masteruser by @kostasrim in #2693
• refactor: block list refactoring #2580 by @BorysTheDev in #2732
• chore: fix DeduceExecMode by @dranikpg in #2733
• fix(cluster): Reply with correct \n / \r\n from CLUSTER sub cmd by @chakaz in #2731
• chore: Introduce fiber stack allocator by @romange in #2730
• fix(cluster): Save replica ID per replica by @chakaz in #2735
• fix(ssl): Proper cleanup by @chakaz in #2742
• chore: add skeleton files for flat_dfs code by @romange in #2738
• chore: better error reporting when connecting to tls with plain socket by @romange in #2740
• chore: Support json paths without root selector by @dranikpg in #2747
• chore: journal cleanup by @romange in #2749
• refactor: remove start-slot-migration cmd #2727 by @BorysTheDev in #2728
• feat(server): Add TLS usage to /metrics and INFO MEMORY by @chakaz in #2755
• chore: preparations for adding flat json support by @romange in #2752
• chore(transaction): Introduce RunCallback by @dranikpg in #2760
• feat(replication): Do not auto replicate different master by @chakaz in #2753
• Improve Helm chart to be rendered locally and on machines where is not the application target by @fafg in #2706
• chore: preparation for basic http api by @romange in #2764
• feat(server): Add metric for RDB save duration. by @chakaz in #2768
• chore: fix flat_dfs read tests. by @romange in #2772
• fix(ci): do not overwrite last_log_file among tests by @kostasrim in #2759
• feat(server): support cluster replication by @adiholden in #2748
• fix: fiber preempts on read path and OnCbFinish() clears fetched_items_ by @kostasrim in #2763
• chore(ci): open last_log_file in append mode by @kostasrim in #2776
• doc(README): fix outdated expiry ranges description by @enjoy-binbin in #2779
• Benchmark runner by @adiholden in #2780
• chore(replication-tests): add cache_mode on test replication all by @kostasrim in #2685
• feat(tiering): DiskStorage by @dranikpg in #2770
• chore: add a boilerplate for bloom filter family by @romange in #2782
• chore: introduce conversion routines between JsonType and FlatJson by @romange in #2785
• chore: Fix memcached flags not updated by @dranikpg in #2787
• chore: remove duplicate code from dash and simplify by @kostasrim in #2765
• chore: disable test_cluster_slot_migration by @kostasrim in #2788
• fix: new[] delete[] missmatch in disk_storage by @kostasrim in #2792
• fix: sanitizers clang build and clean up some warnings by @kostasrim in #2793
• chore: add bloom filter class by @romange in #2791
• chore: add SBF data structure by @romange in #2795
• chore(tiering): Disable compilation for MacOs by @dranikpg in #2799
• chore: fix daily build by @romange in #2798
• chore: expose SBF via compact_object by @romange in #2797
• fix(ci): malloc trim on sanitizers workflow by @kostasrim in #2794
• fix(cluster): Don't miss updates in FLUSHSLOTS by @chakaz in #2783
• feat: add bf.(m)add and bf.(m)exists commands by @romange in #2801
• fix: SBF memory leaks by @kostasrim in #2803
• chore: refactor StringFamily::Set to use CmdArgParser by @romange in #2800
• fix: propagate memcached flags to replica by @romange in #2807
• DFLYMIGRATE ACK refactoring by @BorysTheDev in #2790
• feat: add master lsn and journal_executed dcheck in replica via ping by @kostasrim in #277...

v1.15.1

Dragonfly v1.15.1

This is a patch release.

What's Changed

fix: Workaround for a possible crash during BLPOP calls.

Full Changelog: v1.15.0...v1.15.1

v1.15.0

Dragonfly v1.15.0

After more than a month, we are releasing a new version with even more efficient transactional algorithms, improved lua compatibility, better memory management and lots of stability fixes. For Mac enthusiasts - the project now builds natively on MacOs 💻

Some prominent changes include:

• Lua redis.log(..) is recognized (#2476)
• Lua can optionally return doubles as longs when replying to RESP2 clients using --lua_resp2_legacy_float flag (#2672)
• Added backlog option (#2456)

What's Changed

• chore: improve benchmark for bpptree and zsl tree by @romange in #2450
• feat(transaction): Independent out of order execution by @dranikpg in #2426
• fix(RestoreStreamer): Fix a few glitches by @chakaz in #2452
• feat(cluster): add tx execution in cluster_shard_migration by @BorysTheDev in #2385
• feat(server): Add backlog flag for listen() by @chakaz in #2456
• feat: allow reading offloaded strings without loading to the store by @romange in #2449
• refactor: return OpResult in DbSlice::AddOrFind instead of throwing std::bad_alloc by @kostasrim in #2427
• fix(transaction): Improve ACTIVE flags management by @dranikpg in #2458
• test(cluster): Enable seeder to work against a Dragonfly cluster by @chakaz in #2462
• Update README.md to include dragonfly cloud link by @worldsoup in #2469
• fix: resolves the crash with MGET and duplicate keys by @romange in #2471
• fix: fixes for v1.14.0 by @dranikpg in #2473
• feat(lua): add no-op redis.log command by @kostasrim in #2476
• fix: reduce test_pipeline_batching_while_migrating flakiness by @dranikpg in #2475
• chore: switch json object to pmr allocator by @romange in #2483
• chore(transaction): Simplify PollExecution() by @dranikpg in #2457
• chore: fix clang warnings by @romange in #2485
• chore(transaction): Launder copied keys in multi transactions by @dranikpg in #2478
• chore: lock keys when going through fast-path execution by @romange in #2491
• fix(server): mget crash on same key get by @adiholden in #2474
• chore: Introduce LockKey for LockTable by @romange in #2463
• fix(cluster): fix slot filtration to RestoreStreamer by @BorysTheDev in #2477
• feat(lua): add missing no-op redis.replicate_commands() by @kostasrim in #2472
• chore: remove batch reply statistics by @romange in #2490
• Fix redis->DF replication tests by @romange in #2495
• chore: introduce TaskQueue for executing shard local tasks by @romange in #2498
• fix(cluster): fix incorrect version checking and resource double free by @BorysTheDev in #2499
• fix(server): Fix crash in RENAME by @chakaz in #2503
• test: Disable flaky test test_policy_based_eviction_propagation by @chakaz in #2505
• feat(server): Memory tracker by @chakaz in #2501
• fix(server): update post updater iterator in tiering by @adiholden in #2497
• chore(transaction): Use PhasedBarrier for easier synchronization by @dranikpg in #2455
• feat(server): Implement SCRIPT FLUSH command by @highpon in #2493
• fix: do not crash with inconsistent watch queue by @romange in #2515
• Fix: awakened_keys inconcistency by @romange in #2517
• feat(cluster): add migration finalization by @BorysTheDev in #2507
• fix: fix BLOCKING/REVERSE_MAPPING flags for some commands by @romange in #2516
• fix: handle null length strings in stored commands by @dranikpg in #2518
• feat(server): Add MEMORY TRACK ADDRESS command by @chakaz in #2513
• feat(server): Implement CONFIG HELP command by @highpon in #2510
• fix(transaction): Add special barrier for blocking tx by @dranikpg in #2512
• fix(zset): ignore cancelled op status in BZPopMinMax by @andydunstall in #2527
• fix: fix script replication by @dranikpg in #2531
• chore: remove redis sorted set implementation by @romange in #2522
• fix: add emoji when regression tests timeout by @kostasrim in #2543
• fix(cluster): fix #2532 tests redis client close errors by @BorysTheDev in #2542
• bug: DFLY_PASSWORD environment variable deprecation by @mprimeaux in #2539
• fix: wrongly printing timedout emoji on test failures by @kostasrim in #2546
• feat: move lint test chart to seperate action and trigger it on reg tests by @kostasrim in #2536
• Update helio by @romange in #2538
• fix(server): Do not yield in journal if not allowed by @chakaz in #2540
• fix: fix interpreter acquisition with MULTI by @dranikpg in #2549
• fix(tests): increase interpreter test load, lower notice limit, include 2553 by @dranikpg in #2554
• feat: introduce a skeleton for jsonpath parser by @romange in #2521
• fix(pytest): Disable flaky interpreter test monitoring by @dranikpg in #2558
• feat(cluster): add slots blocking when we finalize migration by @BorysTheDev in #2484
• fix(server): Init tx time for all multi/lua transactions by @chakaz in #2562
• chore: enable ContendExpire by @romange in #2563
• fix(server): do not print warning on replica stop/ takeover by @adiholden in #2560
• chore: parse a simple jsonpath by @romange in #2566
• feat(pytest): Gen2 seeder, part 1 by @dranikpg in #2556
• fix: make sure SCRIPT FLUSH concludes by @romange in #2565
• chore: Remove manual memory management from stats by @dranikpg in #2550
• fix: asan errors on unit tests by @kostasrim in #2564
• chore(transaction): Simplify armed state by @dranikpg in #2508
• test(memory): Unaccounted memory test + add DEBUG POPULATE TYPE <type> by @chakaz in #2561
• chore: Recognize exit status in regression tests by @romange in #2571
• chore(server): replica not accumulate multi commands untill exec by @adiholden in #2557
• chore: use helio MPSCIntrusiveQueue queue by @romange in #2573
• chore: provide plumbing for jsonpath error propagation by @romange in #2567
• test(cluster-migration): Fix some bugs and add cluster migration fuzzy tests by @chakaz in #2572
• fix: do not migrate during connection close by @romange in #2570
• chore: fix fiber types in the codebase by @romange in #2574
• feat(release): Also bundle dragonfly-debug builds by @Pothulapati in #2424
• feat: jsonpath supports index/wildcard expressions by @romange in #2578
• fix(tiering tests): introduce wait until tiering entries num EQ/GT by @adiholden in #2559
• chore(pytest): Refactor snapshot test by @dranikpg in #2583
• fix(test): Apply pytest filter to regression tests as well by @chakaz in #2589
• feat: add descent operator to jsonpath parser by @romange in #2588
• test(cluster): Fix and un...

v1.14.7

Dragonfly v1.14.7

This is a patch release.

What's Changed

• chore: add oom stats to /metrics (#2680)
• feat(server): introduce table_growth_margin flag (#2678)

Full Changelog: v1.14.6...v1.14.7

v1.14.6

Dragonfly v1.14.6

This is a patch release.

What's Changed

• fix: data race during snapshotting with INFO calls (#2637)
• fix: avoid crashing during the connection traversal when TLS is enabled (#2651)

Full Changelog: v1.14.5...v1.14.6

v1.14.5

Dragonfly v1.14.5

This is a patch release.

What's Changed

• fix(bug): server crash info memory while saving (#2637)

Full Changelog: v1.14.4...v1.14.5