v1.36.0

Dragonfly v1.36.0

Important stability fixes and changes:

1. Support Global Vector index #6103
2. Support unicode tags in #6127
3. django-cacheops compatibility fixes: #6123 #6129
4. better memory defragmentation for json objects: #6240 #6276

What's Changed

• actions: Add a volume mount to container by @abhijat in #6048
• chore: remove RecvProvided from dragonfly connection by @kostasrim in #6050
• fix(rdb_load): Store streamed items per rdb lodaer by @dranikpg in #6049
• fix(search): Result set cutoff by @dranikpg in #5906
• fix: remove duplicate expired/evicted metric header by @romange in #6058
• fix(test): ssl test to work with pytest 3.12 by @romange in #6059
• feat(hset): Load tiered values in read-only by @dranikpg in #5996
• chore(tiering): Add throttling metric + more tests by @dranikpg in #6040
• fix: macOS build by @vyavdoshenko in #6065
• server: Parse JSON on destination shard by @abhijat in #6061
• feat: add CSS option for FT.CREATE by @BorysTheDev in #6057
• chore(facade): Handle Socket::Recv() returning 0 explicitly by @romange in #6066
• fix: Remove index docs on eviction and cluster migration by @mkaruza in #6062
• fix(tests): Disable huge val streaming by @dranikpg in #6074
• fix(replication): Add detailed error message for REPLTAKEOVER failures by @vyavdoshenko in #6071
• fix(tiering): Fix upload strategy by @dranikpg in #6067
• fix(replication): use correct replica pointer in takeover catchup loop by @vyavdoshenko in #6068
• fix(replication): add socket timeout to REPLTAKEOVER command by @vyavdoshenko in #6070
• fix(takeover): pick correct replica to reconcile slots by @kostasrim in #6034
• test: fix flaky test_takeover_bug_wrong_replica_checked_in_logs by @vyavdoshenko in #6075
• feat(helm): adds common label support to the helm chart by @gaganhr94 in #6042
• Revert "feat(helm): adds common label support to the helm chart (#6042)" by @romange in #6077
• fix(server): Move LpGetView to core target by @dranikpg in #6076
• feat: adds common label support to the helm chart by @gaganhr94 in #6080
• docs(build-from-source): update LTO instructions to use USE_MOLD option by @glevkovich in #6081
• Update build-from-source.md by @dranikpg in #6087
• feat: add ftsearch result parsing for css by @BorysTheDev in #6090
• chore: pull the latest helio by @romange in #6096
• fix: suppress BUSYGROUP warning logs by @vyavdoshenko in #6095
• feat(tiering): Basic metrics by @dranikpg in #6086
• chore: change the resolution of reply latency to measurement to ns by @romange in #6092
• fix: test_reply_count regression by @romange in #6099
• chore(hooks): add commit-msg hook to enforce Signed-off-by line in commit messages by @glevkovich in #6082
• fix(server): handle INFO command with multiple sections correctly by @glevkovich in #6093
• chore: pull helio by @romange in #6100
• fix: smashing stack on Huffman test by @vyavdoshenko in #6102
• tools: Add scripts and supporting files to generate os packge repo by @abhijat in #5923
• chore(deps): bump the actions group across 1 directory with 4 updates by @dependabot[bot] in #6107
• chore(core): Refactoring to prepare for using stateless allocator by @abhijat in #6111
• tools: Download ARM deb package, disable fedora ARM test by @abhijat in #6110
• fix: replica of self by @kostasrim in #6097
• chore: relax pytest requirements by @romange in #6113
• feat(JSON): Use stateless allocator by @abhijat in #6060
• fix(dfly_bench): fix parsing for memcache by @romange in #6116
• chore: update vector-bench README by @romange in #6117
• chore: add mi_malloc memory test by @romange in #6118
• docs: add copilot-instructions.md for AI coding agent onboarding by @Copilot in #6112
• fix: Handle django-cache script so it could run in dragonfly by @romange in #6123
• chore: replace PMR_NS with StatelessAllocator in dense_set by @romange in #6121
• chore: implement background self-identification of huffman encoding by @romange in #6106
• chore: gcc14 for fedora 30 by @vyavdoshenko in #6124
• chore: expose more information when takeover fails on master. by @romange in #6130
• feat(search): Global HNSW vector index by @mkaruza in #6103
• fix(search): Support unicode for tags by @dranikpg in #6127
• chore: use our own vector for hnsw distance computation by @romange in #6132
• chore: more fixes for django-cacheops compatibility by @romange in #6129
• chore: refactor hnsw index into separate files by @romange in #6134
• chore: move HierarchicalNSW class to our codebase by @romange in #6136
• feat(tiering): Serialize hashes by @dranikpg in #6015
• fix(stream): Allow XINFO in script by @dranikpg in #6128
• fix(core): Remove async delete flag by @dranikpg in #6139
• feat: implement DenseSet::Shrink by @vyavdoshenko in #6094
• chore(core): Move page_usage related code to its own library by @abhijat in #6146
• chore: quietly ignore LATENCY HISTOGRAM command by @romange in #6151
• fix(tiering): missing comma in dashboard json by @goochs in #6150
• chore: add a tool that creates charts based on memtier metrics data by @romange in #6154
• chore: add latency graph to local dashboard by @romange in #6153
• fix: incorrect stalled heartbeat warning by @romange in #6155
• feat(lua): add legacy-float flag for Lua 5.1 number compatibility by @vyavdoshenko in #6149
• chore(dfly_bench): properly handle 0 from Recv by @romange in #6159
• chore: minor cleanups around the eval flow by @romange in #6152
• tests: Use AWS public docker registry by @abhijat in #6161
• fix(search): Fix numeric tree splits by @dranikpg in #6143
• fix(tiering): Small test fixes by @dranikpg in #6133
• chore: introduce common directory by @romange in #6170
• chore: introduce ParsedArguments class by @romange in #6169
• chore: rename WRITE to JOURNALED by @romange in #6172
• chore: fix data corruption bug by @romange in #6174
• chore: add ability for ParsedArgs to reference BackedArguments by @romange in #6171
• chore: use BackedArguments for implementing a pipeline message by @romange in #6167
• feat(search): Defragment search indices by @abhijat in #6144
• chore(core): Refactor SmallString by @dranikpg in #6138
• chore: clean up sanitizers checks by @romange in #6173...

v1.35.1

This is a patch release

What's Changed

• fix(tiering): improvements around tiering stability #6040 #6086 #6067
• fix(server): replica rejects replicating from itself #6097

Full Changelog: v1.35.0...v1.35.1

v1.34.2

This is a patch release

It is strongly advised to update to this version if you use 1.34.0 / 1.34.1 due to regression when Dragonfly runs in cache mode

What's Changed

• fix(evicition): Don't accumulate deleted bytes if there is no RSS evi… by @BorysTheDev in #5908

Full Changelog: v1.34.1...v1.34.2

v1.34.1 (Use with caution - see known issues in description)

This is a patch release

Known issues

#5899
#5891

What's Changed

• fix: prometheus is unable to scrape the /metrics endpoint #5829

Full Changelog: v1.34.0...v1.34.1

v1.34.0 (Use with caution - see known issues in description)

Dragonfly v1.34.0

Known issues

#5899
#5891

Important stability fixes and changes:

• Replication fixes: #5727 #5700
• Geo search fixes: #5714 #5678 #5726 #5734
• SSD tiering improvements: #5802 #5803 #5810

What's Changed

• fix: prevent writeBucket if migration is canceled by @BorysTheDev in #5676
• chore: group memory prometheous metrics by @kostasrim in #5674
• fix: check snapshot_version in correct place by @BorysTheDev in #5687
• fix: stack corruption in MultiCommandSquasher by @romange in #5697
• fix: RdbSaver: destroy SliceSnapshot on its shard thread to fix flaky crash on ARM64 with --force_epoll by @vyavdoshenko in #5685
• server: fix max emission limit in throttle command by @abhijat in #5686
• fix: list ops transformation during replication by @BorysTheDev in #5700
• fix: command parsing in geo family by @kostasrim in #5714
• chore: add queue task index to the log by @romange in #5716
• chore(server): Semi-automatic flag updates by @dranikpg in #5666
• feat(search): Add indexes joining by @BagritsevichStepan in #5604
• feat(search_family): Add LOAD_FROM to the FT.AGGREGATE command by @BagritsevichStepan in #5703
• feat(search_family): Improve sorting and limiting in index joins by @BagritsevichStepan in #5717
• fix: macos and linux build by @vyavdoshenko in #5718
• feat: Implement GEORADIUS_RO command by @EricHayter in #5678
• test: skip failing tests by @vyavdoshenko in #5721
• docs: improve description of --maxmemory parameter (units/tiering/minPerThread) by @frivoire in #5696
• chore: update task queue fiber priority to high by @romange in #5722
• chore: check fail for empty listpacks in streams by @kostasrim in #5704
• feat: improve target node latency during migration by @BorysTheDev in #5715
• chore: Strict bounds when adding uknown command for INFO ALL stats by @mkaruza in #5735
• server: Drop contents of exec_info body on clear by @abhijat in #5740
• feat: Implement GEORADIUSBYMEMBER_RO command by @EricHayter in #5726
• fix: reject geosearch count 0 by @kostasrim in #5734
• fix: jsonPath parser for int values by @BorysTheDev in #5737
• feat(dfly_bench): Generate value with random length by @mkaruza in #5743
• fix: JSON RESP3 compatibility - remove extra nested arrays for most commands by @vyavdoshenko in #5747
• chore: rewrite mimalloc2 external lib dependency by @romange in #5751
• fix: replication for Z...STORE commands by @BorysTheDev in #5728
• server: Enable lua legacy float response for RESP3 by @abhijat in #5754
• fix: bitfield no operations command by @BorysTheDev in #5756
• test: unskip epoll cluster and replication tests by @BorysTheDev in #5757
• Pull helio by @dranikpg in #5760
• fix: search index restore when index names start with ‘:’ (e.g. :Order:index) by @vyavdoshenko in #5755
• Revert "test: unskip epoll cluster and replication tests (#5757)" by @BorysTheDev in #5761
• chore: add more logs for takeover by @BorysTheDev in #5765
• chore: implement the iterative fragmentation check by @romange in #5766
• server: Adds metrics for background defrag task by @abhijat in #5767
• server: metrics to track stored command sizes by @abhijat in #5764
• chore: unlink deletes hashes asynchronously by @romange in #5773
• fix: acl pubsub should only glob match the channel by @kostasrim in #5769
• chore: add self-testing code for heap->full_page_size computation by @romange in #5772
• chore: tune replication/migration settings by @romange in #5779
• facade/server: Use inline instead of extern to reduce loc by @abhijat in #5768
• fix: script error reply by @BorysTheDev in #5776
• chore: Resize vector in StringMap::RandomPairs by @mkaruza in #5781
• dfly_bench: Show error on failing to make address by @abhijat in #5787
• feat: SHUTDOWN: fast path for NOW/FORCE; unify SAVE/SAFE; support NOSAVE by @vyavdoshenko in #5783
• core: Replace sbf with hyperloglog by @abhijat in #5784
• server: Reject caching command if tracking is not enabled by @abhijat in #5786
• fix: data loss in partial sync from same source master by @kostasrim in #5727
• fix: Scan on non-existing key should return zero cursor by @mkaruza in #5780
• fix(snapshot): Replace sleeps with yield by @dranikpg in #5619
• fix(block_list): Speed up Split method by @BagritsevichStepan in #5748
• chore: pass to a string compact_object a key/value flag by @romange in #5795
• feat(range_tree): Add FinalizeInitialization method to the RangeTree by @BagritsevichStepan in #5793
• fix: stale pointer reference during slot migration and flush by @romange in #5798
• chore: Command category cleanup (part 2?) by @dranikpg in #5800
• fix: remove limitation for SSCAN by @BorysTheDev in #5806
• chore: Update helio submodule by @mkaruza in #5807
• chore(tiering): Fix asserts around reads for serialization by @dranikpg in #5802
• feat: prometheus metrics per database by @qlimenoque in #5770
• chore(rdb): Insert big values only after full construction by @dranikpg in #5796
• fix(tiering): Cooldown for DiskStorage::Grow() retries by @dranikpg in #5803
• ci: enable repeat tests to use release and force epoll by @abhijat in #5805
• chore: Clean up string_family by @dranikpg in #5801
• chore(tiering): Make lower bound configurable by @dranikpg in #5810
• fix: increase fiber stack by @BorysTheDev in #5817
• chore: support huffman encoding for string values by @romange in #5815
• chore: increase timeout to 80mins for reg tests by @kostasrim in #5819
• chore(server): Hide internal CO flags from output by @dranikpg in #5814
• feat: Add eviction based on rss memory usage by @BagritsevichStepan in #5791

Huge thanks to all the contributors! ❤️

New Contributors

• @frivoire made their first contribution in #5696
• @qlimenoque made their first contribution in #5770

Full Changelog: v1.33.0...v1.34.0

v1.33.1

This is a patch release.

It is strongly advised to update to this version if you use 1.33 due to regression below.

What's Changed

• fix : stack corruption in MultiCommandSquasher #5697
  It also fixed the io_uring compatibility issue with older kernels (#5693)

v1.33.0

Dragonfly v1.33.0

Important stability fixes and changes:

• Fix stability bugs in georadius #5629 #5634
• Cluster related improvements: #5614 #5621 and #5618

What's Changed

• server: Log warning if server stops accepting connections by @abhijat in #5596
• feat(server): support command help command by @guozhihao-224 in #5601
• chore: refactor JournalItem by @kostasrim in #5585
• feat(linear_search_map): Add keys method by @BagritsevichStepan in #5603
• core: Minor cleanups in bloom filters by @abhijat in #5595
• fix(tiering): Block on concurrent backing file grow by @dranikpg in #5609
• feat: use SET instead of RESTORE for strings during slot migrations by @BorysTheDev in #5614
• chore: introduce shard_thread_busy_polling_usec to control busy polling by @romange in #5624
• fix: UB in georadius by @BorysTheDev in #5629
• fix: test_throttle_on_commands_squashing_replies_bytes by @kostasrim in #5628
• chore: more pipeline latency coverage by @romange in #5632
• fix: ub in GEORADIUSBYMEMBER by @kostasrim in #5634
• chore: more pipeline latency coverage (#5632) by @romange in #5635
• feat: use PXAT flag instead of separate command for strings during migration by @BorysTheDev in #5631
• refactor: CmdArgParser::Error() by @BorysTheDev in #5636
• test(acl): Fix flaky ACL tests with long-running Lua scripts by @vyavdoshenko in #5627
• chore: string_view overload for BaseFamilyTest::Run by @kostasrim in #5637
• fix: FlushSlots test by @BorysTheDev in #5639
• fix: macos build geo_family.cc by @vyavdoshenko in #5640
• fix: preserve null characters in hash values after HEXPIRE by @vyavdoshenko in #5642
• test(acl): make ACL tests with concurrent scripts deterministic by @vyavdoshenko in #5641
• test: skip test_acl_del_user_while_running_lua_script becuase it constantly fails on CI by @vyavdoshenko in #5646
• chore: do not shutdown after takeover only in full cluster mode by @kostasrim in #5621
• chore: add more I/O run-time settings by @romange in #5644
• fix: Prevent race condition during concurrent SAVE operations by @vyavdoshenko in #5633
• core: Track objects skipped during defragmentation by @abhijat in #5647
• chore(tiered_storage): Rename flags, make them updatable by @dranikpg in #5608
• chore: allow replicaof in cluster mode when state is TAKEN_OVER by @kostasrim in #5618
• fix: timeout overflow during migration finalization by @BorysTheDev in #5652
• fix(json): support json keys containing dots in bracket notation by @vyavdoshenko in #5650
• fix: accounting of metrics under squash_stats_latency_lower_limit filter by @romange in #5659
• chore: sort per shard sorted sets for SINTER by @kostasrim in #5617
• fix: macos build tiered_storage.h by @vyavdoshenko in #5663
• fix: RoundRobinSharder for all proactors by @kostasrim in #5654
• workflows: Add a dispatch action which will run pytests on repeat by @abhijat in #5668
• fix(pytest): Don't delete and modify huge value keys in test_replication_all by @mkaruza in #5657
• chore: move tiering flags by @kostasrim in #5665
• fix(json_family): Remove expiry of existing keys in JSON.SET by @mkaruza in #5670
• fix: test_migration_timeout_on_sync by @BorysTheDev in #5667
• fix: refactor save synchronization to prevent deadlocks and crashes by @vyavdoshenko in #5655
• fix(pytest): Increase timeout in test_seeder_key_target by @mkaruza in #5677
• chore: calibrate regression tests running time by @kostasrim in #5658
• chore(lua): Add lua redis.log function by @mkaruza in #5672
• chore: add more detailed timings for slow squash hops by @romange in #5679

Full Changelog: v1.32.0...v1.33.0

v1.32.0

Dragonfly v1.32.0

Important stability fixes and changes:

• Lots of performance improvements around Search #5341#5327 #5350 #5545 #5533
• Add optional latency percentile stats #5372 and #5428
• Update mimalloc version to 2.24
• Added BZMPOP and BLMPOP commands #5436 and #5370
• Added CONFIG REWRITE command #5471

What's Changed

• test(search): add unified benchmarks for prefix/suffix/infix search by @vyavdoshenko in #5319
• chore: get rid of quicklist encoding by @romange in #5282
• feat: add flag pause_wait_timeout by @BorysTheDev in #5328
• test: enable test_multidim_knn python test by @vyavdoshenko in #5329
• test: make random determenistic in test_multidim_knn python test by @vyavdoshenko in #5335
• fix: data race when doing a snapshot in cache mode by @romange in #5336
• chore(search_family): Minor performance improvements to GetAllResults methods. FIRST PR by @BagritsevichStepan in #5341
• fix: JSON.DEL to return 0 for non-existing keys instead of error by @vyavdoshenko in #5343
• chore: improve test_full_sync test by @romange in #5340
• chore: Refactor CompactObject encoding by @dranikpg in #5342
• feat: Added Sort_RO Command. by @H4R5H1T-007 in #5332
• chore: avoid deduplicating docs ids when not needed by @romange in #5344
• chore: add hdr_histogram to the project by @romange in #5330
• chore(tiering_test): tune memory settings to be below 6GB by @romange in #5345
• chore: suffix search by @dranikpg in #5327
• test: Vector operations benchmark was added by @vyavdoshenko in #5353
• feat(replication): Aggregate data until threshold before replication by @mkaruza in #5308
• SimSIMD integration for Vector operations by @vyavdoshenko in #5348
• feat (server) : introduce no point in time replication by @adiholden in #5103
• chore: move 3rd party libraries into a separate cmake file by @romange in #5362
• fix(pytest): Increase allowed diff in test_partial_replication_on_same_source_master by @mkaruza in #5363
• fix(replication): reduce ERROR logging during normal shutdown by @vyavdoshenko in #5366
• chore: remove unneeded mimalloc-specific code by @romange in #5367
• chore: add mimalloc2 third_party library by @romange in #5364
• fix(replication): reduce ERROR to WARNING for connection issues during sync by @vyavdoshenko in #5369
• feat(search): Suffix/Infix search for Tags by @dranikpg in #5350
• fix(zset): expand parsing of rank arguments beyond int32 by @romange in #5377
• test: replication_test.py::test_bug_in_json_memory_tracking enabled by @vyavdoshenko in #5375
• chore: Disable test_replication_all cache mode by @mkaruza in #5385
• chore: Make fakeredis part of the core CI checks by @romange in #5380
• fix(zset_family): Return early if range is expected to be zero by @mkaruza in #5386
• fix: JSON.STRLEN behavior for non-existent keys by @vyavdoshenko in #5389
• chore: add psync metrics for replica by @kostasrim in #5365
• fix(server): fix serialization on move with big values by @adiholden in #5390
• feat(pytest): Set and log random seed used in pytest by @mkaruza in #5391
• chore: remove zset/t_zset files by @romange in #5387
• feat: add support for BLMPOP command by @EricHayter in #5370
• feat(pytest): Initialize random seed for each test by @mkaruza in #5395
• fix(values): fix typos in Helm README and values file by @adimoragbloch in #5400
• fix: no set/hmap expiration during serialization by @BorysTheDev in #5349
• feat(search_family): Add SortedVector<pair<DocId, double>> by @BagritsevichStepan in #5405
• feat: add snapshot-id field into dfs files by @BorysTheDev in #5399
• chore: Enable point_in_time_snapshot by default by @mkaruza in #5403
• chore: Increase BLMPopBlocking sleep duration by @mkaruza in #5408
• chore: skip flaky test_pipeline_cache_size by @kostasrim in #5410
• chore: introduce class scope fixtures by @romange in #5409
• fix(pytest): Increase REPLTAKOVER time in test_take_over_seeder by @mkaruza in #5415
• server_family: Expose command latency stats as part of info by @abhijat in #5372
• fix(search): Cut off results before serializing by @dranikpg in #5412
• test: Add Dragonfly search benchmark tests by @vyavdoshenko in #5406
• feat(search_family): Introduce RangeTree by @BagritsevichStepan in #5414
• fix: search_benchmark_utils on python 3.12 by @romange in #5422
• feat(search_family): Add RangeTree to NumericIndex by @BagritsevichStepan in #5416
• chore: switch to mimalloc2 by @romange in #5368
• feat(search_family): Add optimized merging logic for one and two blocks in RangeResult by @BagritsevichStepan in #5417
• chore(range_tree): Simplify RangeTree code by @BagritsevichStepan in #5418
• Feat config management by @boomballa in #5388
• Lua library Integer overflow can cause the DragonFly crash. (CVE-2020-14147) by @KIMDONGYEON00 in #5421
• chore(search): LIMIT=0 Fast path and remove LOAD parameter from FT.SEARCH by @dranikpg in #5419
• server_family: Support resetting latency histogram by @abhijat in #5428
• test: search benchmark updated to use class fixture for multiple tests by @vyavdoshenko in #5425
• chore: harden checks in _mi_page_malloc_zero by @romange in #5427
• fix(pytest): Wait replication to be in sync in test_stream_approximate_trimming by @mkaruza in #5430
• chore: avoid squashing when squashing_current_reply_size crosses limit by @kostasrim in #4924
• fix: acl log missing return after reply by @kostasrim in #5454
• chore: add acl help by @kostasrim in #5456
• chore: allow initializing a separate huffman table for string values by @romange in #5470
• feat(search_family): Introduce SingleBlockRangeResult and TwoBlocksRangeResult by @BagritsevichStepan in #5424
• chore: add tls_caching flags by @kostasrim in #5485
• chore: add more logs during replication by @BorysTheDev in #5489
• chore(set_family): Fast path SPOP with small count by @dranikpg in #5491
• fix: add a sleep to relax slot migration process by @romange in #5478
• chore: add tls_socket accept error metric by @romange in #5492
• chore: add io_read_cnt metric by @romange in #5494
• chore: add tls handshake statistics by @kostasrim in #5493
• feat: add more auxillary strings to the snapshot by @romange in #5411
• chore: log frequency for IsReplySizeOverLimit by @kostasrim in #5501
• fix: adjust BLMPOP command options and implementation for proper journaling by @EricHayter in #5503
• chore: add more metrics by @romange in https://github.com/dragonflydb/dragonfly/p...

v1.31.2

This is a patch release.

What's Changed

• fix: prevent idle disconnects for replica flow connections (#5506)
• feat: add debug segments command (#5505)
• server: Make memory defrag blocking, accept optional threshold (#5498)
• fix: allow eviction of zero-allocation entries in cache mode (#5500)
• chore: add more metrics (#5504)

Full Changelog: v1.31.1...v1.31.2

v1.31.1

This is a patch release.

What's Changed

• fix: two tls issues
  1. Excessive logging during tls handshake in some cases.
  2. TLS Short read that hinders read performance.
• chore: add tls handshake statistics (#5493)
• chore: add io_read_cnt metric (#5494)
• chore: add tls_socket accept error metric (#5492)
• fix: add sleep to relax slot migration process (#5478)
• fix(set_family): Fast path SPOP with small count (#5491)
• chore: add more logs during replication (#5489)
• chore: add tls_caching flags (#5485)

Full Changelog: v1.31.0...v1.31.1