You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* [C++] [WIP] Hot/cold design (#508)
* [C++] Add force option to record user Delete request
* If force is set to true, then a tombstone will be appended to the log,
irrespective of whether the hash index contains the record itself.
* [C++] Support for defining a Guid for a session externally
* [C++] Replace checkpoint inline callback definition
... with predefined types.
* [C++] Rmw can be configured to not create record
... if one does not exists inside the log.
* [C++] Implement method for conditionally copying to log tail
* [C++] Use minimum number of mutable pages if value is 0
* [C++] Initial implementation of FASTER hot-cold design
* Currently supports reads, upserts, deletes and RMWs.
* [C++] Fix compilation error
* [C++] Initial tests for hot-cold design
* [C++] Lookup-based hybrid log compaction (#487)
* [C++] Log scan can now return record address, along with record
* [C++] Add implementation of Address + operator
* [C++] Add method for finding if a record exists in the hybrid log
* Note that if a tombstone record exists, it will return true.
* [C++] Initial implementation of a better log compaction algorithm
* It leverages the hash index to identify live records, and copy them
to the tail of the log.
* Ensures that if a user performs a concurrent upsert, compaction won't
overwrite their operation.
* Avoids expensive scan of the entire log -- only the relevent log
section is read.
* [C++] Remove unnecessary template typenames
* [C++] Fix several issues in compaction code
* [C++] Several bugfixes in log scan iterator
* Now correctly switches to read from next page if the record didn't
entirely fit in the previous one.
* Fix bug where record address was wrong
* Fix bug where in-disk page wasn't read due to >0 offset in passed
address.
* [C++] Minor bugfixes in lookup-based compaction
* [C++] Add tests for lookup-based compaction algorithm
* [C++] Fix bug in Addres + operator
* [C++] Add a medium-sized value type for tests
* [C++] Compaction context/entry now stores record address
* [C++] Bugfixes in compaction code
* [C++] Update log compaction tests
* Add tests with where other threads perform concurrent insertions &
deletions
* Test actual log truncation correctness (using `ShiftBeginAddress`
method).
* [C++] Fix test compilation error
* [C++] Refactor log compaction code
* [C++] Minor changes
* [C++] Better status handling in RecordExists method
* [C++] Log compaction with multiple threads
* [C++] Unoptimized concurrent page-granularity compaction
* [C++] Fix bug in tests
* [C++] Concurrent compaction \w non-blocking waiting for threads
* [C++] Introduce page- and record-granularity log iterators
* Page-granularity iterator is used with the new lookup-based compaction
method, while the (older) record-granularity one is used by the (old)
compaction algorithm.
* The page log iterator can still be optimized further (i.e. avoid
locking, prefetching, etc).
* [C++] Avoid key/value copying on compaction contexts
* [C++] Improvements on the log compaction method
* + bugfix on sessions start/stop when using multiple threads.
* [C++] Make obsolete write key calls on Read/Exists contexts
* [C++] Add variable-length key tests for log compaction
* [C++] Add delete ops to varlen keys tests
* [C++] Concurrent lock-free log iterator with prefetching
* [C++] Add variable-length value tests for lookup compaction
* [C++] Bugfixes in tests
Co-authored-by: Badrish Chandramouli <[email protected]>
Co-authored-by: Kirk Olynyk <[email protected]>
* Better design FASTER's of copy to tail method
* Implement hot-cold & cold-cold compaction
* Include RMW in compact lookup tests
* Bugfixes in core FASTER
* Bugfixes & preliminary work for retrying RMW ops in hot-cold
* Minor changes in compact lookup tests
* Rework hot-cold implementation to support retries
* Bugfixes in FASTER RMW & log compaction
* Update hot-cold design tests
* Minor change in RMW
* Minor cleanup & better handling of complete pending requests
* Proper handling of deleted records in hot-cold
`Read` method returns `NOT_FOUND` either if no record was found, or if a
tombstone record was found. While there is no point separating the two
cases in the single log case, in the hot-cold design it is important to
know which is the case.
The most useful use-case for that is for the hot-cold `Read` method: if
a tombstone was found in hot log, there is no need to search the cold
log. In other words, `Read` will go through the cold log only if no record
(normal or tombstone) was found in the hot log.
Thus, FASTER `Read` method can now be configured to return a different
status (i.e. `ABORTED`) if it finds a tombstone, insted of `NOT_FOUND`.
We support this, using an additional optional flag `abort_if_tombstone`
in the Read function prototype. By default this is set to `false` --
only hot-cold design this flag, when a Read is issued on the hot log.
* Update hot-cold tests
* Bugfix to guarrantee progress in both stores
Co-authored-by: Badrish Chandramouli <[email protected]>
Co-authored-by: Kirk Olynyk <[email protected]>
* Rename testcases
* Initial implementation of conditional insert
* Cleanup
* Fix hot-cold Read-Modify-Write operation
* Replace hot-cold compaction login with Upserts
* Minor fix
* Add support for variable-length keys in hot-cold
* Minor changes
* Add RMW ops in variable length key tests
* Bugfix on hot-cold read operation
* Add test that exposes Reads-Compaction race condition
* Fix for Read-Compaction race condition
During log compaction (\w lookup), live records are being copied to the
tail of the log. Once the all live records have been copied, the part
of the log that was just compacted is truncated. However, there is a
slim chance that during the log truncation a pending Read operation will
return NOT_FOUND, even thought a record for this key exists.
Specifically this can happen if a live record is being copied to the
tail of the log, but the Read operation has already checked the log
tail, and has issued one (or more) I/O requests to read disk-resident
records. In this case, if we truncate the log before this Read operation
reaches the live record, the Read will return NOT_FOUND.
In order to handle this undesired behavior we keep track of the number
of truncations after performance log compaction (global). Each Read operation
keeps a local copy of this number in its context. If the Read operation
has reached the end (begin) of the log and has not found a live record,
we check if there a log truncation occured due to a log compaction. If
this is the case, this Read op will retry, in order check the newly
introduced log part. This last part is now supported using the
`min_start_address' argument that can be defined in the Read context.
In this case, the Read operation will not go throught the entire log.
* hot-cold: Add test for variable-length value records
* Bugfix in CleanHashTableBuckets
This fixes some spurious error messages, including the following:
`Assertion `idx < size_' failed.'
* Fix wrong assertion, for when RMW entry is deleted
* Working prototype -- suboptimal HC context management
* Better context management memory-wise
* Bugfixes caused by ops inside assertions
* HC RMW operation: proper memory free-ing for contexts
* Minor fix in hot-cold test
* Modify compaction test to use variable number of threads
* Bugfix
* Bugfix on log scan
Fixes a bug that was due to improper calculation of how many bytes to
read from disk.
* Support for hybrid log checkpointing after compaction
* Add tests for checkpointing after compaction
* [WIP] Decoupling index from log
* Move index checkpointing to hash index class
* Fix error from gtest
* Minor changes
* [WIP] Support for index ops that go pending
* [WIP] More work towards decoupling
* Add HashIndex class template to FasterKv
* Minor cleanup
* [WIP] Common interface for index classes
* Update FasterKvHC to accomodate index changes
* [WIP] More work to decouple index-related structures
* Minor fixes in tests
* Use overflow bucket entries only in hot index
* Make root_path accessible from outside of class
* Expand hash index interface with FASTER helper methods
* Parameterize Hot/Cold hash index bucket bit structure
* Auxiliary methods in key hash for cold-index ops
* Contexts for index-related I/O operations
* Store address of newly-created record in async index context
* Initial implementation of cold index
* Initial integration of cold-index to FASTER
* Cold index improvements
* Implement garbage collect
* Used RMW op for FindOrCreateEntry for better performance
* Add cold index test
* Simplify some FASTER contexts
* Integrate cold index to new FASTER design
* Add/fix minor checks
* Modify bitfields to accomodate read-cache integration
* Initial read cache implementation
* Integrate read cache with FASTER
* Several read-cache bugfixes
* In-mem & paging test now check multiple FASTER configs
* Support compaction with Read Cache
* Minor bugfixes
* Improve debug logging
* Bugfix on in-mem hash index
* Multiple fixes in read-cache implementation
* Read cache bugfix when concurrent compaction & read
* Minor bugfixes
* Include thread ID to log messages
* Bugfixes in Read Cache eviction
- Addresses cases where read cache eviction and log truncation occur
concurrently.
* Better support for epoch framework on two logs
* Support concurrent hash index GC with read-cache GC
* Rework ConditionalInsert op for hot-cold design
* New implementation scraps `start_search_entry' in favor of
`expected_hlog_address'.
* Supports all scenarios, including when `min_search_offset' and/or
`expected_hlog_address' are invalid (due to record missing from hot
log during hot-cold RMW copy from cold log operation.
* Code greatly simplified, resulted in additional performance
improvements.
* Fix one memory leak issue in file I/O buffer pool
* Minor fixes
* Move read-cache block allocate to FasterKv
* Support for inserting cold log records to read cache
* Add WARN level in log message module
* Working epoch framework sync on hot-cold design
* Fix bug in FasterIndex when chunk has been updated
During a cold-index operation, it is possible that some other thread
operates on the same chunk. If both ops wish to update the chunk, one of
them will back-off and retry (the one that fails during CAS). This
bugfix fixes incorrect logical behavior wrt cold-log updates/finds.
* Fix bug where epoch framework made no progress
* Add config structs
* Implement single-log auto-compaction
* Fix bug in cold index
* Fix in cold-index garbage collect algorithm
* Minor cold-index fixes
* Minor read cache fix
* Minor changes & cleanup old code
* Make key hash function a class
* Update & modernized tests
* Update hot-cold compaction inner-workings
* Implement unsafe buffer resize method
* Minor bugfix in Read Cache eviction
* Exception is now thrown if savedir not present
* Fix hot-cold compaction memory leaks
* Minor bugfix in cold-index
- Used to return Status::Corruption if HashChunkKey entry was not found
in the hash index chunk log
* Add write counts when measuring I/O statistics
* Call user-provided callback if I/O request failed
* Minor var name changes
* Multi-thread eviction in ReadCache
* Implement FASTER-level actions statistics capturing
* Avoid unecessary IOps when min address is specified
* Bugfix in cold-index key hashing
* Add method to check when compaction is/will be active
* UnsafeBufferResize bugfixes
* Implement concurrent hybrid log scan iterator
Provides a way for multiple threads to operate on the same scanned page
in a thread-safe way, guarranteed by the epoch-based framework. Aims to
replace the old hybrid log iterator, where each thread was reading its
own pages.
* Minor cleanup
* Better logging in cold-index
* Copy read-hot records to read-cache tail
* Cold-index: changed from 4kB chunks to 512B ones
* Better hash index statistics
* Minor organizational changes in hash index
* Minor changes in config
* Minor bugfix in read-cache
* Cold index hash chunk to 256B. More options in comments.
* Enable user threads to participate to compaction
This is trigerred when log size comes close to maximum budget.
* Update hot cold test
* Add benchmark code for YCSB
* Add README.md
* Bump up standard to C++17
* Update CMake gtest installation
* TOML support for FasterKv/FasterKvHC configuration
* Configurable cold-index w/ number of index entries
* Informative messages when initializing hlog
* Helpers to view checkpointing/recovery metadata
* Implement cold-index checkpoint & recovery
* Better handling of file paths
* Add cold-index checkpointing/recovery tests
* Bugfix when creating/normalizing dirs
* Support FasterKv checkpoint with read-cache
During checkpointing, we go through the hash bucket entries and replace
addresses pointing to read-cache, with ones pointing to the hybrid log instead.
* CompactWithLookup: replace static contexts with proper ones
* Checkpoint/Recovery for FasterKvHC (HotCold store)
* Checkponting/Recovery tests for hot-cold store
* Minor fixes in tests
* ShiftBeginAddress: add support for context in callbacks
* Checkpoint: minor fix on callback context behavior
* Minor fixes
* Minor fix
* Minor changes in CMake files
* Minor cleanup
* Add missing copyright header
* Fix F2 performance regression on in-memory tests
Employing a local reference for `thread_ctx()`, instead of calling it
over and over again, results in up to 20% better throughput for
in-memory workloads; F2 performance is now withing 2-3% of original
stable FASTER C++ codebase (i.e., pre-F2 code refactoring).
* Fit pending context struct in a single cache-line
* Minor changes
* Simplify hash function usage
* Support for ops-based benchmarking
Currently benchmarking is time-based: i.e., a workload is run for a X
period of time. We now support ops-based benchmarking: i.e., running a
workload until Y requests have completed.
* Support more benchmarking workloads
* F2: Add hash index class as template arguments
* Add blocking method that waits for log compaction completion.
* F2: Add benchmarking code
* Minor cleanup
* Add missing pragma for header file
* Minor fix in log messages
* Minor index class reordering
* F2: Changes in classes naming
* Naming improvements for index classes
* Remove deprecated benchmark files
* Provide config defaults for FASTER hlog compaction
* Throttle threads when max hlog budget has been reached
For write-intensive workloads, it is possible that even during
compaction, the maximum hlog budget can been reached. For example,
this can occur when the rate of ingesting requests to the hot log is
higher then the rate of compacting rates to the cold log.
To fix, we now allow user threads to participate to the compaction
process, only if we reach the (hard) hlog size limit. Note that
background compaction threads are anyways performing only compaction
work. Once the compaction completes, user threads can resume serving
user requests, as before.
* Minor change
* Bugfix
* Set I/O queue size in a single place
* Add report (i.e., always print) logging level
* Bugfix on thread throttling during compaction
* Add some warnings for return values
* Fix class member reordering warnings
* Fix warnings wrt pointer casting
* Remove unused variables
* String repr for Status enum
* Fix int types due to warnings
* Suppress warnings on unused typedefs.
We would like to keep the typedefs, even if unused, for clarity
purposes.
* Minor changes
* Minor structural change
* Bugfixes
* Fix warnings on tests
* Minor naming changes
* Enhance C++ FasterKv, F2Kv documentation
* Fix minor compilation
* Bump toml11 version to 4.2.0
* Remove unecessary debug messages
* Fix compilation error
* Add (unsafe) index compaction method
* Fix race condition in read-cache insertion method
When attempting to insert a read-cache-resident record to a full read-cache
tail page, the `BlockAllocate` method is called to request a new in-memory
page. However, this operation can trigger a page eviction that invalidates
the last in-memory read-cache page.
If the record being inserted was residing in the now-invalidated page,
this causes a memory access error.
Fix: Implement explicit validation check after `BlockAllocate`, i.e., after
requesting a new page to check whether the record is still in-memory.
* Remove invalid assertion
During garbage-collection, it is possible for a thread to *transiently*
be in a state different that the ones covered; this does not affect
correctness, as the next call to `Refresh()` will move this thread to a
state combination that is covered.
* Fix invalid assertions
Read-cache's `begin_address` always remains unchanged from its initial
value, as no log truncations are performed. As read-cache is an
in-memory structure, the log start is defined by (safe) `head_address` instead.
* Cleanup
* Refactor and simplify F2's index context
* Fix read-cache Evict issue with multiple threads
It is possible that read-cache Evict() callback is called by many
threads, for different [from, to] log address ranges (i.e., different
threads want to evict different pages). This code refactoring makes sure
that in this case, all to-be-evicted read-cache entries do properly
alter the hash chains to avoid any data/system inconsistencies.
* Fix issue in read-cache where invalid record would be inserted
* F2: Fix incorrect read-cache insertion behavior
Read-cache's TryInsert() method now contains the correct checks when
inserting a disk-resident record (from either hot/cold) log to RC.
* Minor fix in read-cache
* Switch to best-effort insertion behavior for read-cache
This solves some race condition issues (e.g., when copying a
read-cache resident record from RC's RO region to its tail, while
allocating a new RC page that triggers eviction, invalidating the source
record). Further, threads are now not blocked inside read-cache (e.g.,
when trying to allocate space for new record).
* Capture statistics for read-cache
* Add more tests for read-cache
* Minor cleanup
* Fix printing statements
* Switch from experimental/filesystem to C++17 filesystem
* Fix MSVC issues with C macros
* Minor fixes for MSVC
* Fix issues with creating/removing folders in tests
* Fix minor MSVC complaints for int types
* Address stack overflow issues in Windows
While in Linux we can increase the program stack size at runtime
(i.e., using `ulimit`), in Windows this is not feasible. Instead,
we need to specify the desired stack size when compiling or linking.
Here, we opt for the latter, increasing the stack size to 64MiB.
* Bump version for TOML parsing library
* Enable struct designated initializers for MSVC
* Update Linux VM image version for CPP in CI
---------
Co-authored-by: Badrish Chandramouli <[email protected]>
Co-authored-by: Kirk Olynyk <[email protected]>
0 commit comments