MDEV-36226 Stall and crash when page cleaner fails to generate free p… #3885
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
dr-m
reviewed
Mar 10, 2025
dr-m
reviewed
Mar 11, 2025
mariadb-DebarunBanerjee
force-pushed
the
10.6-MDEV-36226
branch
from
e190624 to
de96813
Compare
dr-m
reviewed
Mar 13, 2025
storage/innobase/buf/buf0flu.cc
Outdated
Comment on lines
2697
to
2704
| sql_print_information("Innodb: LSN flush parameters\n" | ||
| "-------------------\n" | ||
| "System LSN : %" PRIu64 "\n" | ||
| "Checkpoint LSN: %" PRIu64 "\n" | ||
| "Flush ASync LSN: %" PRIu64 "\n" | ||
| "Flush Sync LSN: %" PRIu64 "\n" | ||
| "-------------------", | ||
| lsn, clsn, buf_flush_async_lsn, buf_flush_sync_lsn); |
There was a problem hiding this comment.
Usually all InnoDB messages are prefixed by InnoDB: (note the case). Do we need this many rows for the output? You need to write uint64_t{buf_flush_async_lsn} or similar to avoid compilation errors:
error: cannot pass object of non-trivial type 'Atomic_relaxed<lsn_t>' (aka 'Atomic_relaxed<unsigned long long>') through variadic function; call will abort at runtime [-Wnon-pod-varargs]
| ulint lru_size= UT_LIST_GET_LEN(LRU); | ||
| ulint dirty_size= UT_LIST_GET_LEN(flush_list); | ||
| ulint free_size= UT_LIST_GET_LEN(free); | ||
| ulint dirty_pct= lru_size ? dirty_size * 100 / (lru_size + free_size) : 0; |
There was a problem hiding this comment.
dirty_pct seems to be redundant information that can be calculated from the rest. It could also be totally misleading, because we were reading these fields without proper mutex or flush_list_mutex protection.
…ages during Async flush During regular iteration the page cleaner does flush from flush list with some flush target and then goes for generating free pages from LRU tail. When asynchronous flush is triggered i.e. when 7/8 th of the LSN margin is filled in the redo log, the flush target for flush list is set to innodb_io_capacity_max. If it could flush all, the flush bandwidth for LRU flush is currently set to zero. If the LRU tail has dirty pages, page cleaner ends up freeing no pages in one iteration. The scenario could repeat across multiple iterations till async flush target is reached. During this time the DB system is starved of free pages resulting in apparent stall and in some cases dict_sys latch fatal error. Fix: In page cleaner iteration, before LRU flush, ensure we provide enough flush limit so that freeing pages is no blocked by dirty pages in LRU tail. Log IO and flush state if double write flush wait is long. Reviewed by: Marko Mäkelä
mariadb-DebarunBanerjee
force-pushed
the
10.6-MDEV-36226
branch
from
de96813 to
3ab4934
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
During regular iteration the page cleaner does flush from flush list with some flush target and then goes for generating free pages from LRU tail. When asynchronous flush is triggered i.e. when 7/8 th of the LSN margin is filled in the redo log, the flush target for flush list is set to innodb_io_capacity_max. If it could flush all, the flush bandwidth for LRU flush is currently set to zero. If the LRU tail has dirty pages, page cleaner ends up freeing no pages in one iteration. The scenario could repeat across multiple iterations till async flush target is reached. During this time the DB system is starved of free pages resulting in apparent stall and in some cases dict_sys latch fatal error.
Fix: In page cleaner iteration, before LRU flush, ensure we provide enough flush limit so that freeing pages is no blocked by dirty pages in LRU tail. Log IO and flush state if double write flush wait is long.
Impact: It could result in increased IO due to LRU flush in specific cases.
Release Notes
None
How can this PR be tested?
Regular Innodb test should cover the path. Performance and stress Test should be run to judge for possible impact.
Reproducing the base issue would require large buffer pool, long run and synchronization between foreground and Innodb background threads.
Basing the PR against the correct MariaDB version
mainbranch.PR quality check