feat: TOOLS-3076 Add info transactions command for MRT metrics (#362)

* feat: Add MRT transaction metrics support in InfoController and templates

- Introduced InfoTransactionsController to handle MRT metrics.
- Updated InfoController to include transactions in the controller map.
- Added MRT metrics template for displaying transaction statistics.
- Enhanced CliView to render MRT transaction metrics with integrated set statistics.

* refactor: Add MRT monitor and provisionals metrics views

Introduces new commands and templates for displaying MRT monitor and provisionals metrics per namespace. Adds 'do_monitors' and 'do_provisionals' methods to InfoTransactionsController, corresponding templates in templates.py, and rendering logic in CliView. This enhances transaction metrics visibility by separating monitor and provisional statistics.

* feat: Remove info_transactions_mrt method from CliView

Deleted the info_transactions_mrt static method and related code from CliView, consolidating MRT set metrics handling into the main namespace stats logic. This simplifies the codebase and removes redundant functionality.

* feat: TOOLS-3076 Remove MRT metrics sheet and update monitor fields

Deleted the info_transactions_mrt_sheet definition and updated field keys in info_transactions_monitors_sheet to use 'mrt_monitors' and 'mrt_data_used_bytes' instead of 'mrt_set_objects' and 'mrt_set_data_used_bytes'. This streamlines MRT metrics handling and aligns field names with current data sources.

* feat: TOOLS-3076 Add MRT metrics display to info controller

Introduces InfoTransactionsController to display Multi-Record Transaction (MRT) metrics for each namespace, including monitors and provisionals. Updates controller_map to include the new transactions controller.

* feat: TOOLS-3076 Add tests for MRT monitor and provisional metrics views

Added unit tests for CliView.info_transactions_monitors and CliView.info_transactions_provisionals, covering normal cases, empty stats, exceptions, and 'with_' modifiers. These tests verify correct merging of stats, handling of exceptions, and proper calls to cluster and render methods.

* Rename MRT metric key in namespace stats

Changed the key 'mrt_data_used_bytes' to 'pseudo_mrt_monitor_used_bytes' when adding set metrics to namespace stats for improved clarity.

* Remove most formatters from transaction sheets

Formatters for alerts have been removed from most fields in the transaction monitor and provisional sheets, except for a yellow alert on 'Active' monitors. Also, the 'Storage' field now uses 'pseudo_mrt_monitor_used_bytes' instead of 'mrt_data_used_bytes'.

* feat: Add red alert formatting for mrt monitor thresholds

Introduces red alert formatters to the monitor count and storage fields in the transactions monitors sheet, highlighting when values exceed stop-writes thresholds. Also updates view logic to include 'stop-writes-count' and 'stop-writes-size' in namespace stats, and adds debug print statements for set and namespace stats.

* fix: Update test data with stop-writes and pseudo MRT fields

Added 'stop-writes-count', 'stop-writes-size', and 'pseudo_mrt_monitor_used_bytes' fields to test data in CliViewTest. Updated assertions to reflect new field names and values for improved test coverage.

* feat: Add yellow alert thresholds and clean up debug prints

Introduces yellow alert formatters for transaction monitor fields at 70% of stop-writes thresholds and adjusts red alert thresholds to 90%. Removes leftover debug print statements from view.py for cleaner output.

* feat: Fetch MRT SET Stats only for info transactions

Moved merging of <ERO~MRT set statistics into namespace stats from the view layer to the controller layer for both collectinfo and live cluster flows. Updated constants to define MRT_SET, simplified view and test signatures, and improved consistency in metric handling.

* feat: TOOLS-3076 Update metrics to use on strong consistency term

Revised command help and logic to clarify that transaction metrics are displayed only for namespaces with 'strong-consistency' enabled. Updated controllers to use get_strong_consistency_namespace and added checks for empty results. Renamed metric titles in the view to reflect transaction focus instead of MRT.

* fix: TOOLS-3076 Update test titles for transaction metrics

Renamed test titles from 'MRT Monitor Metrics' and 'MRT Provisionals Metrics' to 'Transaction Monitor Metrics' and 'Transaction Provisionals Metrics' for clarity and consistency in unit tests.

* feat: Add method to fetch strong consistency namespaces

Introduces get_strong_consistency_namespace to retrieve statistics for namespaces with strong consistency enabled. Supports filtering by nodes and namespace modifiers, and optionally flips the output structure.

* Add method to filter strong consistency namespaces

Introduced get_strong_consistency_namespace to return only namespaces with strong consistency enabled. This helps in isolating and analyzing relevant namespace statistics.

* fix: Refactor strong consistency namespace stats handling

Updated the monitors and provisionals commands to use get_strong_consistency_namespace instead of get_namespace, and improved merging of MRT set metrics into namespace stats. Added logging for cases where no strong consistency namespaces are found.

* fix: Remove unused print_dict import

Eliminated the import of print_dict from lib.health.util in info_controller.py as it was not used in the file.

* fix: Update transaction metrics help text for clarity

Revised help descriptions to specify that transaction metrics apply to 'strong-consistency' enabled namespaces, replacing references to MRT (Multi-Record Transaction) for improved clarity.

* feat: Added tcs around strong-consistency and general fixes

Updated info_controller logic to always add set metrics to namespace stats, defaulting to 0 when set data is missing or invalid. This prevents missing keys and ensures consistent output for downstream consumers and tests.

* feat: Parallelize set statistics retrieval for namespaces

Replaces sequential fetching of <ERO~MRT set statistics for each namespace with concurrent asyncio tasks, improving performance by gathering data from all nodes in parallel and merging results back into namespace statistics.

* Remove unused aggregator arguments from transaction sheets

Eliminated redundant 'aggregator=Aggregators.sum()' parameters from fields in info_transactions_monitors_sheet and info_transactions_provisionals_sheet. These aggregators were not required for the current usage and their removal simplifies the code.

Author: a-spiker

.gitignore

@@ -148,4 +148,7 @@ tramp
 /auto/
 
 # vscode user settings
-.vscode
+.vscode
+
+# cursor
+.cursor

lib/collectinfo_analyzer/get_controller.py

@@ -232,6 +232,23 @@ def get_namespace(self, for_mods=None):
         ] = self.log_handler.info_statistics(stanza=constants.STAT_NAMESPACE)
         return filter_3rd_level_keys(stats, for_mods)
 
+
+    def get_strong_consistency_namespace(self, for_mods=None):
+        stats: TimestampDict[
+            NodeDict[NamespaceDict[dict[str, str]]]
+        ] = self.log_handler.info_statistics(stanza=constants.STAT_NAMESPACE)
+        filtered_stats =  filter_3rd_level_keys(stats, for_mods)
+        
+        for _, nodes_data in filtered_stats.items():
+            for _, namespaces in nodes_data.items():
+                for namespace in list(namespaces.keys()):
+                    # Skip if namespace data is missing or doesn't have strong consistency
+                    if (not namespaces[namespace] or
+                        namespaces[namespace].get("strong-consistency", "false").lower() != 'true'):
+                        del namespaces[namespace]
+        
+        return filtered_stats
+
     def get_sets(self, for_mods=None, flip=False):
         set_filter: list[str] | None = None
         namespaces_filter: list[str] | None = None

lib/collectinfo_analyzer/info_controller.py

@@ -18,6 +18,8 @@
     GetStatisticsController,
 )
 from lib.utils import constants, util, version
+import logging
+logger = logging.getLogger(__name__)
 
 from .collectinfo_command_controller import CollectinfoCommandController
 from lib.base_controller import ShellException
@@ -35,7 +37,10 @@ def __init__(self):
         self.modifiers = set(["for"])
         self.stats_getter = GetStatisticsController(self.log_handler)
         self.config_getter = GetConfigController(self.log_handler)
-        self.controller_map = dict(namespace=InfoNamespaceController)
+        self.controller_map = dict(
+            namespace=InfoNamespaceController,
+            transactions=InfoTransactionsController
+        )
 
     @CommandHelp("Displays network, namespace, and xdr summary information.")
     async def _do_default(self, line):
@@ -297,3 +302,85 @@ def do_object(self, line):
                 timestamp=timestamp,
                 **self.mods,
             )
+
+
+@CommandHelp(
+    "Displays transaction metrics for each 'strong-consistency' enabled namespace.",
+)
+class InfoTransactionsController(CollectinfoCommandController):
+    def __init__(self):
+        self.modifiers = set()
+        self.stats_getter = GetStatisticsController(self.log_handler)
+
+    @CommandHelp(
+        "Displays monitors and provisionals information for transactions in each 'strong-consistency' enabled namespace.",
+    )
+    def _do_default(self, line):
+        self.do_monitors(line)
+        self.do_provisionals(line)
+
+    @CommandHelp(
+        "Displays monitor-related transaction metrics for each 'strong-consistency' enabled namespace.",
+    )
+    def do_monitors(self, line):
+        # Get namespace statistics which contain MRT metrics
+        ns_stats = self.stats_getter.get_strong_consistency_namespace()
+
+        for timestamp in sorted(ns_stats.keys()):
+            if not ns_stats[timestamp]:
+                continue
+
+            namespaces = set()
+            for _, node_stats in ns_stats[timestamp].items():
+                namespaces.update(node_stats.keys())
+
+             # Check if any strong consistency namespaces were found
+            if not namespaces:
+                logger.debug("No namespaces with strong consistency enabled were found at %s", timestamp)
+                continue
+                
+            for namespace in namespaces:
+                set_data = self.stats_getter.get_sets(for_mods=[namespace, constants.MRT_SET])
+
+                if timestamp in set_data:
+                    for node_id, sets_dict in set_data[timestamp].items():
+                        if node_id not in ns_stats[timestamp] or namespace not in ns_stats[timestamp][node_id]:
+                            continue
+                        set_stats = sets_dict.get((namespace, constants.MRT_SET))
+                        # Always add set metrics to namespace stats, defaulting to 0 if not present
+                        ns_stats[timestamp][node_id][namespace]["pseudo_mrt_monitor_used_bytes"] = int(set_stats.get("data_used_bytes", 0) if set_stats else 0)
+                        ns_stats[timestamp][node_id][namespace]["stop-writes-count"] = int(set_stats.get("stop-writes-count", 0) if set_stats else 0)
+                        ns_stats[timestamp][node_id][namespace]["stop-writes-size"] = int(set_stats.get("stop-writes-size", 0) if set_stats else 0)
+
+            self.view.info_transactions_monitors(
+                ns_stats[timestamp],
+                self.log_handler.get_cinfo_log_at(timestamp=timestamp),
+                timestamp=timestamp,
+                **self.mods,
+            )
+
+    @CommandHelp(
+        "Displays provisional-related transaction metrics for each 'strong-consistency' enabled namespace.",
+    )
+    def do_provisionals(self, line):
+        # Get namespace statistics which contain MRT metrics
+        ns_stats = self.stats_getter.get_strong_consistency_namespace()
+
+        for timestamp in sorted(ns_stats.keys()):
+            if not ns_stats[timestamp]:
+                continue
+
+            namespaces = set()
+            for _, node_stats in ns_stats[timestamp].items():
+                namespaces.update(node_stats.keys())
+            
+            if not namespaces:
+                logger.debug("No namespaces with strong consistency enabled were found at %s", timestamp)
+                continue
+
+            self.view.info_transactions_provisionals(
+                ns_stats[timestamp],
+                self.log_handler.get_cinfo_log_at(timestamp=timestamp),
+                timestamp=timestamp,
+                **self.mods,
+            )

lib/live_cluster/get_controller.py

@@ -620,6 +620,49 @@ async def get_namespace(self, flip=False, nodes="all", for_mods=[]):
 
         return ns_stats
 
+    async def get_strong_consistency_namespace(self, flip=False, nodes="all", for_mods: list[str] | None = None):
+        """
+        Get statistics for namespaces with strong consistency enabled.
+        
+        Args:
+            flip: Whether to flip the output structure (default: False)
+            nodes: Nodes to query (default: "all")
+            for_mods: Optional list of namespace filters (default: None)
+            
+        Returns:
+            Dictionary of namespace statistics for strong consistency namespaces
+        """
+        namespace_set = set(await _get_all_namespaces(self.cluster, nodes))
+        namespace_list = list(util.filter_list(namespace_set, for_mods))
+        tasks = [
+            asyncio.create_task(
+                self.cluster.info_namespace_statistics(namespace, nodes=nodes)
+            )
+            for namespace in namespace_list
+        ]
+        ns_stats = {}
+        for namespace, stat_task in zip(namespace_list, tasks):
+            ns_stats[namespace] = await stat_task
+
+            if isinstance(ns_stats[namespace], Exception):
+                continue
+
+            for node in list(ns_stats[namespace].keys()):
+                if not ns_stats[namespace][node] or isinstance(
+                    ns_stats[namespace][node], Exception
+                ):
+                    ns_stats[namespace].pop(node)
+                    continue
+
+                strong_consistency = ns_stats[namespace][node].get("strong-consistency", "false").lower() == 'true'
+                if not strong_consistency:
+                    ns_stats[namespace].pop(node)
+
+        if not flip:
+            return util.flip_keys(ns_stats)
+
+        return ns_stats
+
     async def get_sindex(
         self, flip=False, nodes="all", for_mods: list[str] | None = None
     ):

lib/live_cluster/info_controller.py

@@ -43,7 +43,10 @@
 class InfoController(LiveClusterCommandController):
     def __init__(self):
         self.modifiers = set(["with", "for"])
-        self.controller_map = dict(namespace=InfoNamespaceController)
+        self.controller_map = dict(
+            namespace=InfoNamespaceController,
+            transactions=InfoTransactionsController
+        )
         self.config_getter = GetConfigController(self.cluster)
         self.stat_getter = GetStatisticsController(self.cluster)
 
@@ -299,3 +302,104 @@ async def do_object(self, line):
         return util.callable(
             self.view.info_namespace_object, stats, rack_ids, self.cluster, **self.mods
         )
+
+
+@CommandHelp(
+    "Displays transaction metrics for each 'strong-consistency' enabled namespace.",
+    usage=f"[{ModifierUsageHelp.WITH}]",
+    modifiers=(with_modifier_help,),
+)
+class InfoTransactionsController(LiveClusterCommandController):
+    def __init__(self, get_futures=False):
+        self.modifiers = set(["with"])
+        self.get_futures = get_futures
+        self.stats_getter = GetStatisticsController(self.cluster)
+
+    @CommandHelp(
+        "Displays monitors and provisionals information for transactions in each 'strong-consistency' enabled namespace.",
+    )
+    async def _do_default(self, line):
+        tasks = await asyncio.gather(
+            self.do_monitors(line),
+            self.do_provisionals(line),
+        )
+        if self.get_futures:
+            # Wrapped to prevent base class from calling result.
+            return dict(futures=tasks)
+
+        return tasks
+
+    @CommandHelp(
+        "Displays monitor-related transaction metrics for each 'strong-consistency' enabled namespace.",
+        usage=f"[{ModifierUsageHelp.WITH}]",
+        modifiers=(with_modifier_help,),
+    )
+    async def do_monitors(self, line):
+        # Get namespace statistics which contain MRT metrics
+        ns_stats = await self.stats_getter.get_strong_consistency_namespace(nodes=self.nodes)
+
+        # Collect all namespaces from all nodes
+        namespaces = set()
+        for node_id, node_stats in ns_stats.items():
+            namespaces.update(node_stats.keys())
+        
+        # If no namespaces with strong consistency enabled were found, return
+        if not namespaces:
+            logger.debug("No namespaces with strong consistency enabled were found for do_monitors")
+            return
+
+        # Get <ERO~MRT set statistics for all namespaces from all nodes concurrently
+        set_stats_futures = [
+            asyncio.create_task(
+                self.cluster.info_set_statistics(namespace, constants.MRT_SET, nodes=self.nodes)
+            )
+            for namespace in namespaces
+        ]
+        all_set_data = await asyncio.gather(*set_stats_futures)
+
+        # Map the results back to their namespaces and merge into ns_stats
+        for namespace, set_data in zip(namespaces, all_set_data):
+            for node_id, set_stats in set_data.items():
+                if (
+                    isinstance(set_stats, Exception)
+                    or node_id not in ns_stats
+                    or namespace not in ns_stats[node_id]
+                ):
+                    continue
+
+                # Add set metrics to namespace stats with prefixed names
+                ns_stats[node_id][namespace]["pseudo_mrt_monitor_used_bytes"] = int(set_stats.get("data_used_bytes", 0) if set_stats else 0)
+                ns_stats[node_id][namespace]["stop-writes-count"] = int(set_stats.get("stop-writes-count", 0) if set_stats else 0)
+                ns_stats[node_id][namespace]["stop-writes-size"] = int(set_stats.get("stop-writes-size", 0) if set_stats else 0)
+        
+        return util.callable(
+            self.view.info_transactions_monitors,
+            ns_stats,
+            self.cluster,
+            **self.mods,
+        )
+
+    @CommandHelp(
+        "Displays provisional-related transaction metrics for each 'strong-consistency' enabled namespace.",
+        usage=f"[{ModifierUsageHelp.WITH}]",
+        modifiers=(with_modifier_help,),
+    )
+    async def do_provisionals(self, line):
+        # Get namespace statistics which contain MRT metrics
+        ns_stats = await self.stats_getter.get_strong_consistency_namespace(nodes=self.nodes)
+
+        # Check if any strong consistency namespaces were found
+        namespaces = set()
+        for _, node_stats in ns_stats.items():
+            namespaces.update(node_stats.keys())
+        
+        if not namespaces:
+            logger.debug("No namespaces with strong consistency enabled were found for do_provisionals")
+            return
+        
+        return util.callable(
+            self.view.info_transactions_provisionals,
+            ns_stats,
+            self.cluster,
+            **self.mods,
+        )

lib/utils/constants.py

@@ -75,6 +75,7 @@
 COLLECTINFO_SEPERATOR = "\n====ASCOLLECTINFO====\n"
 COLLECTINFO_PROGRESS_MSG = "Data collection for %s%s  in progress..."
 
+MRT_SET = "<ERO~MRT"
 
 class Enumeration(set):
     def __getattr__(self, name):