⚡️ Speed up function calculate_text_metrics by 109% in PR #11114 (feat/langchain-1.0)

[codeflash-ai]

⚡️ This pull request contains optimizations for PR #11114

If you approve this dependent PR, these changes will be merged into the original PR branch feat/langchain-1.0.

This PR will be automatically closed if the original PR is merged.

---

📄 109% (1.09x) speedup for calculate_text_metrics in src/backend/base/langflow/api/v1/knowledge_bases.py

⏱️ Runtime : 84.7 milliseconds → 40.5 milliseconds (best of 91 runs)

📝 Explanation and details

The optimized code achieves a 108% speedup (from 84.7ms to 40.5ms) by eliminating redundant per-column operations and using more efficient pandas string methods.

Key Optimizations

1. Batch Column Processing via stack()

• Original: Processes each column separately in a loop, creating a new Series for each column (226 iterations in profiler)
• Optimized: Combines all valid columns into a single stacked Series in one operation
• Impact: Reduces intermediate Series allocations from O(n_columns) to O(1), saving ~30% time on astype/fillna operations (line profiler shows 102ms → 89ms for these operations)

2. Regex-based Word Counting

• Original: Uses str.split().str.len() which creates Python lists for every cell, then counts list lengths (145ms in profiler - the slowest operation)
• Optimized: Uses str.count(r'\S+') which counts non-whitespace sequences directly without materializing lists
• Impact: Eliminates expensive list allocations; word counting drops from 42.9% to 18.7% of total runtime

3. Early Exit for Empty Column Lists

• Original: Enters loop even when no valid columns exist
• Optimized: Pre-filters valid columns and returns immediately if none exist
• Impact: Saves 5 test cases (~11% of test runs) from unnecessary processing

Performance Characteristics

The optimization excels when:

• Many columns are analyzed (batch processing reduces overhead multiplicatively)
• Large DataFrames with text data (regex counting scales better than list creation)
• Repeated calls (as suggested by the 45 hits in profiler, typical in data pipelines)

Test results confirm this: the large-scale tests (500 rows, 100 columns) benefit most, while simple single-column cases see modest gains due to the overhead of stacking being comparable to the single-iteration loop.

Correctness Note

Both implementations handle edge cases identically (None → "None", empty strings, Unicode), as confirmed by the comprehensive test suite passing.

✅ Correctness verification report:

Test	Status
⚙️ Existing Unit Tests	🔘 None Found
🌀 Generated Regression Tests	✅ 45 Passed
⏪ Replay Tests	🔘 None Found
🔎 Concolic Coverage Tests	🔘 None Found
📊 Tests Coverage	100.0%

🌀 Click to see Generated Regression Tests

import numpy as np
# function to test
import pandas as pd
# imports
import pytest  # used for our unit tests
from langflow.api.v1.knowledge_bases import calculate_text_metrics

# unit tests

def test_basic_single_column_simple():
    # Basic functionality: simple strings in a single column
    df = pd.DataFrame({
        "text": [
            "hello world",    # 2 words, 11 characters (including space)
            "one",            # 1 word, 3 characters
            "two words here"  # 3 words, 14 characters (including spaces)
        ]
    })

    # Manually compute expected totals to be independent of the function implementation
    expected_words = sum(len(s.split()) for s in df["text"])
    expected_chars = sum(len(s) for s in df["text"])

    # Call the function and verify both counts and types
    words, chars = calculate_text_metrics(df, ["text"])

def test_multiple_columns_with_missing_column():
    # If a requested column is missing, it should be ignored (no error) and other columns processed
    df = pd.DataFrame({
        "a": ["alpha beta", "gamma"],  # 2 and 1 words
        "b": ["one", "two three"]      # 1 and 2 words
    })

    # Include a non-existent column "c" in text_columns; function must simply skip it
    words, chars = calculate_text_metrics(df, ["a", "b", "c"])

    # manual expected totals for columns a and b
    expected_words = sum(len(s.split()) for s in df["a"]) + sum(len(s.split()) for s in df["b"])
    expected_chars = sum(len(s) for s in df["a"]) + sum(len(s) for s in df["b"])

def test_non_string_values_and_nan_behavior():
    # The function calls astype(str) before fillna(""), which means non-string values
    # (including None and np.nan) will be converted to their Python string representation.
    # This test documents and asserts that behavior.
    df = pd.DataFrame({
        "mixed": [123, True, None, np.nan]  # str -> "123", "True", "None", "nan"
    })

    # Manually compute what the function will count:
    # Use Python's str(v) for each item in the same order as in the DataFrame
    values = list(df["mixed"])
    converted = [str(v) for v in values]  # mirrors astype(str) behavior on these types
    expected_chars = sum(len(s) for s in converted)
    expected_words = sum(len(s.split()) for s in converted)

    words, chars = calculate_text_metrics(df, ["mixed"])

def test_empty_and_whitespace_strings():
    # Edge cases: empty strings and strings with only whitespace should behave correctly
    df = pd.DataFrame({
        "txt": ["", "   ", "a  b", "\t\n", " c "]  # includes empty, whitespace-only, multi-space, tabs/newlines
    })

    # Expected: .split() on whitespace-only strings yields [] -> 0 words;
    # characters include whitespace characters.
    expected_chars = sum(len(s) for s in df["txt"])
    expected_words = sum(len(s.split()) for s in df["txt"])

    words, chars = calculate_text_metrics(df, ["txt"])

def test_unicode_and_emoji_handling():
    # Unicode characters should be counted by their Python string length
    df = pd.DataFrame({
        "u": ["café naïve", "東京", "🙂 🙂", "mañana"]  # mix of accented, CJK, emoji, and tilde
    })

    expected_chars = sum(len(s) for s in df["u"])
    expected_words = sum(len(s.split()) for s in df["u"])

    words, chars = calculate_text_metrics(df, ["u"])

def test_empty_dataframe_and_empty_column_list():
    # When the DataFrame is empty or text_columns list is empty, results should be zero
    empty_df = pd.DataFrame(columns=["a", "b"])
    # No columns requested
    words, chars = calculate_text_metrics(empty_df, [])

    # Columns requested exist but DataFrame has no rows -> sums should be 0
    words2, chars2 = calculate_text_metrics(empty_df, ["a", "b"])

def test_all_nonexistent_columns_return_zero():
    # If none of the requested columns exist in the DataFrame, function must return zeros
    df = pd.DataFrame({"x": ["one", "two"]})
    words, chars = calculate_text_metrics(df, ["missing1", "missing2"])

def test_large_scale_within_limits():
    # Large-scale test but keep total elements under 1000 for the test environment.
    # Create 500 rows (single column) -> 500 elements which is within limits.
    rows = 500
    single_cell = "word " * 3  # three words plus trailing space; split() yields 3 words
    df = pd.DataFrame({"bulk": [single_cell for _ in range(rows)]})

    # Compute expected totals: each row has 3 words; characters include trailing space
    per_row_words = len(single_cell.split())
    per_row_chars = len(single_cell)
    expected_words = per_row_words * rows
    expected_chars = per_row_chars * rows

    words, chars = calculate_text_metrics(df, ["bulk"])
# codeflash_output is used to check that the output of the original code is the same as that of the optimized code.
#------------------------------------------------
import pandas as pd
import pytest
from langflow.api.v1.knowledge_bases import calculate_text_metrics


class TestCalculateTextMetricsBasic:
    """Basic functionality tests for calculate_text_metrics."""

    def test_single_column_simple_text(self):
        """Test basic functionality with a single text column."""
        df = pd.DataFrame({"text": ["hello world", "foo bar"]})
        words, characters = calculate_text_metrics(df, ["text"])

    def test_single_word_per_row(self):
        """Test with single words in each row."""
        df = pd.DataFrame({"text": ["hello", "world"]})
        words, characters = calculate_text_metrics(df, ["text"])

    def test_empty_strings(self):
        """Test with empty string values."""
        df = pd.DataFrame({"text": ["", "", ""]})
        words, characters = calculate_text_metrics(df, ["text"])

    def test_multiple_columns(self):
        """Test with multiple text columns."""
        df = pd.DataFrame({
            "col1": ["hello world"],
            "col2": ["foo bar baz"]
        })
        words, characters = calculate_text_metrics(df, ["col1", "col2"])

    def test_mixed_content_columns(self):
        """Test with varied text content across columns."""
        df = pd.DataFrame({
            "text1": ["hello"],
            "text2": ["world test"]
        })
        words, characters = calculate_text_metrics(df, ["text1", "text2"])

    def test_spaces_only(self):
        """Test with strings containing only spaces."""
        df = pd.DataFrame({"text": ["   ", "  "]})
        words, characters = calculate_text_metrics(df, ["text"])

    def test_numeric_values_converted_to_string(self):
        """Test that numeric values are converted to strings."""
        df = pd.DataFrame({"text": [123, 456]})
        words, characters = calculate_text_metrics(df, ["text"])

    def test_single_long_word(self):
        """Test with a single long word."""
        df = pd.DataFrame({"text": ["abcdefghijklmnop"]})
        words, characters = calculate_text_metrics(df, ["text"])


class TestCalculateTextMetricsEdgeCases:
    """Edge case tests for calculate_text_metrics."""

    def test_nonexistent_column(self):
        """Test with a column name that doesn't exist in DataFrame."""
        df = pd.DataFrame({"col1": ["hello world"]})
        words, characters = calculate_text_metrics(df, ["nonexistent"])

    def test_mixed_existent_and_nonexistent_columns(self):
        """Test with some columns that exist and some that don't."""
        df = pd.DataFrame({"col1": ["hello world"]})
        words, characters = calculate_text_metrics(df, ["col1", "nonexistent", "also_missing"])

    def test_empty_column_list(self):
        """Test with empty list of columns to analyze."""
        df = pd.DataFrame({"col1": ["hello world"]})
        words, characters = calculate_text_metrics(df, [])

    def test_none_values(self):
        """Test that None values are handled (converted to 'None' string)."""
        df = pd.DataFrame({"text": [None, "hello", None]})
        words, characters = calculate_text_metrics(df, ["text"])

    def test_mixed_none_and_empty_strings(self):
        """Test with mix of None and empty string values."""
        df = pd.DataFrame({"text": [None, "", "hello", None, ""]})
        words, characters = calculate_text_metrics(df, ["text"])

    def test_special_characters_and_punctuation(self):
        """Test with special characters and punctuation."""
        df = pd.DataFrame({"text": ["hello, world!", "foo@bar#baz"]})
        words, characters = calculate_text_metrics(df, ["text"])

    def test_newlines_and_tabs(self):
        """Test with newlines and tabs (treated as single spaces by split)."""
        df = pd.DataFrame({"text": ["hello\nworld", "foo\tbar"]})
        words, characters = calculate_text_metrics(df, ["text"])

    def test_unicode_characters(self):
        """Test with Unicode characters."""
        df = pd.DataFrame({"text": ["hello café", "señor"]})
        words, characters = calculate_text_metrics(df, ["text"])

    def test_very_long_strings(self):
        """Test with very long text strings."""
        long_text = "word " * 1000  # 1000 repetitions of "word "
        df = pd.DataFrame({"text": [long_text]})
        words, characters = calculate_text_metrics(df, ["text"])

    def test_multiple_spaces_between_words(self):
        """Test that multiple spaces between words count as word separators."""
        df = pd.DataFrame({"text": ["hello    world", "foo  bar  baz"]})
        words, characters = calculate_text_metrics(df, ["text"])

    def test_boolean_values(self):
        """Test that boolean values are converted to strings."""
        df = pd.DataFrame({"text": [True, False]})
        words, characters = calculate_text_metrics(df, ["text"])

    def test_float_values(self):
        """Test that float values are converted to strings."""
        df = pd.DataFrame({"text": [1.5, 2.7]})
        words, characters = calculate_text_metrics(df, ["text"])

    def test_empty_dataframe(self):
        """Test with an empty DataFrame."""
        df = pd.DataFrame({"text": []})
        words, characters = calculate_text_metrics(df, ["text"])

    def test_dataframe_with_no_matching_columns(self):
        """Test DataFrame where none of the requested columns exist."""
        df = pd.DataFrame({"a": [1], "b": [2]})
        words, characters = calculate_text_metrics(df, ["x", "y", "z"])


class TestCalculateTextMetricsLargeScale:
    """Large-scale performance and scalability tests for calculate_text_metrics."""

    def test_large_dataframe_many_rows(self):
        """Test with a large DataFrame containing many rows."""
        # Create DataFrame with 500 rows
        df = pd.DataFrame({
            "text": ["hello world"] * 500
        })
        words, characters = calculate_text_metrics(df, ["text"])

    def test_large_dataframe_many_columns(self):
        """Test with a large DataFrame containing many text columns."""
        # Create DataFrame with 100 text columns
        data = {}
        for i in range(100):
            data[f"col{i}"] = ["hello world"] * 10
        df = pd.DataFrame(data)
        column_names = [f"col{i}" for i in range(100)]
        words, characters = calculate_text_metrics(df, column_names)

    def test_large_text_values(self):
        """Test with very large text values in the DataFrame."""
        # Create a large text string
        large_text = " ".join(["word"] * 200)  # 200 words in one cell
        df = pd.DataFrame({
            "text": [large_text] * 10
        })
        words, characters = calculate_text_metrics(df, ["text"])

    def test_mixed_large_data_varying_sizes(self):
        """Test with large DataFrame containing varying text sizes."""
        df = pd.DataFrame({
            "short": ["a"] * 250,
            "medium": ["hello world"] * 250,
            "long": [" ".join(["word"] * 50)] * 250
        })
        words, characters = calculate_text_metrics(df, ["short", "medium", "long"])

    def test_many_columns_with_missing_values(self):
        """Test performance with many columns containing None values."""
        data = {}
        for i in range(50):
            data[f"col{i}"] = [None, "test", ""] * 20
        df = pd.DataFrame(data)
        column_names = [f"col{i}" for i in range(50)]
        words, characters = calculate_text_metrics(df, column_names)

    def test_large_dataset_with_punctuation_variety(self):
        """Test with a large dataset containing various punctuation."""
        texts = [
            "Hello, world! How are you?",
            "This is a test. It contains... multiple punctuation!!!",
            "Contact us at email@example.com or call 555-1234.",
        ]
        df = pd.DataFrame({
            "text": texts * 100
        })
        words, characters = calculate_text_metrics(df, ["text"])
        # Count words and characters in original texts
        total_text = "".join(texts)
        expected_words = len(total_text.split())
        expected_characters = len(total_text)

    def test_large_dataframe_subset_of_columns(self):
        """Test that only specified columns are analyzed in large dataset."""
        data = {}
        for i in range(60):
            if i % 2 == 0:
                data[f"col{i}"] = ["hello world"] * 10
            else:
                data[f"col{i}"] = ["should be ignored"] * 10
        df = pd.DataFrame(data)
        # Only analyze even-numbered columns
        column_names = [f"col{i}" for i in range(0, 60, 2)]
        words, characters = calculate_text_metrics(df, column_names)

    def test_stress_test_deeply_nested_operations(self):
        """Stress test with multiple large operations."""
        df = pd.DataFrame({
            "col1": ["word1 word2 word3"] * 200,
            "col2": ["test"] * 200,
            "col3": [""] * 200,
            "col4": [None] * 200,
        })
        words, characters = calculate_text_metrics(df, ["col1", "col2", "col3", "col4"])


class TestCalculateTextMetricsReturnTypes:
    """Tests to verify correct return types and tuple structure."""

    def test_return_type_is_tuple(self):
        """Test that the function returns a tuple."""
        df = pd.DataFrame({"text": ["hello world"]})
        codeflash_output = calculate_text_metrics(df, ["text"]); result = codeflash_output

    def test_return_values_are_integers(self):
        """Test that returned values are integers."""
        df = pd.DataFrame({"text": ["hello world"]})
        words, characters = calculate_text_metrics(df, ["text"])

    def test_return_values_are_non_negative(self):
        """Test that returned values are non-negative integers."""
        df = pd.DataFrame({"text": ["hello", "", None]})
        words, characters = calculate_text_metrics(df, ["text"])

    def test_word_count_less_than_or_equal_character_count(self):
        """Test logical relationship: word count is typically <= character count."""
        df = pd.DataFrame({"text": ["hello world test"]})
        words, characters = calculate_text_metrics(df, ["text"])

    def test_exact_word_and_character_count_consistency(self):
        """Test that word and character counts are consistent across calls."""
        df = pd.DataFrame({"text": ["the quick brown fox"]})
        words1, characters1 = calculate_text_metrics(df, ["text"])
        words2, characters2 = calculate_text_metrics(df, ["text"])
# codeflash_output is used to check that the output of the original code is the same as that of the optimized code.

To edit these changes git checkout codeflash/optimize-pr11114-2026-01-19T14.59.04 and push.

[coderabbitai]

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}