Handles single data items correctly

Ensures the mask_data method correctly processes single data items,
as well as lists of data items, and returns the appropriate type.

Previously, the code only worked correctly for lists.

Bump pytest-cov from 6.2.1 to 6.3.0 (#1)


Updates supported version in security policy

Reflects that version 1.0.x is currently supported
with security updates.

Bump mypy from 1.17.1 to 1.18.1 (#4)


Bump pytest-cov from 6.3.0 to 7.0.0 (#3)


Updates pylint target for doubletake

Updates the pylint target in the lint-code command to analyze the 'doubletake' project instead of 'chilo_api'.
This ensures that the linting process focuses on the relevant codebase for the doubletake project.

Adds MetaMatch to pass matching context

Introduces a MetaMatch dataclass to encapsulate pattern matching
context, including the matched pattern, value, replacement, and
breadcrumbs.

This allows custom callback functions to access more comprehensive
information about the match during replacement, enabling more flexible
and context-aware data masking logic.  It replaces passing individual
pattern key/value pairs, and is provided to the user callback.

Refactors extras setting to use a dictionary

Updates the "extras" setting to use a dictionary for custom regex patterns, allowing for named patterns.

This change enhances configurability and readability by allowing users to define custom regex patterns with associated names, making it easier to manage and understand the purpose of each pattern. It also improves validation and type safety.

Adds faker-based address masking with extras

Implements masking for address data using Faker, enhancing data
obfuscation with dynamically generated realistic address values.

Also adds support for idempotent address masking, ensuring consistent
replacement for duplicate address values.

Creates setup.py for package installation

Initializes the setup.py file with metadata and dependencies.

Defines the package name, version, author, description,
required Python version, install dependencies, keywords,
project URLs, classifiers, license, and supported platforms.

This allows the package to be installed using setuptools.

Author: paulcruse3

.circleci/config.yml

@@ -33,7 +33,7 @@ commands:
     lint-code:
         steps:
             - run: pipenv run lint
-            - run: pipenv run pylint chilo_api --output-format=json > ./coverage/lint/pylint_report.json || exit 0
+            - run: pipenv run pylint doubletake --output-format=json > ./coverage/lint/pylint_report.json || exit 0
             - run: pipenv run pylint-json2html ./coverage/lint/pylint_report.json -o ./coverage/lint/pylint_report.html || exit 0
             - store_artifacts:
                 path: ./coverage/lint

Pipfile.lock

@@ -108,15 +108,27 @@ masked_data = db.mask_data(data)
 ### Custom Replacement Logic
 
 ```python
-def custom_replacer(pattern_key: str, pattern_value: str, possible_replacement: str, value: Any):
+# The callback receives:
+#   meta_match: MetaMatch (fields: pattern, value, replacement, breadcrumbs)
+#   faker:      Faker instance (for generating fake data)
+#   value:      The matched value being replaced
+
+def custom_replacer(meta_match, faker, value):
     """Custom replacement with full context"""
-    if pattern_key == 'email':
+    # meta_match.pattern: the pattern key (e.g. 'email', 'ssn', etc.)
+    # meta_match.value:   the regex pattern or extra pattern string
+    # meta_match.replacement: the default replacement value
+    # meta_match.breadcrumbs: set of path keys (for path-aware logic)
+    if meta_match.pattern == 'email':
         return "***REDACTED_EMAIL***"
-    if pattern_key == 'ssn':
+    if meta_match.pattern == 'ssn':
         return "XXX-XX-XXXX"
+    if meta_match.pattern == 'city':
+        # Use Faker to generate a fake city name
+        return faker.city()
     if 'secret' in value:
         return "***CLASSIFIED***"
-    return replacement
+    return meta_match.replacement
 
 db = DoubleTake(callback=custom_replacer)
 ```
@@ -127,7 +139,10 @@ db = DoubleTake(callback=custom_replacer)
 # Only replace certain types, allow others through
 db = DoubleTake(
     allowed=['email'],  # Don't replace emails
-    extras=[r'CUST-\d+', r'REF-[A-Z]{3}-\d{4}']  # Custom patterns
+    extras={
+        'customer_id': r'CUST-\d+',
+        'reference': r'REF-[A-Z]{3}-\d{4}'
+    }  # Custom patterns as a dict
 )
 ```
 
@@ -279,7 +294,7 @@ db = DoubleTake(
     use_faker=False,           # Use fake data vs asterisks
     callback=None,             # Custom replacement function
     allowed=[],                # Pattern types to skip
-    extras=[],                 # Additional regex patterns  
+    extras={},                 # Additional regex patterns as a dict
     safe_values=[],            # Values to protect from replacement
     idempotent=False,          # Prevent double-masking operations
     known_paths=[],            # Specific paths to target
@@ -359,7 +374,7 @@ logs = [
 
 db = DoubleTake(
     safe_values=['[email protected]'],  # Keep official support email visible
-    extras=[r'\+1-555-SUPPORT']          # Keep support phone pattern
+    extras={'phone': r'\+1-555-SUPPORT'}  # Keep support phone pattern
 )
 
 sanitized_logs = db.mask_data(logs)

README.md

@@ -7,7 +7,7 @@ currently being supported with security updates.
 
 | Version | Supported          |
 | ------- | ------------------ |
-| 2.0.x   | :white_check_mark: |
+| 1.0.x   | :white_check_mark: |
 
 ## Reporting a Vulnerability
 

SECURITY.md

@@ -106,11 +106,12 @@ def mask_data(self, data: list[Any]) -> list[Any]:
             >>> result = db.mask_data(data)
             >>> # Emails and phone numbers will be replaced, names and IDs preserved
         """
+        process_data = data if isinstance(data, list) else [data]
         return_data: list[Any] = []
-        for item in data:
+        for item in process_data:
             masked_item = self.__process_data_item(item)
             return_data.append(masked_item)
-        return return_data
+        return return_data if isinstance(data, list) else return_data[0]
 
     def __process_data_item(self, item: Any) -> Any:
         if not self.__use_faker and self.__callback is None:

doubletake/__init__.py

@@ -3,6 +3,7 @@
 
 from doubletake.utils.pattern_manager import PatternManager
 from doubletake.types.settings import Settings
+from doubletake.utils.meta_match import MetaMatch
 
 
 class DataWalker:
@@ -64,18 +65,19 @@ class DataWalker:
     """
 
     def __init__(self, **kwargs: Unpack[Settings]) -> None:
-        self.__breadcrumbs: set[str] = set()
         self.__known_paths: list[str] = kwargs.get('known_paths', [])
+        self.__meta_match: MetaMatch = MetaMatch()
+        kwargs['meta_match'] = self.__meta_match
         self.__pattern_manager: PatternManager = PatternManager(**kwargs)
 
     def walk_and_replace(self, item: dict[str, Any]) -> dict[str, Any]:
-        self.__breadcrumbs = set()
+        self.__meta_match.breadcrumbs = set()
         self.__walk_dict(item, None)
         return item
 
     def __walk_dict(self, item: dict[str, Any], current_key: Optional[str]) -> None:
         if current_key is not None:
-            self.__breadcrumbs.add(current_key)
+            self.__meta_match.breadcrumbs.add(current_key)
         for key in item.keys():
             self.__determine_next_step(item, key)
 
@@ -96,6 +98,6 @@ def __replace_known_paths(self, item: Any) -> None:
         for known_pattern in self.__known_paths:
             known_list = known_pattern.split('.')
             key = known_list.pop()
-            if known_list == list(self.__breadcrumbs):
+            if known_list == list(self.__meta_match.breadcrumbs):
                 if isinstance(item, dict) and key in item:
                     item[key] = self.__pattern_manager.replace_value(item[key], item[key], key, None)

doubletake/searcher/data_walker.py

@@ -5,6 +5,7 @@
 
 from doubletake.utils.pattern_manager import PatternManager
 from doubletake.types.settings import Settings
+from doubletake.utils.meta_match import MetaMatch
 
 
 class JSONGrepper:
@@ -78,6 +79,7 @@ class JSONGrepper:
     """
 
     def __init__(self, **kwargs: Unpack[Settings]) -> None:
+        kwargs['meta_match'] = MetaMatch()
         self.__pattern_manager: PatternManager = PatternManager(**kwargs)
 
     def grep_and_replace(self, item: Any) -> Any:

doubletake/searcher/json_grepper.py

@@ -3,6 +3,7 @@
 
 from doubletake.utils.pattern_manager import PatternManager
 from doubletake.types.settings import Settings
+from doubletake.utils.meta_match import MetaMatch
 
 
 class StringReplacer:
@@ -71,6 +72,7 @@ class StringReplacer:
     """
 
     def __init__(self, **kwargs: Unpack[Settings]) -> None:
+        kwargs['meta_match'] = MetaMatch()
         self.__pattern_manager: PatternManager = PatternManager(**kwargs)
 
     def scan_and_replace(self, item: str) -> Union[str, None]:

doubletake/searcher/string_replacer.py

@@ -1,13 +1,16 @@
 from typing_extensions import TypedDict, NotRequired, Callable
 
+from doubletake.utils.meta_match import MetaMatch
+
 
 class Settings(TypedDict, total=False):
     allowed: NotRequired[list[str]]
     callback: NotRequired[Callable]
-    extras: NotRequired[list[str]]
+    extras: NotRequired[dict[str, str]]
     idempotent: NotRequired[bool]
     known_paths: NotRequired[list[str]]
     maintain_length: NotRequired[bool]
     replace_with: NotRequired[str]
     safe_values: NotRequired[list[str]]
     use_faker: NotRequired[bool]
+    meta_match: NotRequired[MetaMatch]

doubletake/types/settings.py

@@ -101,17 +101,15 @@ def _validate_extras(config: Settings) -> None:
         extras = config.get('extras')
         if extras is None:
             return
-
-        if not isinstance(extras, list):
-            raise ValueError('The "extras" key must be a list of regexstrings if provided.')
-
-        for item in extras:
-            if not isinstance(item, str):
-                raise ValueError('The "extras" key must be a list of regexstrings if provided.')
+        if not isinstance(extras, dict):
+            raise ValueError('The "extras" key must be a dict of {str: regex string} if provided.')
+        for key, value in extras.items():
+            if not isinstance(key, str) or not isinstance(value, str):
+                raise ValueError('Each key and value in "extras" must be a string (key: name, value: regex pattern).')
             try:
-                re.compile(item)
+                re.compile(value)
             except re.error as error:
-                raise ValueError('The "extras" key must be a list of regexstrings if provided.') from error
+                raise ValueError(f'Invalid regex pattern in "extras": {value}') from error
 
     @staticmethod
     def _validate_safe_values(config: Settings) -> None: