Proposal for Snapshot Testing in ERPNext

[blaggacao]

Introduction to Snapshot Testing

Snapshot testing is a type of regression testing that captures the current state of a system and compares it to a previously recorded state to identify changes. It is particularly useful for tracking changes in UI components, JSON data, or any other output format. This approach can simplify test writing, enhance test coverage, and automate manual processes.

Proposed Implementation

1. Snapshot Capture and Storage:

   • Initiate snapshot mode in the UI during development.
   • Perform transactions and capture snapshots of records before and after changes.
   • Store snapshots in test_snapshots.toml files associated with the relevant doctype.

2. Test Stub Creation:

   • Automatically generate a stub class, e.g., SnapshotTestMyDoctype, referencing snapshots as instance attributes.

3. Transaction Recreation:

   • Developers specify transactions (e.g., create, validate) in tests using the captured snapshots.

Example Interaction

class SnapshotMyDoc(SnapshotTestCase):
    def __init__(self):
        self.snapshots = load([f for f in *.toml])

    def test_create_invoice_in_usd(self):
        so = self.snapshots[0]
        si = so.create_ainv()
        self.assertEqual(si, self.snapshots[1])

Challenges and Considerations

• Dependency Management: Ensure all dependencies are captured during snapshot creation to avoid incomplete records.
• File Pollution: This is normal for snapshot tests; of practical importance is a readable diff; maybe use semantic differ.
• Rollback Mechanism: Implement reliable transaction rollback to maintain test isolation and integrity; be immune to db pollution from diverse snapshots (e.g. all recreating thier own Company record).
• Update Mechanism: Implement an update mechanism so that alongside an intentional change, a developer can review the resulting diff in snapshot target records instead of having to manually update each record.

This proposal aims to streamline functional test creation in ERPNext by leveraging the benefits of snapshot testing.

Citations:
[1] Snapshot Testing: Example and Its Benefits - ORIL https://oril.co/blog/snapshot-testing-example-and-its-benefits/
[2] Snapshot Testing in Action: Helpful or Trouble? - Snoopy Developer https://snoopydeveloper.com/snapshot-testing-in-action-helpful-or-trouble
[3] Pros and cons of Jest snapshot testing | TSH.io - The Software House https://tsh.io/blog/pros-and-cons-of-jest-snapshot-tests/
[4] Snapshot Testing: Benefits and Drawbacks - SitePen https://www.sitepen.com/blog/snapshot-testing-benefits-and-drawbacks
[5] When should I use Snapshot testing? - Stack Overflow https://stackoverflow.com/questions/43771602/when-should-i-use-snapshot-testing
[6] Snapshot testing; what are the benefits? : r/reactjs - Reddit https://www.reddit.com/r/reactjs/comments/auyi54/snapshot_testing_what_are_the_benefits/

[uc77]

Great initiative. We are with you on this.