Improve handling of large commits

[hoxbro]

WIP

Note: Running SQLiteDB with :memory: does not have as big of a penalty as when saving to disk. I think the reason I'm seeing such a slowdown is because of my AMD CPU, as reported here. When running on another computer with another CPU, I get the same result running on the main branch.

---

This branch: Commit (size=5,000): 3.187 seconds
Main branch: Commit (size=5,000): 50.559 seconds

from contextlib import contextmanager
from time import perf_counter

import numpy as np
import pandas as pd

from holonote.annotate import Annotator, SQLiteDB


@contextmanager
def catchtime(message):
    start = perf_counter()
    yield
    print(f"{message}: {perf_counter() - start:.3f} seconds")


annotator = Annotator(
    {"TIME": int}, fields=["description"], connector=SQLiteDB(filename="test.db")
)


N = 5000
rng = np.random.default_rng(1337)
start_time = np.arange(0, N * 2, 2)
end_time = start_time + 1
data = pd.DataFrame(
    {
        "start_time": start_time,
        "end_time": end_time,
        "description": rng.choice(["A", "B"], N),
    }
)

annotator.define_annotations(data, TIME=("start_time", "end_time"))

with catchtime(f"Commit (size={N:,})"):
    annotator.commit()

[codspeed-hq]

CodSpeed Performance Report

Merging #77 will not alter performance

_{Comparing large_commits (33119b7) with main (fed5217)}

Summary

✅ 6 untouched benchmarks