Sorting unknowns

Associated directory: 11-sorting-unknowns

Sorting transactions that end up in "expenses:unknown"

If you are like me, you probably go through the periods where you don't update your financial records for several months, and then when you do, you suddenly find yourself face to face with dozens of transactions that your CSV rules failed to classify and they ended up in expenses:unknown.

When it is just a handful of those, you can edit rules.psv by hand, but once you have accumulated a significant number of them, sorting could become a tedious and time-consuming task.

(If you don't know what rules.psv is, check out the section about CSV rules).

Fuzzy finding to the rescue!

Since hledger outputs plain text, we could feed its output into a fuzzy-find program like (I happen to like skim) and use it to narrow the selection and choose bits of output.

Here is the plan:

1. We will use hledger register -O csv to output the list of all transactions in expenses:unknown and feed it into skim. Now we can type-and-find a single transaction that we want to handle (or we could just press Enter to choose the first one)

2. CSV rules will probably contain text from the description of the transaction. But what text, and how much? We could use skim combined with rg (or any other grep-alike tool) to interactively see which CSV lines from our existing import files will be matched by the text from the transaction description, and see what happens if we decide to edit it. This would allow you to create future-proof rules that have a chance to capture future similar transactions.

3. Now we can use hledger accounts + skim to interactively choose the account this transaction should go to.

4. Finally, we can take the comment column from rules.psv and use skim to either choose one of the existing comments or enter a new one.

5. Now we have enough information to add <regexp>|<account>|<comment> to appropriate rules.psv and move one to the next unclassified transaction

I wrote a simple and rough script resolve.sh that does exactly this. Here is how it looks like, applied to our toy project where we have exactly two unclassified transactions:

Session shown above will create new rules in rules.psv and after you run export.sh we can check that they work:

$ tail -n5 import/lloyds/rules.psv

HSBC|liabilities:mortgage|
WIKIMEDIA|expenses:donations|
HLEDGER|expenses:donations|
COSTA COFFEE|expenses:coffee|Regular place was closed
TESCO GROCERIES|expenses:groceries|

$ hledger -f all.journal print desc:COSTA

2017-05-03 (BP) COSTA COFFEE  ; Regular place was closed
    assets:Lloyds:current          £-2.43 = £3220.05
    expenses:coffee

See the changes in 11-sorting-unknowns or diffs/10-to-11.diff.

Next steps

Let's talk about how to maintain file-specific CSV import rules in the next chapter.