Progression

[wolearyc]

In light of the new v3 algorithm, I've been considering ways to gauge progress beyond the lemma/inflection counts available in Anki's main toolbar. Obviously we cannot hope to measure true language acquisition, but what we can do is give the user an idea of their progress in terms of word frequencies. I'd find this feature quite useful, to gauge both my progress and the effectiveness of my sentence mining. I've been brainstorming a few options.

1. Add a new histogram on the stats page plotting percentage of known morphs vs. morph frequency. There seems to be very few addons currently that modify the stats page, but here's one example. Implementing this is beyond me, but it would be beautiful!
2. Same as 1., but use the old stats page. There are more addons available on which to build on, but I worry about introducing technical debt to the project.
3. Add a new dialog box (similar to Generators) presenting frequency progress in a table. This is the easiest option, and comes with the bonus of allowing the user to easily customize the report (for example tracking lemmas vs. inflections, changing the binning, etc.)

I am up for trying to implement option 3, which I think is the simplest and most robust option. I'd hope to have it ready in time for v3.

Let me know what you think!

[mortii]

I'd love more stats!

Same as 1., but use the old stats page. There are more addons available on which to build on, but I worry about introducing technical debt to the project.

I attempted to re-implement the MorphMan stats page in the pre-alpha, and I managed to make this:

but it completely broke after migrating to sqlite, and successfully porting it would probably take me hundreds of hours and had issues:

1. It heavily relied on deprecated api calls, so it could get nixed at any time, potentially making it a complete waste of time.
2. the stats were basically one-to-one with the native anki stats, so it wasn't super useful.

Add a new histogram on the stats page plotting percentage of known morphs vs. morph frequency. There seems to be very few addons currently that modify the stats page, but here's one example. Implementing this is beyond me, but it would be beautiful!

Because of the issues mentioned above, I tried to extend the new stats page instead, but I failed miserably (#64). I'm surprised anyone has managed to do it, but despite knowing that, my last attempt was so discouraging that I won't be trying again any time soon.

Add a new dialog box (similar to Generators) presenting frequency progress in a table. This is the easiest option, and comes with the bonus of allowing the user to easily customize the report (for example tracking lemmas vs. inflections, changing the binning, etc.)

That sounds perfect! I had a similar idea in mind in #242.

I am up for trying to implement option 3, which I think is the simplest and most robust option. I'd hope to have it ready in time for v3.

That would be awesome! No need to rush though, take as much time as you need :)

[wolearyc]

Sad to hear about your stats page breaking, but admittedly relieved to hear the new stats api was giving you trouble as well... The new system seems so complicated, even without considering the search functionality!

[fuquasteve]

So this would allow one to know how far they have moved in terms of frequencies? Compared to whatever frequency a word has in a certain deck? I did not ever use the morphman stats page before.
If you are pulling new cards from several sources, it would be nice to see how far you are along the path of comprehending the source.
If I am barking up the wrong tree, and have completely missed the point of this, please just ignore this message.

[wolearyc]

Yup, you have the right idea. Right now, you can use readability reports to estimate your comprehension of a specific source, be that an .srt, .txt, etc, but as far as I know this just gives a percentage. It would be cool to provide some additional information based on frequency, sentences, etc. My specific proposal is to give progress stats with respect to morph priority, however that has been defined in settings. In my workflow, I use a frequency file to define morph priority, but deck frequency can also be used.

[wolearyc]

So I've got a working prototype which gives stats on which morphs in a priority range are known, learning, unknown, or missing from the collection.

I'd also like to list out all lemmas/inflections in the frequency file and mark them known, learning, etc. Here, I run into a slight problem. In inflection mode, the way _get_morph_priority dictionaries are keyed makes it impossible to reliably separate the lemma and inflection. (In lemma mode, the lemma can be determined by key[:len(key)//2])

@mortii , is there a good reason why the keys aren't delimited? If not, I'd be happy to make the change, maybe to something like f"{lemma}|{inflection}" (or f"{lemma}|{lemma}" in lemma mode).

[mortii]

Wow, that looks sick!

is there a good reason why the keys aren't delimited?

Depending on the quality of the morphemizer and the source text, the morph extraction can fail pretty brutally. I've seen some morphs like this: hello! }, so no characters are completely safe to use as delimiters in a string.

That being said, do you need to use morph priorities this for?

[mortii]

That being said, do you need to use morph priorities this for?

Nvm, of course morph priorities are necessary🤦

I'm busy the rest of the day, but I'll look more into it tomorrow!

[wolearyc]

No rush :) Another possibility is to use (lemma,inflection) tuples at the dictionary keys.

[mortii]

Another possibility is to use (lemma,inflection) tuples at the dictionary keys

That's very clever 👍

just pushed that change: d38093c

[mortii]

@wolearyc

is this okay?

[wolearyc]

Beautiful, it looks way more balanced!Also I’ve rethought “statistics mode” after realizing that our statistics are always discrete 😅 Instead of “statistics mode”, how about: Bin type: []normal []cumulativeAm 7/27/24 um 6:49 AM schrieb mortii ***@***.***>: @wolearyc is this okay? Screenshot.from.2024-07-27.15-33-42.png (view on web) —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[mortii]

like this?

[wolearyc]

Perfect~

[mortii]

Included in the v3 update: https://github.com/mortii/anki-morphs/releases/tag/v3.0.0

Thanks a bunch, this is super cool!