Study Plan feature

[MichaelPetre]

Hi Mortii,

ankiMorphs looks very promising but it seems to be missing the one feature I used the most in MorphMan: the study plan generator. Is this something that is planned in the future?

[mortii]

Hi, @MichaelPetre!

If I recall correctly, the study plan feature is just a very complicated way to superimpose a secondary frequency list on top of a master frequency list. In AnkiMorphs you can easily switch between frequency files in the settings, so I think the study plan feature is redundant.

Let me know if that didn't make sense, or if I misunderstood something.

I could maybe also add it to the FAQ section in the guide since multiple people have asked about this now.

[MichaelPetre]

It reorders your cards to reach a target comprehension rate for specified media. For example, I feed it a light novel series with a total of 266 chapters, set the desired comprehension rate to 95% and cards get shuffled around for the shortest path to reach that target for each successive chapter.

If I understand what you're saying, I could simply generate a frequency list based on the 266 chapters of the light novel series and switch AnkiMorphs to that list to achieve the same result.

[Vilhelm-Ian]

@MichaelPetre I am going to try tomorrow to add this feature

[mortii]

I feed it a light novel series with a total of 266 chapters, set the desired comprehension rate to 95% and cards get shuffled around for the shortest path to reach that target for each successive chapter.

@MichaelPetre Ah, I misunderstand the feature then, I didn't realize it worked consecutively, that is pretty interesting. That being said, I have no idea how MorphMan could make that work, so I'm somewhat skeptical about how strictly those things were achieved/worked as advertised. I would have to dig through the MorphMan code to say for sure.

If you were to recreate that in the current AnkiMorphs version you would have to create 266 separate frequency files, and when one is exhausted, switch to the next one. That would be pretty laborious.

@MichaelPetre I am going to try tomorrow to add this feature

@Vilhelm-Ian Do you want to elaborate on this? I don't see a super obvious way of implementing successive frequency files unless we introduce a completely new mechanic/files.

[mortii]

Btw, the order of the chapters would have an crucial impact on the 'study plan', which adds an exponential amount of complexity to this, which I don't love.

[mortii]

After a little more thought, I have concluded that this part makes implementing a successive 'study plan' unfeasible:

Btw, the order of the chapters would have an crucial impact on the 'study plan', which adds an exponential amount of complexity to this, which I don't love.

Regardless of how you do it, there is no good way set the order of the files without a significant amount of manual work, unless you are lucky and someone else has already numbered/named them in a convenient way.

So the two alternatives are:

1. Create a global frequency files of all the chapters (best for general learning)
2. Switch between the individual frequency files of the chapters

@Vilhelm-Ian and I discussed in #114 the possibility of adding a download button next to each file in the 'frequncy file generator' to download the individual frequency files when generating a global frequency file, and I still think that is a good idea/feature.

I'll add this to the FAQ.

[MichaelPetre]

I have so far remained on Anki 2.1.65 specifically because it puts me in a sweet spot: I can run fsrs, mm-5-alpha and the migaku add-ons at the same time.

So.... thanks to this discussion, I went under the hood to check what really happens with the Study Plan feature...

If you supply a master frequency list, it uses it to filter low frequency morphs from the study plan otherwise it just makes a straight up frequency list of the morphs required to reach the target. It saves the study plan in the morphman dbs folder along with the frequency list (study plan minus the metadata).

The score of a given morph is manipulated by setting it to (morph_instances_in_source^2)*60 + the instance count of the morph in the master frequency list. So morphs that appear many times in the source will get a huge score, and a morph that also has a high frequency in the master frequency list will score higher than a morph with a lower frequency in the master frequency list.

The cards will only get reordered when you hit recalc.

If the study plan was generated from a collection of chapters or subs, both the study plan and the frequency list will contain the chapters/subs in order. So Chapter 001 with its 31 morphs, then Chapter 002 with its 39 morphs, then Chapter 003 with its 67 morphs.

https://web.archive.org/web/20201220134610/https://massimmersionapproach.com/table-of-contents/anki/morphman/
https://sites.google.com/view/jo-mako/readability-list/studying#h.m92ptkxkuou

[mortii]

@MichaelPetre that is an heroic effort, thanks a lot!

Correct me if I'm wrong, but from what I gather, what happens is basically this:

1. A study plan report is created, which is displayed in the readability analyzer and is also saved to a file in the db folder
2. A frequency list is generated (and replaces the previous one I'm assuming)

If that is accurate then I hit the nail on the head with my previous comment:

I have no idea how MorphMan could make that work, so I'm somewhat skeptical about how strictly those things were achieved/worked as advertised.

One plain frequency list provides the weakest form of 'successive' chapter priorities imaginable--it is basically a loose guide.

Again, with the assumption that those two things are accurate, then this feature falls completely into no-man's-land. It is hyper specific in that it needs the following criteria fulfilled:

1. The files have to be sorted in an arbitrary way (implementing something that would switch sorting criteria would be a nightmare)
2. You don't care about strict succession, only that it is somewhat better ordered than a normal general frequency list.

Features like this is the problem with morphman imo; overly-fitted features that results in enormous technical debt, and confusing/overwhelming the users. That is a recipe for a software death spiral. I don't want to reproduce that, sorry.

[MichaelPetre]

Yup I fully understand the pain of tech debt :)

As you said earlier, I can probably get a close enough functionality by doing a frequency list for each chapter and switching to it.

[aleksejrs]

You don't care about strict succession, only that it is somewhat better ordered than a normal general frequency list.

I would say it's a concatenation of individual frequency lists, but ordered somewhat better using the general frequency list.
If you don't like MorphMan's use of a specific formula, you could just order the individual lists using frequencies from the general list, which is very simple if you do two passes.

As you said earlier, I can probably get a close enough functionality by doing a frequency list for each chapter and switching to it.

That leaves the morphs from the other files out (if you have multiple decks, then different cards are next), or you have to concatenate the lists manually.

[mortii]

I would say it's a concatenation of individual frequency lists, but ordered somewhat better using the general frequency list.

Yeah, that is a succinct description.

That leaves the morphs from the other files out (if you have multiple decks, then different cards are next), or you have to concatenate the lists manually.

A study plan feature would certainly be advantageous in some scenarios, but not enough to be worth implementing, imo.

[xofm31]

I too use the study plan in Morphman all the time, and it is the feature that will keep me with Morphman until I'm convinced that there's a viable alternative. What I like about the study plan is that it gives me the words that appear in the first file that appear the most in the collection of files. So if I start with a new TV show, and there are 25 episodes, when I'm doing episode 1, it will give me the words that appear in episode 1 that will help me the most for the whole TV series. (Note: I definitely agree with avoiding unnecessary technical debt, and sometimes I'm not sure if Morphman is doing what it advertises. So I'm hoping that I can find a way to use AnkiMorph that will work just as well for me.)

I wanted to do a comparison of the card order that the Morphman study plan gives, vs Ankimorph, with Ankimorph using a frequency file for just that first episode. Based on the difficulty equation, I was expecting that all of the sentences with a given new morph would show up together in the queue in consecutive order, but they didn't. They also didn't show up in the order that they are listed in the frequency file that I generated. I verified that "Morph Priority" file in AnkiMorph Settings is pointing to the right file. Is this correct?

[mortii]

@xofm31 yeah, having a unified algorithm that works for everyone probably isn't doable. Being able to switch between discrete algorithms might be preferable.

[MichaelPetre]

Since Morphman is (mostly) working for me, I'll probably just continue with it for ordering my cards for now, and hope that by the time it stops working I don't need it anymore. Or, since your code is so much more understandable, I may edit my own copy of your Addon at some point :)

Morphman doesn't work at all after 2.1.66 because the Anki team has fully removed the v1/v2 schedulers support and the v3 scheduler is sufficiently different to break Morphman.

Morphman relies on the scheduler having the _fillNew, _getNew and _answerCard private methods (non-exhaustive list) to interact with the cards, and those do not exist in v3. Morphman's newMorphHelper could possibly be rewritten to use the v3 scheduler... but nobody is volunteering to do it. The last comment on the subject of _fillNew going away suggest waiting for another project (that was abandoned 4 years ago) to solve the issue and copy their fix.

It's also possible that dependencies are broken in other parts... those are just the parts I noticed last night when I tried poking around Morphman's code to get it working with Anki 23.x.

[mortii]

Morphman's newMorphHelper could possibly be rewritten to use the v3 scheduler... but nobody is volunteering to do it.

@MichaelPetre this is off-topic, but the problem is not the lack of volunteers, it's the lack of merging. The repo has been fully abandoned by the maintainers with merging privilages, so if you are planning on creating any pull-requests I would recommend not getting your hopes up.

With that out of the way, let's not discuss the current state of MorphMan anymore 🙏

[MichaelPetre]

Yeah I didn't word it in the best way :) I noticed you were very active on their repo until the last merge/release. I suspected you didn't fork just for the sake of it ;)

[xofm31]

Morphman doesn't work at all after 2.1.66 because the Anki team has fully removed the v1/v2 schedulers support and the v3 scheduler is sufficiently different to break Morphman.

@MichaelPetre I am new to Morphman (in the last year), and never saw a manual on everything it was supposed to do. I din't know that it ever interacted with the scheduler. When I say it "mostly works", what I mean is that I can get it to create reasonable readability reports and study plans, (mostly) find the unknown morphs for the cards, (mostly) put the cards in a reasonable order, and it (mostly) skips the other cards with the same morph once I've studied one card with it. I think all of these things are either in Ankimorphs right now, or considered for the future. If Morphman did complicated things like re-ordering cards to get 2T cards "unstuck", I never saw that.

[mortii]

It occurred to me that in the Readability Report Generator kinda already takes care of my first concern:

1. The files have to be sorted in an arbitrary way (implementing something that would switch sorting criteria would be a nightmare)

In the RRG you can sort by column value:

After that you could just have a 'generate study plan' button to concatenate their frequency files in the order displayed.

If we were to create something like that then we might as well completely combine the Frequency File Generator and the Readability Report generator. I was hoping that separating them would cause less confusion, but with all these features being so intertwined it might ironically end up being the other way around. It would also help make some of the proposed RRG features in #114 be more organically placed.

As for my second concern:

2. You don't care about strict succession, only that it is somewhat better ordered than a normal general frequency list.

It is what it is. As long as it is clearly specified in the docs, then people can decide for themselves if it works for them or not. Alternatively we could maybe add an option in the 'recalc' settings to switch between two distinct algorithms, the current one and an algorithm like @xofm31 described that follows the frequency file strictly (not sure how possible it is to make that kind of algorithm).

[Vilhelm-Ian]

If we were to create something like that then we might as well completely combine the Frequency File Generator and the Readability Report generator.

No. The issue with that is naming. I think even if both opened 90% the same context menu. It's better for them to be seperated. Since there dosen't exist a noun to define both functionality.

We can call it Redabliity Report & Frequncy List Generator but it's a mouth full

[mortii]

@Vilhelm-Ian The naming would be tricky. I liked your suggestion last time of just having a completely abstract name, maybe something like 'AnkiMorphs Tools'.

[mortii]

When I have the time I'll make a quick skeleton UI of how I imagine the combined version would look,

[mortii]

This looks pretty disgusting lmao, but anyway, I was thinking the 'generate' buttons could be grayed out until the user clicks load files, which then populates the 'file' column in the table at the bottom, and then the 'generate' buttons could be activated.

[Vilhelm-Ian]

I might be biased since I've gotten used to the UI. But I think it's okay and far from ugly

[xofm31]

I love this! I hope that there will be an option to write the readability report and the study plan to a file. I think maybe you're already planning this in another issue, but I think that the readability report needs to have both the full count of all of the morphs (what morphman called "instances" - I think this is what you have right now), and the the count of the unique morphs (what morphman called "morphs"). The unique morphs tells you how many words you need to learn. I'm using it for Chinese, so I haven't thought about how you handle different forms of the same word.

[mortii]

I hope that there will be an option to write the readability report and the study plan to a file

@xofm31 I don't really see the appeal of saving the readability report to a file since it's such a tight snapshot of your current knowledge. That being said, we could maybe create a separate folder for it in the anki profile folder, named 'readabililty-reports' or something, basically the equivalent of the 'frequency-files' folder.

As for the more detailed reports, yes, it's definitely something we should add.

[aleksejrs]

I don't really see the appeal of saving the readability report to a file since it's such a tight snapshot of your current knowledge.

@mortii So are PDFs of statistics. Data in a simpler format could be used to track your progress.

[mortii]

@aleksejrs that's fair, people could be more interested in tracking their progress in that way than I am.

[xofm31]

I see that you have made an issue for adding the study plan. I know that @MichaelPetre gave a description of the study plan already, but I'll try to recap here what I think are the main features.

1. You tell it what comprehension percentage you want (80%, 90%, etc). This is the number of tokens that you want to be able to understand.
2. You have an ordered list of files (with Morphman, it is based on alphabetical order, so having them nameed "Chapter001", "Chapter002", etc. worked well.
3. You can also tell it that a morph has to occur a minimum number of times. I never used this, but I guess this would just mean that if it was all words that only are in your file once, it won't put those in the list, even if you won't make it to the desired percentage.
4. Morphman then calculates the "best morph order" for you to get to your desired comprehension percentage. The "best order" is a combination of the number of times the morph is in your whole collection and the number of times it is in the given file. I didn't look at the code, but @MichaelPetre says "(morph_instances_in_source^2)*60 + the instance count of the morph in the master frequency list."
5. Two files are created, the "study plan" and the "frequency file". The study plan consists of a list of the number of words you need to learn from each file, and the list of those words. The frequency file has the list of all of the words in the study plan order, and it doesn't include the morphs that are already known.

For Ankimorphs, if you want to keep the same calculations for card difficulty scoring, you'll need to keep all the known words. I think if you put all of the known words first, followed by the words for the first file, followed by the words for the second file, that would give a frequency file with the desired properties for both @mortii's prefered scoring as well as what I have in mind. Personally, I like to be able to measure my progress, so I do look at the "study plan" file. But a separate file probably isn't that helpful; I think that just having a comment line in the frequency file showing the name of the file and the number of words you need to learn from that file would cover everything that I look at. And maybe that's not really needed - I should be spending less time evaluating my learning, and more time studying.

[mortii]

Here is an early and rough pre-release version:

• for devs, pull the study-plan branch: https://github.com/mortii/anki-morphs/tree/study-plan
• non-devs: ankimorphs-v2-2-0-pre-release.ankiaddon (google drive, github doesn't like .addon files)

Any feedback in #155 is welcome!

Originally posted by @mortii in #194 (comment)

[mortii]

@MichaelPetre amazing, thanks for the feedback! We should definitely get to the bottom of those discrepancies.

Could you run the same comparisons using only a single file with one sentence and gradually change/increase it until you can narrow down where the differences occur?

Also, some caveats:
Use the same morphemizer for everything (recalc and generators) to make sure the comparison is as close as possible, in this case it means that you have to use the built in japanese morphemizer in morphman, and the "AnkiMorphs: Japanese" morphemizer since they are identical. It should be mentioned that I actually fixed some bugs in that morphemizer, so AM might recognize some morphs that MM doesn't:

anki-morphs/ankimorphs/mecab_wrapper.py

Lines 118 to 120 in 577a34f

	# HACK: mecab sometimes does not produce the right morphs if there are no extra characters in the expression,
	# so we just add a whitespace and a japanese punctuation mark "。" at the end to prevent the problem.
	expression += " 。"

According to AnkiMorphs' readability analyzer, I'm at 78.6% comprehension on the first episode of the first show. It has 1790 morphs, 511 of which are unique with 166 unique unknown and 27 unique learning. The study plan built from the episode (with a target of 95%) has 476 morphs

study_plan_frequency_file.csv
study_plan.txt

This doesn't immediately look completely wrong to me. Comparing the frequency list generated by MM might be better than the study plan file, since it's the frequency list that does the actual work.

[mortii]

Also note that this might come into play if you have previously used a spaCy morphemizer:

However, speaking from experience, switching between morphemizers can lead to some cards being marked as known with one morphemizer, and the known tag sticks even if you switch to another morphemizer, so you can get a weird pattern where previously unknown cards/morphs become known. Removing all am-known-automatically tags from all cards should be safe, and should hopefully fix that kind of problem.

Originally posted by @mortii in #193 (comment)

[MichaelPetre]

The frequency list is actually identical to the study plan except for the metadata line :) I'm attaching the " 9 seasons" frequency list below. I have just reinstalled 2.1.66 to perform the comparisons with a smaller data set.
frequency.txt

[mortii]

The frequency list is actually identical to the study plan except for the metadata line :)

That is insane, just when I thought the morphman implementation couldn't get any worse🤦‍♂️

[MichaelPetre]

This is due to the way I generated the study plan: as I didn't supply a global frequency list, it merged the study plan into a blank frequency list (best guess).

[mortii]

It would be pretty useful to know the learning status of the morphs, so I'll add a learning-status column to the csv files.

[mortii]

Here it is:
ankimorphs-v2-2-1-pre-release.ankiaddon

It may or may not be much slower as a result.

[MichaelPetre]

It's only a tiny bit slower, but still way faster than the Morphman version :)

Having changed a few settings (global: "move new cards that only have known morphs to the end of the due queue", generators: ignore numbers), and having gone through 20 cards... I can understand the difference in count. AnkiMorphs did indeed find morphs that Morphman had missed, and did a far better job at parsing.

Here are the unknown for episode 1 (many are actually known but with kanji instead of kana-only) before today's session:

すいません,すいません,unknown,5,Karakai Jozu no Takagi-san.S01E01.JA.srt
やる,やら,unknown,4,Karakai Jozu no Takagi-san.S01E01.JA.srt
ぬう,ぬう,unknown,3,Karakai Jozu no Takagi-san.S01E01.JA.srt
からかう,からかっ,unknown,3,Karakai Jozu no Takagi-san.S01E01.JA.srt
痛い,痛,unknown,2,Karakai Jozu no Takagi-san.S01E01.JA.srt
上履き,上履き,unknown,2,Karakai Jozu no Takagi-san.S01E01.JA.srt
丸見え,丸見え,unknown,2,Karakai Jozu no Takagi-san.S01E01.JA.srt
だい,だい,unknown,2,Karakai Jozu no Takagi-san.S01E01.JA.srt
悔しい,悔し,unknown,2,Karakai Jozu no Takagi-san.S01E01.JA.srt
ふん,ふん,unknown,2,Karakai Jozu no Takagi-san.S01E01.JA.srt
うえ,うえ,unknown,2,Karakai Jozu no Takagi-san.S01E01.JA.srt
アハハ,アハハ,unknown,2,Karakai Jozu no Takagi-san.S01E01.JA.srt
からかう,からかお,unknown,2,Karakai Jozu no Takagi-san.S01E01.JA.srt
なべ,なべ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
ため息,ため息,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
かぎ,かぎ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
べ,べ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
バネ,バネ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
仕込む,仕込め,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
開く,開か,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
落とす,落とし,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
歪,歪,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
じゃ,じゃっ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
一同,一同,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
あす,あす,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
笑い声,笑い声,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
くい,くく,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
まんまと,まんまと,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
助け,助け,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
気付け,気付け,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
ありがと,ありがと,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
両,両,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
想い,想い,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
なれる,なれる,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
おまじない,おまじない,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
ほほ,ほほ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
つく,つか,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
いいかげん,いいかげん,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
しめる,しめる,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
一文字,一文字,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
てこ,てこ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
なる,なりゃ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
思い通り,思い通り,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
かって,かって,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
がる,がら,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
はずす,はずす,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
拭う,拭い,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
日誌,日誌,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
いく,いけ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
やっぱし,やっぱし,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
三文,三文,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
徳,徳,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
しかける,しかけ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
眠気,眠気,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
隙間,隙間,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
拭く,拭こ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
面倒くさい,面倒くさい,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
おく,おこ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
拭く,拭か,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
たくらむ,たくらみ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
残り,残り,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
ここだ,ここだ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
透ける,透ける,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
ちく,ちく,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
する,しょ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
おもしろい,おもしろかっ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
みな,みな,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
共,共,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
勝てる,勝てる,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
見つめる,見つめ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
決め,決め,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
こうか,こうか,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
ところで,ところで,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
ごまかす,ごまかそ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt

After today's session:

やる,やら,unknown,4,Karakai Jozu no Takagi-san.S01E01.JA.srt
ぬう,ぬう,unknown,3,Karakai Jozu no Takagi-san.S01E01.JA.srt
からかう,からかっ,unknown,3,Karakai Jozu no Takagi-san.S01E01.JA.srt
痛い,痛,unknown,2,Karakai Jozu no Takagi-san.S01E01.JA.srt
上履き,上履き,unknown,2,Karakai Jozu no Takagi-san.S01E01.JA.srt
丸見え,丸見え,unknown,2,Karakai Jozu no Takagi-san.S01E01.JA.srt
だい,だい,unknown,2,Karakai Jozu no Takagi-san.S01E01.JA.srt
悔しい,悔し,unknown,2,Karakai Jozu no Takagi-san.S01E01.JA.srt
からかう,からかお,unknown,2,Karakai Jozu no Takagi-san.S01E01.JA.srt
なべ,なべ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
ため息,ため息,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
かぎ,かぎ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
べ,べ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
開く,開か,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
落とす,落とし,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
歪,歪,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
じゃ,じゃっ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
一同,一同,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
あす,あす,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
笑い声,笑い声,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
くい,くく,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
まんまと,まんまと,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
助け,助け,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
両,両,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
想い,想い,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
なれる,なれる,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
おまじない,おまじない,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
ほほ,ほほ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
つく,つか,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
一文字,一文字,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
なる,なりゃ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
かって,かって,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
がる,がら,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
はずす,はずす,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
拭う,拭い,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
日誌,日誌,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
いく,いけ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
やっぱし,やっぱし,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
三文,三文,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
徳,徳,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
しかける,しかけ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
眠気,眠気,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
隙間,隙間,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
拭く,拭こ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
面倒くさい,面倒くさい,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
おく,おこ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
拭く,拭か,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
残り,残り,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
ここだ,ここだ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
透ける,透ける,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
ちく,ちく,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
する,しょ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
おもしろい,おもしろかっ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
みな,みな,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
共,共,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
勝てる,勝てる,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
見つめる,見つめ,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
決め,決め,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt
こうか,こうか,unknown,1,Karakai Jozu no Takagi-san.S01E01.JA.srt

I'll test it more over the weekend, but based on the user experience so far... I think I'm staying with AnkiMorphs :)

[mortii]

Released in v2.2.0!

[xofm31]

Quickly comparing the results from Morphman and AnkiMorphs, I also have AnkiMorphs telling me I need to learn more. I think that when AnkiMorphs calculates how many words I need to learn to get to 90%, it goes through the frequency list until it gets to the 90% point, assuming that all of the words below that in the frequency list are unknown. But since some of those words are already known, my comprehension will be at 90% before that point.

[mortii]

Quickly comparing the results from Morphman and AnkiMorphs, I also have AnkiMorphs telling me I need to learn more.

How are you quantifying that? By number of total rows in the study plan file or by the number of morphs that have the "unknown" learning status? The latter shouldn't really be the case for Chinese, so that might be a bug.

I think that when AnkiMorphs calculates how many words I need to learn to get to 90%, it goes through the frequency list until it gets to the 90% point, assuming that all of the words below that in the frequency list are unknown.

Yes, it's not a one-to-one match with the MM version. The AM version is just concatenated frequency files and removes duplicates--the learning statuses are just additional fun facts.

[xofm31]

How are you quantifying that? By number of total rows in the study plan file or by the number of morphs that have the "unknown" learning status?

Number of morphs that have the "unknown" learning status. The first file in my study plan is called "app73". Here is the number of unknowns from the learning plan set to 90%:

❯ grep app73 study-plan-frequency-file_90.csv | grep unknown | wc
      46      58    1997

Here is the first line of the study plan from Morphman:

'text : app73.txt' study goal: ( 24/  24) morph readability: 75.05 -> 90.39 line readability: 35.71 -> 67.86

When I set them to 100%, they get the same number

❯ grep app73 study-plan-frequency-file.csv| grep unknown | wc
      66      80    2887

'text : app73.txt' study goal: ( 24/  24) morph readability: 75.05 -> 90.39 line readability: 35.71 -> 67.86

I think this must be because there are known words in the file that are not being taken into account my AnkiMorphs, although these are 29 off, and I thought it would be 22.

❯ grep app73 study-plan-frequency-file_90.csv| grep -v unknown | wc
     135     155    5270
❯ grep app73 study-plan-frequency-file.csv| grep -v unknown | wc
     164     184    6420

Both Morphman and AnkiMorphs agree that the readability is 75.1 (In AnkiMorphs, this is: known + learning, Total percentage)
When I set the desired comprehension to 75, I shouldn't get any unknown words, since I've already reached 75. But there are still 24.

❯ more study-plan-frequency-file_70.csv | grep app73 | grep unknown | wc
      24      26    1039

[mortii]

Aha, I'll look into it. Thank you for the amazing feedback!

[xofm31]

I'll make a final plug for the study plan logic that is in Morphman, because I really think that it makes sense. If you're not interested in implementing it, I'll probably get around to doing it for myself, hopefully before Morphman study plan creation stops working for me.

What I see as the best thing about the Morphman study plan is that it orders the unknown words for your current file(s) in the optimal way for your collection as a whole. This is important when I want to learn the most important vocabulary for the whole TV series, but only starting with the words that are in the first episode. When starting with a new episode, there will be a handful of new, really frequent words, but then a large number of words that occur only a few times in the episode. It is much more efficient for learning to prioritize the ones that I'm going to need for upcoming episodes. Since my vocab learning is slower than my TV watching, I don't usually get to the end of the current episode's unknown vocab before moving that file to a different directory so that the next episode is prioritized when I re-run the study plan.

Here is a basic algorithm that I think will accomplish this, and get the desired comprehension rate for each of the files. I'm writing this mostly for me, so that I remember what I was thinking when I get around to implementing it.

1. Get morph_occurrences_by_file as it does now, and also morph_occurences_total for the total counts all of the files together.
2. Get the morphs_learning_status for all of the morphs
3. FOR EACH FILE:
   a. Create a new morph_occurences_for_each_file, which has all the same morph keys as morph_occurrences_by_file, but the values are either the ones from morph_occurrences_total, or to give the occurences in the current file a bit more importance, morph_occurences_for_each_file+morph_occurences_occurences_total.
   b. Calculate the total number of morphs in the file, and multiply that by the desired comprehension rate to get target_known_occurences
   d. Calculate the currently known number of morphs total_known_morph_occurences from the file by looping through morph_occurrences_by_file and adding the value for each morph that is either known or learning.
   d. Sort morph_occurences_for_each_file (the dict that has the total number of occurrences across files) in descending order.
   e. Loop through the keys of morph_occurences_for_each_file
   i. if the key is not already in morph_in_study_plan[key], then morph_writer.writerow; update morph_in_study_plan[key]
   ii. if morphs_learning_status[key]=='unknown'
   1. update morphs_learning_status[key] to 'known' (since it will be once studied, and it should could as already know for subsequent files)
   2. total_known_morph_occurences += morph_occurrences_by_file[key]
   3. if total_known_morph_occurences >= target_known_occurences, end the loop & go to the next file

[xofm31]

@mortii @aleksejrs Thank you for giving this more consideration! I haven't looked at the Morphman code for the study plan, and only started using Morphman once there were already things that didn't work with it. I never had it work with a master frequency file, but maybe it did at one time. Now that I have a big enough list of files, I don't really need the master frequency file anymore, but it would be really helpful at the beginning.

[aleksejrs]

morph_occurences_in_all_input_files[morph] includes morph_occurences_by_file[file][morph] for the current file.

[mortii]

I have a lot of fires to put out, and I'm going to be away for extended periods of time, so I probably won't get around to implementing these kind of changes for at least a month+

morph_occurences_in_all_input_files[morph] includes morph_occurences_by_file[file][morph] for the current file.

@aleksejrs not necessarily a problem since it would be applied consistently for all morphs. It also wouldn't be circular since the former would be calculated first.

[aleksejrs]

@mortii not a problem if it's a percentage of the list length, not the number of occurrences.

[mortii]

@aleksejrs oh right, this does complicate the number of occurrences option. Let's hold off on that for now though, since it is the definition of premature optimization.

[mortii]

Not adding any new features, apologies.