I would like to know everyone's thoughts about having morph highlighting available in "other" cards through a formatter.

[RobHelgeson]

First off, thanks for the amazing plugin! I am new to t+1 learning and discovered MorphMan last week and have since moved up to AnkiMorphs.

One thing that I miss from MorphMan is the ability to use the formatter morphHighlight on card templates to add morph tracking to cards that I am not mining for morphemes. Ala https://github.com/kaegi/MorphMan/blob/master/morph/newMorphHelper.py#L251-L318.

With MorphMan, you did have to register the note type, but you could mark is not read and not write. this would enable the formatter to be used but exclude the cards from any other considerations.

It was not perfect, and I did need to write a little js snippet that would shuffle the attributes around when tags were used. but I eventually got it wired up nicely.

It appears that AnkiMorphs takes a different approach and defers the highlight processing time all to the calculation phase (as opposed to the card draw phase -- names I just made up on the spot, lol). This has not only runtime a performance benefit, but the benefit of not requiring the plug-in on mobile devices. It does, however, have the drawback of requiring that all card types where you'd like to use the morph data to be pre-processed and have static data persisted in the card, making your card data duplicative .

I wonder if this kind of formatter is possible/appropriate for AnkiMorphs, or if I should just not care about card data duplication, bite the bullet, and add am-highlighted to all my "other" cards.

I'd love to hear your thoughts!

[mortii]

should I just not care about card data duplication, bite the bullet, and add am-highlighted to all my "other" cards.

I can't really give any advice on this without more information since it really depends on how many cards you have, whether or not you want your "other" cards to be sorted, etc.

I wonder if this kind of formatter is possible/appropriate for AnkiMorphs

AnkiMorphs does not have a runtime formatter, but you could make your own relatively easily, all you need to do is:

1. find the hook you want trigger on (maybe reviewer_did_show_answer?)
2. check the note fields for the morphHighlight keyword or whatever else you want to use
3. connect to ankimorphs.db to get the learning intervals of the respective morphs
4. highlight accordingly

If anyone made that into a separate add-on and called it AnkiMorphs Runtime Highlighter or something that would be super cool :)

[RobHelgeson]

Thanks for the quick response, I might just play around here in my upcoming time off from work :D

[RobHelgeson]

Ok, I looked at this a little more this evening. I think I have a game plan together.

from anki import hooks

hooks.field_filter.append(am_highlight)

I'll need your https://github.com/mortii/anki-morphs/blob/main/ankimorphs/ankimorphs_db.py#L181-L197

and text_highlighting.get_highlighted_text(...)

With those the only piece I would be missing to connect the dots would be having the formatter get the card id. I'm still getting up to speed on writing plugins for Anki, still looking to see if that data is given to me in a prop when the filter is called. (i.e. I am trying to find the datatype for the second param to the formatter (field) if that's a complex type that includes card id, then I am golden. if not I may have to go back to the drawing board a bit.

I'm also loathe to copy/paste so much of your code into the extension. Both for the fact that it seems crappy to do that, but also for the maint aspect. (OTOH it'd be nice to have the exact implications as you so that we'd be 100% in sync) Would you entertain pushing some of your code to pipy or some other repo, or should I just suck it up and paste it in?

I think the db stuff is so small scoped and internal that having a copy of that would be fine, but the text manipulation would suck to have to re-write, and having a different implementations means that we'd have different bugs than each other and keeping the plugin data in sync with "the real thing" would be a PITA.

Anyway I'm still a long way out since I have all kinds of other things to figure out in how to make an Anki extension in general.

[RobHelgeson]

Ok, that was a naive plan.... Of course a card will not have a card id in the Ankimorphs db if it's not included at Recalc time.

New plan is to see how you morph-ize the cards in batch mode, and do it one-off. then use those morphs as the input to text_highlighting.get_highlighted_text(...).

P.S. FWIW, to answer my own question from before.... the card info is passed into formatting funcs in the final arg, of type: ‎TemplateRenderContext that has (among other things) a card() getter.

[mortii]

Ah, I overlooked the fact that the text has to be morphized during runtime as well to make the formatting accurate. Importing the morphemizers is a lot of heavy lifting, so it would probably be better to just to modify/extend AM instead of creating a new add-on.

To get the morphs during runtime you can probably use much of the code in text_preprocessing.py and generators_text_processing directly, or with slight adjustments.

[mortii]

Excuse my inaccurate use of 'runtime', it's just the huristic I always use for 'on-the-fly'.

[RobHelgeson]

Yea, after digging I do see that you have 99.9% of the stuff needed to make it all work, it's just a matter of shuffling some things around and registering them in the right places.

Right now I think that the only piece that is not "perfect" with the way I am thinking about going at it is that since it would be nice to not have to add a note type config in AM for all cards where you want to use the formatter, selecting the global morphemizers[-1] as it appears that they're adding by increasing accuracy (ref: https://github.com/mortii/anki-morphs/blob/main/ankimorphs/morphemizers/morphemizer.py#L50-L76).

I did want to get your thoughts on this shortcoming. After all if I'm going to raise a PR into your repo, it better be up to your standards. 😄

So, here are a couple of options:

1. As above, do not require the user to add a configuration (aka filter) for all note types where they would like to use the am-highlight-morphs (working name) field filter in their card templates, but default to the [-1]th element of the global morphemizers.
2. As was the way in MorphMan, require the user to create a configuration for each note type where they would like to use the filter (they can choose to disable read and write and this will not have any other impacts) so that we can deterministically read which morphemizer the user intends to use through get_morphemizer_by_description
3. The best of both worlds (?) - See if there is a config for the note type that am-highlight-morphs was used on. If so, use that morphemizer from the config. if not, use the [-1]th element of the global as in # 1.

I have not looked at your tests / testing strategy, so I'd like to do that this evening after work to make sure I'm not going to run into any snags there either.

P.S. If you haven't noticed I'm largely writing this out to get my head in the right place and gather the req's before I pull down the repo and make the first incision. The goal is that all the thinking is done up front and that execution becomes quick with few surprises 😄

P.P.S. Read you loud and clear on the 'runtime' thing, I think you can see I'm usually pretty imprecise with my words 😬 -- as long as I get my point across.

[mortii]

After all if I'm going to raise a PR into your repo, it better be up to your standards.

I don't know if you've seen the latest release notes, but this project is essentially in a bug fix only phase, although I left a tiny gap open for new features, so any additions would have to be minimally invasive.

do not require the user to add a configuration [...] but default to the [-1]th element of the global morphemizers

Guessing the morphemizer is definitely not a good approach. It would probably be fine most of the time, but when it fails it will likely fail spectacularly.

As was the way in MorphMan, require the user to create a configuration for each note type where they would like to use the filter (they can choose to disable read and write and this will not have any other impacts)

That seems reasonable, reusing the current note-filter system would reduce additional code 👍

[RobHelgeson]

I don't know if you've seen the latest release notes, but this project is essentially in a bug fix only phase, although I left a tiny gap open for new features, so any additions would have to be minimally invasive.

Yes, I did see that. I'm sad to hear it, but I understand what it's like to be so into something and then have a critical dependency change drastically out from under you.

If this means you're loathe to approve my PR, that's totally fine. While I do believe there should be little to no impact on the overall plugin, I do understand, any change is a change.

Guessing the morphemizer is definitely not a good approach. It would probably be fine most of the time, but when it fails it will likely fail spectacularly.

I didn't feel great about this either and your opinion on the matter makes it a no-brainer to go with # 2 above.

[RobHelgeson]

#308 is ready for review. I am all ears for any changes.

[RobHelgeson]

Just noting that I'm still on this. I worked on it a bunch over the weekend, and a little last night. I have a branch, where I ended up chasing features until I had completely reimplemented the highlight_text function. I don't know if I even want to propose/pr it here. or if I do, maybe it's just for informational purposes, and or if anyone is curious.

I was really pleased to see your tests for highlight_text and I have stubbed in my implementation and run through them. I am especially thankful for the regex escape one, as that was def overlooked by me! The German one still has some work before I pass.

Contrasting the implementations:
Mine takes a lot more bytes to get the job done (I have not optimized for this yet).
Mine allows correct display of rubies in languages that use them.
Mine tags unprocessed sections as unprocessed instead of unknown, as I believe them to be different, and worth differentiating.
I'm trying to understand the perf differences between the 2 implementations. (if you have any ideas for perf testing, I am all ears, but my plan will be to create a giant deck and use wall clock time of the 2 impls not great, but it's what we got)

I didn't look at your implementation in any detail while I was implementing mine, so I'm going to do that when the tests are fully passing. From what I did see, the difference in our impls is that I rely more heavily on regexes. I expect that they will be pretty similar to one another in the long run.

At the end of the day, it's a fun and interesting exercise, and I certainly upped my regex game while tinkering around in here. I doubt this code should see the light of day beyond a pr for exposure, but fun none the less. Should have a pr in a couple days.

I'll still need to go back to the "real" original pr and make sure that I tidy up loose ends so that it's quality enough to merge.

[mortii]

if you have any ideas for perf testing, I am all ears

Here is a link to the profiling section that used to be in the dev guide. py-spy can get very granular if that's something you want.

reimplemented the highlight_text function

I expect that any change to the output will cause the integration tests relying on card collections to fail, but I can update those if necessary.

In general, a lot of the tests are there to prevent regression, they aren't necessarily testing against perfection, so don't be afraid of changing them if you make something better 👍

[RobHelgeson]

I'm still alive.

here's a tease :D

Tests fully pass. Code is done but needs a cleanup pass. Will have a PR in a day or so.

Main:
Recalc duration: 15.572 seconds
Recalc duration: 13.090 seconds
Recalc duration: 13.011 seconds
Recalc duration: 12.880 seconds
Recalc duration: 12.760 seconds

New:
Recalc duration: 13.772 seconds
Recalc duration: 12.886 seconds
Recalc duration: 12.116 seconds
Recalc duration: 12.075 seconds
Recalc duration: 12.317 seconds

[RobHelgeson]

Mac m1 on a 35.6K card collection - Spacy Japanese large

[mortii]

Exciting!

[mortii]

@RobHelgeson I'll try to review the PRs before the weekend, hopefully!

[RobHelgeson]

This has been implemented and merged in #313!