Image translate.

[gety9]

Sometimes when visiting website parts of text are image (tables, buttons, banners, etc) so what i do then:
1, use screenshot app to screenshot selected area
2, use https://translate.yandex.com/?source_lang=zh&target_lang=en > camera to transalte the screenshot

it works fine, but requires so many steps (change between tabs, save screenshot, select file, etc)

it will be very useful to have it in einkbro
1, press toolbar icon "image translate"
2, select area
3, show result (with same layout on top of selected area)

Question is how difficult it's to do?
It seems to me we have everything what's needed:
1, lots of open source OCR projects to recognize text
2, for translation we can use google translate or also open source library/api
3, ideally show translation in same layout/font size/etc like original selection

[plateaukao]

@gety9
If free API is available, then it's not difficult to implement this feature.
If not, about your proposed tasks:

1. lots of open source OCR projects: most of them are built for desktop environment. The only offline (on-device) library I could find is Google's ML-kit with basic text recognition capability; by adding only two languages' data model, EinkBro will expand to around 25MB.

2., 3. Yes, if text could be extracted perfectly, the translation part is easy. However, to identify how the original text is aligned, which part of the text sections are of the same paragraph; and try to re-render image with translated string at the similar position -- this is difficult, and need both computing power and AI tech to achieve this. It's not the scope of one-man casual project can dream of.

Naver Papago did very well on image translation, much better than Google Translate and Yandex. You can see the effect below: left side is original image; right side is what's rendered into Chinese after translation. You can see how well it extracts the text, and blending into correct background color when painting new translated strings. However, the API is not free; I could not integrate it freely into EinkBro. :)

[gety9]

@plateaukao Daniel hi,

I noticed "Key for image translation feature" setting. Does it use Papago api?

Could you pleas tell the cost (per request?) and how to apply for the key?

I tried searching "Papago API key" but can't find amything useful.

[plateaukao]

You can check pricing here.
https://www.ncloud.com/product/aiService/papagoTranslation

[gety9]

closing