East tflite on Pixel emulator

[akshyasingh-mset]

I am trying to run EAST tflite model provided by @sayakpaul on tf-hub https://tfhub.dev/sayakpaul/lite-model/east-text-detector/fp16/1 to detect text on an image on Pixel emulator. I am able to use custrom model https://developers.google.com/codelabs/tflite-object-detection-android#6 on sample images to detect salads. I am a complete novice on running tflite on android devices. So i just replaced the above mentioned model with EAST tflite float 16 model and replaced sample image with the image i want east to detect text in. But I am receiving following error :

E/TaskJniUtils: Error getting native address of native library: task_vision_jni
java.lang.RuntimeException: Error occurred when initializing ObjectDetector: Input tensor has type kTfLiteFloat32: it requires specifying NormalizationOptions metadata to preprocess input images.
at org.tensorflow.lite.task.vision.detector.ObjectDetector.initJniWithModelFdAndOptions(Native Method)
at org.tensorflow.lite.task.vision.detector.ObjectDetector.access$000(ObjectDetector.java:88)
at org.tensorflow.lite.task.vision.detector.ObjectDetector$1.createHandle(ObjectDetector.java:156)
at org.tensorflow.lite.task.vision.detector.ObjectDetector$1.createHandle(ObjectDetector.java:149)
at org.tensorflow.lite.task.core.TaskJniUtils$1.createHandle(TaskJniUtils.java:70)
at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromLibrary(TaskJniUtils.java:91)
at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromFdAndOptions(TaskJniUtils.java:66)
at org.tensorflow.lite.task.vision.detector.ObjectDetector.createFromFileAndOptions(ObjectDetector.java:147)
at org.tensorflow.codelabs.objectdetection.MainActivity.runObjectDetection(MainActivity.kt:130)
at org.tensorflow.codelabs.objectdetection.MainActivity.access$runObjectDetection(MainActivity.kt:49)
at org.tensorflow.codelabs.objectdetection.MainActivity$setViewAndDetect$1.invokeSuspend(MainActivity.kt:184)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
E/AndroidRuntime: FATAL EXCEPTION: DefaultDispatcher-worker-1
Process: org.tensorflow.codelabs.objectdetection, PID: 6881
java.lang.IllegalStateException: Error getting native address of native library: task_vision_jni
at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromLibrary(TaskJniUtils.java:95)
at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromFdAndOptions(TaskJniUtils.java:66)
at org.tensorflow.lite.task.vision.detector.ObjectDetector.createFromFileAndOptions(ObjectDetector.java:147)
at org.tensorflow.codelabs.objectdetection.MainActivity.runObjectDetection(MainActivity.kt:130)
at org.tensorflow.codelabs.objectdetection.MainActivity.access$runObjectDetection(MainActivity.kt:49)
at org.tensorflow.codelabs.objectdetection.MainActivity$setViewAndDetect$1.invokeSuspend(MainActivity.kt:184)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Caused by: java.lang.RuntimeException: Error occurred when initializing ObjectDetector: Input tensor has type kTfLiteFloat32: it requires specifying NormalizationOptions metadata to preprocess input images.
at org.tensorflow.lite.task.vision.detector.ObjectDetector.initJniWithModelFdAndOptions(Native Method)
at org.tensorflow.lite.task.vision.detector.ObjectDetector.access$000(ObjectDetector.java:88)
at org.tensorflow.lite.task.vision.detector.ObjectDetector$1.createHandle(ObjectDetector.java:156)
at org.tensorflow.lite.task.vision.detector.ObjectDetector$1.createHandle(ObjectDetector.java:149)
at org.tensorflow.lite.task.core.TaskJniUtils$1.createHandle(TaskJniUtils.java:70)
at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromLibrary(TaskJniUtils.java:91)
at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromFdAndOptions(TaskJniUtils.java:66) 
at org.tensorflow.lite.task.vision.detector.ObjectDetector.createFromFileAndOptions(ObjectDetector.java:147) 
at org.tensorflow.codelabs.objectdetection.MainActivity.runObjectDetection(MainActivity.kt:130) 
at org.tensorflow.codelabs.objectdetection.MainActivity.access$runObjectDetection(MainActivity.kt:49) 
at org.tensorflow.codelabs.objectdetection.MainActivity$setViewAndDetect$1.invokeSuspend(MainActivity.kt:184) 
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) 
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) 
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571) 
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750) 
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678) 
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) 
D/EGL_emulation: eglMakeCurrent: 0xcd693de0: ver 3 0 (tinfo 0xe8846f20)
W/BaseTaskApi: Closing an already closed native lib

Please help me, I am learning to deploy tflite on mobile devices. Sorry for the english

[sayakpaul]

@farmaker47 could you help here?

[akshyasingh-mset]

@sayakpaul @farmaker47 I used the link : https://www.tensorflow.org/lite/convert/metadata_writer_tutorial to convert the above mentioned float16 tflite EAST model to include metadata. But now I am receiving an altogether different error. It seems I am doing something wrong here. Please help

E/TaskJniUtils: Error getting native address of native library: task_vision_jni
java.lang.IllegalArgumentException: Error occurred when initializing ObjectDetector: Mobile SSD models are expected to have exactly 4 outputs, found 2
at org.tensorflow.lite.task.vision.detector.ObjectDetector.initJniWithModelFdAndOptions(Native Method)
at org.tensorflow.lite.task.vision.detector.ObjectDetector.access$000(ObjectDetector.java:88)
at org.tensorflow.lite.task.vision.detector.ObjectDetector$1.createHandle(ObjectDetector.java:156)
at org.tensorflow.lite.task.vision.detector.ObjectDetector$1.createHandle(ObjectDetector.java:149)
at org.tensorflow.lite.task.core.TaskJniUtils$1.createHandle(TaskJniUtils.java:70)
at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromLibrary(TaskJniUtils.java:91)
at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromFdAndOptions(TaskJniUtils.java:66)
at org.tensorflow.lite.task.vision.detector.ObjectDetector.createFromFileAndOptions(ObjectDetector.java:147)
at org.tensorflow.codelabs.objectdetection.MainActivity.runObjectDetection(MainActivity.kt:130)
at org.tensorflow.codelabs.objectdetection.MainActivity.access$runObjectDetection(MainActivity.kt:49)
at org.tensorflow.codelabs.objectdetection.MainActivity$setViewAndDetect$1.invokeSuspend(MainActivity.kt:183)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
E/AndroidRuntime: FATAL EXCEPTION: DefaultDispatcher-worker-1
Process: org.tensorflow.codelabs.objectdetection, PID: 8734
java.lang.IllegalStateException: Error getting native address of native library: task_vision_jni
at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromLibrary(TaskJniUtils.java:95)
at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromFdAndOptions(TaskJniUtils.java:66)
at org.tensorflow.lite.task.vision.detector.ObjectDetector.createFromFileAndOptions(ObjectDetector.java:147)
at org.tensorflow.codelabs.objectdetection.MainActivity.runObjectDetection(MainActivity.kt:130)
at org.tensorflow.codelabs.objectdetection.MainActivity.access$runObjectDetection(MainActivity.kt:49)
at org.tensorflow.codelabs.objectdetection.MainActivity$setViewAndDetect$1.invokeSuspend(MainActivity.kt:183)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Caused by: java.lang.IllegalArgumentException: Error occurred when initializing ObjectDetector: Mobile SSD models are expected to have exactly 4 outputs, found 2
at org.tensorflow.lite.task.vision.detector.ObjectDetector.initJniWithModelFdAndOptions(Native Method)
at org.tensorflow.lite.task.vision.detector.ObjectDetector.access$000(ObjectDetector.java:88)
at org.tensorflow.lite.task.vision.detector.ObjectDetector$1.createHandle(ObjectDetector.java:156)
at org.tensorflow.lite.task.vision.detector.ObjectDetector$1.createHandle(ObjectDetector.java:149)
at org.tensorflow.lite.task.core.TaskJniUtils$1.createHandle(TaskJniUtils.java:70)
at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromLibrary(TaskJniUtils.java:91)
at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromFdAndOptions(TaskJniUtils.java:66) 
at org.tensorflow.lite.task.vision.detector.ObjectDetector.createFromFileAndOptions(ObjectDetector.java:147) 
at org.tensorflow.codelabs.objectdetection.MainActivity.runObjectDetection(MainActivity.kt:130) 
at org.tensorflow.codelabs.objectdetection.MainActivity.access$runObjectDetection(MainActivity.kt:49) 
at org.tensorflow.codelabs.objectdetection.MainActivity$setViewAndDetect$1.invokeSuspend(MainActivity.kt:183) 
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) 
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) 
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571) 
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750) 
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678) 
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) 

[farmaker47]

@farmaker47 could you help here?

Yes sure.. Let me take a look in the next days.

[farmaker47]

@singh-aymcommerce Unfortunately object detector applications and EAST tflite file do not match. You have to use a custom project with an Interpreter. Check the guide at Tensorflow Lite https://www.tensorflow.org/lite/guide/inference#load_and_run_a_model_in_java
You can firstly use the Python API to run inference..so you can understand how you are going to acomplish that in Java inside android project. https://www.tensorflow.org/lite/guide/inference#load_and_run_a_model_in_python

You have to see the original Python notebook to understand the outputs of the model. there are 2 of them. One of 1,80,80,1 and the other of 1,80,80,5 You have to understand what are those...and how they are converted to desired object

[akshyasingh-mset]

@farmaker47 thank you for your response. I will try your above proposed solution to write custom interpreter in java. Give me couple of days and i will get back if I am able to do it. Thanks a lot @farmaker47 @sayakpaul :D

[akshyasingh-mset]

@sayakpaul @farmaker47 I am successfully able to import tflite model for EAST with your help.

i am coming across one weird thing that while adding metadata i added a labels.txt with only one line "text" as only one class is detected as label

but there are all null values in the geometry array when i try to inference an image with text PFA. But above tflite file is working when i try to detect text boxes using @sayakpaul Python script. The EAST tflite model is expecting input in float32 format. Am i doing something wrong, please suggest

dtype = {DataType@10768} "FLOAT32"
nativeHandle = 3391991168
quantizationParamsCopy = {Tensor$QuantizationParams@10769}
shapeCopy = {int[4]@10770} [1, 320, 320, 3]
shapeSignatureCopy = {int[4]@10771} [1, 320, 320, 3]
shadow$klass = {Class@10688} "class org.tensorflow.lite.Tensor"
shadow$monitor = 0

[akshyasingh-mset]

@sayakpaul @farmaker47 I have solved the above issue :D

I have come across another roadblock on how to implement nonMaxSuppresion on android. Can you help me please?

[10770]

Boa noite, eu acho que você está enviando um e-mail para a pessoa errada. Sou José Felipe, sou do Brasil. Em qua., 29 de dez. de 2021 às 05:17, singh-aymcommerce < ***@***.***> escreveu:

…

@sayakpaul <https://github.com/sayakpaul> @farmaker47 <https://github.com/farmaker47> I have solved the above issue :D I have come across another roadblock on how to implement nonMaxSuppresion on android. Can you help me please? — Reply to this email directly, view it on GitHub <#56 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AV2VUNDEYVTRS7BE3YF64B3UTK727ANCNFSM5JQ6BQFA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

-- *Cordialmente, * *José Felipe. * Universidade Federal de Sergipe (UFS), Campus São Cristóvão. Departamento de Química (DQI) Laboratório de Corrosão e Nanotecnologia (LCNT/NUPEG). Discente do curso de Química licenciatura. Segundo Secretário Geral do CALIQ. Contato: (79) 999889473, ***@***.*** Currículo Lattes: http://lattes.cnpq.br/5234169420338145