[Fix] Tensorflow MASTER implementation #949

felixdittrich92 · 2022-06-13T12:03:55Z

This PR:

refactored tensorflow transformer implementation
fix MASTER Decoder

Any feedback is welcome 🤗
@charlesmindee @frgfm feel free to take a first look some improvements are very welcome :D

Todo:

toy benchmark (train: 500k MJSynth / val: FUNSD/CORD)
fix decoding stage while inference (Note: training works well only decoding troublesome)
replace numpy cast in positional encoding
Issue:
Cannot train pytorch sar_resnet31 and master recognition model #802

codecov · 2022-06-13T12:23:38Z

Codecov Report

Merging #949 (760ceb3) into main (82d6e7e) will increase coverage by 0.26%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #949      +/-   ##
==========================================
+ Coverage   94.89%   95.16%   +0.26%     
==========================================
  Files         134      134              
  Lines        5542     5520      -22     
==========================================
- Hits         5259     5253       -6     
+ Misses        283      267      -16

Flag	Coverage Δ
unittests	`95.16% <100.00%> (+0.26%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...tr/models/classification/magc_resnet/tensorflow.py	`100.00% <ø> (ø)`
doctr/models/recognition/master/pytorch.py	`95.32% <ø> (ø)`
doctr/models/recognition/transformer/pytorch.py	`100.00% <ø> (ø)`
doctr/models/recognition/master/tensorflow.py	`100.00% <100.00%> (+2.80%)`	⬆️
doctr/models/recognition/transformer/tensorflow.py	`100.00% <100.00%> (+11.88%)`	⬆️
doctr/transforms/functional/base.py	`95.65% <0.00%> (-1.45%)`	⬇️
doctr/transforms/modules/base.py	`94.59% <0.00%> (ø)`
doctr/transforms/modules/tensorflow.py	`83.51% <0.00%> (+1.09%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 82d6e7e...760ceb3. Read the comment docs.

felixdittrich92 · 2022-06-13T19:32:28Z

@frgfm If you should get to it before me, please take a look at the decoding and code improvements in advance would also be quite good 👍

felixdittrich92 · 2022-06-15T08:33:03Z

MJSynth 500K train 100K val
Train set loaded in 311.6s (509998 samples in 3984 batches)
Validation loss decreased inf --> 0.000531279: saving state...
Epoch 1/1 - Validation loss: 0.000531279 (Exact: 99.86% | Partial: 99.86%)

FUNSD:
Validation loss: 1.01889 (Exact: 71.46% | Partial: 71.46%)
CORD:
Validation loss: 0.8728 (Exact: 63.35% | Partial: 63.35%)

But decoding while inference is still broken :
single (recognition_predictor):
[('8555555ySIXIyddy', 0.30775195360183716)]

complete pipe:

Block(
        (lines): [Line(
          (words): [
            Word(value='85555555SIXI5Xd', confidence=0.2),
            Word(value='8555555ySIXIyddy', confidence=0.3),
            Word(value='85555555SIXI5Xyyy', confidence=0.19),
            Word(value='85555555SIXI5Xyyy', confidence=0.19),
            Word(value='85555555SIXI5Xyyy', confidence=0.19),
          ]
        )]

@frgfm @charlesmindee i don't see what i missed, so i think a second pair of eyes would be quite good 👀

I would recommend that one of you two takes this stand and validates it again, because there is a clear difference to the toy benchmark on the pyTorch side implementation which works fine 😅

felixdittrich92 · 2022-06-23T22:22:06Z

@frgfm this PR would be very happy to get your help 🤣

felixdittrich92 · 2022-06-30T11:04:05Z

Again 500K MJSynth toy run:
Validation loss decreased inf --> 0.237682: saving state...
Epoch 1/10 - Validation loss: 0.237682 (Exact: 71.12% | Partial: 74.52%)
FUNSD: Validation loss: 2.02109 (Exact: 41.82% | Partial: 44.07%)
CORD: Validation loss: 2.76744 (Exact: 28.25% | Partial: 29.00%)

Inference:

Line(
(words): [
Word(value='Die', confidence=0.57),
Word(value='jeweils', confidence=0.61),
Word(value='gewahlte', confidence=0.49),
Word(value='Ausfuhrung', confidence=0.99),
]
),

felixdittrich92 · 2022-06-30T11:09:40Z

doctr/models/recognition/transformer/tensorflow.py

+        )
+        pe = pe.numpy()
+        pe[:, 0::2] = tf.math.sin(position * div_term)
+        pe[:, 1::2] = tf.math.cos(position * div_term)


@frgfm Any idea how to do this in TF and remove the numpy cast? 😅

Hi @felixdittrich92, I know this but I think it is still a bit experimental

felixdittrich92 · 2022-06-30T11:28:40Z

Note:
mypy will be fixed with #966
train-char-classification ref. #949 @frgfm (it isn't fixed with the upgrade 😅) i think best would be as mentioned to replace mobilenet with keras.applications implementation soon

charlesmindee

Thanks, LGTM!

charlesmindee · 2022-07-01T15:01:39Z

doctr/models/recognition/transformer/tensorflow.py

+        )
+        pe = pe.numpy()
+        pe[:, 0::2] = tf.math.sin(position * div_term)
+        pe[:, 1::2] = tf.math.cos(position * div_term)


Hi @felixdittrich92, I know this but I think it is still a bit experimental

charlesmindee

Thanks! LGTM

felixdittrich92 added this to the 0.5.2 milestone Jun 13, 2022

felixdittrich92 added type: bug Something isn't working module: models Related to doctr.models framework: tensorflow Related to TensorFlow backend topic: text recognition Related to the task of text recognition labels Jun 13, 2022

felixdittrich92 mentioned this pull request Jun 13, 2022

Cannot train pytorch sar_resnet31 and master recognition model #802

Closed

4 tasks

felixdittrich92 requested a review from frgfm June 13, 2022 19:32

felixdittrich92 added the help wanted Extra attention is needed label Jun 15, 2022

felixdittrich92 force-pushed the fix-master-tf branch from d00adc6 to 74318ff Compare June 16, 2022 10:23

felixdittrich92 self-assigned this Jun 27, 2022

felixdittrich92 force-pushed the fix-master-tf branch from 5e4299a to 86433e4 Compare June 28, 2022 06:34

felixdittrich92 removed the help wanted Extra attention is needed label Jun 30, 2022

felixdittrich92 force-pushed the fix-master-tf branch from 86433e4 to 2d7a01c Compare June 30, 2022 11:08

felixdittrich92 commented Jun 30, 2022

View reviewed changes

felixdittrich92 requested a review from charlesmindee June 30, 2022 11:10

felixdittrich92 marked this pull request as ready for review June 30, 2022 11:11

felixdittrich92 changed the title ~~[WIP][Fix] Tensorflow MASTER implementation~~ [Fix] Tensorflow MASTER implementation Jun 30, 2022

charlesmindee previously approved these changes Jul 1, 2022

View reviewed changes

felixdittrich92 added 6 commits July 1, 2022 17:45

first version trainable

cb86db7

fix masking

a96c8ce

make flake8 happy

bc541a1

update decoding - still broken

a54843b

update decoding - still broken

71f7965

update masking

f82e7bf

felixdittrich92 added 4 commits July 1, 2022 17:45

minor docstring update

9b755f0

rebase and update train label check

72262d4

fix masking

2d8d834

update magc mean std and small improvements

b77e6b6

felixdittrich92 dismissed charlesmindee’s stale review via b77e6b6 July 1, 2022 15:46

felixdittrich92 force-pushed the fix-master-tf branch from 2d7a01c to b77e6b6 Compare July 1, 2022 15:46

felixdittrich92 requested a review from charlesmindee July 1, 2022 15:47

make mypy happy

760ceb3

charlesmindee approved these changes Jul 1, 2022

View reviewed changes

felixdittrich92 merged commit 9530f81 into mindee:main Jul 1, 2022

felixdittrich92 deleted the fix-master-tf branch July 1, 2022 18:07

felixdittrich92 modified the milestones: 0.5.2, 0.6.0 Sep 26, 2022

felixdittrich92 mentioned this pull request Sep 26, 2022

Release tracker - v0.6.0 #791

Closed

85 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Fix] Tensorflow MASTER implementation #949

[Fix] Tensorflow MASTER implementation #949

Uh oh!

felixdittrich92 commented Jun 13, 2022 •

edited

Loading

Uh oh!

codecov bot commented Jun 13, 2022 •

edited

Loading

Uh oh!

felixdittrich92 commented Jun 13, 2022

Uh oh!

felixdittrich92 commented Jun 15, 2022 •

edited

Loading

Uh oh!

felixdittrich92 commented Jun 23, 2022

Uh oh!

felixdittrich92 commented Jun 30, 2022 •

edited

Loading

Uh oh!

felixdittrich92 Jun 30, 2022

Uh oh!

charlesmindee Jul 1, 2022

Uh oh!

felixdittrich92 commented Jun 30, 2022

Uh oh!

charlesmindee left a comment

Uh oh!

charlesmindee Jul 1, 2022

Uh oh!

charlesmindee left a comment

Uh oh!

Uh oh!

[Fix] Tensorflow MASTER implementation #949

[Fix] Tensorflow MASTER implementation #949

Uh oh!

Conversation

felixdittrich92 commented Jun 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jun 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

felixdittrich92 commented Jun 13, 2022

Uh oh!

felixdittrich92 commented Jun 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

felixdittrich92 commented Jun 23, 2022

Uh oh!

felixdittrich92 commented Jun 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

felixdittrich92 Jun 30, 2022

Choose a reason for hiding this comment

Uh oh!

charlesmindee Jul 1, 2022

Choose a reason for hiding this comment

Uh oh!

felixdittrich92 commented Jun 30, 2022

Uh oh!

charlesmindee left a comment

Choose a reason for hiding this comment

Uh oh!

charlesmindee Jul 1, 2022

Choose a reason for hiding this comment

Uh oh!

charlesmindee left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

felixdittrich92 commented Jun 13, 2022 •

edited

Loading

codecov bot commented Jun 13, 2022 •

edited

Loading

felixdittrich92 commented Jun 15, 2022 •

edited

Loading

felixdittrich92 commented Jun 30, 2022 •

edited

Loading