Synthtiger: korean dataset for doctr training

[khawar-islam]

Hi @felixdittrich92 , thank you for you suggestion. After your suggestion, I have worked on https://github.com/clovaai/synthtiger repo and generate a new korean synthetic dataset. I have attached the .zip for your confirmation then i will be able to generate full 5 Million images dataset. I have changed some paramters for doctr training, please have a look on configuration and .zip file

While preparing training data, we do not need masks and coords.txt information. Am i right?
Can i generate the vertical and multi line images for doctr training? Is that OK

vertical: false
quality: [60, 95]
visibility_check: true

midground: 0.25
midground_offset:
  percents: [[-0.5, 0.5], [-0.5, 0.5]]
foreground_mask_pad: 5

corpus:
  weights: [1, 0]
  args:
    # length augmentable corpus
    - paths: [resources/corpus/ko.txt]
      weights: [1]
      min_length: 1
      max_length: 10
      textcase: [lower, upper, capitalize]
      augmentation: 0.1
      augmentation_length: [1, 10]
    # char augmentable corpus
    - paths: []
      weights: []
      min_length: 1
      max_length: 10
      textcase: [lower, upper, capitalize]
      augmentation: 0
      augmentation_charset: resources/charset/alphanum_special.txt

font:
  paths: [resources/font]
  weights: [1]
  size: [40, 80]
  bold: 0.5

texture:
  prob: 0.5
  args:
    paths: [resources/image]
    weights: [1]
    alpha: [0, 1]
    grayscale: 0
    crop: 1

colormap2:
  paths: [resources/colormap/iiit5k_gray.txt]
  weights: [1]
  k: 2
  alpha: [1, 1]
  colorize: 1

colormap3:
  paths: [resources/colormap/iiit5k_gray.txt]
  weights: [1]
  k: 3
  alpha: [1, 1]
  colorize: 1

color:
  gray: [0, 255]
  alpha: [1, 1]
  colorize: 1

shape:
  prob: 1
  args:
    weights: [1, 1]
    args:
      # elastic distortion
      - alpha: [15, 20]
        sigma: [4, 12]
      # elastic distortion
      - alpha: [0, 2]
        sigma: [0, 0.6]

layout:
  weights: [4, 1]
  args:
    # flow layout
    - space: [-2, 5]
      line_align: [middle]
    # curve layout
    - curve: [20, 40]
      space: [-2, 5]
      convex: 0.5
      upward: 0.5

style:
  prob: 0.25
  args:
    weights: [1, 2, 2]
    args:
      # text border
      - size: [1, 12]
        alpha: [1, 1]
        grayscale: 0
      # text shadow
      - distance: [1, 6]
        angle: [0, 360]
        alpha: [0.3, 0.7]
        grayscale: 0
      # text extrusion
      - length: [1, 12]
        angle: [0, 360]
        alpha: [1, 1]
        grayscale: 0

transform:
  prob: 1
  args:
    weights: [1, 1, 1, 1, 1, 1, 2]
    args:
      # perspective x
      - percents: [[0.5, 1], [1, 1]]
        aligns: [[0, 0], [0, 0]]
      # perspective y
      - percents: [[1, 1], [0.5, 1]]
        aligns: [[0, 0], [0, 0]]
      # trapezoidate x
      - weights: [1, 0, 1, 0]
        percent: [0.75, 1]
        align: [-1, 1]
      # trapezoidate y
      - weights: [0, 1, 0, 1]
        percent: [0.5, 1]
        align: [-1, 1]
      # skew x
      - weights: [1, 0]
        angle: [0, 30]
        ccw: 0.5
      # skew y
      - weights: [0, 1]
        angle: [0, 10]
        ccw: 0.5
      # rotate
      - angle: [0, 10]
        ccw: 0.5

pad:
  prob: 1
  args:
    pxs: [[0, 10], [0, 10], [0, 10], [0, 10]]

postprocess:
  args:
    # gaussian noise
    - prob: 1
      args:
        scale: [4, 8]
        per_channel: 0
    # gaussian blur
    - prob: 0.9
      args:
        sigma: [0, 2]
    # resample
    - prob: 0.1
      args:
        size: [0.4, 0.4]
    # median blur
    - prob: 1
      args:
        k: [1, 1]

results.zip

[felixdittrich92]

Hi @khawar-islam no need to keep masks and coords. Mh vertical could be used multiline is Not supported in doctr.

[khawar-islam]

thank you @felixdittrich92 , I am generating new dataset and still worry about some images, i look on some images and images are very distorted and even i cannot understand. Are these images fine for model?

[felixdittrich92]

Puh i think you need to play a bit with the config.

[khawar-islam]

Hello @felixdittrich92 Just for checking purpose, I have generated a dataset and start training with blurry images. In some extend, results are good but the confidence of each word is very low.

Train set loaded in 7.78s (1200000 samples in 75000 batches)
Validation loss decreased inf --> 0.595284: saving state...                                                                                  
Epoch 1/15 - Validation loss: 0.595284 (Exact: 67.42% | Partial: 67.42%)
Validation loss decreased 0.595284 --> 0.466508: saving state...                                                                             
Epoch 2/15 - Validation loss: 0.466508 (Exact: 74.72% | Partial: 74.72%)
Validation loss decreased 0.466508 --> 0.395294: saving state...                                                                             
Epoch 3/15 - Validation loss: 0.395294 (Exact: 78.02% | Partial: 78.02%)
Validation loss decreased 0.395294 --> 0.344752: saving state...                                                                             
Epoch 4/15 - Validation loss: 0.344752 (Exact: 80.45% | Partial: 80.45%)
Validation loss decreased 0.344752 --> 0.317908: saving state...                                                                             
Epoch 5/15 - Validation loss: 0.317908 (Exact: 82.11% | Partial: 82.11%)
Validation loss decreased 0.317908 --> 0.288069: saving state...                                                                             
Epoch 6/15 - Validation loss: 0.288069 (Exact: 83.32% | Partial: 83.32%)
Validation loss decreased 0.288069 --> 0.261786: saving state...                                                                             
Epoch 7/15 - Validation loss: 0.261786 (Exact: 84.40% | Partial: 84.40%)
Validation loss decreased 0.261786 --> 0.236861: saving state...                                                                             
Epoch 8/15 - Validation loss: 0.236861 (Exact: 85.48% | Partial: 85.48%)
Validation loss decreased 0.236861 --> 0.221139: saving state...                                                                             
Epoch 9/15 - Validation loss: 0.221139 (Exact: 86.16% | Partial: 86.16%)
Validation loss decreased 0.221139 --> 0.201975: saving state...                                                                             
Epoch 10/15 - Validation loss: 0.201975 (Exact: 86.99% | Partial: 86.99%)
Validation loss decreased 0.201975 --> 0.194313: saving state...                                                                             
Epoch 11/15 - Validation loss: 0.194313 (Exact: 87.41% | Partial: 87.41%)
Validation loss decreased 0.194313 --> 0.179732: saving state...                                                                             
Epoch 12/15 - Validation loss: 0.179732 (Exact: 87.91% | Partial: 87.91%)
Validation loss decreased 0.179732 --> 0.168934: saving state...                                                                             
Epoch 13/15 - Validation loss: 0.168934 (Exact: 88.27% | Partial: 88.27%)
Validation loss decreased 0.168934 --> 0.167898: saving state...                                                                             
Epoch 14/15 - Validation loss: 0.167898 (Exact: 88.35% | Partial: 88.35%)
Validation loss decreased 0.167898 --> 0.166855: saving state...                                                                             
Epoch 15/15 - Validation loss: 0.166855 (Exact: 88.38% | Partial: 88.38%)

I asked to author of synthtiger repo clovaai/synthtiger#36 they told me that the engine does not support to correct these images and sometime it generates bad images.

Solution:
If i will generate a single character per image, I mean that each image contains one hangul character. Would it be good for training a model? The size of dataset maybe 10M. But, when we use below command to visualize it then it will show 70% black image because our images have a single character. Does it create any problem for training? Why i am asking that because image are very distorted and blurry.

python3 train_pytorch.py master --train_path /media/cvpr/CM_22/KR/train --val_path /media/cvpr/CM_22/KR/val --show-samples
Second Solution
I have increased min_length and max_length of words. Now each image contains six to ten hangul character and we visualize correctly on above command.

Which one is better for model training? 1 character per image or 6 to 10 character per image without black portion?

[chintan-eonian]

hey @khawar-islam . Can you tell me how you prepared Corpus Text file for generating Korean Language Dataset using synthtiger?
I am confused about the step. Do I have to write it manually?