Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add index correction internal logic #2194

Merged
merged 103 commits into from
Oct 3, 2023
Merged

Add index correction internal logic #2194

merged 103 commits into from
Oct 3, 2023

Commits on Sep 15, 2023

  1. implement the initail framework

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    a1ee9d8 View commit details
    Browse the repository at this point in the history

  2. add corrector configuration

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    8a1221d View commit details
    Browse the repository at this point in the history

  3. add corrector logic

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    fa1eb75 View commit details
    Browse the repository at this point in the history

  4. add build make command for index correction binary

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    39e7d71 View commit details
    Browse the repository at this point in the history

  5. add Dockerfile for index correction

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    2bd41e9 View commit details
    Browse the repository at this point in the history

  6. add Docker image for index job correction

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    6ef29e3 View commit details
    Browse the repository at this point in the history

  7. add timer

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    c7e1ff2 View commit details
    Browse the repository at this point in the history

  8. fix tag align

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    22dcaaa View commit details
    Browse the repository at this point in the history

  9. tmp

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    d51e26b View commit details
    Browse the repository at this point in the history

  10. fix log

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    6aa8fde View commit details
    Browse the repository at this point in the history

  11. temporally implement two versions of correct function

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    41faffd View commit details
    Browse the repository at this point in the history

  12. set eg limit from config

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    8f5bcf0 View commit details
    Browse the repository at this point in the history

  13. add stream list concurrency config

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    065e9e2 View commit details
    Browse the repository at this point in the history

  14. implement index id caching

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    8562a99 View commit details
    Browse the repository at this point in the history

  15. add config to use cache or not

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    81345f3 View commit details
    Browse the repository at this point in the history

  16. style: Format code with prettier and gofumpt

    @deepsource-autofix @ykadowak
    deepsource-autofix[bot] authored and ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    77c33f0 View commit details
    Browse the repository at this point in the history

  17. refactor availableAddrs

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    72c3956 View commit details
    Browse the repository at this point in the history

  18. add kvs range duration

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    5ffd9b3 View commit details
    Browse the repository at this point in the history

  19. add leftAgentAddrs for performance

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    5d3c6c7 View commit details
    Browse the repository at this point in the history

  20. Revert "add kvs range duration" 

    This reverts commit 5b647be.
    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    3b3710b View commit details
    Browse the repository at this point in the history

  21. refactor

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    6b3934e View commit details
    Browse the repository at this point in the history

  22. fix without cache bug

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    bc591ae View commit details
    Browse the repository at this point in the history

  23. enable observability

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    5209a72 View commit details
    Browse the repository at this point in the history

  24. refactor

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    a0cb7aa View commit details
    Browse the repository at this point in the history

  25. SIGTERM after complete

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    b453e61 View commit details
    Browse the repository at this point in the history

  26. add metrics server

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    db6868d View commit details
    Browse the repository at this point in the history

  27. add pcache

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    7a630b3 View commit details
    Browse the repository at this point in the history

  28. remove comment

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    ca84f56 View commit details
    Browse the repository at this point in the history

  29. [TEMP] use pcache

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    feda428 View commit details
    Browse the repository at this point in the history

  30. [TMP] use pcache

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    c229a87 View commit details
    Browse the repository at this point in the history

  31. fix empty shard returns error

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    fa0b68b View commit details
    Browse the repository at this point in the history

  32. fix to use local map

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    06b312b View commit details
    Browse the repository at this point in the history

  33. [TMP] add prestop for pcache

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    490345f View commit details
    Browse the repository at this point in the history

  34. [TEMP] add pcache config

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    05838c3 View commit details
    Browse the repository at this point in the history

  35. style: Format code with prettier and gofumpt

    @deepsource-autofix @ykadowak
    deepsource-autofix[bot] authored and ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    9805e7e View commit details
    Browse the repository at this point in the history

  36. [TEMP] add pcache log

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    a0d698e View commit details
    Browse the repository at this point in the history

  37. fix map alloc size

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    9e5f2ae View commit details
    Browse the repository at this point in the history

  38. [TMP] Add bbolt cache

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    3728276 View commit details
    Browse the repository at this point in the history

  39. update bbolt

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    1678ff7 View commit details
    Browse the repository at this point in the history

  40. fix bbolt bug

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    4f87601 View commit details
    Browse the repository at this point in the history

  41. add bbolt test

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    01225aa View commit details
    Browse the repository at this point in the history

  42. [TEMP] use bbolt as persistent cache

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    428ee35 View commit details
    Browse the repository at this point in the history

  43. style: Format code with prettier and gofumpt

    @deepsource-autofix @ykadowak
    deepsource-autofix[bot] authored and ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    81b800e View commit details
    Browse the repository at this point in the history

  44. add SetBatch to bbolt

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    bfc73c4 View commit details
    Browse the repository at this point in the history

  45. use batch to write map to disk

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    d4f3695 View commit details
    Browse the repository at this point in the history

  46. style: Format code with prettier and gofumpt

    @deepsource-autofix @ykadowak
    deepsource-autofix[bot] authored and ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    9975237 View commit details
    Browse the repository at this point in the history

  47. delete the map elements on finalize

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    2b00e05 View commit details
    Browse the repository at this point in the history

  48. manually call GC after the map shrink

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    a611e88 View commit details
    Browse the repository at this point in the history

  49. add limit to SetBatch goroutine number

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    e21441c View commit details
    Browse the repository at this point in the history

  50. stop unnecesarry GC

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    66bc79c View commit details
    Browse the repository at this point in the history

  51. increase eg limit to the MaxBatchSize

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    3a85405 View commit details
    Browse the repository at this point in the history

  52. use ch to set batch bbolt

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    16d2986 View commit details
    Browse the repository at this point in the history

  53. fix servers shutdown properly

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    919c9f9 View commit details
    Browse the repository at this point in the history

  54. use internal/kvs/bbolt

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    9bd57c0 View commit details
    Browse the repository at this point in the history

  55. refactor

    @ykadowak
    ykadowak committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    55c6586 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2023

  1. always use bbolt cache for correction

    @ykadowak
    ykadowak committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    bc92686 View commit details
    Browse the repository at this point in the history

  2. update sample.yaml for correction

    @ykadowak
    ykadowak committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    319ec8b View commit details
    Browse the repository at this point in the history

  3. style: format code with Prettier and Gofumpt 

    This commit fixes the style issues introduced in 319ec8b according to the output
from Prettier and Gofumpt.

Details: #2152
    @deepsource-autofix
    deepsource-autofix[bot] authored Sep 19, 2023
    Configuration menu
    Copy the full SHA
    eb5cdb0 View commit details
    Browse the repository at this point in the history

  4. use go std slices pkg

    @ykadowak
    ykadowak committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    d3616e5 View commit details
    Browse the repository at this point in the history

  5. refactor

    @ykadowak
    ykadowak committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    89e2339 View commit details
    Browse the repository at this point in the history

  6. add comment

    @ykadowak
    ykadowak committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    e2eeabc View commit details
    Browse the repository at this point in the history

  7. remove valdsync

    @ykadowak
    ykadowak committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    9a9e622 View commit details
    Browse the repository at this point in the history

  8. use vald errgroup

    @ykadowak
    ykadowak committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    4c22bcc View commit details
    Browse the repository at this point in the history

  9. Merge branch 'main' into feature/index/correction-job

    @ykadowak
    ykadowak authored Sep 19, 2023
    Configuration menu
    Copy the full SHA
    e0ef7a8 View commit details
    Browse the repository at this point in the history

  10. refactor

    @ykadowak
    ykadowak committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    4dc4801 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2023

  1. Define ErrNoAvailableAgentToInsert

    @ykadowak
    ykadowak committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    705dc29 View commit details
    Browse the repository at this point in the history

  2. update comment in English

    @ykadowak
    ykadowak committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    952a6e7 View commit details
    Browse the repository at this point in the history

  3. Apply new actions yaml format

    @ykadowak
    ykadowak committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    5f93398 View commit details
    Browse the repository at this point in the history

  4. Disable godox

    @ykadowak
    ykadowak committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    c860ddc View commit details
    Browse the repository at this point in the history

  5. style: format code with Prettier and Gofumpt 

    This commit fixes the style issues introduced in c860ddc according to the output
from Prettier and Gofumpt.

Details: #2194
    @deepsource-autofix
    deepsource-autofix[bot] authored Sep 20, 2023
    Configuration menu
    Copy the full SHA
    2c2bb09 View commit details
    Browse the repository at this point in the history

  6. remove comment

    @ykadowak
    ykadowak committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    2ecf8cf View commit details
    Browse the repository at this point in the history

  7. Apply format

    @ykadowak
    ykadowak committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    8bb271d View commit details
    Browse the repository at this point in the history

  8. Add type check for type assertion

    @ykadowak
    ykadowak committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    e5f73a3 View commit details
    Browse the repository at this point in the history

  9. use const to specify filemode

    @ykadowak
    ykadowak committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    d3e4d78 View commit details
    Browse the repository at this point in the history

  10. Add bbolt concurrency as config

    @ykadowak
    ykadowak committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    7e5cd71 View commit details
    Browse the repository at this point in the history

  11. fix var style

    @ykadowak
    ykadowak committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    a184b22 View commit details
    Browse the repository at this point in the history

  12. Suppress linter

    @ykadowak
    ykadowak committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    4972294 View commit details
    Browse the repository at this point in the history

  13. fix comment

    @ykadowak
    ykadowak committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    ef93123 View commit details
    Browse the repository at this point in the history

  14. add test template

    @ykadowak
    ykadowak committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    375c04e View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2023

  1. Refactor parameters for index correction

    @ykadowak
    ykadowak committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    33ad603 View commit details
    Browse the repository at this point in the history

  2. Refactor config

    @ykadowak
    ykadowak committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    55bc821 View commit details
    Browse the repository at this point in the history

  3. Add corrector test

    @ykadowak
    ykadowak committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    004bf81 View commit details
    Browse the repository at this point in the history

  4. style: format code with Prettier and Gofumpt 

    This commit fixes the style issues introduced in 004bf81 according to the output
from Prettier and Gofumpt.

Details: #2194
    @deepsource-autofix
    deepsource-autofix[bot] authored Sep 25, 2023
    Configuration menu
    Copy the full SHA
    97ff3c0 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into feature/index/correction-job

    @ykadowak
    ykadowak authored Sep 25, 2023
    Configuration menu
    Copy the full SHA
    df88705 View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2023

  1. Add timestamp check

    @ykadowak
    ykadowak committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    8b0540d View commit details
    Browse the repository at this point in the history

  2. Apply format

    @ykadowak
    ykadowak committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    5430f67 View commit details
    Browse the repository at this point in the history

  3. fix schema type

    @ykadowak
    ykadowak committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    bdc9c70 View commit details
    Browse the repository at this point in the history

  4. Fix DeepSource errors

    @ykadowak
    ykadowak committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    f7ffd6a View commit details
    Browse the repository at this point in the history

  5. Fix misspell

    @ykadowak
    ykadowak committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    5e73905 View commit details
    Browse the repository at this point in the history

  6. Add type check

    @ykadowak
    ykadowak committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    ce78ccf View commit details
    Browse the repository at this point in the history

  7. Remove unused config

    @ykadowak
    ykadowak committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    ef2e20d View commit details
    Browse the repository at this point in the history

  8. Fix DeepSource error

    @ykadowak
    ykadowak committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    93d6ab7 View commit details
    Browse the repository at this point in the history

  9. Add required go:build e2e tag

    @ykadowak
    ykadowak committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    31058f5 View commit details
    Browse the repository at this point in the history

  10. Remove memo

    @ykadowak
    ykadowak committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    6f79736 View commit details
    Browse the repository at this point in the history

  11. Refactor comment

    @ykadowak
    ykadowak committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    0d4d9d2 View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2023

  1. Remove TODO comment that is already done

    @ykadowak
    ykadowak committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    0eebe24 View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2023

  1. Remove unused config

    @ykadowak
    ykadowak committed Oct 2, 2023
    Configuration menu
    Copy the full SHA
    0eba31e View commit details
    Browse the repository at this point in the history

  2. Add comment to errors

    @ykadowak
    ykadowak committed Oct 2, 2023
    Configuration menu
    Copy the full SHA
    5a73044 View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2023

  1. change app name

    @ykadowak
    ykadowak committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    5117332 View commit details
    Browse the repository at this point in the history

  2. replace filepath pkg with internal file 

    replace filepath pkg with internal file

refactor
    @ykadowak
    ykadowak committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    74bd1c6 View commit details
    Browse the repository at this point in the history

  3. Refactor 

    refactor
    @ykadowak
    ykadowak committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    d687013 View commit details
    Browse the repository at this point in the history

  4. Fix gRPC spelling

    @ykadowak
    ykadowak committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    ba241cb View commit details
    Browse the repository at this point in the history

  5. Remove memo

    @ykadowak
    ykadowak committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    436da6e View commit details
    Browse the repository at this point in the history