Add index correction internal logic #2194
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This reverts commit 5b647be.
ykadowak
force-pushed
the
feature/index/correction-job
branch
2 times, most recently
from
6be6278
to
9503475
Compare
replace filepath pkg with internal file refactor
ykadowak
force-pushed
the
feature/index/correction-job
branch
from
a38f985
to
45856a9
Compare
ykadowak
force-pushed
the
feature/index/correction-job
branch
from
45856a9
to
d687013
Compare
ykadowak
added a commit
that referenced
this pull request
Oct 10, 2023
* implement the initail framework * add corrector configuration * add corrector logic * add build make command for index correction binary * add Dockerfile for index correction * add Docker image for index job correction * add timer * fix tag align * tmp * fix log * temporally implement two versions of correct function * set eg limit from config * add stream list concurrency config * implement index id caching * add config to use cache or not * style: Format code with prettier and gofumpt * refactor availableAddrs * add kvs range duration * add leftAgentAddrs for performance * Revert "add kvs range duration" This reverts commit 5b647be. * refactor * fix without cache bug * enable observability * refactor * SIGTERM after complete * add metrics server * add pcache * remove comment * [TEMP] use pcache * [TMP] use pcache * fix empty shard returns error * fix to use local map * [TMP] add prestop for pcache * [TEMP] add pcache config * style: Format code with prettier and gofumpt * [TEMP] add pcache log * fix map alloc size * [TMP] Add bbolt cache * update bbolt * fix bbolt bug * add bbolt test * [TEMP] use bbolt as persistent cache * style: Format code with prettier and gofumpt * add SetBatch to bbolt * use batch to write map to disk * style: Format code with prettier and gofumpt * delete the map elements on finalize * manually call GC after the map shrink * add limit to SetBatch goroutine number * stop unnecesarry GC * increase eg limit to the MaxBatchSize * use ch to set batch bbolt * fix servers shutdown properly * use internal/kvs/bbolt * refactor * always use bbolt cache for correction * update sample.yaml for correction * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in 319ec8b according to the output from Prettier and Gofumpt. Details: #2152 * use go std slices pkg * refactor * add comment * remove valdsync * use vald errgroup * refactor * Define ErrNoAvailableAgentToInsert * update comment in English * Apply new actions yaml format * Disable godox * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in c860ddc according to the output from Prettier and Gofumpt. Details: #2194 * remove comment * Apply format * Add type check for type assertion * use const to specify filemode * Add bbolt concurrency as config * fix var style * Suppress linter * fix comment * add test template * Refactor parameters for index correction * Refactor config * Add corrector test * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in 004bf81 according to the output from Prettier and Gofumpt. Details: #2194 * Add timestamp check * Apply format * fix schema type * Fix DeepSource errors * Fix misspell * Add type check * Remove unused config * Fix DeepSource error * Add required go:build e2e tag * Remove memo * Refactor comment * Add index job correction helm templates * Add more fields * Add index correction job E2E test * Add e2e action for job * [REVERT THIS] Temporally change version * Fix name and command * Apply format * update crd * Revert "[REVERT THIS] Temporally change version" This reverts commit 1801a63. * Remove unused pkg * Remove experimental file * remove old workflow * Fix cron job name to new one * Update sample.yaml * fix build path * Fix corrector name * add e2e-jobs to slack notification * Update crds --------- Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
ykadowak
added a commit
that referenced
this pull request
Oct 16, 2023
…s for operator to deploy index correction (#2205) * implement the initail framework * add corrector configuration * add corrector logic * add build make command for index correction binary * add Dockerfile for index correction * add Docker image for index job correction * add timer * fix tag align * tmp * fix log * temporally implement two versions of correct function * set eg limit from config * add stream list concurrency config * implement index id caching * add config to use cache or not * style: Format code with prettier and gofumpt * refactor availableAddrs * add kvs range duration * add leftAgentAddrs for performance * Revert "add kvs range duration" This reverts commit 5b647be. * refactor * fix without cache bug * enable observability * refactor * SIGTERM after complete * add metrics server * add pcache * remove comment * [TEMP] use pcache * [TMP] use pcache * fix empty shard returns error * fix to use local map * [TMP] add prestop for pcache * [TEMP] add pcache config * style: Format code with prettier and gofumpt * [TEMP] add pcache log * fix map alloc size * [TMP] Add bbolt cache * update bbolt * fix bbolt bug * add bbolt test * [TEMP] use bbolt as persistent cache * style: Format code with prettier and gofumpt * add SetBatch to bbolt * use batch to write map to disk * style: Format code with prettier and gofumpt * delete the map elements on finalize * manually call GC after the map shrink * add limit to SetBatch goroutine number * stop unnecesarry GC * increase eg limit to the MaxBatchSize * use ch to set batch bbolt * fix servers shutdown properly * use internal/kvs/bbolt * refactor * always use bbolt cache for correction * update sample.yaml for correction * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in 319ec8b according to the output from Prettier and Gofumpt. Details: #2152 * use go std slices pkg * refactor * add comment * remove valdsync * use vald errgroup * refactor * Define ErrNoAvailableAgentToInsert * update comment in English * Apply new actions yaml format * Disable godox * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in c860ddc according to the output from Prettier and Gofumpt. Details: #2194 * remove comment * Apply format * Add type check for type assertion * use const to specify filemode * Add bbolt concurrency as config * fix var style * Suppress linter * fix comment * add test template * Refactor parameters for index correction * Refactor config * Add corrector test * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in 004bf81 according to the output from Prettier and Gofumpt. Details: #2194 * Add timestamp check * Apply format * fix schema type * Fix DeepSource errors * Fix misspell * Add type check * Remove unused config * Fix DeepSource error * Add required go:build e2e tag * Remove memo * Refactor comment * Add index job correction helm templates * Add more fields * Add index correction job E2E test * Add e2e action for job * [REVERT THIS] Temporally change version * Fix name and command * Apply format * update crd * Revert "[REVERT THIS] Temporally change version" This reverts commit 1801a63. * Remove unused pkg * Remove experimental file * remove old workflow * Fix cron job name to new one * Update sample.yaml * fix build path * Fix corrector name * add e2e-jobs to slack notification * Update crds * Add StreamListObject to LB * Add E2E for StreamListObject * Update error handling * Fix StreamListObject e2e verification * Add StreamListObject to LB * Add E2E for StreamListObject * Update error handling * Fix StreamListObject e2e verification * Update index correction e2e to verify correction result with StramListObject * Make it possible to deploy index correction cronjob from operator * Update operator manifests * Make schedule field empty so that a user has to specify manually * add default schedule of index correction --------- Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Merged
kmrmt
pushed a commit
that referenced
this pull request
Dec 12, 2023
* implement the initail framework * add corrector configuration * add corrector logic * add build make command for index correction binary * add Dockerfile for index correction * add Docker image for index job correction * add timer * fix tag align * tmp * fix log * temporally implement two versions of correct function * set eg limit from config * add stream list concurrency config * implement index id caching * add config to use cache or not * style: Format code with prettier and gofumpt * refactor availableAddrs * add kvs range duration * add leftAgentAddrs for performance * Revert "add kvs range duration" This reverts commit 5b647be. * refactor * fix without cache bug * enable observability * refactor * SIGTERM after complete * add metrics server * add pcache * remove comment * [TEMP] use pcache * [TMP] use pcache * fix empty shard returns error * fix to use local map * [TMP] add prestop for pcache * [TEMP] add pcache config * style: Format code with prettier and gofumpt * [TEMP] add pcache log * fix map alloc size * [TMP] Add bbolt cache * update bbolt * fix bbolt bug * add bbolt test * [TEMP] use bbolt as persistent cache * style: Format code with prettier and gofumpt * add SetBatch to bbolt * use batch to write map to disk * style: Format code with prettier and gofumpt * delete the map elements on finalize * manually call GC after the map shrink * add limit to SetBatch goroutine number * stop unnecesarry GC * increase eg limit to the MaxBatchSize * use ch to set batch bbolt * fix servers shutdown properly * use internal/kvs/bbolt * refactor * always use bbolt cache for correction * update sample.yaml for correction * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in 319ec8b according to the output from Prettier and Gofumpt. Details: #2152 * use go std slices pkg * refactor * add comment * remove valdsync * use vald errgroup * refactor * Define ErrNoAvailableAgentToInsert * update comment in English * Apply new actions yaml format * Disable godox * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in c860ddc according to the output from Prettier and Gofumpt. Details: #2194 * remove comment * Apply format * Add type check for type assertion * use const to specify filemode * Add bbolt concurrency as config * fix var style * Suppress linter * fix comment * add test template * Refactor parameters for index correction * Refactor config * Add corrector test * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in 004bf81 according to the output from Prettier and Gofumpt. Details: #2194 * Add timestamp check * Apply format * fix schema type * Fix DeepSource errors * Fix misspell * Add type check * Remove unused config * Fix DeepSource error * Add required go:build e2e tag * Remove memo * Refactor comment * Remove TODO comment that is already done * Remove unused config * Add comment to errors * change app name * replace filepath pkg with internal file replace filepath pkg with internal file refactor * Refactor refactor * Fix gRPC spelling * Remove memo --------- Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
kmrmt
pushed a commit
that referenced
this pull request
Dec 12, 2023
* implement the initail framework * add corrector configuration * add corrector logic * add build make command for index correction binary * add Dockerfile for index correction * add Docker image for index job correction * add timer * fix tag align * tmp * fix log * temporally implement two versions of correct function * set eg limit from config * add stream list concurrency config * implement index id caching * add config to use cache or not * style: Format code with prettier and gofumpt * refactor availableAddrs * add kvs range duration * add leftAgentAddrs for performance * Revert "add kvs range duration" This reverts commit 5b647be. * refactor * fix without cache bug * enable observability * refactor * SIGTERM after complete * add metrics server * add pcache * remove comment * [TEMP] use pcache * [TMP] use pcache * fix empty shard returns error * fix to use local map * [TMP] add prestop for pcache * [TEMP] add pcache config * style: Format code with prettier and gofumpt * [TEMP] add pcache log * fix map alloc size * [TMP] Add bbolt cache * update bbolt * fix bbolt bug * add bbolt test * [TEMP] use bbolt as persistent cache * style: Format code with prettier and gofumpt * add SetBatch to bbolt * use batch to write map to disk * style: Format code with prettier and gofumpt * delete the map elements on finalize * manually call GC after the map shrink * add limit to SetBatch goroutine number * stop unnecesarry GC * increase eg limit to the MaxBatchSize * use ch to set batch bbolt * fix servers shutdown properly * use internal/kvs/bbolt * refactor * always use bbolt cache for correction * update sample.yaml for correction * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in 319ec8b according to the output from Prettier and Gofumpt. Details: #2152 * use go std slices pkg * refactor * add comment * remove valdsync * use vald errgroup * refactor * Define ErrNoAvailableAgentToInsert * update comment in English * Apply new actions yaml format * Disable godox * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in c860ddc according to the output from Prettier and Gofumpt. Details: #2194 * remove comment * Apply format * Add type check for type assertion * use const to specify filemode * Add bbolt concurrency as config * fix var style * Suppress linter * fix comment * add test template * Refactor parameters for index correction * Refactor config * Add corrector test * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in 004bf81 according to the output from Prettier and Gofumpt. Details: #2194 * Add timestamp check * Apply format * fix schema type * Fix DeepSource errors * Fix misspell * Add type check * Remove unused config * Fix DeepSource error * Add required go:build e2e tag * Remove memo * Refactor comment * Add index job correction helm templates * Add more fields * Add index correction job E2E test * Add e2e action for job * [REVERT THIS] Temporally change version * Fix name and command * Apply format * update crd * Revert "[REVERT THIS] Temporally change version" This reverts commit 1801a63. * Remove unused pkg * Remove experimental file * remove old workflow * Fix cron job name to new one * Update sample.yaml * fix build path * Fix corrector name * add e2e-jobs to slack notification * Update crds --------- Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
kmrmt
pushed a commit
that referenced
this pull request
Dec 12, 2023
…s for operator to deploy index correction (#2205) * implement the initail framework * add corrector configuration * add corrector logic * add build make command for index correction binary * add Dockerfile for index correction * add Docker image for index job correction * add timer * fix tag align * tmp * fix log * temporally implement two versions of correct function * set eg limit from config * add stream list concurrency config * implement index id caching * add config to use cache or not * style: Format code with prettier and gofumpt * refactor availableAddrs * add kvs range duration * add leftAgentAddrs for performance * Revert "add kvs range duration" This reverts commit 5b647be. * refactor * fix without cache bug * enable observability * refactor * SIGTERM after complete * add metrics server * add pcache * remove comment * [TEMP] use pcache * [TMP] use pcache * fix empty shard returns error * fix to use local map * [TMP] add prestop for pcache * [TEMP] add pcache config * style: Format code with prettier and gofumpt * [TEMP] add pcache log * fix map alloc size * [TMP] Add bbolt cache * update bbolt * fix bbolt bug * add bbolt test * [TEMP] use bbolt as persistent cache * style: Format code with prettier and gofumpt * add SetBatch to bbolt * use batch to write map to disk * style: Format code with prettier and gofumpt * delete the map elements on finalize * manually call GC after the map shrink * add limit to SetBatch goroutine number * stop unnecesarry GC * increase eg limit to the MaxBatchSize * use ch to set batch bbolt * fix servers shutdown properly * use internal/kvs/bbolt * refactor * always use bbolt cache for correction * update sample.yaml for correction * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in 319ec8b according to the output from Prettier and Gofumpt. Details: #2152 * use go std slices pkg * refactor * add comment * remove valdsync * use vald errgroup * refactor * Define ErrNoAvailableAgentToInsert * update comment in English * Apply new actions yaml format * Disable godox * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in c860ddc according to the output from Prettier and Gofumpt. Details: #2194 * remove comment * Apply format * Add type check for type assertion * use const to specify filemode * Add bbolt concurrency as config * fix var style * Suppress linter * fix comment * add test template * Refactor parameters for index correction * Refactor config * Add corrector test * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in 004bf81 according to the output from Prettier and Gofumpt. Details: #2194 * Add timestamp check * Apply format * fix schema type * Fix DeepSource errors * Fix misspell * Add type check * Remove unused config * Fix DeepSource error * Add required go:build e2e tag * Remove memo * Refactor comment * Add index job correction helm templates * Add more fields * Add index correction job E2E test * Add e2e action for job * [REVERT THIS] Temporally change version * Fix name and command * Apply format * update crd * Revert "[REVERT THIS] Temporally change version" This reverts commit 1801a63. * Remove unused pkg * Remove experimental file * remove old workflow * Fix cron job name to new one * Update sample.yaml * fix build path * Fix corrector name * add e2e-jobs to slack notification * Update crds * Add StreamListObject to LB * Add E2E for StreamListObject * Update error handling * Fix StreamListObject e2e verification * Add StreamListObject to LB * Add E2E for StreamListObject * Update error handling * Fix StreamListObject e2e verification * Update index correction e2e to verify correction result with StramListObject * Make it possible to deploy index correction cronjob from operator * Update operator manifests * Make schedule field empty so that a user has to specify manually * add default schedule of index correction --------- Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
This PR implements an internal logic of index correction feature and adds Docker image for the correction job. The main logic exists in
service/corrector.go.Implementation of Helm templates and E2E tests will follow this PR.
Related Issue:
Versions:
Checklist:
Special notes for your reviewer: