[WIP] Feature/index/correction job #2152
Conversation
|
[WARNING:INTCFG] Changes in |
|
[CHATOPS:HELP] ChatOps commands.
|
github-actions
bot
added
area/internal
area/helm
size/XL
type/feature
Deploying with
|
| Latest commit: |
9cbf664
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://efbb2846.vald.pages.dev |
| Branch Preview URL: | https://feature-index-correction-job.vald.pages.dev |
ykadowak
force-pushed
the
feature/index/correction-job
branch
from
b899be6
to
bb037ab
Compare
Codecov ReportPatch has no changes to coverable lines.
📢 Thoughts on this report? Let us know!. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| // Indexer represent agent auto indexing service configuration | ||
| Corrector *config.Corrector `json:"corrector" yaml:"corrector"` | ||
|
|
||
| // FIXME: ここから読み込むときLB側の設定とのconsistencyをどう担保するのか |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/config/config.go:41: Line contains TODO/BUG/FIXME: "FIXME: ここから読み込むときLB側の設定とのconsistencyをどう担..." (godox)
|
|
||
| if res.GetVector() == nil { | ||
| st := res.GetStatus() | ||
| // TODO: errors.Join? |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:129: Line contains TODO/BUG/FIXME: "TODO: errors.Join?" (godox)
| }, | ||
| addrs, | ||
| ); err != nil { | ||
| // TODO: errors.Join? |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:143: Line contains TODO/BUG/FIXME: "TODO: errors.Join?" (godox)
| } | ||
|
|
||
| // when there are less replicas than the correct number, add the extra replicas | ||
| // TODO: refine this logic. pretty complicated |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:284: Line contains TODO/BUG/FIXME: "TODO: refine this logic. pretty complica..." (godox)
| log.Infof("replica shortage of vector %s. inserting to other agents...", | ||
| targetReplica.vec.GetId()) | ||
| if len(availableAddrs) == 0 { | ||
| // TODO: define errors in errors pkg |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:289: Line contains TODO/BUG/FIXME: "TODO: define errors in errors pkg" (godox)
| return vald.NewInsertClient(conn).Insert(ctx, &payload.Insert_Request{ | ||
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation | ||
| Config: &payload.Insert_Config{ |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
SkipStrictExistCheck, Filters are missing in Insert_Config (exhaustruct)
| func (c *correct) deleteObject(ctx context.Context, addr string, vector *payload.Object_Vector) error { | ||
| res, err := c.discoverer.GetClient(). | ||
| Do(grpc.WithGRPCMethod(ctx, "core.v1.Vald/Delete"), addr, func(ctx context.Context, conn *grpc.ClientConn, copts ...grpc.CallOption) (interface{}, error) { | ||
| return vald.NewRemoveClient(conn).Remove(ctx, &payload.Remove_Request{ |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
Config is missing in Remove_Request (exhaustruct)
| return cfg, &cfg.GlobalConfig, nil | ||
| }), | ||
| runner.WithDaemonInitializer(func(cfg interface{}) (runner.Runner, error) { | ||
| return usecase.New(cfg.(*config.Data)) |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
type assertion must be checked (forcetypeassert)
|
|
||
| // FIXME: 以下をやめてシンプルにStartを抜けたらself SIGTERMで終了させる方がいいかも | ||
| // その場合echは無視する | ||
| ech := make(chan error, 100) |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
mnd: Magic number: 100, in detected (gomnd)
| }) | ||
|
|
||
| latest := allReplicas[0] | ||
| latestTs := latest.vec.GetTimestamp() |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
ST1003: var latestTs should be latestTS (stylecheck)
| ENV APP_NAME correction | ||
|
|
||
| # skipcq: DOK-DL3008 | ||
| RUN apt-get update && apt-get install -y --no-install-recommends \ |
| } | ||
|
|
||
| func (c *run) Start(ctx context.Context) (<-chan error, error) { | ||
| // TODO: timeoutはconfigから指定 |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/usecase/corrector.go:102: Line contains TODO/BUG/FIXME: "TODO: timeoutはconfigから指定" (godox)
| end := time.Since(start) | ||
| log.Infof("correction finished in %v", end) | ||
|
|
||
| // FIXME: 以下をやめてシンプルにStartを抜けたらself SIGTERMで終了させる方がいいかも |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/usecase/corrector.go:116: Line contains TODO/BUG/FIXME: "FIXME: 以下をやめてシンプルにStartを抜けたらself SIGTERM..." (godox)
|
|
||
| if res.GetVector() == nil { | ||
| st := res.GetStatus() | ||
| // TODO: errors.Join? |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:136: Line contains TODO/BUG/FIXME: "TODO: errors.Join?" (godox)
| }, | ||
| addrs, | ||
| ); err != nil { | ||
| // TODO: errors.Join? |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:150: Line contains TODO/BUG/FIXME: "TODO: errors.Join?" (godox)
| } | ||
|
|
||
| // when there are less replicas than the correct number, add the extra replicas | ||
| // TODO: refine this logic. pretty complicated |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:291: Line contains TODO/BUG/FIXME: "TODO: refine this logic. pretty complica..." (godox)
| log.Infof("replica shortage of vector %s. inserting to other agents...", | ||
| targetReplica.vec.GetId()) | ||
| if len(availableAddrs) == 0 { | ||
| // TODO: define errors in errors pkg |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:296: Line contains TODO/BUG/FIXME: "TODO: define errors in errors pkg" (godox)
| } | ||
|
|
||
| // availableAddrsからdiff個選んでinsert処理する | ||
| // TODO: どのagentにinsertするのが最適化のロジックを考える |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:301: Line contains TODO/BUG/FIXME: "TODO:\u3000どのagentにinsertするのが最適化のロジックを考える..." (godox)
| Do(grpc.WithGRPCMethod(ctx, "core.v1.Vald/Update"), addr, func(ctx context.Context, conn *grpc.ClientConn, copts ...grpc.CallOption) (interface{}, error) { | ||
| return vald.NewUpdateClient(conn).Update(ctx, &payload.Update_Request{ | ||
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:356: Line contains TODO/BUG/FIXME: "FIXME: this should be deleted after Conf..." (godox)
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation | ||
| Config: &payload.Update_Config{ | ||
| // TODO: Decrementing because it's gonna be incremented befor being pushed |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:358: Line contains TODO/BUG/FIXME: "TODO: Decrementing because it's gonna be..." (godox)
| Do(grpc.WithGRPCMethod(ctx, "core.v1.Vald/Insert"), addr, func(ctx context.Context, conn *grpc.ClientConn, copts ...grpc.CallOption) (interface{}, error) { | ||
| return vald.NewInsertClient(conn).Insert(ctx, &payload.Insert_Request{ | ||
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:381: Line contains TODO/BUG/FIXME: "FIXME: this should be deleted after Conf..." (godox)
| } | ||
|
|
||
| func (c *correct) loadInfos(ctx context.Context) (err error) { | ||
| // FIXME: o11yは最後に整える |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:419: Line contains TODO/BUG/FIXME: "FIXME: o11yは最後に整える" (godox)
| end := time.Since(start) | ||
| log.Infof("correction finished in %v", end) | ||
|
|
||
| // FIXME: 以下をやめてシンプルにStartを抜けたらself SIGTERMで終了させる方がいいかも |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/usecase/corrector.go:120: Line contains TODO/BUG/FIXME: "FIXME: 以下をやめてシンプルにStartを抜けたらself SIGTERM..." (godox)
|
|
||
| if res.GetVector() == nil { | ||
| st := res.GetStatus() | ||
| // TODO: errors.Join? |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:140: Line contains TODO/BUG/FIXME: "TODO: errors.Join?" (godox)
| }, | ||
| addrs, | ||
| ); err != nil { | ||
| // TODO: errors.Join? |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:154: Line contains TODO/BUG/FIXME: "TODO: errors.Join?" (godox)
| } | ||
|
|
||
| // when there are less replicas than the correct number, add the extra replicas | ||
| // TODO: refine this logic. pretty complicated |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:298: Line contains TODO/BUG/FIXME: "TODO: refine this logic. pretty complica..." (godox)
| log.Infof("replica shortage of vector %s. inserting to other agents...", | ||
| targetReplica.vec.GetId()) | ||
| if len(availableAddrs) == 0 { | ||
| // TODO: define errors in errors pkg |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:303: Line contains TODO/BUG/FIXME: "TODO: define errors in errors pkg" (godox)
| } | ||
|
|
||
| // availableAddrsからdiff個選んでinsert処理する | ||
| // TODO: どのagentにinsertするのが最適化のロジックを考える |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:308: Line contains TODO/BUG/FIXME: "TODO:\u3000どのagentにinsertするのが最適化のロジックを考える..." (godox)
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation | ||
| Config: &payload.Update_Config{ | ||
| // TODO: Decrementing because it's gonna be incremented befor being pushed |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:365: Line contains TODO/BUG/FIXME: "TODO: Decrementing because it's gonna be..." (godox)
| Do(grpc.WithGRPCMethod(ctx, "core.v1.Vald/Insert"), addr, func(ctx context.Context, conn *grpc.ClientConn, copts ...grpc.CallOption) (interface{}, error) { | ||
| return vald.NewInsertClient(conn).Insert(ctx, &payload.Insert_Request{ | ||
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:388: Line contains TODO/BUG/FIXME: "FIXME: this should be deleted after Conf..." (godox)
| } | ||
|
|
||
| func (c *correct) loadInfos(ctx context.Context) (err error) { | ||
| // FIXME: o11yは最後に整える |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:426: Line contains TODO/BUG/FIXME: "FIXME: o11yは最後に整える" (godox)
| return err | ||
| } | ||
|
|
||
| log.Infof("starting to retreive index from agent(%s)", addr) |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶retreive is a misspelling of retrieve (misspell)
| } | ||
|
|
||
| func New(cfg *config.Data, discoverer discoverer.Client, eg errgroup.Group) (Corrector, error) { | ||
| return &correct{ |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
indexInfos, uuidsCount, uncommittedUUIDsCount are missing in correct (exhaustruct)
ykadowak
force-pushed
the
feature/index/correction-job
branch
from
19e7ba3
to
38336bf
Compare
|
|
||
| if res.GetVector() == nil { | ||
| st := res.GetStatus() | ||
| // TODO: errors.Join? |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:147: Line contains TODO/BUG/FIXME: "TODO: errors.Join?" (godox)
| }, | ||
| addrs, | ||
| ); err != nil { | ||
| // TODO: errors.Join? |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:162: Line contains TODO/BUG/FIXME: "TODO: errors.Join?" (godox)
| } | ||
|
|
||
| // when there are less replicas than the correct number, add the extra replicas | ||
| // TODO: refine this logic. pretty complicated |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:306: Line contains TODO/BUG/FIXME: "TODO: refine this logic. pretty complica..." (godox)
| log.Infof("replica shortage of vector %s. inserting to other agents...", | ||
| targetReplica.vec.GetId()) | ||
| if len(availableAddrs) == 0 { | ||
| // TODO: define errors in errors pkg |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:311: Line contains TODO/BUG/FIXME: "TODO: define errors in errors pkg" (godox)
| } | ||
|
|
||
| // availableAddrsからdiff個選んでinsert処理する | ||
| // TODO: どのagentにinsertするのが最適化のロジックを考える |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:316: Line contains TODO/BUG/FIXME: "TODO:\u3000どのagentにinsertするのが最適化のロジックを考える..." (godox)
| Do(grpc.WithGRPCMethod(ctx, "core.v1.Vald/Update"), addr, func(ctx context.Context, conn *grpc.ClientConn, copts ...grpc.CallOption) (interface{}, error) { | ||
| return vald.NewUpdateClient(conn).Update(ctx, &payload.Update_Request{ | ||
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:371: Line contains TODO/BUG/FIXME: "FIXME: this should be deleted after Conf..." (godox)
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation | ||
| Config: &payload.Update_Config{ | ||
| // TODO: Decrementing because it's gonna be incremented befor being pushed |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:373: Line contains TODO/BUG/FIXME: "TODO: Decrementing because it's gonna be..." (godox)
| Do(grpc.WithGRPCMethod(ctx, "core.v1.Vald/Insert"), addr, func(ctx context.Context, conn *grpc.ClientConn, copts ...grpc.CallOption) (interface{}, error) { | ||
| return vald.NewInsertClient(conn).Insert(ctx, &payload.Insert_Request{ | ||
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:396: Line contains TODO/BUG/FIXME: "FIXME: this should be deleted after Conf..." (godox)
| } | ||
|
|
||
| func (c *correct) loadInfos(ctx context.Context) (err error) { | ||
| // FIXME: o11yは最後に整える |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:434: Line contains TODO/BUG/FIXME: "FIXME: o11yは最後に整える" (godox)
| } | ||
|
|
||
| // when there are less replicas than the correct number, add the extra replicas | ||
| // TODO: refine this logic. pretty complicated |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:303: Line contains TODO/BUG/FIXME: "TODO: refine this logic. pretty complica..." (godox)
| log.Infof("replica shortage of vector %s. inserting to other agents...", | ||
| targetReplica.vec.GetId()) | ||
| if len(availableAddrs) == 0 { | ||
| // TODO: define errors in errors pkg |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:308: Line contains TODO/BUG/FIXME: "TODO: define errors in errors pkg" (godox)
| } | ||
|
|
||
| // availableAddrsからdiff個選んでinsert処理する | ||
| // TODO: どのagentにinsertするのが最適化のロジックを考える |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:313: Line contains TODO/BUG/FIXME: "TODO:\u3000どのagentにinsertするのが最適化のロジックを考える..." (godox)
| Do(grpc.WithGRPCMethod(ctx, "core.v1.Vald/Update"), addr, func(ctx context.Context, conn *grpc.ClientConn, copts ...grpc.CallOption) (interface{}, error) { | ||
| return vald.NewUpdateClient(conn).Update(ctx, &payload.Update_Request{ | ||
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:368: Line contains TODO/BUG/FIXME: "FIXME: this should be deleted after Conf..." (godox)
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation | ||
| Config: &payload.Update_Config{ | ||
| // TODO: Decrementing because it's gonna be incremented befor being pushed |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:370: Line contains TODO/BUG/FIXME: "TODO: Decrementing because it's gonna be..." (godox)
| Do(grpc.WithGRPCMethod(ctx, "core.v1.Vald/Insert"), addr, func(ctx context.Context, conn *grpc.ClientConn, copts ...grpc.CallOption) (interface{}, error) { | ||
| return vald.NewInsertClient(conn).Insert(ctx, &payload.Insert_Request{ | ||
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:393: Line contains TODO/BUG/FIXME: "FIXME: this should be deleted after Conf..." (godox)
| } | ||
|
|
||
| func (c *correct) loadInfos(ctx context.Context) (err error) { | ||
| // FIXME: o11yは最後に整える |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:431: Line contains TODO/BUG/FIXME: "FIXME: o11yは最後に整える" (godox)
| } | ||
|
|
||
| seg, ctx := stdeg.WithContext(ctx) | ||
| seg.SetLimit(100) // FIXME: server settingsをそのまま流用で良いのか？ |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:120: Line contains TODO/BUG/FIXME: "FIXME: server settingsをそのまま流用で良いのか？..." (godox)
| case <-streamEnd: | ||
| return nil | ||
| default: | ||
| // TODO: when vald internal errgroup is changed to block when eg limitation is reached, |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:145: Line contains TODO/BUG/FIXME: "TODO: when vald internal errgroup is cha..." (godox)
| }, | ||
| addrs, | ||
| ); err != nil { | ||
| // TODO: valdとstdでerrorの処理が違うので注意 |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:183: Line contains TODO/BUG/FIXME: "TODO: valdとstdでerrorの処理が違うので注意..." (godox)
| } | ||
|
|
||
| // when there are less replicas than the correct number, add the extra replicas | ||
| // TODO: refine this logic. pretty complicated |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:398: Line contains TODO/BUG/FIXME: "TODO: refine this logic. pretty complica..." (godox)
| log.Infof("replica shortage of vector %s. inserting to other agents...", | ||
| targetReplica.vec.GetId()) | ||
| if len(availableAddrs) == 0 { | ||
| // TODO: define errors in errors pkg |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:403: Line contains TODO/BUG/FIXME: "TODO: define errors in errors pkg" (godox)
| Do(grpc.WithGRPCMethod(ctx, "core.v1.Vald/Update"), addr, func(ctx context.Context, conn *grpc.ClientConn, copts ...grpc.CallOption) (interface{}, error) { | ||
| return vald.NewUpdateClient(conn).Update(ctx, &payload.Update_Request{ | ||
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:463: Line contains TODO/BUG/FIXME: "FIXME: this should be deleted after Conf..." (godox)
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation | ||
| Config: &payload.Update_Config{ | ||
| // TODO: Decrementing because it's gonna be incremented befor being pushed |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:465: Line contains TODO/BUG/FIXME: "TODO: Decrementing because it's gonna be..." (godox)
| Do(grpc.WithGRPCMethod(ctx, "core.v1.Vald/Insert"), addr, func(ctx context.Context, conn *grpc.ClientConn, copts ...grpc.CallOption) (interface{}, error) { | ||
| return vald.NewInsertClient(conn).Insert(ctx, &payload.Insert_Request{ | ||
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:488: Line contains TODO/BUG/FIXME: "FIXME: this should be deleted after Conf..." (godox)
| } | ||
|
|
||
| func (c *correct) loadInfos(ctx context.Context) (err error) { | ||
| // FIXME: o11yは最後に整える |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:526: Line contains TODO/BUG/FIXME: "FIXME: o11yは最後に整える" (godox)
| if err := c.discoverer.GetClient().OrderedRange(ctx, addrs, | ||
| func(ctx context.Context, addr string, conn *grpc.ClientConn, copts ...grpc.CallOption) error { | ||
| eg, ctx := errgroup.New(ctx) | ||
| eg.Limitation(100) |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
mnd: Magic number: 100, in detected (gomnd)
| } | ||
|
|
||
| seg, ctx := stdeg.WithContext(ctx) | ||
| seg.SetLimit(c.cfg.Server.GetGRPCStreamConcurrency()) // FIXME: server settingsをそのまま流用で良いのか？ |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:120: Line contains TODO/BUG/FIXME: "FIXME: server settingsをそのまま流用で良いのか？..." (godox)
|
|
||
| seg, ctx := stdeg.WithContext(ctx) | ||
| concurrency := c.cfg.Corrector.GetStreamListConcurrency() | ||
| seg.SetLimit(concurrency) // FIXME: server settingsをそのまま流用で良いのか？ |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:121: Line contains TODO/BUG/FIXME: "FIXME: server settingsをそのまま流用で良いのか？..." (godox)
| case <-streamEnd: | ||
| return nil | ||
| default: | ||
| // TODO: when vald internal errgroup is changed to block when eg limitation is reached, |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:148: Line contains TODO/BUG/FIXME: "TODO: when vald internal errgroup is cha..." (godox)
| }, | ||
| addrs, | ||
| ); err != nil { | ||
| // TODO: valdとstdでerrorの処理が違うので注意 |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:186: Line contains TODO/BUG/FIXME: "TODO: valdとstdでerrorの処理が違うので注意..." (godox)
| } | ||
|
|
||
| // when there are less replicas than the correct number, add the extra replicas | ||
| // TODO: refine this logic. pretty complicated |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:401: Line contains TODO/BUG/FIXME: "TODO: refine this logic. pretty complica..." (godox)
| log.Infof("replica shortage of vector %s. inserting to other agents...", | ||
| targetReplica.vec.GetId()) | ||
| if len(availableAddrs) == 0 { | ||
| // TODO: define errors in errors pkg |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:406: Line contains TODO/BUG/FIXME: "TODO: define errors in errors pkg" (godox)
| } | ||
|
|
||
| // availableAddrsからdiff個選んでinsert処理する | ||
| // TODO: どのagentにinsertするのが最適化のロジックを考える |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:411: Line contains TODO/BUG/FIXME: "TODO:\u3000どのagentにinsertするのが最適化のロジックを考える..." (godox)
| Do(grpc.WithGRPCMethod(ctx, "core.v1.Vald/Update"), addr, func(ctx context.Context, conn *grpc.ClientConn, copts ...grpc.CallOption) (interface{}, error) { | ||
| return vald.NewUpdateClient(conn).Update(ctx, &payload.Update_Request{ | ||
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:466: Line contains TODO/BUG/FIXME: "FIXME: this should be deleted after Conf..." (godox)
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation | ||
| Config: &payload.Update_Config{ | ||
| // TODO: Decrementing because it's gonna be incremented befor being pushed |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:468: Line contains TODO/BUG/FIXME: "TODO: Decrementing because it's gonna be..." (godox)
| Do(grpc.WithGRPCMethod(ctx, "core.v1.Vald/Insert"), addr, func(ctx context.Context, conn *grpc.ClientConn, copts ...grpc.CallOption) (interface{}, error) { | ||
| return vald.NewInsertClient(conn).Insert(ctx, &payload.Insert_Request{ | ||
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:491: Line contains TODO/BUG/FIXME: "FIXME: this should be deleted after Conf..." (godox)
| } | ||
|
|
||
| func (c *correct) loadInfos(ctx context.Context) (err error) { | ||
| // FIXME: o11yは最後に整える |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:529: Line contains TODO/BUG/FIXME: "FIXME: o11yは最後に整える" (godox)
| } | ||
|
|
||
| // when there are less replicas than the correct number, add the extra replicas | ||
| // TODO: refine this logic. pretty complicated |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:402: Line contains TODO/BUG/FIXME: "TODO: refine this logic. pretty complica..." (godox)
| log.Infof("replica shortage of vector %s. inserting to other agents...", | ||
| targetReplica.vec.GetId()) | ||
| if len(availableAddrs) == 0 { | ||
| // TODO: define errors in errors pkg |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:407: Line contains TODO/BUG/FIXME: "TODO: define errors in errors pkg" (godox)
| } | ||
|
|
||
| // availableAddrsからdiff個選んでinsert処理する | ||
| // TODO: どのagentにinsertするのが最適化のロジックを考える |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:412: Line contains TODO/BUG/FIXME: "TODO:\u3000どのagentにinsertするのが最適化のロジックを考える..." (godox)
| Do(grpc.WithGRPCMethod(ctx, "core.v1.Vald/Update"), addr, func(ctx context.Context, conn *grpc.ClientConn, copts ...grpc.CallOption) (interface{}, error) { | ||
| return vald.NewUpdateClient(conn).Update(ctx, &payload.Update_Request{ | ||
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:467: Line contains TODO/BUG/FIXME: "FIXME: this should be deleted after Conf..." (godox)
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation | ||
| Config: &payload.Update_Config{ | ||
| // TODO: Decrementing because it's gonna be incremented befor being pushed |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:469: Line contains TODO/BUG/FIXME: "TODO: Decrementing because it's gonna be..." (godox)
| Do(grpc.WithGRPCMethod(ctx, "core.v1.Vald/Insert"), addr, func(ctx context.Context, conn *grpc.ClientConn, copts ...grpc.CallOption) (interface{}, error) { | ||
| return vald.NewInsertClient(conn).Insert(ctx, &payload.Insert_Request{ | ||
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:492: Line contains TODO/BUG/FIXME: "FIXME: this should be deleted after Conf..." (godox)
| } | ||
|
|
||
| func (c *correct) loadInfos(ctx context.Context) (err error) { | ||
| // FIXME: o11yは最後に整える |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:530: Line contains TODO/BUG/FIXME: "FIXME: o11yは最後に整える" (godox)
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
ykadowak
force-pushed
the
feature/index/correction-job
branch
from
e941495
to
2f7d0b3
Compare
ykadowak
force-pushed
the
feature/index/correction-job
branch
from
cf9fa5f
to
16d2986
Compare
| case <-streamEnd: | ||
| return nil | ||
| default: | ||
| // TODO: when vald internal errgroup is changed to block when eg limitation is reached, |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:287: Line contains TODO/BUG/FIXME: "TODO: when vald internal errgroup is cha..." (godox)
| }, | ||
| leftAgentAddrs, | ||
| ); err != nil { | ||
| // TODO: valdとstdでerrorの処理が違うので注意 |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:340: Line contains TODO/BUG/FIXME: "TODO: valdとstdでerrorの処理が違うので注意..." (godox)
| return nil // continue other processes | ||
| } | ||
|
|
||
| // TODO: define error group |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:347: Line contains TODO/BUG/FIXME: "TODO: define error group" (godox)
| } | ||
|
|
||
| // when there are less replicas than the correct number, add the extra replicas | ||
| // TODO: refine this logic. pretty complicated |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:498: Line contains TODO/BUG/FIXME: "TODO: refine this logic. pretty complica..." (godox)
| log.Infof("replica shortage of vector %s. inserting to other agents...", | ||
| targetReplica.vec.GetId()) | ||
| if len(availableAddrs) == 0 { | ||
| // TODO: define errors in errors pkg |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:503: Line contains TODO/BUG/FIXME: "TODO: define errors in errors pkg" (godox)
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation | ||
| Config: &payload.Update_Config{ | ||
| // TODO: Decrementing because it's gonna be incremented befor being pushed |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:561: Line contains TODO/BUG/FIXME: "TODO: Decrementing because it's gonna be..." (godox)
| Do(grpc.WithGRPCMethod(ctx, "core.v1.Vald/Insert"), addr, func(ctx context.Context, conn *grpc.ClientConn, copts ...grpc.CallOption) (interface{}, error) { | ||
| return vald.NewInsertClient(conn).Insert(ctx, &payload.Insert_Request{ | ||
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:584: Line contains TODO/BUG/FIXME: "FIXME: this should be deleted after Conf..." (godox)
| d := filepath.Join(os.TempDir(), "bbolt") | ||
| file.MkdirAll(d, os.ModePerm) | ||
| dbfile := filepath.Join(d, "checkedid.db") | ||
| bolt, err := bbolt.New(dbfile, "", os.FileMode(0o600)) |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
mnd: Magic number: 0o600, in detected (gomnd)
| seg.SetLimit(concurrency) | ||
|
|
||
| bolteg, ctx := stdeg.WithContext(ctx) | ||
| bolteg.SetLimit(2048) |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
mnd: Magic number: 2048, in detected (gomnd)
| indexInfos valdsync.Map[string, *payload.Info_Index_Count] | ||
| uuidsCount uint32 | ||
| uncommittedUUIDsCount uint32 | ||
| checkedId bbolt.Bbolt |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
ST1003: struct field checkedId should be checkedID (stylecheck)
| case <-streamEnd: | ||
| return nil | ||
| default: | ||
| // TODO: when vald internal errgroup is changed to block when eg limitation is reached, |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:182: Line contains TODO/BUG/FIXME: "TODO: when vald internal errgroup is cha..." (godox)
| }, | ||
| leftAgentAddrs, | ||
| ); err != nil { | ||
| // TODO: valdとstdでerrorの処理が違うので注意 |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:235: Line contains TODO/BUG/FIXME: "TODO: valdとstdでerrorの処理が違うので注意..." (godox)
| return nil // continue other processes | ||
| } | ||
|
|
||
| // TODO: define error group |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:242: Line contains TODO/BUG/FIXME: "TODO: define error group" (godox)
| } | ||
|
|
||
| // when there are less replicas than the correct number, add the extra replicas | ||
| // TODO: refine this logic. pretty complicated |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:393: Line contains TODO/BUG/FIXME: "TODO: refine this logic. pretty complica..." (godox)
| log.Infof("replica shortage of vector %s. inserting to other agents...", | ||
| targetReplica.vec.GetId()) | ||
| if len(availableAddrs) == 0 { | ||
| // TODO: define errors in errors pkg |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:398: Line contains TODO/BUG/FIXME: "TODO: define errors in errors pkg" (godox)
| Do(grpc.WithGRPCMethod(ctx, "core.v1.Vald/Update"), addr, func(ctx context.Context, conn *grpc.ClientConn, copts ...grpc.CallOption) (interface{}, error) { | ||
| return vald.NewUpdateClient(conn).Update(ctx, &payload.Update_Request{ | ||
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:454: Line contains TODO/BUG/FIXME: "FIXME: this should be deleted after Conf..." (godox)
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation | ||
| Config: &payload.Update_Config{ | ||
| // TODO: Decrementing because it's gonna be incremented befor being pushed |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:456: Line contains TODO/BUG/FIXME: "TODO: Decrementing because it's gonna be..." (godox)
| Do(grpc.WithGRPCMethod(ctx, "core.v1.Vald/Insert"), addr, func(ctx context.Context, conn *grpc.ClientConn, copts ...grpc.CallOption) (interface{}, error) { | ||
| return vald.NewInsertClient(conn).Insert(ctx, &payload.Insert_Request{ | ||
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:479: Line contains TODO/BUG/FIXME: "FIXME: this should be deleted after Conf..." (godox)
| return nil, err | ||
| } | ||
|
|
||
| return &correct{ |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
service.correct is missing fields agentAddrs, indexInfos, uuidsCount, uncommittedUUIDsCount (exhaustruct)
| }, | ||
| leftAgentAddrs, | ||
| ); err != nil { | ||
| // TODO: valdとstdでerrorの処理が違うので注意 |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:232: Line contains TODO/BUG/FIXME: "TODO: valdとstdでerrorの処理が違うので注意..." (godox)
| } | ||
|
|
||
| // when there are less replicas than the correct number, add the extra replicas | ||
| // TODO: refine this logic. pretty complicated |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:391: Line contains TODO/BUG/FIXME: "TODO: refine this logic. pretty complica..." (godox)
| log.Infof("replica shortage of vector %s. inserting to other agents...", | ||
| targetReplica.vec.GetId()) | ||
| if len(availableAddrs) == 0 { | ||
| // TODO: define errors in errors pkg |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:396: Line contains TODO/BUG/FIXME: "TODO: define errors in errors pkg" (godox)
| func (c *correct) updateObject(ctx context.Context, addr string, vector *payload.Object_Vector) error { | ||
| res, err := c.discoverer.GetClient(). | ||
| Do(grpc.WithGRPCMethod(ctx, "core.v1.Vald/Update"), addr, func(ctx context.Context, conn *grpc.ClientConn, copts ...grpc.CallOption) (interface{}, error) { | ||
| // TODO: use UpdateTimestamp when it's implemented because here we just want to update only the timestamp but not the vector |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:450: Line contains TODO/BUG/FIXME: "TODO: use UpdateTimestamp when it's impl..." (godox)
| // TODO: use UpdateTimestamp when it's implemented because here we just want to update only the timestamp but not the vector | ||
| return vald.NewUpdateClient(conn).Update(ctx, &payload.Update_Request{ | ||
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:453: Line contains TODO/BUG/FIXME: "FIXME: this should be deleted after Conf..." (godox)
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation | ||
| Config: &payload.Update_Config{ | ||
| // TODO: Decrementing because it's gonna be incremented befor being pushed |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:455: Line contains TODO/BUG/FIXME: "TODO: Decrementing because it's gonna be..." (godox)
| Do(grpc.WithGRPCMethod(ctx, "core.v1.Vald/Insert"), addr, func(ctx context.Context, conn *grpc.ClientConn, copts ...grpc.CallOption) (interface{}, error) { | ||
| return vald.NewInsertClient(conn).Insert(ctx, &payload.Insert_Request{ | ||
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:478: Line contains TODO/BUG/FIXME: "FIXME: this should be deleted after Conf..." (godox)
| log.Infof("starting correction for agent %s, concurrency: %d", addr, concurrency) | ||
|
|
||
| // 事前にRecvすべき件数は事前にわからない。なぜなら処理中に新規でinsertされる可能性があるため | ||
| // TODO: そういうものはtimestampで判断して弾かないといけない |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:157: Line contains TODO/BUG/FIXME: "TODO: そういうものはtimestampで判断して弾かないといけない..." (godox)
| } | ||
|
|
||
| // when there are less replicas than the correct number, add the extra replicas | ||
| // TODO: refine this logic. pretty complicated |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:385: Line contains TODO/BUG/FIXME: "TODO: refine this logic. pretty complica..." (godox)
| log.Infof("replica shortage of vector %s. inserting to other agents...", | ||
| targetReplica.vec.GetId()) | ||
| if len(availableAddrs) == 0 { | ||
| // TODO: define errors in errors pkg |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:390: Line contains TODO/BUG/FIXME: "TODO: define errors in errors pkg" (godox)
| func (c *correct) updateObject(ctx context.Context, addr string, vector *payload.Object_Vector) error { | ||
| res, err := c.discoverer.GetClient(). | ||
| Do(grpc.WithGRPCMethod(ctx, "core.v1.Vald/Update"), addr, func(ctx context.Context, conn *grpc.ClientConn, copts ...grpc.CallOption) (interface{}, error) { | ||
| // TODO: use UpdateTimestamp when it's implemented because here we just want to update only the timestamp but not the vector |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:444: Line contains TODO/BUG/FIXME: "TODO: use UpdateTimestamp when it's impl..." (godox)
| // TODO: use UpdateTimestamp when it's implemented because here we just want to update only the timestamp but not the vector | ||
| return vald.NewUpdateClient(conn).Update(ctx, &payload.Update_Request{ | ||
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:447: Line contains TODO/BUG/FIXME: "FIXME: this should be deleted after Conf..." (godox)
| Id: targetReplica.vec.GetId(), | ||
| }, | ||
| }) | ||
|
|
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
File is not gofumpt-ed (gofumpt)
| return vald.NewUpdateClient(conn).Update(ctx, &payload.Update_Request{ | ||
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation | ||
| Config: &payload.Update_Config{ |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
payload.Update_Config is missing fields SkipStrictExistCheck, Filters, DisableBalancedUpdate (exhaustruct)
| return vald.NewInsertClient(conn).Insert(ctx, &payload.Insert_Request{ | ||
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation | ||
| Config: &payload.Insert_Config{ |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
payload.Insert_Config is missing fields SkipStrictExistCheck, Filters (exhaustruct)
| func (c *correct) deleteObject(ctx context.Context, addr string, vector *payload.Object_Vector) error { | ||
| res, err := c.discoverer.GetClient(). | ||
| Do(grpc.WithGRPCMethod(ctx, "core.v1.Vald/Delete"), addr, func(ctx context.Context, conn *grpc.ClientConn, copts ...grpc.CallOption) (interface{}, error) { | ||
| return vald.NewRemoveClient(conn).Remove(ctx, &payload.Remove_Request{ |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
payload.Remove_Request is missing field Config (exhaustruct)
| }) | ||
|
|
||
| latest := allReplicas[0] | ||
| latestTs := latest.vec.GetTimestamp() |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
ST1003: var latestTs should be latestTS (stylecheck)
| } | ||
|
|
||
| // when there are less replicas than the correct number, add the extra replicas | ||
| // TODO: refine this logic. pretty complicated |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:384: Line contains TODO/BUG/FIXME: "TODO: refine this logic. pretty complica..." (godox)
| log.Infof("replica shortage of vector %s. inserting to other agents...", | ||
| targetReplica.vec.GetId()) | ||
| if len(availableAddrs) == 0 { | ||
| // TODO: define errors in errors pkg |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:389: Line contains TODO/BUG/FIXME: "TODO: define errors in errors pkg" (godox)
| func (c *correct) updateObject(ctx context.Context, addr string, vector *payload.Object_Vector) error { | ||
| res, err := c.discoverer.GetClient(). | ||
| Do(grpc.WithGRPCMethod(ctx, "core.v1.Vald/Update"), addr, func(ctx context.Context, conn *grpc.ClientConn, copts ...grpc.CallOption) (interface{}, error) { | ||
| // TODO: use UpdateTimestamp when it's implemented because here we just want to update only the timestamp but not the vector |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:443: Line contains TODO/BUG/FIXME: "TODO: use UpdateTimestamp when it's impl..." (godox)
| // TODO: use UpdateTimestamp when it's implemented because here we just want to update only the timestamp but not the vector | ||
| return vald.NewUpdateClient(conn).Update(ctx, &payload.Update_Request{ | ||
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:446: Line contains TODO/BUG/FIXME: "FIXME: this should be deleted after Conf..." (godox)
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation | ||
| Config: &payload.Update_Config{ | ||
| // TODO: Decrementing because it's gonna be incremented befor being pushed |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:448: Line contains TODO/BUG/FIXME: "TODO: Decrementing because it's gonna be..." (godox)
| Do(grpc.WithGRPCMethod(ctx, "core.v1.Vald/Insert"), addr, func(ctx context.Context, conn *grpc.ClientConn, copts ...grpc.CallOption) (interface{}, error) { | ||
| return vald.NewInsertClient(conn).Insert(ctx, &payload.Insert_Request{ | ||
| Vector: vector, | ||
| // FIXME: this should be deleted after Config.Timestamp deprecation |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
pkg/index/job/correction/service/corrector.go:471: Line contains TODO/BUG/FIXME: "FIXME: this should be deleted after Conf..." (godox)
|
Will continue with new PR |
* implement the initail framework * add corrector configuration * add corrector logic * add build make command for index correction binary * add Dockerfile for index correction * add Docker image for index job correction * add timer * fix tag align * tmp * fix log * temporally implement two versions of correct function * set eg limit from config * add stream list concurrency config * implement index id caching * add config to use cache or not * style: Format code with prettier and gofumpt * refactor availableAddrs * add kvs range duration * add leftAgentAddrs for performance * Revert "add kvs range duration" This reverts commit 5b647be. * refactor * fix without cache bug * enable observability * refactor * SIGTERM after complete * add metrics server * add pcache * remove comment * [TEMP] use pcache * [TMP] use pcache * fix empty shard returns error * fix to use local map * [TMP] add prestop for pcache * [TEMP] add pcache config * style: Format code with prettier and gofumpt * [TEMP] add pcache log * fix map alloc size * [TMP] Add bbolt cache * update bbolt * fix bbolt bug * add bbolt test * [TEMP] use bbolt as persistent cache * style: Format code with prettier and gofumpt * add SetBatch to bbolt * use batch to write map to disk * style: Format code with prettier and gofumpt * delete the map elements on finalize * manually call GC after the map shrink * add limit to SetBatch goroutine number * stop unnecesarry GC * increase eg limit to the MaxBatchSize * use ch to set batch bbolt * fix servers shutdown properly * use internal/kvs/bbolt * refactor * always use bbolt cache for correction * update sample.yaml for correction * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in 319ec8b according to the output from Prettier and Gofumpt. Details: #2152 * use go std slices pkg * refactor * add comment * remove valdsync * use vald errgroup * refactor * Define ErrNoAvailableAgentToInsert * update comment in English * Apply new actions yaml format * Disable godox * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in c860ddc according to the output from Prettier and Gofumpt. Details: #2194 * remove comment * Apply format * Add type check for type assertion * use const to specify filemode * Add bbolt concurrency as config * fix var style * Suppress linter * fix comment * add test template * Refactor parameters for index correction * Refactor config * Add corrector test * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in 004bf81 according to the output from Prettier and Gofumpt. Details: #2194 * Add timestamp check * Apply format * fix schema type * Fix DeepSource errors * Fix misspell * Add type check * Remove unused config * Fix DeepSource error * Add required go:build e2e tag * Remove memo * Refactor comment * Remove TODO comment that is already done * Remove unused config * Add comment to errors * change app name * replace filepath pkg with internal file replace filepath pkg with internal file refactor * Refactor refactor * Fix gRPC spelling * Remove memo --------- Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
* implement the initail framework * add corrector configuration * add corrector logic * add build make command for index correction binary * add Dockerfile for index correction * add Docker image for index job correction * add timer * fix tag align * tmp * fix log * temporally implement two versions of correct function * set eg limit from config * add stream list concurrency config * implement index id caching * add config to use cache or not * style: Format code with prettier and gofumpt * refactor availableAddrs * add kvs range duration * add leftAgentAddrs for performance * Revert "add kvs range duration" This reverts commit 5b647be. * refactor * fix without cache bug * enable observability * refactor * SIGTERM after complete * add metrics server * add pcache * remove comment * [TEMP] use pcache * [TMP] use pcache * fix empty shard returns error * fix to use local map * [TMP] add prestop for pcache * [TEMP] add pcache config * style: Format code with prettier and gofumpt * [TEMP] add pcache log * fix map alloc size * [TMP] Add bbolt cache * update bbolt * fix bbolt bug * add bbolt test * [TEMP] use bbolt as persistent cache * style: Format code with prettier and gofumpt * add SetBatch to bbolt * use batch to write map to disk * style: Format code with prettier and gofumpt * delete the map elements on finalize * manually call GC after the map shrink * add limit to SetBatch goroutine number * stop unnecesarry GC * increase eg limit to the MaxBatchSize * use ch to set batch bbolt * fix servers shutdown properly * use internal/kvs/bbolt * refactor * always use bbolt cache for correction * update sample.yaml for correction * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in 319ec8b according to the output from Prettier and Gofumpt. Details: #2152 * use go std slices pkg * refactor * add comment * remove valdsync * use vald errgroup * refactor * Define ErrNoAvailableAgentToInsert * update comment in English * Apply new actions yaml format * Disable godox * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in c860ddc according to the output from Prettier and Gofumpt. Details: #2194 * remove comment * Apply format * Add type check for type assertion * use const to specify filemode * Add bbolt concurrency as config * fix var style * Suppress linter * fix comment * add test template * Refactor parameters for index correction * Refactor config * Add corrector test * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in 004bf81 according to the output from Prettier and Gofumpt. Details: #2194 * Add timestamp check * Apply format * fix schema type * Fix DeepSource errors * Fix misspell * Add type check * Remove unused config * Fix DeepSource error * Add required go:build e2e tag * Remove memo * Refactor comment * Add index job correction helm templates * Add more fields * Add index correction job E2E test * Add e2e action for job * [REVERT THIS] Temporally change version * Fix name and command * Apply format * update crd * Revert "[REVERT THIS] Temporally change version" This reverts commit 1801a63. * Remove unused pkg * Remove experimental file * remove old workflow * Fix cron job name to new one * Update sample.yaml * fix build path * Fix corrector name * add e2e-jobs to slack notification * Update crds --------- Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
…s for operator to deploy index correction (#2205) * implement the initail framework * add corrector configuration * add corrector logic * add build make command for index correction binary * add Dockerfile for index correction * add Docker image for index job correction * add timer * fix tag align * tmp * fix log * temporally implement two versions of correct function * set eg limit from config * add stream list concurrency config * implement index id caching * add config to use cache or not * style: Format code with prettier and gofumpt * refactor availableAddrs * add kvs range duration * add leftAgentAddrs for performance * Revert "add kvs range duration" This reverts commit 5b647be. * refactor * fix without cache bug * enable observability * refactor * SIGTERM after complete * add metrics server * add pcache * remove comment * [TEMP] use pcache * [TMP] use pcache * fix empty shard returns error * fix to use local map * [TMP] add prestop for pcache * [TEMP] add pcache config * style: Format code with prettier and gofumpt * [TEMP] add pcache log * fix map alloc size * [TMP] Add bbolt cache * update bbolt * fix bbolt bug * add bbolt test * [TEMP] use bbolt as persistent cache * style: Format code with prettier and gofumpt * add SetBatch to bbolt * use batch to write map to disk * style: Format code with prettier and gofumpt * delete the map elements on finalize * manually call GC after the map shrink * add limit to SetBatch goroutine number * stop unnecesarry GC * increase eg limit to the MaxBatchSize * use ch to set batch bbolt * fix servers shutdown properly * use internal/kvs/bbolt * refactor * always use bbolt cache for correction * update sample.yaml for correction * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in 319ec8b according to the output from Prettier and Gofumpt. Details: #2152 * use go std slices pkg * refactor * add comment * remove valdsync * use vald errgroup * refactor * Define ErrNoAvailableAgentToInsert * update comment in English * Apply new actions yaml format * Disable godox * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in c860ddc according to the output from Prettier and Gofumpt. Details: #2194 * remove comment * Apply format * Add type check for type assertion * use const to specify filemode * Add bbolt concurrency as config * fix var style * Suppress linter * fix comment * add test template * Refactor parameters for index correction * Refactor config * Add corrector test * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in 004bf81 according to the output from Prettier and Gofumpt. Details: #2194 * Add timestamp check * Apply format * fix schema type * Fix DeepSource errors * Fix misspell * Add type check * Remove unused config * Fix DeepSource error * Add required go:build e2e tag * Remove memo * Refactor comment * Add index job correction helm templates * Add more fields * Add index correction job E2E test * Add e2e action for job * [REVERT THIS] Temporally change version * Fix name and command * Apply format * update crd * Revert "[REVERT THIS] Temporally change version" This reverts commit 1801a63. * Remove unused pkg * Remove experimental file * remove old workflow * Fix cron job name to new one * Update sample.yaml * fix build path * Fix corrector name * add e2e-jobs to slack notification * Update crds * Add StreamListObject to LB * Add E2E for StreamListObject * Update error handling * Fix StreamListObject e2e verification * Add StreamListObject to LB * Add E2E for StreamListObject * Update error handling * Fix StreamListObject e2e verification * Update index correction e2e to verify correction result with StramListObject * Make it possible to deploy index correction cronjob from operator * Update operator manifests * Make schedule field empty so that a user has to specify manually * add default schedule of index correction --------- Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
* implement the initail framework * add corrector configuration * add corrector logic * add build make command for index correction binary * add Dockerfile for index correction * add Docker image for index job correction * add timer * fix tag align * tmp * fix log * temporally implement two versions of correct function * set eg limit from config * add stream list concurrency config * implement index id caching * add config to use cache or not * style: Format code with prettier and gofumpt * refactor availableAddrs * add kvs range duration * add leftAgentAddrs for performance * Revert "add kvs range duration" This reverts commit 5b647be. * refactor * fix without cache bug * enable observability * refactor * SIGTERM after complete * add metrics server * add pcache * remove comment * [TEMP] use pcache * [TMP] use pcache * fix empty shard returns error * fix to use local map * [TMP] add prestop for pcache * [TEMP] add pcache config * style: Format code with prettier and gofumpt * [TEMP] add pcache log * fix map alloc size * [TMP] Add bbolt cache * update bbolt * fix bbolt bug * add bbolt test * [TEMP] use bbolt as persistent cache * style: Format code with prettier and gofumpt * add SetBatch to bbolt * use batch to write map to disk * style: Format code with prettier and gofumpt * delete the map elements on finalize * manually call GC after the map shrink * add limit to SetBatch goroutine number * stop unnecesarry GC * increase eg limit to the MaxBatchSize * use ch to set batch bbolt * fix servers shutdown properly * use internal/kvs/bbolt * refactor * always use bbolt cache for correction * update sample.yaml for correction * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in 319ec8b according to the output from Prettier and Gofumpt. Details: #2152 * use go std slices pkg * refactor * add comment * remove valdsync * use vald errgroup * refactor * Define ErrNoAvailableAgentToInsert * update comment in English * Apply new actions yaml format * Disable godox * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in c860ddc according to the output from Prettier and Gofumpt. Details: #2194 * remove comment * Apply format * Add type check for type assertion * use const to specify filemode * Add bbolt concurrency as config * fix var style * Suppress linter * fix comment * add test template * Refactor parameters for index correction * Refactor config * Add corrector test * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in 004bf81 according to the output from Prettier and Gofumpt. Details: #2194 * Add timestamp check * Apply format * fix schema type * Fix DeepSource errors * Fix misspell * Add type check * Remove unused config * Fix DeepSource error * Add required go:build e2e tag * Remove memo * Refactor comment * Remove TODO comment that is already done * Remove unused config * Add comment to errors * change app name * replace filepath pkg with internal file replace filepath pkg with internal file refactor * Refactor refactor * Fix gRPC spelling * Remove memo --------- Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
* implement the initail framework * add corrector configuration * add corrector logic * add build make command for index correction binary * add Dockerfile for index correction * add Docker image for index job correction * add timer * fix tag align * tmp * fix log * temporally implement two versions of correct function * set eg limit from config * add stream list concurrency config * implement index id caching * add config to use cache or not * style: Format code with prettier and gofumpt * refactor availableAddrs * add kvs range duration * add leftAgentAddrs for performance * Revert "add kvs range duration" This reverts commit 5b647be. * refactor * fix without cache bug * enable observability * refactor * SIGTERM after complete * add metrics server * add pcache * remove comment * [TEMP] use pcache * [TMP] use pcache * fix empty shard returns error * fix to use local map * [TMP] add prestop for pcache * [TEMP] add pcache config * style: Format code with prettier and gofumpt * [TEMP] add pcache log * fix map alloc size * [TMP] Add bbolt cache * update bbolt * fix bbolt bug * add bbolt test * [TEMP] use bbolt as persistent cache * style: Format code with prettier and gofumpt * add SetBatch to bbolt * use batch to write map to disk * style: Format code with prettier and gofumpt * delete the map elements on finalize * manually call GC after the map shrink * add limit to SetBatch goroutine number * stop unnecesarry GC * increase eg limit to the MaxBatchSize * use ch to set batch bbolt * fix servers shutdown properly * use internal/kvs/bbolt * refactor * always use bbolt cache for correction * update sample.yaml for correction * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in 319ec8b according to the output from Prettier and Gofumpt. Details: #2152 * use go std slices pkg * refactor * add comment * remove valdsync * use vald errgroup * refactor * Define ErrNoAvailableAgentToInsert * update comment in English * Apply new actions yaml format * Disable godox * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in c860ddc according to the output from Prettier and Gofumpt. Details: #2194 * remove comment * Apply format * Add type check for type assertion * use const to specify filemode * Add bbolt concurrency as config * fix var style * Suppress linter * fix comment * add test template * Refactor parameters for index correction * Refactor config * Add corrector test * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in 004bf81 according to the output from Prettier and Gofumpt. Details: #2194 * Add timestamp check * Apply format * fix schema type * Fix DeepSource errors * Fix misspell * Add type check * Remove unused config * Fix DeepSource error * Add required go:build e2e tag * Remove memo * Refactor comment * Add index job correction helm templates * Add more fields * Add index correction job E2E test * Add e2e action for job * [REVERT THIS] Temporally change version * Fix name and command * Apply format * update crd * Revert "[REVERT THIS] Temporally change version" This reverts commit 1801a63. * Remove unused pkg * Remove experimental file * remove old workflow * Fix cron job name to new one * Update sample.yaml * fix build path * Fix corrector name * add e2e-jobs to slack notification * Update crds --------- Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
…s for operator to deploy index correction (#2205) * implement the initail framework * add corrector configuration * add corrector logic * add build make command for index correction binary * add Dockerfile for index correction * add Docker image for index job correction * add timer * fix tag align * tmp * fix log * temporally implement two versions of correct function * set eg limit from config * add stream list concurrency config * implement index id caching * add config to use cache or not * style: Format code with prettier and gofumpt * refactor availableAddrs * add kvs range duration * add leftAgentAddrs for performance * Revert "add kvs range duration" This reverts commit 5b647be. * refactor * fix without cache bug * enable observability * refactor * SIGTERM after complete * add metrics server * add pcache * remove comment * [TEMP] use pcache * [TMP] use pcache * fix empty shard returns error * fix to use local map * [TMP] add prestop for pcache * [TEMP] add pcache config * style: Format code with prettier and gofumpt * [TEMP] add pcache log * fix map alloc size * [TMP] Add bbolt cache * update bbolt * fix bbolt bug * add bbolt test * [TEMP] use bbolt as persistent cache * style: Format code with prettier and gofumpt * add SetBatch to bbolt * use batch to write map to disk * style: Format code with prettier and gofumpt * delete the map elements on finalize * manually call GC after the map shrink * add limit to SetBatch goroutine number * stop unnecesarry GC * increase eg limit to the MaxBatchSize * use ch to set batch bbolt * fix servers shutdown properly * use internal/kvs/bbolt * refactor * always use bbolt cache for correction * update sample.yaml for correction * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in 319ec8b according to the output from Prettier and Gofumpt. Details: #2152 * use go std slices pkg * refactor * add comment * remove valdsync * use vald errgroup * refactor * Define ErrNoAvailableAgentToInsert * update comment in English * Apply new actions yaml format * Disable godox * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in c860ddc according to the output from Prettier and Gofumpt. Details: #2194 * remove comment * Apply format * Add type check for type assertion * use const to specify filemode * Add bbolt concurrency as config * fix var style * Suppress linter * fix comment * add test template * Refactor parameters for index correction * Refactor config * Add corrector test * style: format code with Prettier and Gofumpt This commit fixes the style issues introduced in 004bf81 according to the output from Prettier and Gofumpt. Details: #2194 * Add timestamp check * Apply format * fix schema type * Fix DeepSource errors * Fix misspell * Add type check * Remove unused config * Fix DeepSource error * Add required go:build e2e tag * Remove memo * Refactor comment * Add index job correction helm templates * Add more fields * Add index correction job E2E test * Add e2e action for job * [REVERT THIS] Temporally change version * Fix name and command * Apply format * update crd * Revert "[REVERT THIS] Temporally change version" This reverts commit 1801a63. * Remove unused pkg * Remove experimental file * remove old workflow * Fix cron job name to new one * Update sample.yaml * fix build path * Fix corrector name * add e2e-jobs to slack notification * Update crds * Add StreamListObject to LB * Add E2E for StreamListObject * Update error handling * Fix StreamListObject e2e verification * Add StreamListObject to LB * Add E2E for StreamListObject * Update error handling * Fix StreamListObject e2e verification * Update index correction e2e to verify correction result with StramListObject * Make it possible to deploy index correction cronjob from operator * Update operator manifests * Make schedule field empty so that a user has to specify manually * add default schedule of index correction --------- Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Description:
Related Issue:
Versions:
Checklist:
Special notes for your reviewer: