fix cloud provider #191
fix cloud provider #191
Conversation
fix task parsing
WalkthroughThe changes update validation logic in the MongoDB task repository to strip unknown properties and relax strictness, and modify the task queue repository to use a hardcoded cloud provider value ('gcp') instead of dynamically referencing it from the task payload. No structural or control flow changes are introduced. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant TaskQueueBullMQRepository
participant JobNode
User->>TaskQueueBullMQRepository: Submit RDB import/export task
TaskQueueBullMQRepository->>JobNode: Enqueue job with cloudProvider='gcp'
JobNode-->>TaskQueueBullMQRepository: Job status/progress
TaskQueueBullMQRepository-->>User: Progress/notification updates
Assessment against linked issues
Assessment against linked issues: Out-of-scope changesNo out-of-scope changes found. Poem
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
Tofu Plan Output - observability_stack_ctrl_plane_infra |
Tofu Plan Output - observability_stack_ctrl_plane_k8s |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
backend/services/db-importer-worker/src/repositories/tasks/TasksDBMongoRepository.ts(1 hunks)backend/services/db-importer/src/repositories/tasksQueue/TaskQueueBullMQRepository.ts(6 hunks)
🧰 Additional context used
🧬 Code Graph Analysis (1)
backend/services/db-importer-worker/src/repositories/tasks/TasksDBMongoRepository.ts (1)
backend/services/db-importer-worker/src/schemas/rdb-task.ts (2)
RDBTask(87-120)RDBTaskType(122-122)
🔇 Additional comments (2)
backend/services/db-importer/src/repositories/tasksQueue/TaskQueueBullMQRepository.ts (2)
214-214:✅ Verification successful
Verify hardcoded cloud provider for control plane operations.
The multi-shard export flow now uses hardcoded
'gcp'instead oftask.payload.cloudProviderfor control plane operations (RDB merge monitoring and requesting). This suggests these operations always run on GCP regardless of the original task's cloud provider.Please verify that this hardcoding aligns with the system architecture where control plane operations are exclusively on GCP:
Also applies to: 229-229
🏁 Script executed:
#!/bin/bash # Description: Verify control plane operations and cloud provider usage patterns # Expected: Confirm control plane jobs always use GCP # Search for other control plane related cloud provider usage rg -A 3 -B 3 "CTRL_PLANE" --type ts # Search for cloudProvider usage in schemas to understand expected values rg -A 5 -B 5 "cloudProvider.*gcp|gcp.*cloudProvider" --type tsLength of output: 83097
Control plane operations correctly hardcode GCP
I’ve verified that every control-plane job node in
TaskQueueBullMQRepository.ts—and its associated tests—consistently uses'gcp'forcloudProvider, matching the architecture where control-plane tasks run exclusively on GCP. No further changes are needed.
393-393:✅ Verification successful
Verify hardcoded cloud provider for validation jobs.
The import RDB flow now uses hardcoded
'gcp'for format and size validation jobs that run on the control plane. This change is consistent with the export flow modifications.Please confirm that validation jobs (format and size validation) are designed to run exclusively on the GCP control plane:
Also applies to: 410-410, 430-430, 448-448
🏁 Script executed:
#!/bin/bash # Description: Verify validation job schemas expect GCP cloud provider # Expected: Validation job schemas should accept 'gcp' as cloudProvider # Search for validation job schemas and their cloudProvider requirements ast-grep --pattern 'RdbImportValidateRDBFormat: $_' ast-grep --pattern 'RdbImportValidateRDBSize: $_' ast-grep --pattern 'RdbImportMonitorFormatValidationProgress: $_' ast-grep --pattern 'RdbImportMonitorSizeValidationProgress: $_'Length of output: 239
I’m not seeing any schema definitions for those job types—let’s locate where they’re declared and how
cloudProvideris consumed. Running these searches should clarify whether they accept only'gcp':
🏁 Script executed:
#!/bin/bash # Find references/definitions of the validation jobs rg -n 'RdbImportValidateRDBFormat' -C5 rg -n 'RdbImportValidateRDBSize' -C5 rg -n 'RdbImportMonitorFormatValidationProgress' -C5 rg -n 'RdbImportMonitorSizeValidationProgress' -C5 # Inspect the enqueue calls in TaskQueueBullMQRepository around where cloudProvider is set rg -n "cloudProvider" -C3 backend/services/db-importer/src/repositories/tasksQueue/TaskQueueBullMQRepository.ts sed -n '380,480p' backend/services/db-importer/src/repositories/tasksQueue/TaskQueueBullMQRepository.tsLength of output: 64580
Validation jobs scheduled on GCP control plane
I’ve confirmed that all four validation tasks in TaskQueueBullMQRepository.ts are explicitly queued with
cloudProvider: 'gcp' clusterId: process.env.CTRL_PLANE_CLUSTER_ID region: process.env.CTRL_PLANE_REGIONand nested under the same control-plane environment (including
projectIdandnamespacewhere applicable). Their corresponding TypeBox schemas (RdbImportValidateRDBFormatProcessorDataSchema,RdbImportValidateRDBSizeProcessorDataSchema,RdbImportMonitorFormatValidationProgressProcessorDataSchema,RdbImportMonitorSizeValidationProgressProcessorDataSchema) all useSupportedCloudProviderSchema, which includes'gcp'.No changes needed here.
| return RDBTask.validateSync(task, { | ||
| stripUnknown: true, | ||
| strict: false | ||
| }) as RDBTaskType; |
There was a problem hiding this comment.
💡 Verification agent
🧩 Analysis chain
Approve relaxed validation to handle legacy data.
The change to use { stripUnknown: true, strict: false } makes task validation more permissive, which should resolve parsing issues with existing database records that contain extra fields not defined in the current schema.
Please verify that this change doesn't introduce data integrity risks by ensuring critical fields are still validated:
🏁 Script executed:
#!/bin/bash
# Description: Verify the RDBTask schema still validates required fields properly
# Expected: Required fields should still be enforced despite relaxed validation
# Check the RDBTask schema definition for required fields
rg -A 20 -B 5 "export const RDBTask.*=.*Yup\.object" backend/services/db-importer-worker/src/schemas/
# Look for any test cases that verify validation behavior
fd "test.*task.*ts$|spec.*task.*ts$" --exec rg -l "validateSync|RDBTask" {}Length of output: 2647
Relaxed validation approved; ensure required fields are enforced
Verification confirms that the RDBTask schema still marks all critical properties as required:
taskId(required)type(one ofTaskTypes, required)createdAt(required)updatedAt(required)status(default + required)
However, there are no existing tests exercising RDBTask.validateSync to verify that missing required fields still cause validation errors under the new { stripUnknown: true, strict: false } options.
Please add or update unit tests to confirm that:
- A task missing any required field is rejected by
validateSync. - Unknown or extra properties are stripped without impacting required-field enforcement.
🤖 Prompt for AI Agents
In
backend/services/db-importer-worker/src/repositories/tasks/TasksDBMongoRepository.ts
around lines 33 to 36, the validation call uses relaxed options { stripUnknown:
true, strict: false } which may allow missing required fields to pass unnoticed.
To fix this, add or update unit tests for RDBTask.validateSync that verify tasks
missing any required fields are rejected and that extra unknown properties are
stripped without affecting required field validation. This ensures data
integrity while allowing legacy data compatibility.
fix task parsing
fix #172
Summary by CodeRabbit