|
| 1 | +# OpenProject-GitHub Two-Way Sync Service |
| 2 | + |
| 3 | +## Project Overview |
| 4 | + |
| 5 | +A Deno-based webhook service that maintains bidirectional synchronization between OpenProject work packages and GitHub repository issues. The service ensures both platforms stay in sync for title, description, assignee, and status fields. |
| 6 | + |
| 7 | +## Architecture |
| 8 | + |
| 9 | +### Service Type |
| 10 | +- **Event-driven webhook receiver** (not polling) |
| 11 | +- Listens for webhooks from both GitHub and OpenProject |
| 12 | +- Performs full reconciliation sync on startup |
| 13 | +- Runs as a persistent HTTP server |
| 14 | + |
| 15 | +### Technology Stack |
| 16 | +- **Runtime**: Deno (TypeScript) |
| 17 | +- **HTTP Server**: Deno's native HTTP server |
| 18 | +- **APIs**: GitHub REST API v3, OpenProject API v3 |
| 19 | +- **Environment**: Docker-ready, configurable via environment variables |
| 20 | + |
| 21 | +## Data Flow |
| 22 | + |
| 23 | +``` |
| 24 | +GitHub Issue ←→ Sync Service ←→ OpenProject Work Package |
| 25 | + ↓ ↓ |
| 26 | + Webhook Webhook |
| 27 | + ↓ ↓ |
| 28 | + HTTP Server (this service) |
| 29 | + ↓ |
| 30 | + Sync Logic |
| 31 | + ↓ |
| 32 | + API Calls to update both sides |
| 33 | +``` |
| 34 | + |
| 35 | +## Field Mapping |
| 36 | + |
| 37 | +### Synced Fields |
| 38 | + |
| 39 | +| Field | GitHub | OpenProject | Notes | |
| 40 | +|-------|--------|-------------|-------| |
| 41 | +| Title | `title` | `subject` | GitHub prefixed with `[OP#<id>]` | |
| 42 | +| Description | `body` | `description.raw` | Plain text/markdown | |
| 43 | +| Assignee | `assignee.login` | `assignee` (user ID) | Mapped via config | |
| 44 | +| Status | `state` (open/closed) | `status` | See status mapping below | |
| 45 | + |
| 46 | +### Status Mapping |
| 47 | + |
| 48 | +**GitHub → OpenProject:** |
| 49 | +- `open` → `new` (status ID to be determined from OP API) |
| 50 | +- `closed` → `closed` (status ID to be determined from OP API) |
| 51 | + |
| 52 | +**OpenProject → GitHub:** |
| 53 | +- `new`, `in specification`, `specified`, `developed`, `in testing`, `tested`, `test failed`, `on hold` → `open` |
| 54 | +- `closed`, `rejected` → `closed` |
| 55 | + |
| 56 | +### Assignee Mapping |
| 57 | + |
| 58 | +Configured via environment variable as comma-separated pairs: |
| 59 | + |
| 60 | +``` |
| 61 | +ASSIGNEE_MAP=github_user1:op_user_id1,github_user2:op_user_id2 |
| 62 | +``` |
| 63 | + |
| 64 | +Example: |
| 65 | +``` |
| 66 | +ASSIGNEE_MAP=johndoe:42,janedoe:123 |
| 67 | +``` |
| 68 | + |
| 69 | +If no mapping exists, the assignee field is not synced for that user. |
| 70 | + |
| 71 | +## Linking Strategy |
| 72 | + |
| 73 | +### OpenProject → GitHub |
| 74 | +- Use custom field "GitHub Issue" (integer field) to store GitHub issue number |
| 75 | +- Field stores the issue number only (e.g., `123` for issue #123) |
| 76 | + |
| 77 | +### GitHub → OpenProject |
| 78 | +- Prefix issue title with `[OP#<id>]` (e.g., `[OP#456] Fix login bug`) |
| 79 | +- Parse this prefix to identify linked work packages |
| 80 | + |
| 81 | +### Detecting Links |
| 82 | +1. For OP work packages: Check if "GitHub Issue" field is populated |
| 83 | +2. For GitHub issues: Check if title starts with `[OP#\d+]` pattern |
| 84 | + |
| 85 | +## Webhook Endpoints |
| 86 | + |
| 87 | +### Authentication |
| 88 | +Simple token-based auth using URL path parameter: |
| 89 | + |
| 90 | +``` |
| 91 | +POST /webhook/:token/github |
| 92 | +POST /webhook/:token/openproject |
| 93 | +``` |
| 94 | + |
| 95 | +The `:token` must match `SECRET_TOKEN` environment variable. |
| 96 | + |
| 97 | +### GitHub Webhook |
| 98 | +- **Events to subscribe**: `issues` (opened, edited, closed, reopened, assigned, unassigned) |
| 99 | +- **Payload**: Standard GitHub issue webhook payload |
| 100 | + |
| 101 | +### OpenProject Webhook |
| 102 | +- **Events to subscribe**: Work package created, updated |
| 103 | +- **Payload**: OpenProject webhook payload with work package data |
| 104 | + |
| 105 | +## Startup Behavior |
| 106 | + |
| 107 | +On service startup, perform full reconciliation sync: |
| 108 | + |
| 109 | +1. **Fetch all issues/work packages** for configured repo/project pairs |
| 110 | +2. **Identify unlinked items**: |
| 111 | + - GitHub issues without `[OP#]` prefix |
| 112 | + - OP work packages without "GitHub Issue" field populated |
| 113 | +3. **Create corresponding items**: |
| 114 | + - For unlinked GitHub issues → Create OP work package |
| 115 | + - For unlinked OP work packages → Create GitHub issue |
| 116 | +4. **Do NOT attempt to match** existing items (always create new counterparts) |
| 117 | + |
| 118 | +## Conflict Resolution |
| 119 | + |
| 120 | +**Last-write-wins strategy:** |
| 121 | +- Compare timestamps from both systems |
| 122 | +- Apply the most recent change |
| 123 | +- Timestamps to compare: |
| 124 | + - GitHub: `issue.updated_at` |
| 125 | + - OpenProject: `work_package.updatedAt` |
| 126 | + |
| 127 | +## Configuration |
| 128 | + |
| 129 | +### Environment Variables |
| 130 | + |
| 131 | +```bash |
| 132 | +# Authentication tokens |
| 133 | +GH_TOKEN=<github_pat> # GitHub Personal Access Token |
| 134 | +OP_TOKEN=<openproject_api_key> # OpenProject API Key |
| 135 | +SECRET_TOKEN=<webhook_secret> # Webhook URL authentication |
| 136 | + |
| 137 | +# Service URLs |
| 138 | +OP_URL=https://op.stoatinternal.com # OpenProject instance URL |
| 139 | + |
| 140 | +# Repository to Project mapping (comma-separated) |
| 141 | +REPO_PROJECT_MAP=stoatchat/for-web:8,stoatchat/my-repo:999 |
| 142 | + |
| 143 | +# User mapping (optional, comma-separated github_user:op_user_id pairs) |
| 144 | +ASSIGNEE_MAP=johndoe:42,janedoe:123 |
| 145 | + |
| 146 | +# OpenProject custom field ID for "GitHub Issue" field |
| 147 | +OP_GITHUB_ISSUE_FIELD=customField123 |
| 148 | +``` |
| 149 | + |
| 150 | +### Multiple Repository Support |
| 151 | + |
| 152 | +The `REPO_PROJECT_MAP` format: |
| 153 | +``` |
| 154 | +owner/repo:project_id,owner/repo2:project_id2 |
| 155 | +``` |
| 156 | + |
| 157 | +Each mapping creates a bidirectional sync between that GitHub repository and OpenProject project. |
| 158 | + |
| 159 | +## Error Handling |
| 160 | + |
| 161 | +### Strategy |
| 162 | +- **Log all errors** to stdout/stderr with context |
| 163 | +- **Do not retry** webhook processing (webhooks can be re-delivered) |
| 164 | +- **Continue processing** other items if one fails |
| 165 | +- **Return HTTP 500** if webhook processing fails entirely |
| 166 | + |
| 167 | +### Error Logging Format |
| 168 | +``` |
| 169 | +[ERROR] [timestamp] [source] Message |
| 170 | + Context: {additional context as JSON} |
| 171 | +``` |
| 172 | + |
| 173 | +## API Interactions |
| 174 | + |
| 175 | +### GitHub API |
| 176 | +- **Authentication**: Bearer token (`Authorization: Bearer ${GH_TOKEN}`) |
| 177 | +- **Endpoints used**: |
| 178 | + - `GET /repos/:owner/:repo/issues` - List issues |
| 179 | + - `GET /repos/:owner/:repo/issues/:number` - Get issue |
| 180 | + - `POST /repos/:owner/:repo/issues` - Create issue |
| 181 | + - `PATCH /repos/:owner/:repo/issues/:number` - Update issue |
| 182 | + |
| 183 | +### OpenProject API |
| 184 | +- **Authentication**: Basic auth (`Authorization: Basic ${btoa('apikey:' + OP_TOKEN)}`) |
| 185 | +- **Endpoints used**: |
| 186 | + - `GET /api/v3/projects/:id/work_packages` - List work packages |
| 187 | + - `GET /api/v3/work_packages/:id` - Get work package |
| 188 | + - `POST /api/v3/projects/:id/work_packages` - Create work package |
| 189 | + - `PATCH /api/v3/work_packages/:id` - Update work package |
| 190 | + - `GET /api/v3/statuses` - Get available statuses |
| 191 | + - `GET /api/v3/projects/:id/available_assignees` - Get assignees |
| 192 | + |
| 193 | +### Rate Limiting |
| 194 | +- Implement basic rate limit handling (retry with exponential backoff if 429 received) |
| 195 | +- GitHub: 5000 requests/hour for authenticated requests |
| 196 | +- OpenProject: Check response headers for limits |
| 197 | + |
| 198 | +## Implementation Phases |
| 199 | + |
| 200 | +### Phase 1: Core Infrastructure |
| 201 | +1. Set up Deno HTTP server with webhook endpoints |
| 202 | +2. Environment configuration loading |
| 203 | +3. Webhook authentication middleware |
| 204 | +4. Basic logging infrastructure |
| 205 | + |
| 206 | +### Phase 2: API Client Layer |
| 207 | +1. GitHub API client with authentication |
| 208 | +2. OpenProject API client with authentication |
| 209 | +3. Type definitions for both APIs |
| 210 | +4. Error handling for API calls |
| 211 | + |
| 212 | +### Phase 3: Data Mapping |
| 213 | +1. Status mapping logic (with OP status ID lookup on startup) |
| 214 | +2. Assignee mapping logic |
| 215 | +3. Title formatting (add/remove `[OP#]` prefix) |
| 216 | +4. Link detection utilities |
| 217 | + |
| 218 | +### Phase 4: Sync Logic |
| 219 | +1. GitHub → OpenProject sync function |
| 220 | +2. OpenProject → GitHub sync function |
| 221 | +3. Conflict resolution (timestamp comparison) |
| 222 | +4. Create missing items on both sides |
| 223 | + |
| 224 | +### Phase 5: Webhook Handlers |
| 225 | +1. GitHub webhook parser and handler |
| 226 | +2. OpenProject webhook parser and handler |
| 227 | +3. Event filtering (only process relevant events) |
| 228 | + |
| 229 | +### Phase 6: Startup Sync |
| 230 | +1. Fetch all issues/work packages for configured mappings |
| 231 | +2. Identify unlinked items |
| 232 | +3. Create corresponding items on both sides |
| 233 | +4. Link newly created items |
| 234 | + |
| 235 | +### Phase 7: Testing & Deployment |
| 236 | +1. Manual testing with real GitHub/OP instances |
| 237 | +2. Docker build and deployment |
| 238 | +3. Webhook registration on both platforms |
| 239 | +4. Monitoring and logging verification |
| 240 | + |
| 241 | +## File Structure |
| 242 | + |
| 243 | +``` |
| 244 | +/ |
| 245 | +├── main.ts # Entry point, HTTP server setup |
| 246 | +├── src/ |
| 247 | +│ ├── config.ts # Environment configuration parsing |
| 248 | +│ ├── server.ts # HTTP server and routing |
| 249 | +│ ├── middleware/ |
| 250 | +│ │ └── auth.ts # Webhook authentication |
| 251 | +│ ├── clients/ |
| 252 | +│ │ ├── github.ts # GitHub API client |
| 253 | +│ │ └── openproject.ts # OpenProject API client |
| 254 | +│ ├── types/ |
| 255 | +│ │ ├── github.ts # GitHub type definitions |
| 256 | +│ │ └── openproject.ts # OpenProject type definitions |
| 257 | +│ ├── mappers/ |
| 258 | +│ │ ├── status.ts # Status mapping logic |
| 259 | +│ │ ├── assignee.ts # Assignee mapping logic |
| 260 | +│ │ └── link.ts # Link detection/formatting |
| 261 | +│ ├── sync/ |
| 262 | +│ │ ├── github-to-op.ts # GitHub → OP sync |
| 263 | +│ │ ├── op-to-github.ts # OP → GitHub sync |
| 264 | +│ │ └── reconcile.ts # Startup reconciliation |
| 265 | +│ ├── handlers/ |
| 266 | +│ │ ├── github-webhook.ts # GitHub webhook handler |
| 267 | +│ │ └── openproject-webhook.ts # OP webhook handler |
| 268 | +│ └── utils/ |
| 269 | +│ ├── logger.ts # Logging utilities |
| 270 | +│ └── errors.ts # Error definitions |
| 271 | +├── deno.json # Deno configuration |
| 272 | +├── deno.lock # Dependency lock file |
| 273 | +├── .env.example # Example environment file |
| 274 | +├── Dockerfile # Docker build configuration |
| 275 | +├── CLAUDE.md # This file |
| 276 | +└── README.md # User documentation |
| 277 | +``` |
| 278 | + |
| 279 | +## TypeScript Type Definitions |
| 280 | + |
| 281 | +### Core Types |
| 282 | + |
| 283 | +```typescript |
| 284 | +// Configuration |
| 285 | +interface Config { |
| 286 | + githubToken: string; |
| 287 | + opToken: string; |
| 288 | + opUrl: string; |
| 289 | + secretToken: string; |
| 290 | + repoProjectMap: Map<string, number>; // "owner/repo" -> project_id |
| 291 | + assigneeMap: Map<string, number>; // github_username -> op_user_id |
| 292 | + opGithubIssueField: string; // Custom field ID |
| 293 | +} |
| 294 | + |
| 295 | +// Link information |
| 296 | +interface LinkedPair { |
| 297 | + githubRepo: string; |
| 298 | + githubIssueNumber: number; |
| 299 | + opProjectId: number; |
| 300 | + opWorkPackageId: number; |
| 301 | +} |
| 302 | + |
| 303 | +// Sync direction |
| 304 | +type SyncDirection = 'github-to-op' | 'op-to-github'; |
| 305 | +``` |
| 306 | + |
| 307 | +## Success Criteria |
| 308 | + |
| 309 | +The service is considered complete when: |
| 310 | + |
| 311 | +1. ✅ Webhooks from GitHub update OpenProject work packages |
| 312 | +2. ✅ Webhooks from OpenProject update GitHub issues |
| 313 | +3. ✅ All specified fields (title, description, assignee, status) sync correctly |
| 314 | +4. ✅ Status mapping works according to specification |
| 315 | +5. ✅ Assignee mapping works (with graceful handling of missing mappings) |
| 316 | +6. ✅ Links are maintained via custom field and title prefix |
| 317 | +7. ✅ Startup sync creates missing items on both sides |
| 318 | +8. ✅ Conflicts are resolved using last-write-wins |
| 319 | +9. ✅ Service runs reliably in Docker |
| 320 | +10. ✅ Errors are logged appropriately |
| 321 | + |
| 322 | +## Open Questions / Future Enhancements |
| 323 | + |
| 324 | +- **Deletion handling**: What happens when an issue/work package is deleted? |
| 325 | +- **Comment sync**: May be valuable in future |
| 326 | +- **Label/tag sync**: Could enhance categorization |
| 327 | +- **Bidirectional search**: Web UI to search linked items |
| 328 | +- **Metrics/monitoring**: Track sync success rates |
| 329 | +- **Database**: Consider storing sync state for better conflict resolution |
0 commit comments