Skip to content

Commit 20947cd

Browse files
authored
feat(cli): Add dotnet experimentframework CLI tool for configuration validation and diagnostics (#61)
1 parent e4a9a03 commit 20947cd

File tree

9 files changed

+1199
-0
lines changed

9 files changed

+1199
-0
lines changed

docs/user-guide/cli.md

Lines changed: 315 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,315 @@
1+
# CLI Tool
2+
3+
The ExperimentFramework CLI tool (`dotnet experimentframework`) provides commands for validating configurations, diagnosing issues, and exporting registration plans.
4+
5+
## Installation
6+
7+
Install the CLI tool as a global .NET tool:
8+
9+
```bash
10+
dotnet tool install -g ExperimentFramework.Cli
11+
```
12+
13+
Or install it locally in your project:
14+
15+
```bash
16+
dotnet tool install ExperimentFramework.Cli
17+
```
18+
19+
## Commands
20+
21+
### `doctor`
22+
23+
Validates experiment configuration and dependency injection wiring without starting the full application.
24+
25+
**Usage:**
26+
27+
```bash
28+
dotnet experimentframework doctor [options]
29+
```
30+
31+
**Options:**
32+
33+
- `--config <path>` - Path to the configuration file (JSON)
34+
- `--assembly <path>` - Path to the application assembly containing the host configuration
35+
36+
**Examples:**
37+
38+
```bash
39+
# Validate a configuration file
40+
dotnet experimentframework doctor --config appsettings.json
41+
42+
# Validate an assembly
43+
dotnet experimentframework doctor --assembly bin/Debug/net10.0/MyApp.dll
44+
45+
# Validate both configuration and assembly
46+
dotnet experimentframework doctor --config appsettings.json --assembly bin/Debug/net10.0/MyApp.dll
47+
```
48+
49+
**What it checks:**
50+
51+
- Configuration file syntax and schema compliance
52+
- Trial definitions (control/condition compatibility)
53+
- Duplicate condition keys
54+
- Missing required fields (service types, implementation types)
55+
- Selection provider registrations (when assembly is provided)
56+
57+
**Exit codes:**
58+
59+
- `0` - All checks passed
60+
- `1` - Validation errors found
61+
62+
**Example output:**
63+
64+
```
65+
ExperimentFramework Doctor
66+
==========================
67+
68+
Checking configuration file: appsettings.json
69+
✓ Configuration file is valid
70+
✓ Trial definitions are valid
71+
72+
✓ All checks passed!
73+
```
74+
75+
### `config validate`
76+
77+
Validates a configuration file against the ExperimentFramework schema.
78+
79+
**Usage:**
80+
81+
```bash
82+
dotnet experimentframework config validate <path>
83+
```
84+
85+
**Arguments:**
86+
87+
- `<path>` - Path to the configuration file to validate
88+
89+
**Examples:**
90+
91+
```bash
92+
# Validate a configuration file
93+
dotnet experimentframework config validate appsettings.json
94+
95+
# Validate a standalone experiment config
96+
dotnet experimentframework config validate experiments.json
97+
```
98+
99+
**What it checks:**
100+
101+
- JSON syntax
102+
- Required fields (serviceType, selectionMode, control)
103+
- Valid selection mode types
104+
- Valid decorator types
105+
- Duplicate condition keys
106+
- Error policy references
107+
- Activation settings
108+
109+
**Exit codes:**
110+
111+
- `0` - Configuration is valid
112+
- `1` - Validation errors found or file not found
113+
114+
**Example output (valid):**
115+
116+
```
117+
Validating configuration file: appsettings.json
118+
119+
✓ Configuration is valid
120+
```
121+
122+
**Example output (invalid):**
123+
124+
```
125+
Validating configuration file: appsettings.json
126+
127+
✗ Configuration validation failed with 2 error(s):
128+
129+
✗ [ERROR] trials[0].serviceType
130+
Service type is required
131+
132+
✗ [ERROR] trials[0].control.implementationType
133+
Implementation type is required
134+
```
135+
136+
### `plan export`
137+
138+
Exports a human-readable or JSON plan of experiment registrations and configuration.
139+
140+
**Usage:**
141+
142+
```bash
143+
dotnet experimentframework plan export --config <path> [options]
144+
```
145+
146+
**Options:**
147+
148+
- `--config <path>` (required) - Path to the configuration file (JSON)
149+
- `--format <json|text>` - Output format (default: text)
150+
- `--out <path>` - Output file path (default: stdout)
151+
152+
**Examples:**
153+
154+
```bash
155+
# Export plan to console (text format)
156+
dotnet experimentframework plan export --config appsettings.json
157+
158+
# Export plan to file (JSON format)
159+
dotnet experimentframework plan export --config appsettings.json --format json --out plan.json
160+
161+
# Export detailed text report
162+
dotnet experimentframework plan export --config experiments.json --format text --out plan.txt
163+
```
164+
165+
**Output includes:**
166+
167+
- Global decorators
168+
- Standalone trials with selection modes
169+
- Control and condition implementations
170+
- Named experiments with their trials
171+
172+
**Exit codes:**
173+
174+
- `0` - Plan exported successfully
175+
- `1` - Configuration file not found or parsing error
176+
177+
**Example output (text format):**
178+
179+
```
180+
Loading configuration from: appsettings.json
181+
✓ Configuration loaded successfully
182+
183+
=== ExperimentFramework Configuration Plan ===
184+
185+
Generated: 2026-01-15 04:07:27 UTC
186+
187+
--- Global Decorators (2) ---
188+
• logging
189+
• metrics
190+
191+
--- Standalone Trials (1) ---
192+
Service: IMyService
193+
Selection Mode: featureFlag
194+
Control: control -> MyControlService
195+
Conditions:
196+
- variant-a -> MyVariantAService
197+
- variant-b -> MyVariantBService
198+
199+
=== End of Plan ===
200+
```
201+
202+
**Example output (JSON format):**
203+
204+
```json
205+
{
206+
"generatedAt": "2026-01-15T04:07:27Z",
207+
"decorators": [
208+
{
209+
"type": "logging",
210+
"typeName": null
211+
},
212+
{
213+
"type": "metrics",
214+
"typeName": null
215+
}
216+
],
217+
"trials": [
218+
{
219+
"serviceType": "IMyService",
220+
"selectionMode": "featureFlag",
221+
"control": {
222+
"key": "control",
223+
"implementation": "MyControlService"
224+
},
225+
"conditions": [
226+
{
227+
"key": "variant-a",
228+
"implementation": "MyVariantAService"
229+
},
230+
{
231+
"key": "variant-b",
232+
"implementation": "MyVariantBService"
233+
}
234+
]
235+
}
236+
],
237+
"experiments": []
238+
}
239+
```
240+
241+
## Common Use Cases
242+
243+
### CI/CD Pipeline Validation
244+
245+
Add configuration validation to your CI/CD pipeline:
246+
247+
```bash
248+
# In your CI script
249+
dotnet experimentframework config validate appsettings.json
250+
if [ $? -ne 0 ]; then
251+
echo "Configuration validation failed!"
252+
exit 1
253+
fi
254+
```
255+
256+
### Pre-deployment Checks
257+
258+
Validate both configuration and assembly before deployment:
259+
260+
```bash
261+
dotnet build
262+
dotnet experimentframework doctor \
263+
--config appsettings.Production.json \
264+
--assembly bin/Release/net10.0/MyApp.dll
265+
```
266+
267+
### Documentation Generation
268+
269+
Export configuration plans for documentation:
270+
271+
```bash
272+
# Generate human-readable plan for documentation
273+
dotnet experimentframework plan export \
274+
--config appsettings.json \
275+
--format text \
276+
--out docs/experiment-plan.txt
277+
278+
# Generate JSON for tooling/analysis
279+
dotnet experimentframework plan export \
280+
--config appsettings.json \
281+
--format json \
282+
--out docs/experiment-plan.json
283+
```
284+
285+
### Troubleshooting
286+
287+
Use the doctor command to diagnose configuration issues:
288+
289+
```bash
290+
# Check what's wrong with your configuration
291+
dotnet experimentframework doctor --config appsettings.json
292+
293+
# The output will show specific errors with paths to fix
294+
```
295+
296+
## Tips
297+
298+
1. **Always validate before deployment** - Use `config validate` in your CI/CD pipeline to catch configuration errors early.
299+
300+
2. **Use deterministic output** - The `plan export` command generates deterministic output, making it suitable for diffing and version control.
301+
302+
3. **Check exit codes** - All commands exit with code 0 on success and 1 on failure, making them scriptable.
303+
304+
4. **Combine commands** - Use multiple commands in sequence for comprehensive validation:
305+
```bash
306+
dotnet experimentframework config validate config.json && \
307+
dotnet experimentframework doctor --config config.json && \
308+
dotnet experimentframework plan export --config config.json --format json --out plan.json
309+
```
310+
311+
## See Also
312+
313+
- [Configuration Guide](configuration.md) - Learn how to configure experiments
314+
- [Getting Started](getting-started.md) - Quick start guide
315+
- [Service Registration Safety](../SERVICE_REGISTRATION_SAFETY.md) - Understanding registration plans

docs/user-guide/toc.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@
77
href: configuration.md
88
- name: Core Concepts
99
href: core-concepts.md
10+
- name: CLI Tool
11+
href: cli.md
1012

1113
# Traffic Routing
1214
- name: Selection & Routing

0 commit comments

Comments
 (0)