Improved prompting for agents

[nbren12]

Cursor struggles with using this API, even on the example prompts listed in the README. It makes many formatting error and struggles to debug the options. I asked it why and it complained about the lack of clear schema for graphql queries---I've had the same experience TBH when using the python API.

It performs better with the guide below. I had it generate this based on the wandb code base. Maybe something like this should be baked in to the MCP as a default.

Details

# Guide to Using W&B GraphQL API via MCP

Key Learnings from wandb GitHub Repository

1. Order Parameter Syntax

From wandb/apis/public/runs.py (line 237-239):

order: (str) Order can be `created_at`, `heartbeat_at`, `config.*.value`, or `summary_metrics.*`.
    If you prepend order with a + order is ascending (default).
    If you prepend order with a - order is descending.

Examples:

• +created_at - oldest to newest (default)
• -created_at - newest to oldest
• -summary_metrics.eval/accuracy - highest to lowest accuracy
• +config.learning_rate.value - lowest to highest learning rate

2. GraphQL Query Structure

From runs.py lines 123-154, the GraphQL query structure is:

query Runs($project: String!, $entity: String!, $cursor: String, $perPage: Int = 50, $order: String, $filters: JSONString) {
    project(name: $project, entityName: $entity) {
        runCount(filters: $filters)
        runs(filters: $filters, after: $cursor, first: $perPage, order: $order) {
            edges {
                node {
                    ...RunFragment
                }
                cursor
            }
            pageInfo {
                endCursor
                hasNextPage
            }
        }
    }
}

3. Run Fragment Fields

Full Run Fragment (lines 69-93):

fragment RunFragment on Run {
    id
    tags
    name
    displayName
    sweepName
    state
    config
    group
    jobType
    commit
    readOnly
    createdAt
    heartbeatAt
    description
    notes
    systemMetrics
    summaryMetrics  # <-- This is the key field for metrics
    historyLineCount
    user {
        name
        username
    }
    historyKeys
}

Important: The summaryMetrics field is returned as a JSON string, not structured data. You must parse it client-side.

4. Filters Structure

From api.py lines 1082-1085, filters use MongoDB query syntax and are passed as a JSONString:

filters: (dict) queries for specific runs using the MongoDB query language.
    You can filter by run properties such as config.key, summary_metrics.key, state, entity, createdAt, etc.
    For example: {"config.experiment_name": "foo"} would find runs with a config entry
        of experiment name set to "foo"

Supported MongoDB operators (lines 1063-1076):

• $and, $or, $nor
• $eq, $ne
• $gt, $gte, $lt, $lte
• $in, $nin
• $exists
• $regex

5. How Sorting by Nested Metrics Works

From the code analysis:

1. The order parameter accepts dot notation for nested fields:

   • summary_metrics.eval/accuracy - sorts by the eval/accuracy metric
   • config.learning_rate.value - sorts by config value

2. The sorting happens server-side when you pass the order parameter

3. You can use both order AND filters together

6. Correct MCP Query Pattern

Based on the wandb SDK implementation:

query GetTopRunsByAccuracy($entity: String!, $project: String!, $order: String!, $perPage: Int!) {
  project(entityName: $entity, name: $project) {
    runs(first: $perPage, order: $order) {
      edges {
        node {
          id
          name
          displayName
          state
          summaryMetrics
          createdAt
        }
      }
      pageInfo {
        endCursor
        hasNextPage
      }
    }
  }
}

Variables:

{
  "entity": "wandb-smle",
  "project": "hiring-agent-demo-public",
  "order": "-summary_metrics.eval/accuracy",
  "perPage": 5
}

7. Why My Previous Queries Failed

1. Wrong parameter order in query definition - I used $project: String!, $entity: String! but W&B expects them in GraphQL in a specific order
2. Used order as a parameter without defining it in the query signature
3. Didn't realize summaryMetrics is returned as a JSON string that needs parsing

8. Client-Side vs Server-Side Sorting

Server-side sorting (preferred):

• Use the order parameter: order: "-summary_metrics.eval/accuracy"
• More efficient, less data transferred
• Only works if the metric exists in summaryMetrics

Client-side sorting (fallback):

• Fetch all runs with summaryMetrics
• Parse the JSON string for each run
• Sort in code
• Necessary when:
  • Metric doesn't exist in all runs
  • Need complex sorting logic
  • Need to handle missing values

9. Best Practices for MCP Queries

1. Always include required pagination structure:

   edges {
     node { ...fields... }
   }
   pageInfo {
     endCursor
     hasNextPage
   }

2. Define all variables in the query signature:

   query Name($var1: Type!, $var2: Type) { ... }

3. Use proper variable types:

   • String! for required strings
   • Int for optional integers
   • JSONString for filters (passed as JSON-stringified dict)

4. When sorting by metrics:

   • Use summary_metrics.metric_name in the order parameter
   • Prepend with - for descending, + for ascending
   • The metric name can include special characters like / (e.g., eval/accuracy)

5. Handle missing data:

   • Not all runs will have all metrics
   • Parse summaryMetrics JSON and check for key existence
   • Consider filtering with filters: {"summary_metrics.eval/accuracy": {"$exists": true}}

10. Common Pitfalls to Avoid

1. ❌ Don't use order: "-summaryMetrics.eval/accuracy" (wrong - uses capital M)

2. ✅ Use order: "-summary_metrics.eval/accuracy" (correct - uses underscore and lowercase)

3. ❌ Don't forget to JSON.stringify the filters dict

4. ✅ Always pass filters as a JSON string in variables

5. ❌ Don't assume summaryMetrics is an object

6. ✅ Parse it as JSON: JSON.parse(run.summaryMetrics)

7. ❌ Don't query for metrics that don't exist in many runs without filtering first

8. ✅ Use $exists filter to only get runs with the metric you need

Example: Complete Working Query

query GetTopRunsByMetric($entity: String!, $project: String!, $order: String, $perPage: Int, $filters: JSONString) {
  project(entityName: $entity, name: $project) {
    runCount(filters: $filters)
    runs(first: $perPage, order: $order, filters: $filters) {
      edges {
        node {
          id
          name
          displayName
          state
          summaryMetrics
          config
          createdAt
          heartbeatAt
        }
      }
      pageInfo {
        endCursor
        hasNextPage
      }
    }
  }
}

Variables for top 5 by eval/accuracy:

{
  "entity": "wandb-smle",
  "project": "hiring-agent-demo-public",
  "order": "-summary_metrics.eval/accuracy",
  "perPage": 5,
  "filters": "{\"summary_metrics.eval/accuracy\": {\"$exists\": true}}"
}

Note: The filters value is a JSON-stringified string, not a JSON object!

[NiWaRe]

Thanks a lot for the feedback @nbren12 - I'll have a look at this this week!