You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# AI labeling block: Bounding box validation Using GPT-4o
2
2
3
-
Let GPT-4o look at every image in your object detection dataset and check against up to three validation prompts you provide. If images are not compliant they will be disabled and reasoning will be put into the metadata. The block passes each image's bounding boxes and labels to GPT-4o so you can for example check if "The bounding boxes and labels do not correspond to to the objects in the image" among other things.
3
+
Let GPT-4o look at every image in your object detection dataset and check against a validation prompt you provide. If images are not compliant they will be disabled and reasoning will be put into the metadata. The block passes each image's bounding boxes and labels to GPT-4o so you can for example check if "The bounding boxes and labels do not correspond to to the objects in the image" among other things.
4
4
5
-
You must pass an OPENAI API key either in the field below or via a secret which you can add from the Custom Blocks->Transformation page in your org with the name OPENAI_API_KEY.
1. Go to an Enterprise project, upload an Object Detection dataset which you wish to validatae
11
-
2. Choose **Data acquisition->Data Sources->Add new data source**.
12
-
3. Select Transformation Block and the 'Validate Object Detection Datasets Using GPT-4o' block, fill in your prompts and and run the block.
9
+
If you just want to use GPT-4o as a data validation tool in your Edge Impulse project you don't need this repo. Just go to any project, select **Data acquisition > AI labeling**, choose **Bounding box validation Using GPT-4o**.
13
10
14
-
> You need an OPENAI API Key to run the GPT4o model
11
+
> You'll need an OpenAI API Key, see [OpenAI Platform > API Keys](https://platform.openai.com/api-keys).
15
12
16
-
4. Any items which are invalid will be disabled. Reasoning will be provided in the metadata for each data item
You can use this repository to develop your own block that uses GPT-4o (or some other LLM) to help you do data curation (or other tasks).
19
16
20
-
### Customizing this repository (enterprise only)
17
+
1. Create a new Edge Impulse project, and add some images.
18
+
2. Create a file called `ids.json` and add the IDs of the samples you want to label. You can find the sample ID by clicking the 'expand' button on **Data acquisiton**.
21
19
22
-
You can modify this repository and push it as a new custom transformation block.
20
+
23
21
24
-
1. Install the [Edge Impulse CLI](https://docs.edgeimpulse.com/docs/tools/edge-impulse-cli).
25
-
2. Open a command prompt or terminal, and navigate to this folder.
26
-
3. Create a new transformation block:
22
+
Add these IDs to the `ids.json` file as an array of numbers, e.g.:
27
23
24
+
```json
25
+
[1299267659, 1299267609, 1299267606]
28
26
```
29
-
$ edge-impulse-blocks init
30
27
31
-
? Choose a type of block: Transformation block
32
-
? Choose an option: Create a new block
33
-
? Enter the name of your block: Custom Validate Object Detection Datasets Using GPT-4o
34
-
? Enter the description of your block: Use GPT4o to validate a dataset
35
-
? What type of data does this block operate on? Standalone (runs the container, but no files / data items passed in)
36
-
? Which buckets do you want to mount into this block (will be mounted under /mnt/s3fs/BUCKET_NAME, you can change these mount points in the St
37
-
udio)?
38
-
? Would you like to download and load the example repository? no
28
+
3. Load your API keys (both Edge Impulse and OpenAI):
29
+
39
30
```
31
+
export OPENAI_API_KEY=sk-M...
32
+
export EI_PROJECT_API_KEY=ei_44...
33
+
```
34
+
35
+
> You can find your OpenAI API key on the [OpenAI API Keys](https://platform.openai.com/api-keys) page. You can find your Edge Impulse API Key via **Dashboard > Keys**.
40
36
41
-
4. Push the block:
37
+
4. Install Node.js 20.
38
+
5. Build and run this project to label your data:
42
39
43
40
```
41
+
npm run build
42
+
node build/llm-validation.js \
43
+
--validation-prompt "- The bounding boxes and labels do not correspond to to the objects in the image\n- The image is not clear enough to determine the objects in the image" \
44
+
--concurrency 10 \
45
+
--data-ids-file ids.json
46
+
```
47
+
48
+
6. Afterwards you'll have labeled data in your project.
49
+
50
+
### Pushing block to Edge Impulse (enterprise only)
51
+
52
+
If you've modified this block, you can push it back to Edge Impulse so it's available to everyone in your organization.
53
+
54
+
1. Update `parameters.json` to update the name and description of your block.
55
+
2. Initialize and push the block:
56
+
57
+
```
58
+
$ edge-impulse-blocks init
44
59
$ edge-impulse-blocks push
45
60
```
61
+
62
+
3. Afterwards, you can run your block through **Data acquisition > AI labeling** in any Edge Impulse project.
63
+
64
+
### Proposed changes
65
+
66
+
AI labeling blocks should be able to run in 'preview' mode (triggered when you click *Label preview data* in the Studio) - where changes are _staged_ but not directly applied. If this is the case `--propose-actions <job-id>` is passed into your block. When you see this flag you should not apply changes directly (e.g. via `api.rawData.editLabel`) but rather use the `setSampleProposedChanges` API. Search for this API in [llm-validation.ts](llm-validation.ts) to see how this should be used.
Copy file name to clipboardExpand all lines: parameters.json
+11-35
Original file line number
Diff line number
Diff line change
@@ -1,16 +1,12 @@
1
1
{
2
2
"version": 1,
3
-
"type": "transform",
3
+
"type": "ai-action",
4
4
"info": {
5
-
"name": "Validate Object Detection Datasets Using GPT-4o",
6
-
"description": "Let GPT-4o look at every image in your object detection dataset and check against up to three validation prompts you provide. If images are not compliant they will be disabled and reasoning will be put into the metadata. The block passes each image's bounding boxes and labels to GPT-4o so you can for example check if \"The bounding boxes and labels do not correspond to the objects in the image\".",
7
-
"operatesOn": "standalone",
8
-
"transformMountpoints": [],
9
-
"allowExtraCliArguments": false,
10
-
"cliArguments": "",
11
-
"indMetadata": true,
12
-
"showInCreateTransformationJob": false,
13
-
"showInDataSources": true
5
+
"name": "Bounding box validation Using GPT-4o",
6
+
"description": "Use GPT-4o to validate that your bounding boxes are correct. If images are not compliant they will be disabled and reasoning will be put into the metadata. The block passes each image's bounding boxes and labels to GPT-4o so you can for example check if \"The bounding boxes and labels do not correspond to the objects in the image\".",
7
+
"operatesOn": [
8
+
"images_object_detection"
9
+
]
14
10
},
15
11
"parameters": [
16
12
{
@@ -21,33 +17,13 @@
21
17
"param": "OPENAI_API_KEY"
22
18
},
23
19
{
24
-
"name": "Validation Prompt 1",
25
-
"value": "The bounding boxes and labels do not correspond to to the objects in the image",
20
+
"name": "Validation prompt",
21
+
"value": "- The bounding boxes and labels do not correspond to to the objects in the image\n- The image is not clear enough to determine the objects in the image\n- There is text overlay visible on this image",
26
22
"type": "string",
27
23
"help": "Disable the sample if this prompt is true",
28
-
"param": "validation1"
29
-
},
30
-
{
31
-
"name": "Validation Prompt 2",
32
-
"value": "The image is not clear enough to determine the objects in the image.",
33
-
"type": "string",
34
-
"help": "Disable the sample if this prompt is true",
35
-
"param": "validation2"
36
-
},
37
-
{
38
-
"name": "Validation Prompt 3",
39
-
"value": "There is text overlay visible on this image.",
40
-
"type": "string",
41
-
"help": "Disable the sample if this prompt is true",
42
-
"param": "validation3"
43
-
},
44
-
{
45
-
"name": "Max. no. of samples to label",
46
-
"value": "",
47
-
"type": "int",
48
-
"optional": true,
49
-
"help": "Number of samples to label",
50
-
"param": "limit"
24
+
"param": "validation-prompt",
25
+
"multiline": true,
26
+
"hint": "Put each prompt on a new line, and start with a -"
0 commit comments