Object cluster & image color segment matching for object point cloud refinement #138
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…tected point clouds from the image to the point cloud.
…nto pliam1105/improved_perception
…de each detected object's bounding box (when projected onto the image), with the respective labels and probabilities, colcon build succesful, didn't test yet
…throws some out of range errors but I catched them and made them warnings and it seems to work fine)
…ions in resize, may be the wrong order in terms of channels so may need to change the order both in resize() and in at() again), seems to correctly select the points in the objects detected
…l meaningful clusters based on the detections
…on, need to find a euclidean clustering algorithm for n-dimensional data or with customizable distance function, or just one that takes color into account (there are some in PCL but they mostly rely on color thresholds or difference limits rather than adding their squared difference to the distance) or implement it from scratch (and KD-tree from scratch?)
…nce based condition)
… from the bounding box ones
… to publish it now
…izing them in RViz (not tested or built yet)
… test & troubleshoot/tune the matching
…formulas for the choosing & matching metrics
…nd those projected to cluster points)
…n the actual object, but more testing/tuning would be good)
… by publishing unorganized point clouds (channels don't matter anyways in visualization)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The code clusters (based on color difference and euclidean distance) the points that are in each object's bounding box (detected by the object detection node) when projected onto the image, then performs color segmentation on the image depending on the color that YOLOv8/color segmentation algorithm detected, creating multiple clusters and contours, between which it then takes the optimal pair based on an optimality metric that takes into account the similarity to the color detected by YOLOv8, the distance between the cluster points when projected onto the image and the color segment points, and the size of the cluster and the color segments, thus selecting the cluster and color segment that best correspond to the object detected.