[ENHANCEMENT] Proposal to allow detection from different perspectives

[TimWalter]

Hey,

I've been tinkering around the problem of being able to detect the color-checker when its shot was taken, so that the approximate rectangle of the checker is distorted.

I am unsure if the detection of such a contour is currently possible with this package, but I would like to contribute in order to enable such a functionality.

As the masking of the checker should stay fixed, the idea is to warp the detected contour to fit the mask.
When the contour is warped even the size of the mask can stay fixed.
This can be done as follows:

# the dimensions are somewhat arbitrary but should fit the aspect ratio of the checker 

def correct_distortion(distorted_cnt, x_dim, y_dim, img):
    desired_cnt = [[0,0],[x_dim,0],[x_dim,y_dim],[0, y_dim]]
    M = cv2.getPerspectiveTransform(np.float32(distorted_cnt), np.float32(desired_cnt))
    return cv2.warpPerspective(img, M,(x_dim,y_dim))

As the checker orientation matters, it is necessary to ensure that the points of the detected contour are ordered in the same way as the new coordinates e.g.

My attempt to ensure such an ordering are the following functions:

def order_clockwise(cnt):
    centroid = contour_centroid(cnt)
    angles = [math.atan2(c[1]-centroid[0][1], c[0]-centroid[0][0])*180//np.pi for corner in cnt]
    clockwise_contours = [c for c, a in sorted(zip(unpacked_cnt, angles), key=lambda pair: pair[1])]

    return clockwise_contours 

def order_cnt(cnt):
    clockwise_contour = order_clockwise(cnt)

    lowest_point_index = np.argmax([pt[0][1] for pt in clockwise_contour])
    next_index = (lowest_point_index+1) % len(clockwise_contour)
    previous_index = lowest_point_index-1

    
    lowest_point = clockwise_contour[lowest_point_index]
    next_point = clockwise_contour[next_index]
    previous_point = clockwise_contour[previous_index]
    
    next_diff_x = np.abs(lowest_point[0][0] - next_point[0][0])
    next_diff_y = np.abs(lowest_point[0][1] - next_point[0][1])
    previous_diff_x = np.abs(lowest_point[0][0] - previous_point[0][0])
    previous_diff_y = np.abs(lowest_point[0][1] - previous_point[0][1])
    
    anti_clockwise_angle = math.atan2(next_diff_y, next_diff_x) * 180 //np.pi
    clockwise_angle = math.atan2(previous_diff_y, previous_diff_x) * 180 //np.pi
    
    # Lower line is already aligned in the image
    if next_diff_y == 0 or previous_diff_y == 0:
        next_angle = previous_angle = "Aligned"
    
    # "Falling" to the right, Less rotation needed to align when rotating clockwise
    if anti_clockwise_angle > clockwise_angle: 
        first_index = next_index       
    
    # "Falling" to the left, Less rotation needed to align when rotating anti-clockwise
    if anti_clockwise_angle == "Aligned" or anti_clockwise_angle <= clockwise_angle: 
        first_index = next_index+1 if next_index+1 < len(clockwise_contour) else 0 
        if previous_diff_y == 0:
            first_index -= 1
        
    return clockwise_contour[first_index:] + clockwise_contour[:first_index]
    

This works under the assumption that the checker is not flipped but was recorded as the aspect ratio suggests. It is required to perform the ordering before the wrapping.

I've attached a notebook, with a demonstration of the described pipeline
Demo.zip

[KelSolaar]

Hi @TimWalter,

I am unsure if the detection of such a contour is currently possible with this package, but I would like to contribute in order to enable such a functionality.

Not right now, I mean you could have a bit of perspective but the expectation is to shoot the chart normal to the optical axis as much as possible, and run distortion and vignette correction possibly before detection.

I don't have too much cycles to look into that right now but any improvement is appreciated! I would like to roll in the proposal from #53 too!

Cheers,

Thomas

[TimWalter]

Hey,
it's been a while but lately I've been back at the problem. Over the last month, I have been working on revising the checker detection strategy to improve detection for multiple checkers and different perspectives. I changed the strategy quiet a bit from the initial idea described earlier in this discussion, which was based on too many heuristics in hindsight. The new approach is theoretically more sound and so far in my testing works much more reliably. It basically codes the swatches centroids and tries to map the resulting graph to the one of a reference checker (which is similar to the mask generated in the current approach).

I believe that my revised strategy is far superior to the current implementation in terms of robustness of detection, although it is slower (approximately 5x). Nonetheless, being able to detect multiple checkers in one image and detecting them even if they are not strictly facing the camera, or if not all swatches are detected, is vital for certain applications

I have done significant integration work on my local branch into the directory, including integrating it into segmentation.py, documenting it, and adapting the unit test and demo. The demo is a Jupyter notebook that I have attached to this comment for discussion. However, you may need to adapt the image and template paths to use it.

The benefits of my revised strategy are significant, as it improves the robustness of checker detection. I would love to hear your feedback on the code and any further testing that you might be able to do. My hope is that we can integrate this into the library.

Demo.zip

Cheers Tim

[KelSolaar]

Hi @TimWalter,

Thanks a lot! It is certainly on my radar to implement your updates, I will try to look at that in the coming weeks. Something I would like to do maybe, is use it as a fail-over:

1. We try to detect with the opinionated current approach which is faster
2. Checkers are detected, good, we move on
3. Nothing it detected, we try again with your approach

Cheers,

Thomas

[TimWalter]

Sounds reasonable for the robustness for sure!

But how would you handle the multiple checkers in one image approach?

In any case let me know if I can help

[KelSolaar]

But how would you handle the multiple checkers in one image approach?

The current code should be able to detect many already.

[KelSolaar]

Or do you mean if one is detected but not the other?

[TimWalter]

Multiple in one image, do you have to call the function multiple times with same image to get multiple? Or how would you detect multiple checkers with similar aspect ratio and discriminate between them to apply the correct mask.

As an example we specifically had the problem that some checkerboards (e.g. calibrite passport2) have 2 boards with basically the same aspect ratio so discriminating between them with the current code is hard if not all swatches are detected (as Swatch count is the only other metric left)

[KelSolaar]

The code should already be doing swatches analysis to isolate them, we thus reject the white balance section of the passport.

[KelSolaar]

Hey @TimWalter,

I haven't forgotten about that, been busy, very busy on many different projects so this got at the bottom of the stack. I look at your Notebook though, it is great but will take some effort to get added. I also saw that there is a dependency on sklearn.DBSCAN nothing dramatic but thought I would point out!

Cheers,

Thomas

[TimWalter]

Yes that is true, I just needed some clustering algorithm to remove outliers and DBSCAN seemed to work most robustly

[mexicantexan]

Not sure if this is helpful but here is a script I wrote a while back for a project. It is not production-ready by any means, but might be helpful as a reference. It is a method for finding macbeth cards built on top of @KelSolaar 's awesome library. We found it to work really well for our use case, allowing us to find rotated colorcards (even upside down/flipped). Hopefully it helps?

macBeth_color_finder.zip

[KelSolaar]

Hey @TimWalter,

I'm working on this and I have a few notes / questions:

• is_convex_quadrilateral definition is missing (it is not critical that said).
• What value do you use for settings.greedy_heuristic?
• I'm not quite sure how you generated the templates, however, I'm noting that the correspondences tables are massive with multiple dozen of thousands of elements. This is only for a ColorChecker Classic, I cannot imagine how many there would be for a ColorChecker SG or DC and the performance hit.

[KelSolaar]

Just deleted a bunch of comments, I was not thinking clearly, long week... :)

[KelSolaar]

Looking at the are_three_collinear definition should not the comparison be as follows in generate_template definition: if np.all(angle_difference > 0) and not are_three_collinear_2(points), i.e. we add a correspondence only if it has no collinear points?

Note that this is a bit faster:

def are_three_collinear_2(points):
    non_collinear = True
    for p in combinations(points, 3):
        non_collinear *= (
            p[0][0] * (p[1][1] - p[2][1])
            + p[1][0] * (p[2][1] - p[0][1])
            + p[2][0] * (p[0][1] - p[1][1])
            != 0
        )
    return not bool(non_collinear)