Dealing with puzzle captcha using Selenium that has a limit of 5 attempts

Question

I have been scraping this site: https://www.immobilienscout24.de

When opening the site with selenium, the site sometimes gives you to solve a puzzle captcha after you have clicked you are not a robot captcha. The thing is that the site lets you only move the puzzle 6 times. If you fail to put the puzzle in the right place after the 6th time is bans you from entering the site.

The code moves the puzzle, but how would I arrange the for loop to split the screen into 6 exact pieces? Or if someone has some kind of different approach?

The code for the puzzle captcha is the following:

try:
    slider = driver.find_element(By.CLASS_NAME, 'geetest_slider_button')
    for x in range(0, 200, 10):
        actions.move_to_element(slider).click_and_hold().move_by_offset(x, 0).release().perform()
        time.sleep(0.1)
except:
    print('No slider found. Continuing with the code.')

Thank you.

Answer 1

based on my understanding of your question, your goal is to validate a picture captcha within six attempts.

This is a very complex issue, and I believe the best solution would be to directly hand the images over to AI for image sequencing, which would be easier. However, I do not have the qualifications to call the relevant AI APIs. Maybe you can try to activate them. As for the code to call the AI, I don't think that would be difficult.

Excluding the possibility of using AI, I believe that attempting to achieve your goal through just simulated drag-and-drop will not be successful.

First, you need to capture the captcha area by whatever means necessary, eventually obtaining six disordered images. For instance, you could capture a single disordered image and then slice it locally, or directly fetch the image resources from the web page's source code.

Assuming you've reached this step and obtained six images, you can then proceed to merge them and perform image sampling. There are many ways to sample, such as first setting the images to grayscale and then performing contour extraction. This way, you get a contour map. Generally, as there is a gradient in brightness in a single image, there will be clear dividing lines between images. Of course, to get these distinct feature lines, you may need to constantly change the parameters during contour extraction. However, since the images have already been downloaded locally, you can still brute force this problem.

Mostly, we cannot achieve a 100% match in edge testing, so you need to establish feature values for each edge matching result. After completing all permutations and combinations of the sequence, use the highest feature value to move the images. If there are still attempts left, you can continue to use a lower-ranked sequence.

You might ask how to establish feature values when you don't have an absolutely correct image for edge matching comparison. My thinking is that you could reverse your approach. For example, manually create a blank image and artificially draw contrasting lines at the positions of the dividing lines. After that, extract the contours of this image to serve as a reference for your feature values.

Additionally, another thought is to apply an edge matching algorithm directly to the six images, match them in pairs, and calculate the final matching value.

The above is my thought process, and below could be the potential code involved in that process.

1. edge matching

def match_edges(image1, image2):
    # Convert images to grayscale
    gray1 = cv2.cvtColor(image1, cv2.COLOR_BGR2GRAY)
    gray2 = cv2.cvtColor(image2, cv2.COLOR_BGR2GRAY)

    # Detect edges in the images using the Canny algorithm
    edges1 = cv2.Canny(gray1, 50, 150)
    edges2 = cv2.Canny(gray2, 50, 150)

    # Initialize variables for matching edges
    # This example simplifies the matching process for demonstration purposes
    height, width = edges1.shape
    best_match_value = float('inf')
    best_match_position = None
    
    # Assuming edge matching is performed side by side (left/right edges comparison)
    for y in range(height):
        # Calculate the difference between edges of the two images
        match_value = np.sum(np.bitwise_xor(edges1[y,:], edges2[y,:]))
        # Update the best match if the current match is better
        if match_value < best_match_value:
            best_match_value = match_value
            best_match_position = y

    # Return the position and value of the best match
    return best_match_position, best_match_value

2. Feature extraction

# Restore picture 
f = merge(sum_rows,sum_cols,channels,part1,part2,part3,part4)
show_images([f],'Restore picture[1,2,3,4]')

# Gray
gray = cv2.cvtColor(f, cv2.COLOR_BGRA2GRAY)
show_images([gray],'Gray')

# Extract contour
edges = cv2.Canny(gray, 35, 80, apertureSize=3)
show_images([edges],'Extract contour')

3. Draw feature lines

# f is current unsorted merge-image 
f = merge(sum_rows,sum_cols,channels,part1,part2,part3,part4)
lf = f.copy()
cv2.line(lf, (0, 75), (300, 75), (0, 0, 255), 2)
cv2.line(lf, (150, 0), (150, 150), (0, 0, 255), 2)
show_images([lf],'out of order，The gradient becomes the cross feature line.')

4. Feature matching

f = merge(sum_rows,sum_cols,channels,part1,part2,part3,part4)
gray = cv2.cvtColor(f, cv2.COLOR_BGRA2GRAY)
edges = cv2.Canny(gray, 35, 80, apertureSize=3)
show_images([edges],'Extract contour')
# find the split line.
lines = cv2.HoughLinesP(edges,0.01,np.pi/360,60,minLineLength=50,maxLineGap=10)
if lines is None:
    print('No cross feature line found')
else:
    lf = f.copy()
    for line in lines:
        x1, y1, x2, y2 = line[0]
        cv2.line(lf, (x1, y1), (x2, y2), (0, 0, 255), 2)
    show_images([lf])

Reference：

platform.openai.com/docs/guides/function-calling
zhihu.com/p/434851845    
geeksforgeeks.org/feature-detection-and-matching-with-opencv-python