Skip to content

Computer Vision

Using the methods below you will be able to use computer vision to interact with the elements that appear on the screen.

Add Image

By default, element images are fetched from the project's resources folder and added to the image map. However, it is possible to add an image that is in another directory in the image map and thus access it normally through the defined label.

Note

This operation is not necessary for images that are already in the project's default resources folder.

# Adding an image to the image map that is in a specific directory.
bot.add_image("element", "/my_directory/images/element.png")

// Adding an image to the image map that is in a specific directory.
addImage("element", "/my_directory/images/element.png");

Get Image from Map

You can get a specific image from the image map using a defined label.

# Getting the image that have the "element" label.
img = bot.get_image_from_map("element")

// Not yet implemented.

Finding Elements

Using the find methods it is possible to search for an element on the screen using computer vision until a timeout happens. You can use the return of the find method to check if an element was found or not and based on that perform the necessary treatment.

# Print if the first_element was NOT found on the screen.
if not bot.find(label="first_element", matching=0.97, waiting_time=10000):
    print("This element was not found.")

# Print if the "second element" was found on the screen.
if bot.find(label="second_element", matching=0.97, waiting_time=10000):
    print("This element was found!")

# Using find_until will have the same effect as using find.
if bot.find_until(label="another_element", matching=0.97, waiting_time=10000):
    print("Found!")

// Print if the first_element was NOT found on the screen.
if(!find("first_element", 0.97, 10000)) {
    System.out.println("This element was not found");
}

// Print if the second_element was found on the screen.
if(find("second_element", 0.97, 10000)) {
    System.out.println("This element was found!");
}

The find method above is the most common to use and will basically work for all cases where we need to search for an element on the screen. However, it is possible to use find methods for specific situations.

Find Text

# Searching for an element that contains only text.
if bot.find_text(label="txt_element", matching=0.97, waiting_time=10000):
    print("This text was found on screen!")

// Searching for an element that contains only text.
if(findText("txt_element", 0.97, 10000)){
    System.out.println("This text was found on screen!");
}

Find All

In addition to performing operations with single elements, it is also possible to search for several different elements or for all occurrences of the same element.

# Search for all occurrences of the element.
elements = bot.find_all(label="test_element", matching=0.97, waiting_time=10000)

# For each element found, print the coordinates.
for ele in elements:
    print(ele)

// Search for all occurrences of the element.
List<State> elements = findAll("test_element", 0.97, 20000);

// For each element found, print the coordinates.
for(State ele : elements) {
    System.out.println(ele.getX());
    System.out.println(ele.getY());
    System.out.println(ele.getWidth());
    System.out.println(ele.getHeight());
}

Find Multiple

# List with the elements label.
elements_to_find = ["ele1", "ele2", "ele3", "ele4"]

# Searching for all elements will return a dictionary with the label and coordinates of each one.
elements = bot.find_multiple(labels=elements_to_find, matching=0.97, waiting_time=20000)

print(elements)

// List with the elements label.
List<String> elementsToFind = new ArrayList<String>();
elementsToFind.add("ele1");
elementsToFind.add("ele2");
elementsToFind.add("ele3");
elementsToFind.add("ele4");

// Searching for all elements will return a Map with the label and coordinates of each one.
Map<String, State> elements = findMultiple(elementsToFind, 0.97, 10000);

// Printing the coordinates of the element "ele2".
System.out.println(elements.get("ele2"));

Set Current Element

You can set the current element to perform operations on it. This can be useful when we are looking for multiple elements but want to work with a specific one.

# Search for all occurrences of an element.
elements = bot.find_all(label="button", matching=0.97, waiting_time=10000, as_list=True)

# Setting the second occurrence as the current element.
bot.set_current_element(elements[1])

# Clicking on the defined element.
bot.click()

// Search for all occurrences of an element.
List<State> elements = findAll("button", 0.97, 10000);

// Setting the second occurrence as the current element.
setCurrentElement(elements.get(1));

// Clicking on the defined element.
click();

Getting Elements

You can get information about the last element found and also the coordinates of a specific element.

Get Last Element

# Get the last element found on the screen.
last_element = bot.get_last_element()

# Printing the element coords.
print(last_element)

// Get the last element found on the screen.
State lastElement = getLastElement();

// Printing the element coords.
System.out.println(lastElement.getX());
System.out.println(lastElement.getY());
System.out.println(lastElement.getWidth());
System.out.println(lastElement.getHeight());

Get Element Coordinates

# This method will search for the element and return its X and Y.
ele = bot.get_element_coords(label="test_element", matching=0.97)
print(ele)

# This method will search for the element and return its X and Y centered.
element_centered = bot.get_element_coords_centered(label="test_element", matching=0.97)
print(element_centered)

// This method will search for the element and return its X and Y.
Point ele = getElementCoords("test_element", 0.97);
System.out.println(ele);

// This method will search for the element and return its X and Y centered.
Point elementCentered = getElementCoordsCentered("test_element", 0.97);
System.out.println(elementCentered);