Skip to content

Automating a Windows Application

In some cases, we can automate Windows applications through element properties, as an alternative to computer vision.

This can be a useful alternative when we don't want to worry about resolution issues. We can simply interact with the application structure through the attributes of elements and windows.

This tutorial will guide you through the process of creating a simple automation that interacts with the elements of a Windows application.

Creating a New Project

The first step is to create a project for a desktop automation, you can follow the same steps defined in the previous section.

See how to create a Desktop Bot project.

Target Application and Spy Tools

For this simple example, we are going to use the standard Windows wordpad.

To help us inspect this application, we are going to use the Accessibility Insights for Windows spy tool.

See more details about the spy tools usage.

Interacting with the Application

Connecting

With the project open, let's insert the first step in the bot.py file, which is to establish the connection with the target application.

from botcity.core import DesktopBot, Backend
...
def main():
    bot = DesktopBot()

    # Application path
    app_path = "wordpad.exe"

    # Launching the app
    bot.execute(app_path)

    # Connecting to the application using 'path' selector
    bot.connect_to_app(backend=Backend.UIA, path=app_path)
...

With the application open, we can use the spy tool to see some information about the elements.

Spy tool inspect

Finding Windows and Elements

With the app connected, we can now use the reference from the main window and collect the element referring to the text editor, for example.

from botcity.core import DesktopBot, Backend
...
def main():
    bot = DesktopBot()

    # Application path
    app_path = "wordpad.exe"

    # Launching the app
    bot.execute(app_path)

    # Connecting to the application using 'path' selector
    bot.connect_to_app(backend=Backend.UIA, path=app_path)

    # Getting the Wordpad main window context using 'title_re',  'class_name' and 'control_type' selector
    main_window = bot.find_app_window(title_re='Document - WordPad', class_name="WordPadClass",
                                        control_type="Window")

    # Getting the edit text area using the element 'class_name' and 'control_type'
    # We were able to view this property through the spy tool
    edit = bot.find_app_element(from_parent_window=main_window, control_type="Document",
                                    class_name='RICHEDIT50W')
...

Note that we used the class_name and control_type property that we got from the spy tool. You can basically use any information that appears to filter a certain element.

Element properties

Tip

Another way to inspect the application's control elements is to use the print_control_identifiers method. After opening the dialog box and assigning it to a variable, the print_control_identifiers method should print all available controls. See more details about this here.

Performing Actions

With the element found, we can perform the operations. As this element is of type Document, we can insert a text content.

from botcity.core import DesktopBot, Backend
...
def main():
    bot = DesktopBot()

    # Application path
    app_path = "wordpad.exe"

    # Launching the app
    bot.execute(app_path)

    # Connecting to the application using 'path' selector
    bot.connect_to_app(backend=Backend.UIA, path=app_path)

    # Getting the Wordpad main window context using 'title_re',  'class_name' and 'control_type' selector
    main_window = bot.find_app_window(title_re='Document - WordPad', class_name="WordPadClass",
                                         control_type="Window")

    # Getting the edit text area using the element 'class_name' and 'control_type'
    # We were able to view this property through the spy tool
    edit = bot.find_app_element(from_parent_window=main_window, control_type="Document",
                                     class_name='RICHEDIT50W')

    # Typing some text
    edit.type_keys("Hello! Welcome to BotCity!", with_spaces=True)
...

Info

See more details about the methods available for each different type of element at this link.

Running the Bot

Just the code above would be enough for us to start Wordpad and insert a text content.

We have established a connection with the application and are performing operations in the context of a specific element.

Running the code, the final result will look like this:

Result

Accessing other elements

With the same approach as above, using the control_type and title_re property, let's access other Wordpad elements.

Accesing Print Preview

# Accessing WordPad's 'Print Preview' option through the `control_type` and` title_re` selectors
print_preview = bot.find_app_element(from_parent_window=main_window, control_type='Button',
                                        title_re='Print preview')
print_preview.click()
...

Then we will select the Page Setup option

# Accessing the 'Page Setup' option through selectors `control_type` and` title_re`
page_setup = bot.find_app_element(from_parent_window=main_window, control_type='Button',
                                        title_re='Page setup')
page_setup.click()
...

Using select to choose an option on the box of selection

We will interact with the Size element, as we see below:

Paper-Setup

Before any manipulation, we will enter the context of the Page Setup window. It is a window within the scope of main_window but its attributes are not directly exposed to it. This way, to find the elements and attributes of the window Page Setup, it will be necessary to return it first.

# Obtaining context of the 'Page Setup' window through the 'Title_re' selector
page_setup_window = bot.find_app_element(from_parent_window=main_window, title_re='Page Setup')
...

Now yes, to select the paper size, let's map control_type and title from the size element so we can select them through theselect method.

This time we will use another approach to get the property of the element, the print_control_identifiers, as we see below:

# Priting the control identifiers from the "Page Setup" window.
page_setup_window.print_control_identifiers()
...
Running the code above, the controls of the Page Setup window will be generated in your terminal.

Control-Identifiers

Where we can see the title and control_type of element, which will be used below, as parameters of find_app_element method.

# Finding element to select page size
size = bot.find_app_element(from_parent_window=main_window, control_type="ComboBox",         
                                title="Size:")
# Selecting page type
size.select("A5")
...

Using invoke to choose an the radio button

Here, we will use invoke to select the Landscape radio button.

Radio-button

Still using controls identifiers.

Control-Identifiers

Let's get the title and control_type of the Landscape element to find it below:

# Finding element to select radio button "Landscape"
orientation = bot.find_app_element(from_parent_window=main_window, control_type="RadioButton",
                                        title="Landscape")
# Method `click` is an alias of `invoke`
orientation.invoke()

# Finding element to click on button "OK"
btn_ok = bot.find_app_element(from_parent_window=main_window, title="OK")
# Clicking on button
btn_ok.click()
...

Info

In this case, it was not possible to use the methods click, and click_input to select the radio button, because they only have control over elements with property control_type="Button".

Info

invoke uses the so called InvokePattern that represents a default action for the control. When using Application(backend="uia"), the click method is an alias of invoke, where the InvokePattern action brings the window to the foreground, allowing interaction with the element.

After the above actions, the page configuration looked like this:

Page-setup

Using type_keys and click to save the Wordpad in the file explorer.

To finish, we will access the Save option in the menu.

    # Accessing 'Save' icon of the top menu through the 'Control_Type' and 'Title' selectors
    save = bot.find_app_element(from_parent_window=main_window, control_type="Button",         
                                    title="Save")
    save.click()
...

This action will open the file explorer window, as we see below:

Save-As

Thus, we will define a name for the file, in this case it will be tutorial.txt, and we will click on the save button, through the click method.

# Finding element to type the name of file
file_name = bot.find_app_element(from_parent_window=main_window, class_name="Edit", 
                                    title="File name:")
# Typing the file name
file_name.type_keys("tutorial.txt")

# Finding element to clicks on button save
btn_save = bot.find_app_element(from_parent_window=main_window, class_name="Button", 
                                    title="Save")
# Clicking on button save
btn_save.click()
...

Complete Code

from botcity.core import DesktopBot, Backend
...
def main():
    bot = DesktopBot()

    # Application path
    app_path = "wordpad.exe"

    # Launching the app
    bot.execute(app_path)

    # Connecting to the application using 'path' selector
    bot.connect_to_app(backend=Backend.UIA, path=app_path)

    # Getting the Wordpad main window context using 'title_re',  'class_name' and 'control_type' selector
    main_window = bot.find_app_window(title_re='Document - WordPad', class_name="WordPadClass",
                                        control_type="Window")

    # Getting the edit text area using the element 'class_name' and 'control_type'
    # We were able to view this property through the spy tool
    edit = bot.find_app_element(from_parent_window=main_window, control_type="Document",
                                    class_name='RICHEDIT50W')

    # Typing some text
    edit.type_keys("Hello! Welcome to BotCity!", with_spaces=True)

    # Accessing WordPad's 'print preview' option through the `control_type` and` title_re` selectors
    print_preview = bot.find_app_element(from_parent_window=main_window, control_type='Button',
                                            title_re='Print preview')
    print_preview.click()

    # Accessing the 'Page Setup' option through selectors `control_type` and` title_re`
    page_setup = bot.find_app_element(from_parent_window=main_window, control_type='Button',
                                         title_re='Page setup')
    page_setup.click()

    # Obtaining context of the 'Page Setup' window through the 'Title_re' selector
    page_setup_window = bot.find_app_element(from_parent_window=main_window, title_re='Page Setup')

    # Priting the control identifiers from the "Page Setup" window.
    page_setup_window.print_control_identifiers()

    # Finding element to select page size
    size = bot.find_app_element(from_parent_window=main_window, class_name="ComboBox", 
                                    title="Size:")
    # Selecting page type
    size.select("A5")

    # Finding element to select radio button "Landscape"
    orientation = bot.find_app_element(from_parent_window=main_window, 
                                           control_type="RadioButton", title="Landscape")
    # Method click() is an alias of invoke()
    orientation.invoke()

    # Finding element to click on button "OK"
    btn_ok = bot.find_app_element(from_parent_window=main_window, title="OK")
    # Clicking on button
    btn_ok.click()

    # Accessing 'Save' icon of the top menu through the 'Control_Type' and 'Title' selectors
    save = bot.find_app_element(from_parent_window=main_window, control_type="Button",         
                                    title="Save")
    save.click()

    # Finding element to type the name of file
    file_name = bot.find_app_element(from_parent_window=main_window, class_name="Edit", 
                                        title="File name:")
    # Typing the file name
    file_name.type_keys("tutorial.txt")

    # Finding element to clicks on button save
    btn_save = bot.find_app_element(from_parent_window=main_window, class_name="Button", 
                                        title="Save")
    # Clicking on button save
    btn_save.click()
...

Important

Some applications may be more difficult to inspect and find information about elements.

Use the spy tools as a reference to determine if this is the best alternative for your automation.

Conclusion

Under this tutorial you learned:

  • How to use a spy tool to inspect a Windows application.

  • How to connect to an application and view its attributes and properties.

  • How to interact with an element that has been found.