Automating Windows Applications with BotCity Inspector¶

In some cases, we can automate Windows applications through element properties, as an alternative to computer vision.

This can be a useful alternative when we don't want to worry about resolution issues. We can simply interact with the application structure through the attributes of elements and windows.

This tutorial will guide you through the process of creating a simple automation that interacts with the elements of a Windows application.

Creating a New Project¶

The first step is to create a project for a desktop automation, you can follow the same steps defined in the previous section.

See how to create a Desktop Bot project using the project template.

Spy Tools¶

The attributes and properties of a Windows application can be viewed using spy tools.

Currently, there are several tools that allow for the inspection of Desktop applications, such as Accessibility Insights for Windows.

However, these tools generally offer basic features for inspecting applications and do not necessarily focus on automation development, requiring additional work to identify properties that can be used in the code when searching for elements.

In light of this scenario, BotCity provides a tool focused on inspecting Desktop applications and generating Python code integrated with the development framework, making the use of these resources more simplified and intuitive when building automation.

BotCity Windows Inspector¶

To use the BotCity Windows Inspector, simply install the plugin for BotCity Studio for Visual Studio Code.

After installing the extension and loading a Python project, you will have access to the tool to start inspecting applications on Windows.

Next, you will see more details about using this feature and also best practices when inspecting applications and generating code.

Interacting with the application¶

For this simple example, we will use the standard Wordpad of Windows.

With the application open and the help of BotCity Windows Inspector, we can start mapping the elements we wish to interact with.

Mapping the initial elements¶

Initially, let's assume the intention is to interact with the buttons related to text font configuration.

For example, we can map the Bold, Italic, and Underline buttons of Wordpad, as well as the button to center the text.

Windows Inspector resources¶

Mapped Elements:

Each element that is mapped in the context of an application is added to the list of mapped elements.

From this list, you can click on an element to view its properties or remove an element you no longer wish to interact with.

Inspection Tree:

By clicking on an element in the list, the corresponding inspection tree will be generated.

This way, it is possible to visualize the "full path" from the main window of the application to the target element that has been mapped.

Properties Table:

In addition to the inspection tree, the BotCity Windows Inspector also displays all existing properties of a mapped element.

The values in this table can be used as selectors in the code when searching for these elements.

Continuing the mapping¶

Before we move on to the code generation stage, let's map a few more elements.

Going back to Wordpad, we can click the Start button on the Windows Inspector again to continue the mapping.

This time, we will map, for example, the area of the document where the text is inserted and also the button to save the document.

Generating code from mapped elements¶

With the initial elements mapped, we can now generate the Python code that will be used as the basis for the automation.

In this section, we can define what the Backend will be for this application and also the appropriate code snippets for the current context.

• Generate code to launch the app: Generates the code snippet responsible for launching the application being mapped (assuming that every time the automation is executed, the application will start from scratch).
• Generate code to connect to the app: Generates the code snippet responsible for establishing a connection with the open instance of the application. This option can be unchecked if you already have this code snippet in your automation and are only mapping new elements.
• Generate code only for mapped elements: Generates the code using the selectors for each element in the list of mapped elements. This option does not consider the reference of parent elements in the generated code, so it may be necessary to adjust the parameters according to your needs.
• Generate code for mapped elements + parents: Generates the code using the selectors for each element in the list of mapped elements considering the parent elements. This option is recommended if you want the Inspector to generate code as close as possible to the final version.

After configuring the code to be generated, just click the Generate Code button for BotCity Windows Inspector to generate the code directly in the .py file.

Saving the document and finishing the process¶

As the last step of the automation, we will map the window that opens when we click the "Save" button. We can continue using exactly the same strategy we used before.

In this case, instead of mapping a specific element of the "Save as" window, we can map only the region of the main window itself.

In the code, we will later use a strategy to insert the path and save the document using this window reference.

Since the base code has already been generated previously, we can only check the option to generate the code related to the newly mapped element.

Final adjustments and code execution¶

With the base code generated by the BotCity Windows Inspector, we can refactor and make any necessary adjustments for the process.

In this case, we will simply include in the code the content that will be inserted into the document and also the path to save the file at the end.

Inserting text content in the document area:

We can simply adjust the type_keys method of the element related to the document, passing a string with the content as a parameter.

Identify in your code the section where this element is being manipulated and make the following modification:

...
target_element = bot.find_app_element(waiting_time=10000, from_parent_window=main_window, auto_id="59648", class_name="RICHEDIT50W", control_type="Document")
## Write content in the edit
target_element.type_keys("Hello! Welcome to BotCity!", with_spaces=True)
## Get the text of the edit
# target_element.text_block()

Passing the path to save the file:

To save the file, we will use a very interesting strategy using the "Save As" window that we mapped earlier.

With the reference of the window, we can again use type_keys to insert the file path as well as to execute an Enter action in the window.

Identify in your code the section where this element related to the "Save As" window is being manipulated and make the following modification:

...
target_element = bot.find_app_element(waiting_time=10000, from_parent_window=main_window, best_match="Save As", class_name="#32770", control_type="Window")
## Set the focus to this element
target_element.set_focus()
target_element.type_keys(r"C:\Users\Administrator\Documents\document.rtf")
target_element.type_keys("{ENTER}")

Exploring the code generated by the Inspector¶

When we run the final code after the adjustments, the bot will automatically perform the following actions:

• Open Wordpad
• Establish a connection with the application
• Perform the defined interactions: center the paragraph, click the Bold, Italic, and Underline buttons
• Type the text in the document area and save the file in the defined path

As mentioned earlier, the code generated by the Windows Inspector uses the Windows Applications module of the BotCity desktop development framework.

By default, the BotCity Windows Inspector will always try to generate the most complete code possible. However, adjustments and treatments may be necessary to ensure that the process works as expected.

Searching for elements¶

The Windows Inspector uses some key selectors in the code it generates to find a specific element.

The table below better describes some selectors that can be used:

If you wish to replace a parameter generated by the Inspector, or want to make a more complex combination, just use the properties table as a reference to obtain the information for each element.

With these values in hand, simply pass them as parameters to the find_app_element method.

target_element = bot.find_app_element(
    waiting_time=10000,
    from_parent_window=None,
    auto_id="59648",
    class_name="RICHEDIT50W",
    control_type="Document"
)

## Replacing 'auto_id' with the element's 'title' and removing 'class_name'
## It would also be possible to use them together if necessary
target_element = bot.find_app_element(
    waiting_time=10000,
    from_parent_window=None,
    title="Rich Text Window",
    control_type="Document"
)

Another interesting alternative is to access a specific element by its index, in relation to its parent element.

This type of approach becomes quite useful when there are no unique properties to filter by an element, or when multiple identical elements share the same properties.

In the scenario above, we have several 'Edit' type elements with shared properties and no unique identifiers.

This would make it difficult to search for a specific element since several elements could match the same filter.

# This code would end up being generic
# as it would not be filtering for a specific element
target_element = bot.find_app_element(
    waiting_time=10000,
    from_parent_window=parent_form,
    class_name="ThunderRT6TextBox",
    control_type="Edit"
)

In this scenario, we could use the parent element as a reference, and from it access the target element through its index.

# With the reference of the parent element,
# we can access an element by its index
target_element = parent_form.Edit3
target_element.type_keys("", with_spaces=True)

Visualizing the application structures in the code¶

If for any reason you need to take a deeper look at the structure of a specific window or element in the application directly in the code, you can use specific methods from pywinauto.

With the element reference in hand, simply call the dump_tree() or print_control_identifiers() method to display the element's structure in the terminal.

This alternative is quite interesting if you are having trouble searching for specific elements and want to debug.

# Obtaining the reference of the element,
# for example, the main window of the application
main_window = current_app.top_window()
# Displaying the component structure directly in the terminal
main_window.dump_tree()

Control Identifiers:

Dialog - 'Document - WordPad'    (L48, T42, R821, B918)
['Document - WordPadDialog', 'Dialog', 'Document - WordPad']
child_window(title="Document - WordPad", control_type="Window")
| 
| Pane - 'UIRibbonDockTop'    (L56, T73, R813, B189)
| ['UIRibbonDockTop', 'Pane', 'UIRibbonDockTopPane', 'Pane0', 'Pane1']
| child_window(title="UIRibbonDockTop", control_type="Pane")
|    | 
|    | Pane - 'Ribbon'    (L56, T73, R813, B189)
|    | ['RibbonPane', 'Ribbon', 'Pane2', 'RibbonPane0', 'RibbonPane1', 'Ribbon0', 'Ribbon1']
|    | child_window(title="Ribbon", control_type="Pane")
|    |    | 
|    |    | Pane - 'Ribbon'    (L56, T73, R813, B189)
|    |    | ['RibbonPane2', 'Ribbon2', 'Pane3']
|    |    | child_window(title="Ribbon", control_type="Pane")
|    |    |    | 
|    |    |    | Pane - ''    (L56, T73, R813, B220)
|    |    |    | ['Pane4']
|    |    |    |    | 
|    |    |    |    | Pane - 'Ribbon'    (L56, T42, R813, B189)
|    |    |    |    | ['RibbonPane3', 'Ribbon3', 'Pane5']
|    |    |    |    | child_window(title="Ribbon", control_type="Pane")
|    |    |    |    |    | 
|    |    |    |    |    | Toolbar - 'Quick Access'    (L94, T47, R160, B69)
|    |    |    |    |    | ['Toolbar', 'Quick AccessToolbar', 'Quick Access', 'Toolbar0', 'Toolbar1']
|    |    |    |    |    | child_window(title="Quick Access", control_type="ToolBar")
|    |    |    |    |    |    | 
|    |    |    |    |    |    | Button - 'Save'    (L94, T45, R116, B69)
|    |    |    |    |    |    | ['Save', 'Button', 'SaveButton', 'Button0', 'Button1']
|    |    |    |    |    |    | child_window(title="Save", control_type="Button")
|    |    |    |    |    |    | 
|    |    |    |    |    |    | Button - 'Undo'    (L116, T45, R138, B69)
|    |    |    |    |    |    | ['Button2', 'UndoButton', 'Undo']
|    |    |    |    |    |    | child_window(title="Undo", control_type="Button")
|    |    |    |    |    |    | 
|    |    |    |    |    |    | Button - 'Redo'    (L138, T45, R160, B69)
|    |    |    |    |    |    | ['Redo', 'Button3', 'RedoButton']
|    |    |    |    |    |    | child_window(title="Redo", control_type="Button")

...

Performing actions with elements¶

For certain types of elements, such as buttons and text fields, the Windows Inspector will also generate some action suggestions that can be performed with that element along with the code.

## Perform a default click action on the button
target_element.click()
## Perform a click action with a mouse move on the button
target_element.click_input()

## Write content in the edit
target_element.type_keys("", with_spaces=True) 
## Get the text of the edit
target_element.text_block()

## Select an item from the combo box, item can be either a 0 based index of the item to select, or it can be the string that you want to select
target_element.select()
## Return the selected item text from the combo box
target_element.selected_text()
## Return the text of all items in the combo box
target_element.texts()

## Set the focus to this element.
target_element.set_focus()
## Maximize the window
target_element.maximize()
## Close the window
target_element.close()

## Select a menu item using the path to the item
target_element.item_by_path("{Item name} -> {Item name} -> ...", exact=True).select()
## Return all menu items
target_element.items()

As we saw earlier, it is also possible to perform the action of a key or keyboard shortcut directly in the context of an element.

Having the reference of the target element, we can use the type_keys or send_keys method with the following combinations:

# Executes the shortcut Ctrl + A in the context of the element
target_element.type_keys("^a")
# Executes the shortcut Ctrl + C in the context of the element
target_element.type_keys("^c")

# Executes the shortcut Alt + F4 in the context of the element
target_element.type_keys('%{F4}')
# Executes the shortcut Alt + F + S in the context of the element
target_element.type_keys("%{f}{s}")

# Presses the ENTER key in the context of the element
target_element.type_keys('{ENTER}')
# Presses the TAB key in the context of the element
target_element.type_keys('{TAB}')
# Presses the SHIFT key in the context of the element
target_element.type_keys('{VK_SHIFT}')

Best practices when using BotCity Windows Inspector¶

Below, we list some tips and important points that can facilitate the use of Windows Inspector during the development of your automations.

If you have questions or want to see more about this type of content, feel free to explore the BotCity community channels.