nut.js logonutjs.dev

Window element inspection plugins

@nut-tree/element-inspector

Read this first!

This plugin is currently released in beta. Please report any issues you encounter.

Installation

npm i @nut-tree/element-inspector

Buy

@nut-tree/element-inspector is included in the Solo and Team plans.

Description

@nut-tree/element-inspector is a window element inspection plugin for nut.js. It provides an implementation of the ElementInspectionProviderInterface to enable inspection of GUI elements of a window.

Usage: Retrieve elements of a window

Let's dive right into an example:

import {
    useConsoleLogger,
    ConsoleLogLevel,
    screen,
    windowWithTitle
} from "@nut-tree/nut-js";
import {useBolt} from "@nut-tree/bolt";
import "@nut-tree/element-inspector";
import {elements} from "@nut-tree/element-inspector/win";
// Similar for macOS
// import {elements} from "@nut-tree/element-inspector/macos";

useConsoleLogger({logLevel: ConsoleLogLevel.DEBUG});
useBolt();

const vs = await screen.find(windowWithTitle(/Visual Studio Code/));
await vs.focus();
// We can configure the max depths of the search tree 
// const items = await vs.getElements(); // <== By default, it will search through up to 100 levels deep
const items = await vs.getElements(5); // <== This will search 5 levels deep
console.log(JSON.stringify(items, null, 2));

The above example is a great way to examine the elements of a window.

Here's an excerpt of the output:

{
  "id": "",
  "role": "",
  "title": "",
  "type": "Group",
  "children": [
    {
      "id": "",
      "role": "toolbar",
      "title": "",
      "type": "ToolBar",
      "children": [
        {
          "id": "",
          "role": "button",
          "title": "Go Back (Alt+LeftArrow)",
          "type": "Button",
          "children": [
            {
              ...
            }
          ]
        }
      ]
    },
    {
      ...
    }
  ]
}

As you can see, the output is a JSON object that represents hierarchy of elements of the window. Each entry in the JSON object is an element description which looks like this:

interface ShortWindowElementInfo {
    id?: string,
    role?: string,
    sublRole?: string,
    title?: string,
    type?: string,
    children?: ShortWindowElementInfo[]
}

This element tree is useful to understand the structure of the window and to identify elements that you want to interact with.

Usage: Search for a specific element of a window

import {
    useConsoleLogger,
    ConsoleLogLevel,
    screen,
    windowWithTitle,
    mouse,
    Button,
    straightTo,
    centerOf
} from "@nut-tree/nut-js";
import {useBolt} from "@nut-tree/bolt";
import "@nut-tree/element-inspector";
import {elements} from "@nut-tree/element-inspector/win";
// Similar for macOS
// import {elements} from "@nut-tree/element-inspector/macos";

useConsoleLogger({logLevel: ConsoleLogLevel.DEBUG});
useBolt();

const vs = await screen.find(windowWithTitle(/Visual Studio Code/));
await vs.focus();
// We can configure the max depths of the search tree 
// const items = await vs.getElements(); // <== By default, it will search through up to 100 levels deep
const items = await vs.getElements(); // <== This will search 5 levels deep
console.log(JSON.stringify(items, null, 2));

const fileMenu = await vs.find(elements.menuItem({title: "File"}));
if (fileMenu.region != null) {
    await screen.highlight(fileMenu.region);
    await mouse.move(straightTo(centerOf(fileMenu.region)));
    await mouse.click(Button.LEFT);
}

Searching for a specific element is a great way to interact with a window.

Window elements searches are based on WindowElementQuery objects.

{
    id: string;
    type: "window-element";
    by: {
        description: WindowElementDescription;
    }
}

These queries search for an element described by a WindowElementDescription object.

interface WindowElementDescription {
    id?: string | RegExp;
    type?: string;
    title?: string | RegExp;
    value?: string | RegExp;
    selectedText?: string | RegExp;
    role?: string;
}

A WindowElementDescription describes an element by a varying set of properties. However, it requires at least one property to be set to be valid.

When comparing the WindowElementDescription object to an entry in the getElements() tree you'll notice that you can describe elements by their properties returned by the getElements() method. So to search for a specific element, you can use the properties of the element you're looking for to create a query which specifies the element you're looking for.

To avoid cumbersome object creation, the elements object provides a set of factory functions to create WindowElementQuery objects. Element types are platform dependent, so the factory functions are grouped by platform.

To use Windows element types, import the elements object from @nut-tree/element-inspector/win.

If you either do not know the type of element you're looking for, or if you want to search for an element with a specific title, no matter the type, you can use the windowElementDescribedBy factory function to construct a WindowElementQuery.

import {windowWithTitle} from "@nut-tree/nut-js";
import {useBolt} from "@nut-tree/bolt";
import "@nut-tree/element-inspector";
import {windowElementDescribedBy} from "@nut-tree/element-inspector/win";

useBolt();

const vs = await screen.find(windowWithTitle(/Visual Studio Code/));
await vs.focus();

// This line searches for any element with the title "File", no matter the type
const elementWithTitle = await vs.find(windowElementDescribedBy({title: "File"}));

Usage: Search for multiple instances of an element in a window

The element inspection API follows the same pattern as the screen API.

So besides find, there is also a findAll method that returns all instances of an element that match the query.

import {
    useConsoleLogger,
    ConsoleLogLevel,
    screen,
    windowWithTitle,
} from "@nut-tree/nut-js";
import {useBolt} from "@nut-tree/bolt";
import "@nut-tree/element-inspector";
import {elements} from "@nut-tree/element-inspector/win";
// Similar for macOS
// import {elements} from "@nut-tree/element-inspector/macos";

useConsoleLogger({logLevel: ConsoleLogLevel.DEBUG});
useBolt();

const vs = await screen.find(windowWithTitle(/Visual Studio Code/));
const menuItems = await vs.findAll(elements.menuItem({}));

const itemTitles = await Promise.all(menuItems.map(item => item.title));
console.log(itemTitles);

As you can see, the findAll method returns an array of elements that match the query. If there are no elements that match the query, an empty array is returned.

Usage: Waiting for an element in a window

Since the element inspection API follows the same pattern as the screen API, it's also possible to wait for an element to appear in a window.

import {
    useConsoleLogger,
    ConsoleLogLevel,
    screen,
    windowWithTitle,
} from "@nut-tree/nut-js";
import {useBolt} from "@nut-tree/bolt";
import "@nut-tree/element-inspector";
import {elements} from "@nut-tree/element-inspector/win";
// Similar for macOS
// import {elements} from "@nut-tree/element-inspector/macos";

useConsoleLogger({logLevel: ConsoleLogLevel.DEBUG});
useBolt();

const vs = await screen.find(windowWithTitle(/Visual Studio Code/));
const newFileMenuItem = await vs.waitFor(elements.menu({}));
console.log(newFileMenuItem);

The above snippet will repeatedly search for a menu element in the window until it appears. This way you can wait for elements to appear in a window, like menus do after a click on a menu bar.

Stable elements

Even though it is possible to wait for an element to appear in a window, one still has to be careful when interacting with elements that are not stable. For example, a menu that appears after a click on a menu bar is in a moving state, because as the menu unfolds, its size changes.

This can lead to issues when trying to interact with the menu while it is still moving, because even though the menu might already exist, positions of its child elements might still change.

Element relations

Sometimes it is not obvious which element we're targeting. There might be multiple elements with the same title, or the element we're looking for might be a child of another.

In order to precisely target the element we're looking for, we can use the in relation to specify an element's relation to its parent element(s).

import { windowWithTitle, screen, straightTo, centerOf, mouse, Button, keyboard, Key } from "@nut-tree/nut-js";
import "@nut-tree/element-inspector";
import { elements } from "@nut-tree/element-inspector/win"
import { useBoltWindowFinder } from "@nut-tree/bolt";

useBoltWindowFinder();
mouse.config.autoDelayMs = 100;

const explorer = await screen.find(windowWithTitle("This PC"));

await explorer.focus();

const dvdItem = await explorer.find(elements.treeItem({
    title: "boot",
    in: elements.treeItem({
        title: /DVD.*/,
        in: elements.treeItem({
            title: "Desktop"
        })
    })
}));

await mouse.move(straightTo(centerOf(dvdItem.region)));
await mouse.click(Button.LEFT);

const bootFiles = await explorer.findAll(elements.listItem({
    title: /.*boot.*/,
    in: elements.group({
        title: "Files Currently on the Disc"
    })
}));

In this example, we're looking for a treeItem that is nested withing two parent treeItems.

const dvdItem = await explorer.find(elements.treeItem({
    title: "boot",
    in: elements.treeItem({
        title: /DVD.*/,
        in: elements.treeItem({
            title: "Desktop"
        })
    })
}));

Relation resolving

Relations are resolve in reverse order, so the innermost relation is resolved first.

Why so?

Resolving relations gives us an additional level of control over the search process. By resolving relations in reverse order we're able to immediately discard elements that do not match the innermost relation. So in case of very deep element hierarchies, we can avoid searching through the entire hierarchy by specifying a relation path from the root element to the element we're looking for.

Troubleshooting

Element inspection is a bit of a complex task and relies on the underlying platform's accessibility API. This means that the results of element inspection can vary depending on the platform and the application being inspected.

Here are some common issues and how to solve them:

No/only a few elements are returned when calling getElements:

If you did not manually limit the element search depth, it might be an application-specific issue. Some applications have a dedicated accessibility mode that needs to be enabled to expose elements to the accessibility.

Visual Studio Code, for example, has an accessibility mode that needs to be enabled:

"editor.accessibilitySupport": "on"

However, some applications might not have such a mode, or it might not be enough to enable it. In this case, you might need to use a different approach to interact with the application.

Detected element has no size/position

In scenarios where elements are located within dynamic containers, the element's position might not be stable. One example of such a scenario is a menu that appears after a click on a menu bar. You might have to add a short delay before querying the element to ensure that it has settled in its final position.

Buy

@nut-tree/element-inspector is included in the Solo and Team plans.

Previous
@nut-tree/plugin-azure
Next
@nut-tree/element-inspector/macos