Examples

Element Inspector

Using the nut.js Element Inspector for native UI element discovery and automation

element-inspectoraccessibilityui-automation

The Element Inspector lets you automate native applications by interacting with their UI element tree rather than matching pixels. This approach is more reliable across themes, resolutions, and minor UI changes.

When to Use Element Inspector

  • Native desktop apps where you need to interact with menus, buttons, text fields
  • Dynamic content that changes appearance but keeps the same structure
  • Accessibility testing to verify proper element labeling
  • Complex workflows involving nested UI hierarchies

For web automation, consider the Playwright Bridge or Selenium Bridge instead.

Installation

bash
npm i @nut-tree/element-inspector

Supported on Windows, macOS, and Linux.

Prerequisites

The target application must have accessibility features enabled. For example, Visual Studio Code requires:

json
"editor.accessibilitySupport": "on"

Exploring an Application's UI Tree

Before automating, you'll want to see what elements are available:

js
import {
    useConsoleLogger,
    ConsoleLogLevel,
    screen,
    windowWithTitle
} from "@nut-tree/nut-js";
import {useBolt} from "@nut-tree/bolt";
import { useElementInspector } from "@nut-tree/element-inspector";

useElementInspector();
import {elements} from "@nut-tree/element-inspector/win";
// For macOS:
// import {elements} from "@nut-tree/element-inspector/macos";
// For Linux:
// import {elements} from "@nut-tree/element-inspector/linux";

useConsoleLogger({logLevel: ConsoleLogLevel.DEBUG});
useBolt();

const app = await screen.find(windowWithTitle(/Visual Studio Code/));
await app.focus();

// Limit depth to avoid huge output (default is 100 levels)
const tree = await app.getElements(5);
console.log(JSON.stringify(tree, null, 2));

This outputs a tree structure showing each element's id, role, title, and type:

json
{
  "type": "Group",
  "children": [
    {
      "role": "toolbar",
      "type": "ToolBar",
      "children": [
        {
          "role": "button",
          "title": "Go Back (Alt+LeftArrow)",
          "type": "Button"
        }
      ]
    }
  ]
}

Finding and Clicking an Element

Once you know what elements exist, you can find and interact with them:

js
import {
    screen,
    windowWithTitle,
    mouse,
    Button,
    straightTo,
    centerOf
} from "@nut-tree/nut-js";
import {useBolt} from "@nut-tree/bolt";
import { useElementInspector } from "@nut-tree/element-inspector";

useElementInspector();
import {elements} from "@nut-tree/element-inspector/win";

useBolt();

const app = await screen.find(windowWithTitle(/Visual Studio Code/));
await app.focus();

// Find the File menu by title
const fileMenu = await app.find(elements.menuItem({title: "File"}));

if (fileMenu.region != null) {
    await screen.highlight(fileMenu.region);
    await mouse.move(straightTo(centerOf(fileMenu.region)));
    await mouse.click(Button.LEFT);
}

Finding Elements by Any Property

Use windowElementDescribedBy when you don't know the element type but know other properties:

js
import {screen, windowWithTitle} from "@nut-tree/nut-js";
import {useBolt} from "@nut-tree/bolt";
import { useElementInspector } from "@nut-tree/element-inspector";

useElementInspector();
import {windowElementDescribedBy} from "@nut-tree/element-inspector/win";

useBolt();

const app = await screen.find(windowWithTitle(/Visual Studio Code/));
await app.focus();

// Find any element with title "File", regardless of type
const element = await app.find(windowElementDescribedBy({title: "File"}));

You can query by any combination of these properties:

ts
interface WindowElementDescription {
    id?: string | RegExp;
    type?: string;
    title?: string | RegExp;
    value?: string | RegExp;
    selectedText?: string | RegExp;
    role?: string;
}

Finding All Matching Elements

Get all elements of a type, for example all menu items:

js
import {screen, windowWithTitle} from "@nut-tree/nut-js";
import {useBolt} from "@nut-tree/bolt";
import { useElementInspector } from "@nut-tree/element-inspector";

useElementInspector();
import {elements} from "@nut-tree/element-inspector/win";

useBolt();

const app = await screen.find(windowWithTitle(/Visual Studio Code/));
const menuItems = await app.findAll(elements.menuItem({}));

// Get all titles
const titles = await Promise.all(menuItems.map(item => item.title));
console.log("Menu items:", titles);

Waiting for Elements to Appear

Wait for UI elements that appear after some action:

js
import {screen, windowWithTitle} from "@nut-tree/nut-js";
import {useBolt} from "@nut-tree/bolt";
import { useElementInspector } from "@nut-tree/element-inspector";

useElementInspector();
import {elements} from "@nut-tree/element-inspector/win";

useBolt();

const app = await screen.find(windowWithTitle(/Visual Studio Code/));

// Click something that opens a menu, then wait for it
const menu = await app.waitFor(elements.menu({}));
console.log("Menu appeared");

Use the in: property to find elements within specific parent elements:

js
import {screen, windowWithTitle, mouse, Button, straightTo, centerOf} from "@nut-tree/nut-js";
import { useElementInspector } from "@nut-tree/element-inspector";

useElementInspector();
import {elements} from "@nut-tree/element-inspector/win";
import {useBoltWindowFinder} from "@nut-tree/bolt";

useBoltWindowFinder();

const explorer = await screen.find(windowWithTitle("This PC"));
await explorer.focus();

// Find "boot" folder inside a DVD drive, inside Desktop
const bootFolder = await explorer.find(elements.treeItem({
    title: "boot",
    in: elements.treeItem({
        title: /DVD.*/,
        in: elements.treeItem({
            title: "Desktop"
        })
    })
}));

await mouse.move(straightTo(centerOf(bootFolder.region)));
await mouse.click(Button.LEFT);

// Find files within a specific group
const files = await explorer.findAll(elements.listItem({
    title: /.*boot.*/,
    in: elements.group({
        title: "Files Currently on the Disc"
    })
}));

Ancestor and Descendant Relations

Use descendantOf and ancestorOf for flexible tree traversal that doesn't require the element to be a direct child or parent:

js
import {screen, windowWithTitle} from "@nut-tree/nut-js";
import {useBolt} from "@nut-tree/bolt";
import { useElementInspector } from "@nut-tree/element-inspector";

useElementInspector();
import {elements} from "@nut-tree/element-inspector/win";

useBolt();

const app = await screen.find(windowWithTitle(/My App/));

// Find a button anywhere inside a dialog (not just direct children)
const okButton = await app.find(elements.button({
    title: "OK",
    descendantOf: elements.dialog({ title: "Confirm" }),
}));

// Find the group that contains a specific button
const container = await app.find(elements.group({
    ancestorOf: elements.button({ title: "Submit" }),
}));

Sibling Relations

Target elements relative to their siblings using after, before, and siblingOf:

js
import {screen, windowWithTitle} from "@nut-tree/nut-js";
import {useBolt} from "@nut-tree/bolt";
import { useElementInspector } from "@nut-tree/element-inspector";

useElementInspector();
import {elements} from "@nut-tree/element-inspector/win";

useBolt();

const app = await screen.find(windowWithTitle(/My App/));

// Find the text field that appears after a "Username" label
const usernameField = await app.find(elements.textField({
    after: elements.staticText({ title: "Username" }),
}));

// Find the button that appears before "Cancel"
const submitBtn = await app.find(elements.button({
    before: elements.button({ title: "Cancel" }),
}));

// Find any button that is a sibling of a specific checkbox
const relatedBtn = await app.find(elements.button({
    siblingOf: elements.checkbox({ title: "Remember me" }),
}));

Position Modifiers

Use nthChild, firstChild, and lastChild to select elements by their position among siblings:

js
import {screen, windowWithTitle} from "@nut-tree/nut-js";
import {useBolt} from "@nut-tree/bolt";
import { useElementInspector } from "@nut-tree/element-inspector";

useElementInspector();
import {elements} from "@nut-tree/element-inspector/win";

useBolt();

const app = await screen.find(windowWithTitle(/My App/));

// Get the third tab (0-indexed)
const thirdTab = await app.find(elements.tab({
    nthChild: 2,
    in: elements.tabGroup({}),
}));

// Get the first item in a list
const firstItem = await app.find(elements.listItem({
    firstChild: true,
    in: elements.list({}),
}));

// Get the last menu item
const lastItem = await app.find(elements.menuItem({
    lastChild: true,
    in: elements.menu({}),
}));

Excluding Elements

Use notMatching to filter out unwanted matches:

js
import {screen, windowWithTitle} from "@nut-tree/nut-js";
import {useBolt} from "@nut-tree/bolt";
import { useElementInspector } from "@nut-tree/element-inspector";

useElementInspector();
import {elements} from "@nut-tree/element-inspector/win";

useBolt();

const app = await screen.find(windowWithTitle(/My App/));

// Find all enabled buttons (exclude disabled ones)
const enabledButtons = await app.findAll(elements.button({
    notMatching: elements.button({ isEnabled: false }),
}));

Tips

  • Start with getElements() to understand the app's structure before writing automation
  • Use shallow depth (e.g., getElements(3)) for faster exploration
  • Regex patterns work for title, id, and value properties
  • Element regions can be null if the element isn't currently visible - always check before interacting
  • Different apps expose different elements - what's available depends on how the app implements accessibility

For the complete API, see the Element Inspector plugin page.

Was this page helpful?