On-Screen Search

Element Search

Find UI elements using accessibility APIs for reliable automation.

Overview

Element search uses the operating system's accessibility APIs to find UI elements. Unlike image or text search, element search works with the actual UI structure, making it more reliable across different themes, resolutions, and screen configurations.

Plugin Required

Element search requires @nut-tree/element-inspector. Available for macOS, Windows, and Linux.

Find by Type

Find buttons, inputs, checkboxes, etc.

elements.button({})

Find by Title

Find elements by accessible title or label

elements.menuItem({ title: "File" })

Platform-Specific

Uses native accessibility APIs

import { elements } from ".../macos"

Installation

typescript
npm install @nut-tree/element-inspector

Setup

Activate the plugin and import platform-specific element factories:

typescript
import { getActiveWindow } from "@nut-tree/nut-js";
import { useElementInspector } from "@nut-tree/element-inspector";

useElementInspector();

// For macOS:
import { elements } from "@nut-tree/element-inspector/macos";

// For Windows:
// import { elements } from "@nut-tree/element-inspector/win";

// For Linux:
// import { elements } from "@nut-tree/element-inspector/linux";

const win = await getActiveWindow();

Quick Reference

find (with element)

window.find(elements.button({ title: ... }))
Promise<WindowElement>

Find an element matching the query

findAll (with element)

window.findAll(elements.menuItem({}))
Promise<WindowElement[]>

Find all elements matching the query

waitFor (with element)

window.waitFor(elements.dialog({}), timeout)
Promise<WindowElement>

Wait until an element appears

Finding Elements

Element search uses factory functions from the elements object. Each factory corresponds to a UI element type and returns a query descriptor.

Find by Element Type

Use element factory functions to find specific UI element types.

typescript
import { getActiveWindow } from "@nut-tree/nut-js";
import { useElementInspector } from "@nut-tree/element-inspector";

useElementInspector();
import { elements } from "@nut-tree/element-inspector/macos";

const win = await getActiveWindow();

// Find a button
const button = await win.find(elements.button({}));

// Find all menu items
const menuItems = await win.findAll(elements.menuItem({}));

// Available element types vary by platform:
// macOS: button, menuItem, menu, textField, checkBox, etc.
// Windows: button, menuItem, menu, edit, checkBox, etc.

Find by Title

Find elements by their accessible title or label.

typescript
import { getActiveWindow, mouse, centerOf, straightTo, Button as MouseButton } from "@nut-tree/nut-js";
import { useElementInspector } from "@nut-tree/element-inspector";

useElementInspector();
import { elements } from "@nut-tree/element-inspector/macos";

const win = await getActiveWindow();

// Find menu item by title
const fileMenu = await win.find(elements.menuItem({ title: "File" }));

// Click on it using the element's region
await mouse.move(straightTo(centerOf(fileMenu.region)));
await mouse.click(MouseButton.LEFT);

Use windowElementDescribedBy() to search by title across element types.

typescript
import { getActiveWindow } from "@nut-tree/nut-js";
import { useElementInspector } from "@nut-tree/element-inspector";

useElementInspector();
import { windowElementDescribedBy } from "@nut-tree/element-inspector/macos";

const win = await getActiveWindow();

// Find any element with this title (regardless of type)
const element = await win.find(windowElementDescribedBy({ title: "Save" }));

Waiting for Elements

Wait for UI elements to appear, useful for dynamic content or after actions.

typescript
import { getActiveWindow } from "@nut-tree/nut-js";
import { useElementInspector } from "@nut-tree/element-inspector";

useElementInspector();
import { elements } from "@nut-tree/element-inspector/macos";

const win = await getActiveWindow();

// Wait for a menu to appear
const menu = await win.waitFor(elements.menu({}));

// Wait for a specific dialog with timeout
try {
    await win.waitFor(elements.dialog({}), 10000);
    console.log("Dialog appeared");
} catch (error) {
    console.log("Dialog did not appear in time");
}

Element Relations

Element queries support relation modifiers to target elements based on their position in the UI tree. This lets you disambiguate elements that share the same type or title.

Hierarchical Relations

Target elements by their parent-child relationship in the element tree:

typescript
// Direct parent with 'in' (resolves innermost to outermost)
const item = await win.find(elements.treeItem({
    title: "boot",
    in: elements.treeItem({
        title: /DVD.*/,
        in: elements.treeItem({ title: "Desktop" }),
    }),
}));

// Any ancestor with 'descendantOf'
const button = await win.find(elements.button({
    title: "OK",
    descendantOf: elements.dialog({ title: "Confirm" }),
}));

// Any descendant with 'ancestorOf'
const container = await win.find(elements.group({
    ancestorOf: elements.button({ title: "Submit" }),
}));

Sibling Relations

Target elements relative to their siblings:

typescript
// Find a text field after a specific label
const input = await win.find(elements.textField({
    after: elements.staticText({ title: "Username" }),
}));

// Find a button that is a sibling of a checkbox
const btn = await win.find(elements.button({
    siblingOf: elements.checkbox({ title: "Remember me" }),
}));

Position Modifiers

Narrow down matches by position among siblings:

typescript
// Get the third tab in a tab bar
const tab = await win.find(elements.tab({
    nthChild: 2,
    in: elements.tabGroup({}),
}));

// Get the first item in a list
const first = await win.find(elements.listItem({
    firstChild: true,
    in: elements.list({}),
}));

Exclusion

Exclude elements from results with notMatching:

typescript
// Find all buttons except disabled ones
const buttons = await win.findAll(elements.button({
    notMatching: elements.button({ isEnabled: false }),
}));

For the full list of relation modifiers and options, see the element-inspector plugin page.

Best Practices

Element Search Tips

  • Use element search when UI appearance might change (themes, scaling)
  • Works well for standard UI controls (buttons, menus, dialogs)
  • Use platform accessibility inspector tools to discover element properties
  • Elements have a .region property for mouse interactions

Platform Considerations

  • Element types and property names differ between platforms
  • Some applications may not expose accessibility information

Element Inspector Example

Complete guide to element-based automation with real-world scenarios

Was this page helpful?