Element Inspection

element-inspector

Inspect and interact with GUI elements using platform accessibility APIs.

Overview

The @nut-tree/element-inspector plugin implements the ElementInspectionProviderInterface to enable inspection and interaction with GUI elements within windows. It uses platform accessibility APIs to find, query, and interact with UI elements.

Find Elements

Locate UI elements by role, title, type, or value

win.find(elements.menuItem({title: "File"}))

Element Hierarchy

Inspect the full element tree of a window

win.getElements(5)

Element Relations

Target nested elements with parent-child relations

in: elements.treeItem({title: ...})

Installation

typescript
npm i @nut-tree/element-inspector

Subscription Required

This package is included in Solo and Team subscription plans.

Setup

Activate the plugin with useElementInspector() and import elements from the platform-specific subpath:

typescript
import { useElementInspector } from "@nut-tree/element-inspector";

// macOS
import { elements } from "@nut-tree/element-inspector/macos";

// Windows
import { elements } from "@nut-tree/element-inspector/win";

// Linux
import { elements } from "@nut-tree/element-inspector/linux";

useElementInspector();

Retrieving Window Elements

Examine a window's element hierarchy using getElements():

typescript
const items = await win.getElements();
console.log(JSON.stringify(items, null, 2));

Depth Limit

By default, getElements() will return a maximum of 1000 elements. Pass a number to modify the amount of elements for better performance or more details.

Element Structure

Each element contains:

typescript
interface WindowElement {
  id?: string;
  type?: string;
  title?: string;
  value?: string;
  role?: string;
  subRole?: string;
  className?: string;
  helpText?: string;
  selectedText?: string;
  clickablePoint?: Point;
  region?: Region;
  expandCollapsedState?: ExpandCollapseState;
  isFocused?: boolean;
  isEnabled?: boolean;
  isVisible?: boolean;
  isSelected?: boolean;
  isChecked?: boolean;
  isRequiredForForm?: boolean;
  isKeyboardFocusable?: boolean;
  isReadOnly?: boolean;
  children?: WindowElement[];
};

Finding Elements

Locate elements using WindowElementQuery objects with WindowElementDescription properties. Properties accept strings or RegExp patterns:

typescript
// Find a single element
const fileMenu = await win.find(elements.menuItem({ title: "File" }));

// Find all matching elements (returns empty array if none found)
const menuItems = await win.findAll(elements.menuItem({}));

// Wait for an element to appear
const menu = await win.waitFor(elements.menu({}));

Property Matchers

Match elements by their properties. String properties also accept RegExp patterns:

id

id?: string | RegExp

Element identifier

role

role?: string

Accessibility role of the element

type

type?: string

Element type

className

className?: string | RegExp

CSS class name of the element

title

title?: string | RegExp

Element title or label

value

value?: string | RegExp

Element value

selectedText

selectedText?: string | RegExp

Currently selected text in the element

helpText

helpText?: string | RegExp

Help or tooltip text associated with the element

State Matchers

Match elements by their current state:

expandCollapsedState

expandCollapsedState?: "EXPANDED" | "COLLAPSED" | "TRANSITIONING"

Expand/collapse state of the element

isFocused

isFocused?: boolean

Whether the element currently has focus

isEnabled

isEnabled?: boolean

Whether the element is enabled

isVisible

isVisible?: boolean

Whether the element is visible

isChecked

isChecked?: boolean

Whether the element is checked (checkboxes, radio buttons)

isSelected

isSelected?: boolean

Whether the element is selected

isRequiredForForm

isRequiredForForm?: boolean

Whether the element is required for form submission

isDataValidForForm

isDataValidForForm?: boolean

Whether the element's data is valid for form submission

isKeyboardFocusable

isKeyboardFocusable?: boolean

Whether the element can receive keyboard focus

isReadOnly

isReadOnly?: boolean

Whether the element is read-only

typescript
// Find an enabled, visible checkbox that is checked
const checkbox = await win.find(elements.checkbox({
    title: "Accept Terms",
    isEnabled: true,
    isVisible: true,
    isChecked: true,
}));

// Find an expanded tree item
const expanded = await win.find(elements.treeItem({
    title: "Documents",
    expandCollapsedState: "EXPANDED",
}));

Hierarchical Relations

Target elements based on their position in the element tree:

in

in?: WindowElementQuery

Direct parent element (resolves innermost to outermost)

descendantOf

descendantOf?: WindowElementQuery

Any ancestor element (not just direct parent)

ancestorOf

ancestorOf?: WindowElementQuery

Any descendant element (not just direct child)

typescript
// Direct parent with 'in'
const dvdItem = await explorer.find(
    elements.treeItem({
        title: "boot",
        in: elements.treeItem({
            title: /DVD.*/,
            in: elements.treeItem({ title: "Desktop" }),
        }),
    })
);

// Any ancestor with 'descendantOf'
const button = await win.find(elements.button({
    title: "OK",
    descendantOf: elements.dialog({ title: "Confirm" }),
}));

Sibling Relations

Target elements relative to their siblings:

after

after?: WindowElementQuery

The element must appear after this sibling

before

before?: WindowElementQuery

The element must appear before this sibling

siblingOf

siblingOf?: WindowElementQuery

The element must be a sibling of this element

typescript
// Find a text field that appears after a label
const input = await win.find(elements.textField({
    after: elements.staticText({ title: "Username" }),
}));

// Find the menu item before "Exit"
const item = await win.find(elements.menuItem({
    before: elements.menuItem({ title: "Exit" }),
}));

// Find any button that is a sibling of a specific checkbox
const btn = await win.find(elements.button({
    siblingOf: elements.checkbox({ title: "Remember me" }),
}));

Position Modifiers

Narrow down matches by position among siblings:

nthChild

nthChild?: number

Match the nth child (0-indexed)

firstChild

firstChild?: boolean

Match only the first child

lastChild

lastChild?: boolean

Match only the last child

typescript
// Get the third tab in a tab bar
const tab = await win.find(elements.tab({
    nthChild: 2,
    in: elements.tabGroup({}),
}));

// Get the first item in a list
const first = await win.find(elements.listItem({
    firstChild: true,
    in: elements.list({}),
}));

// Get the last menu item
const last = await win.find(elements.menuItem({
    lastChild: true,
    in: elements.menu({}),
}));

Logic

Exclude elements from results:

notMatching

notMatching?: WindowElementQuery

Exclude elements that match this query

typescript
// Find all buttons except disabled ones
const buttons = await win.findAll(elements.button({
    notMatching: elements.button({ isEnabled: false }),
}));

// Find menu items that are not separators
const items = await win.findAll(elements.menuItem({
    notMatching: elements.menuItem({ role: "separator" }),
}));

Factory Functions

The elements object provides platform-specific factory functions for creating element queries. Common functions include:

typescript
elements.menuItem({ title: "File" })
elements.menu({})
elements.button({ title: "OK" })
elements.textField({ value: "..." })
elements.treeItem({ title: "Documents" })
elements.checkbox({ title: "Enable" })

For type-agnostic searching, use windowElementDescribedBy() instead.

Troubleshooting

Few or No Elements Returned

Some applications require dedicated accessibility modes. For example, Visual Studio Code needs "editor.accessibilitySupport": "on" in its settings.

Unstable Element Positions

Dynamic containers (like menus in transition) may require brief delays before querying to ensure position stability.

Element Search

Learn about element search capabilities in nut.js

bolt

Required native platform provider for element inspection

Was this page helpful?