Introducing the @nut-tree/element-inspector Plugin for Windows

I'm thrilled to announce the release of the latest nut.js plugin, @nut-tree/element-inspector, now available in beta for Windows users. This powerful new tool is designed to enhance the capabilities of nut.js by providing an implementation of the ElementInspectionProviderInterface. This release marks a significant step forward in our commitment to developing robust tools for GUI automation and testing.

Key Features and Availability

@nut-tree/element-inspector enables developers to delve into the GUI elements of a window, providing a detailed JSON output of the element hierarchy. This not only aids in understanding the structure of the window but also facilitates precise interaction with its elements.

Current Availability

• Beta Release: The plugin is currently in beta. We encourage you to try it out and provide feedback to help us improve.
• Windows Exclusive: At this time, the plugin is available exclusively for Windows platforms. We plan to expand support to other platforms in future updates.

Getting Started

To get started with @nut-tree/element-inspector, simply install it using npm:

npm i @nut-tree/element-inspector

Pricing Plans

@nut-tree/element-inspector is included in both our Solo and Team plans, making it accessible to individual developers and teams alike.

Deep Dive into Usage

The plugin's functionality can be explored through various use cases:

1. Retrieve and Inspect Window Elements: Quickly fetch and inspect elements up to a specified depth within the window hierarchy.

2. Search for Specific Elements: Utilize the WindowElementQuery to find specific elements based on various properties like type, title, and role.

3. Handle Multiple Elements: Efficiently manage multiple instances of elements and perform collective operations with them.

4. Dynamic Element Handling: Employ methods to wait for elements to appear or stabilize within the window, which is crucial for dynamic GUIs.

Here's a quick snippet to demonstrate the new possibilities to search and interact with window elements:

import {
    useConsoleLogger,
    ConsoleLogLevel,
    screen,
    windowWithTitle,
    mouse,
    Button,
    straightTo,
    centerOf
} from "@nut-tree/nut-js";
import {useBolt} from "@nut-tree/bolt";
import "@nut-tree/element-inspector";
import {elements} from "@nut-tree/element-inspector/win";

useConsoleLogger({logLevel: ConsoleLogLevel.DEBUG});
useBolt();

const vs = await screen.find(windowWithTitle(/Visual Studio Code/));
await vs.focus();

const fileMenu = await vs.find(elements.menuItem({title: "File"}));
if (fileMenu.region != null) {
    await screen.highlight(fileMenu.region);
    await mouse.move(straightTo(centerOf(fileMenu.region)));
    await mouse.click(Button.LEFT);
}

Documentation

Head over to the element-inspector documentation for detailed information on the plugin's capabilities.

Troubleshooting and Tips

Given the complex nature of element inspection, here are a few tips for troubleshooting common issues:

• Enable Accessibility Modes: Some applications require specific accessibility settings to be enabled for full functionality.
• Stability of Dynamic Elements: When interacting with dynamic elements like unfolding menus, adding slight delays can ensure accurate element captures.

Future Plans

Looking ahead, we are excited to expand the capabilities of @nut-tree/element-inspector with more features and broader platform support. Your feedback during this beta phase is invaluable as we strive to refine and enhance the plugin.

Stay tuned for updates, and happy inspecting!

All the best

Simon