Screen Support

nut.js allows searching and waiting for images on your screen to either verify certain conditions, or use them for further processing.

Configuration

The nut.js screen comes with a config object which allows configuration of its behaviour.

confidence

screen.config.confidence specifies the required matching percentage required to mark a possible candidate for a given template image a match.

autoHighlight

screen.config.autoHighlight is a boolean toggle which enables automated highlighting of image search results. This will highlight the matching Region by showing an opaque window.

highlightDurationMs

screen.config.highlightDurationMs configures the millisecond duration a highlight window is shown.

highlightOpacity

screen.config.highlightOpacity configures the opacity of highlight windows. Ranges from 0 (fully transparent) to 1 ( fully opaque).

resourceDirectory

screen.config.resourceDirectory configures the location to load assets via imageResource. This allows configuration of resource locations depending on e.g. the current operating system.

One could provide multiple folders containing platform specific template images and choose the correct resource directory at runtime. Following this scheme, loading of platform specific images would be possible without changes to the source.

capture

capture allows you to capture a screenshot and store it to your filesystem.

captureRegion

captureRegion allows you to capture a screenshot of a desktop region and store it to your filesystem.

grab

grab allows you to retrieve an Image containing the current screen content.

grabRegion

grabRegion allows you to retrieve an Image containing the current content of a desktop region.

find

find takes a find query and tries to find a match on the main screen. It is possible to override the configured matching confidence and search region providing OptionalSearchParameters. In case of a match, the corresponding Region on screen is returned.

await mouse.move(straightTo(centerOf(screen.find(imageResource("image.png")))));

findAll

In contrast to find returning only the most probable match, findAll takes a find query and returns a list of all matched occurrences on the main screen. It is possible to override the configured matching confidence and search region providing OptionalSearchParameters. In case of a match, the corresponding Regions on screen is returned.

const matches = await screen.findAll(imageResource("image.png"));
for (const match of matches) {
    await mouse.move(straightTo(centerOf(match)));
}

highlight

When working with template images to e.g. move the mouse to certain positions it can be quite cumbersome to follow along without visual clues.

highlight allows you to display an opaque window overlay which makes it easier to visually follow detection / movement.

await screen.highlight(screen.find(imageResource("image.png")));

on

on allows you to register callbacks which will be executed once find, findAll or waitFor returns a match for a given template image.

This way it's possible to repeatedly execute actions whenever a certain query is detected on screen.

Example

import {screen, RGBA, pixelWithColor, mouse, straightTo, Point} from '@nut-tree/nut-js';

const colorQuery = pixelWithColor(new RGBA(0, 0, 0, 255));
const secondQuery = pixelWithColor(new RGBA(43, 45, 48, 255));

screen.on(colorQuery, async (matchResult) => {
    await mouse.move(straightTo(matchResult.location));
});

screen.on(secondQuery, async (matchResult) => {
    console.log('Second query found');
});

await screen.find(colorQuery); // This will trigger the on callback registered for colorQuery
await mouse.move(straightTo(new Point(100, 100)));
await screen.find(secondQuery); // This will trigger the on callback registered for secondQuery
await screen.find(colorQuery); // This will trigger the on callback registered for colorQuery again

waitFor

Similar to find, waitFor will search for a find query on a system's main screen.

While find will fail immediately if no match is found, waitFor allows configuration of a timeout in milliseconds during which the screen will repeatedly be scanned for the query. The interval in milliseconds, in which the image search is carried out, is configurable as well. Its default value is set to 500ms. Once the configured timeout is reached with no match, waitFor will fail.

await mouse.move(straightTo(centerOf(screen.waitFor(imageResource("image.png"), 3000, 500))));

colorAt

colorAt will return RGBA color information at a specified pixel location within the system's main screen.

For example, a black pixel would be represented as:

RGBA { R: 0, G: 0, B: 0, A: 255 }

width

width returns the main screen's width in pixels.

height

height returns the main screen's height in pixels.