On-Screen Search

Image Search

Find UI elements by searching for reference images on screen.

Overview

Image search is one of the most powerful features of nut.js. Instead of hardcoding screen coordinates, you can search for images and interact with UI elements visually. This makes your automation scripts more robust and maintainable across different screen resolutions and UI layouts.

Plugin Required

Image search requires the @nut-tree/nl-matcher package. Install it with npm install @nut-tree/nl-matcher and import it in your code.

Find Single

Locate the first occurrence of an image

screen.find(imageResource("button.png"))

Find All

Find all occurrences of an image

screen.findAll(imageResource("icon.png"))

Wait For

Wait until an image appears

screen.waitFor(imageResource("dialog.png"))

Quick Reference

find

screen.find(needle: Image, options?: FindOptions)
Promise<Region>

Search for an image on screen and return its location

findAll

screen.findAll(needle: Image, options?: FindOptions)
Promise<Region[]>

Find all occurrences of an image on screen

waitFor

screen.waitFor(needle: Image, timeout?: number, interval?: number, options?: FindOptions)
Promise<Region>

Wait until an image appears on screen

Basic Usage

Finding Images

Use screen.find() to locate an image on screen. First, capture a screenshot of the element you want to find and save it as an image file.

typescript
import { screen, imageResource } from "@nut-tree/nut-js";
import { useNlMatcher } from "@nut-tree/nl-matcher";

useNlMatcher();

// Find a button on screen
const buttonLocation = await screen.find(imageResource("button.png"));
console.log(`Found button at: ${buttonLocation.left}, ${buttonLocation.top}`);

// The returned Region contains:
// - left: X coordinate of the top-left corner
// - top: Y coordinate of the top-left corner
// - width: Width of the found region
// - height: Height of the found region

Click on Found Images

Combine image search with mouse actions to click on UI elements. Use centerOf() to get the center point of a found region, and straightTo() to create a movement path.

typescript
import { screen, mouse, imageResource, centerOf, straightTo, Button } from "@nut-tree/nut-js";
import { useNlMatcher } from "@nut-tree/nl-matcher";

useNlMatcher();

// Find and click a button
const button = await screen.find(imageResource("submit-button.png"));
await mouse.move(straightTo(centerOf(button)));
await mouse.click(Button.LEFT);

// Or as a one-liner
await mouse.move(straightTo(centerOf(await screen.find(imageResource("submit-button.png")))));
await mouse.click(Button.LEFT);

Find Multiple Occurrences

Use screen.findAll() to find all instances of an image on screen.

typescript
import { screen, mouse, imageResource, centerOf, straightTo, Button } from "@nut-tree/nut-js";
import { useNlMatcher } from "@nut-tree/nl-matcher";

useNlMatcher();

// Find all checkboxes on screen
const checkboxes = await screen.findAll(imageResource("checkbox.png"));
console.log(`Found ${checkboxes.length} checkboxes`);

// Click each checkbox
for (const checkbox of checkboxes) {
    await mouse.move(straightTo(centerOf(checkbox)));
    await mouse.click(Button.LEFT);
}

Waiting for Images

Use screen.waitFor() to wait until an image appears on screen. This is essential for handling loading states, animations, or asynchronous UI changes.

typescript
import { screen, imageResource } from "@nut-tree/nut-js";
import { useNlMatcher } from "@nut-tree/nl-matcher";

useNlMatcher();

// Wait for a dialog to appear (default timeout)
await screen.waitFor(imageResource("dialog.png"));

// Wait with custom timeout (in milliseconds)
await screen.waitFor(imageResource("loading-complete.png"), 10000);

// Handle timeout gracefully
try {
    await screen.waitFor(imageResource("success.png"), 5000);
    console.log("Success dialog appeared!");
} catch (error) {
    console.log("Timed out waiting for success dialog");
}

Timeout Behavior

If the image is not found within the timeout period, waitFor will throw an error. Always use try/catch when there's a possibility the image might not appear.

Limit your search to a specific region for dramatically improved performance. Searching a small region is orders of magnitude faster than searching the entire screen.

typescript
import { screen, imageResource, Region } from "@nut-tree/nut-js";
import { useNlMatcher } from "@nut-tree/nl-matcher";

useNlMatcher();

// Define search area (e.g., left sidebar)
const sidebar = new Region(0, 0, 300, 800);

// Search only within the sidebar
const icon = await screen.find(imageResource("icon.png"), {
    searchRegion: sidebar
});

// This is much faster than searching the entire screen!

Performance Optimization

Always use regions when you know approximately where an element will be. Searching a 300x300 region is ~100x faster than searching a 4K display.

Configuration

Confidence Threshold

The confidence threshold determines how closely an image must match. A value of 1.0 requires a perfect match, while lower values allow for slight variations (anti-aliasing, compression artifacts, etc.).

typescript
import { screen } from "@nut-tree/nut-js";

// Strict matching (default)
screen.config.confidence = 0.99;

// Allow slight variations (anti-aliasing, compression)
screen.config.confidence = 0.9;

// Very fuzzy matching (use with caution)
screen.config.confidence = 0.8;

Resource Directory

Set a base directory for your image files to use shorter paths.

typescript
import { screen, imageResource } from "@nut-tree/nut-js";
import { useNlMatcher } from "@nut-tree/nl-matcher";

useNlMatcher();

// Set resource directory for images
screen.config.resourceDirectory = "./images";

// Now you can use relative paths
const button = await screen.find(imageResource("button.png"));
// Loads from: ./images/button.png

Confidence Trade-offs

Lower confidence values may cause false positives (finding images that aren't actually there). Start with high confidence and lower it only if needed.

Best Practices

Image Preparation

  • Capture images at the same resolution you'll search at
  • Avoid including variable content (dynamic text, timestamps)
  • Use unique, distinctive parts of the UI
  • Keep images small and focused for faster searches
  • Test on the same display scaling (100%, 125%, etc.)

Common Pitfalls

  • Different screen resolutions will cause images not to match
  • Theme changes (light/dark mode) will affect image matching
  • Animations can cause timing issues—wait for stable states
  • JPEG compression can introduce artifacts that affect matching

Image Matching Example

Complete guide to image matching with real-world scenarios

Was this page helpful?