On-Screen Search

Color Search

Find pixels and regions by color on screen.

Overview

Color search allows you to find pixels or regions based on their color. This is useful for detecting UI states (active/inactive, success/error), finding colored indicators, or verifying visual feedback.

Read Color

Get the color of a pixel at a specific position

screen.colorAt(new Point(100, 200))

Find by Color

Search for pixels of a specific color

screen.find(pixelWithColor(red))

Quick Reference

colorAt

screen.colorAt(point: Point)

Promise<RGBA>

Get the RGBA color of a pixel at a specific position

find (with color)

screen.find(pixelWithColor(color: RGBA))

Promise<Region>

Find the first pixel matching a color

findAll (with color)

screen.findAll(pixelWithColor(color: RGBA))

Promise<Region[]>

Find all pixels matching a color

waitFor (with color)

screen.waitFor(pixelWithColor(color: RGBA), timeout?, interval?)

Promise<Region>

Wait until a pixel of the specified color appears

Reading Colors

Use screen.colorAt() to read the color of a pixel at a specific position.

typescript

import { screen, Point } from "@nut-tree/nut-js";

// Get color at a specific position
const color = await screen.colorAt(new Point(500, 300));

// RGBA object with values 0-255
console.log(`Red: ${color.R}`);
console.log(`Green: ${color.G}`);
console.log(`Blue: ${color.B}`);
console.log(`Alpha: ${color.A}`);

Checking for Specific Colors

typescript

import { screen, Point } from "@nut-tree/nut-js";

async function isPixelRed(x: number, y: number): Promise<boolean> {
    const color = await screen.colorAt(new Point(x, y));

    // Check if predominantly red
    return color.R > 200 && color.G < 50 && color.B < 50;
}

async function isPixelGreen(x: number, y: number): Promise<boolean> {
    const color = await screen.colorAt(new Point(x, y));

    // Check if predominantly green
    return color.R < 50 && color.G > 200 && color.B < 50;
}

// Check a status indicator
const statusX = 100, statusY = 50;
if (await isPixelGreen(statusX, statusY)) {
    console.log("Status: OK");
} else if (await isPixelRed(statusX, statusY)) {
    console.log("Status: Error");
}

Finding by Color

Search for pixels of a specific color on screen.

typescript

import { screen, RGBA } from "@nut-tree/nut-js";
import { pixelWithColor } from "@nut-tree/nut-js";

// Define the color to search for
const targetColor: RGBA = { R: 255, G: 0, B: 0, A: 255 };

// Find first pixel of that color
const location = await screen.find(pixelWithColor(targetColor));
console.log(`Found red pixel at: ${location.left}, ${location.top}`);

// Find all pixels of that color
const allRed = await screen.findAll(pixelWithColor(targetColor));
console.log(`Found ${allRed.length} red pixels`);

Color Tolerance

Exact color matching can be too strict for real-world use. Consider allowing a tolerance range when comparing colors, especially for anti-aliased elements or varying display settings.

Best Practices

Color Search Tips

Allow tolerance when comparing colors (colors vary slightly)
Use color search for simple state detection (status indicators)
Sample multiple nearby pixels for more reliable detection
Combine with region limiting for faster searches

Limitations

Display color profiles affect actual RGB values
Anti-aliasing creates color variations at edges
Dark mode vs light mode will have different colors
Full-screen color search can be slow—use regions

Color Search Example

Complete guide to color-based automation with real-world scenarios