@nut-tree/playwright-bridge

Installation

npm i @nut-tree/playwright-bridge

Buy

@nut-tree/playwright-bridge is included in the Solo and Team plans, as well as the OCR and nl-matcher packages.

Description

@nut-tree/playwright-bridge is not a classical nut.js plugin like e.g. @nut-tree/nl-matcher. It is a context provider that allows you to use Playwright in your nut.js scripts.

By passing either a BrowserContext or Page object to the Playwright bridge, nut.js can interact with the browser controlled by Playwright. This allows you to switch the nut.js context from your whole desktop to just the browser viewport, making it easy to use the nut.js image search or OCR features in the context of your Playwright tests.

Usage

Simply require / import the package to wire up the provider using either usePage or useContext:

import {
    straightTo,
    centerOf,
    mouse,
    Button,
    screen
} from "@nut-tree/nut-js";
import "@nut-tree/nl-matcher";

import {chromium, selectors} from 'playwright';
import {useContext, usePage} from "@nut-tree/playwright-bridge";

const browser = await chromium.launch({headless: true});
const ctx = await browser.newContext();

// Pass the Playwright context to the useContext function of the Playwright bridge
// From here on we're no longer using the whole desktop, but just the browser viewport
// Every action we carry out will be in the context of the browser, image search, mouse movement, keyboard input etc.
useContext({context: ctx});

const firstPage = await ctx.newPage();
await firstPage.goto('https://www.google.com');

// Alternatively, you can pass a Playwright page object to the usePage function
// This will make the Playwright page the active context for nut.js
// This is useful if you want to interact with a specific page in your Playwright test
//usePage({page: firstPage});

// This will scroll the browser viewport, even in headless mode with other applications focused
await mouse.scrollDown(40);

// Same goes for image search and mouse movement.
// All of this happens in the context of your Playwright test, not on your desktop.
// So even in headless mode, image search is targeting only your browser viewport.
await mouse.move(
    straightTo(
        centerOf(
            screen.find(imageResource("cookies.png"))
        )
    )
);
await mouse.click(Button.LEFT);

await ctx.close();
await browser.close();

Configuration

The only additional piece of configuration that both useContext and usePage accept is the trackPageChanges property.

export type PlaywrightBridgePageOptions = {
    page: Page;
    trackPageChanges?: boolean;
}
export type PlaywrightBridgeContextConfigOptions = {
    context: BrowserContext;
    trackPageChanges?: boolean;
}

When set to true, the Playwright bridge will track changes to the page and context and update the nut.js context accordingly. This way you don't have to call usePage with a new page object every time you navigate to a new page in your Playwright test, like in this Playwright example.

Playwright locators

@nut-tree/playwright-bridge also provides a custom selector engine that allows you to locate elements on a Playwright page by their position. This is useful when you want to interact with elements on a page that you can't easily locate by their attributes.

To use this custom locator, it has to be registered BEFORE a page is created! See Playwright documentation for reference

Let's revisit the previous example and see how it would look like with the custom locator:

import {
    centerOf,
    screen
} from "@nut-tree/nut-js";
import "@nut-tree/nl-matcher";

import {chromium, selectors} from 'playwright';
import {useContext, locateByPosition} from "@nut-tree/playwright-bridge";

// Register the custom locator by name `atPosition`
await selectors.register('atPosition', locateByPosition);

const browser = await chromium.launch({headless: true});
const ctx = await browser.newContext();

useContext({context: ctx});

const firstPage = await ctx.newPage();
await firstPage.goto('https://www.google.com');
await mouse.scrollDown(40);

// Search for the image and retrieve its center
const imagePosition = await centerOf(screen.find(imageResource("cookies.png")));

// Instead of using the typical nut.js way of mouse.move(...), mouse.click(...), we can just use the custom locator and continue using the Playwright syntax for clicking, typing etc.
await firstPage.locator(`atPosition=${firstPostion}`).click();

await ctx.close();
await browser.close();

Usage example

import {chromium, selectors} from 'playwright';
import {locateByPosition, useContext} from "@nut-tree/playwright-bridge";
import {centerOf, ConsoleLogLevel, imageResource, keyboard, mouse, screen, useConsoleLogger} from "@nut-tree/nut-js";
import "@nut-tree/nl-matcher";
import {useBoltWindowFinder} from "@nut-tree/bolt";

await selectors.register('atPosition', locateByPosition);

useConsoleLogger({logLevel: ConsoleLogLevel.DEBUG});
useBoltWindowFinder();

const runHeadless = false;

const browser = await chromium.launch({headless: runHeadless});
const ctx = await browser.newContext();

if (runHeadless) {
    screen.config.resourceDirectory = './headless';
} else {
    screen.config.resourceDirectory = './non-headless';
}

keyboard.config.autoDelayMs = 10;
mouse.config.mouseSpeed = 3000;
screen.config.autoHighlight = true;

useContext({context: ctx});

const firstPage = await ctx.newPage();
await firstPage.goto('https://www.google.com');
await mouse.scrollDown(40);

const firstPostion = await centerOf(screen.find(imageResource("target.png"), {
    confidence: 0.9,
    providerData: {validateMatches: true}
}));
await firstPage.locator(`atPosition=${firstPostion}`).click();
const searchBox = await screen.find(imageResource('search.png'));
const searchBoxPosition = await centerOf(searchBox);
const box = await firstPage.locator('atPosition=' + searchBoxPosition);
await box.click();
await box.fill('nut.js');
await box.press('Enter');
const result = await screen.waitFor(imageResource('google.png'), 7000, 1000);
await firstPage.screenshot({
    path: 'result.png',
    clip: {x: result.left, y: result.top, width: result.width, height: result.height}
})

await ctx.close();
await browser.close();

Usage with @playwright/test

@nut-tree/playwright-bridge can be used in combination with @playwright/test to integrate nut.js features like image search or OCR into your Playwright tests. To do so in the most efficient way, nut.js provides a set of custom matchers for @playwright/test that allow you to use nut.js features as part of your Playwright tests.

To set up the custom matchers, you need to import playwrightMatchers from @nut-tree/playwright-bridge and pass it on to expect.extend(playwrightMatchers);.

import { expect, test } from "@playwright/test";
import { playwrightMatchers } from "@nut-tree/playwright-bridge";

// From here on, you can use the custom matchers provided by nut.js
expect.extend(playwrightMatchers);

Once set up, you can make use of nut.js matchers to write your test expectations.

Available matchers

toBeAt

toBeAt is a custom matcher that allows you to verify that your mouse is at a specific position on screen.

await expect(mouse).toBeAt(new Point(100, 100));
await expect(mouse).not.toBeAt(new Point(10, 10));

toBeIn

toBeIn is a custom matcher that allows you to verify that your mouse is within a specific region on screen.

await expect(mouse).toBeIn(new Region(10, 25, 70, 30));
await expect(mouse).not.toBeIn(new Region(0, 0, 10, 10));

toShow

toShow is a versatile custom matcher that allows you to verify that an image is shown on screen, a text is displayed on screen, or a specific color is present on screen.

You can pass any search query supported by find and it'll verify whether the queried item is present on screen.

await expect(screen).toShow(imageResource("image.png"));
await expect(screen).toShow(textLine("Hello, World!"));
await expect(screen).toShow(pixelWithColor(new RGBA(255, 0, 0, 255)));
await expect(screen).not.toShow(windowWithTitle(/Explorer/));

toWaitFor

toWaitFor is to toShow what screen.waitFor is to screen.find. It allows you to wait any search query supported by find to appear on screen within a specified timeout.

Its signature is similar to screen.waitFor, so you can specify both a timeout and an update interval.

await expect(screen).toWaitFor(imageResource("image.png"), 6000, 300);
await expect(screen).toWaitFor(textLine("Hello, World!"), 7000, 500, {providerData: {preprocessConfig: {binarize: false}}});
await expect(screen).toWaitFor(pixelWithColor(new RGBA(255, 0, 0, 255)));
await expect(screen).not.toWaitFor(windowWithTitle(/Explorer/));

The default values for timeoutMs and updateIntervalMs are 5000 and 500, respectively.

toHaveColor

toHaveColor allows you to verify that a specific color is present at a given position on screen.

const pointInQuestion = new Point(100, 100);
await expect(pointInQuestion).toHaveColor(pixelWithColor(new RGBA(255, 0, 0, 255)));
await expect(pointInQuestion).not.toHaveColor(pixelWithColor(new RGBA(255, 0, 255, 255)));

Full example

import "@nut-tree/nl-matcher";

import {
    Button,
    Key,
    centerOf,
    imageResource,
    keyboard,
    mouse,
    screen,
    sleep,
    textLine,
} from "@nut-tree/nut-js";
import { expect, test } from "@playwright/test";
import { playwrightMatchers, usePage } from "@nut-tree/playwright-bridge";

// From here on, you can use the custom matchers provided by nut.js
expect.extend(playwrightMatchers);

test("create new program", async ({ page }) => {
    // Pass the Playwright page object to the usePage function to make the Playwright page the active context for nut.js
    usePage({ page });

    const newButton = await screen.waitFor(
        imageResource("new_button.png"),
        7000,
        500
    );
    await mouse.setPosition(centerOf(newButton));
    await mouse.click(Button.LEFT);

    const newActionButton = await screen.waitFor(
        imageResource("new_action.png"),
        7000,
        500,
        { providerData: { validateMatches: true } }
    );
    await mouse.setPosition(centerOf(newActionButton));
    await mouse.click(Button.LEFT);
    await sleep(300);
    await keyboard.type("my_new_test");
    await keyboard.type(Key.Enter);
    
    // The following test expectations are using the custom matchers provided by nut.js
    await expect(screen).toWaitFor(
        textLine(/^my_new_test . Ready$/),
        10000,
        500,
        { confidence: 0.8 }
    );
    await expect(screen).not.toShow(imageResource("error.png"), {
        providerData: { validateMatches: true },
    });
});

Buy

@nut-tree/playwright-bridge is included in the Solo and Team plans, as well as the OCR and nl-matcher packages.