Automate outside the box - nut.js remote plugins

It all started two years ago with nut.js v2.0.0 - and now I'm finally able to show you how this plan has turned out!

Design For Flexibility

With release v2.0.0 nut.js received a big overhaul of its internal architecture. What used to be a rather tightly coupled framework has been changed into a tight user-facing API and a loose core framework.

This architecture gives nut.js the ability to be extensible and allows users to tweak it to their specific needs. A perfect example for this are two existing plugins, @nut-tree/playwright-bridge and @nut-tree/selenium-bridge.

Instead of re-inventing the wheel by coming up with a custom web testing DSL for nut.js, we make use of the framework's extensibility to integrate nut.js with e.g. Playwright.

Custom provider packages make it possible to attach a Playwright context or a Selenium driver to nut.js. This allows you to use nut.js features like image search or OCR in your web tests, re-using the well-known public APIs of nut.js.

Thinking Outside Your Box

All of this is possible because nut.js essentially relies on four core building blocks:

• It uses a screen to capture screen content.
• It uses a mouse to control the mouse cursor.
• It uses a keyboard to simulate keyboard input.
• It uses a clipboard to copy and paste text.

These four core building blocks can be implemented in a number of different ways. They might be provided by your local machine, but they can just as well be "virtualized" by e.g. a headless Playwright session.

But if you take another look at the four core building blocks, maybe some other sources for these building blocks might come to your mind?

Let's look at the following code sample which was used in the recording at the top of this page:

import "@nut-tree/nl-matcher"

import { Key, imageResource, keyboard, screen, sleep } from "@nut-tree/nut-js"

await sleep(3000);

await keyboard.type(Key.LeftWin);
await screen.waitFor(imageResource("start.png"), 10000, 1000);
await keyboard.type("notepad");
await keyboard.type(Key.Enter);
await screen.waitFor(imageResource("notepad.png"), 10000, 1000);
await keyboard.type("Hello world! nut.js ❤️ you!");

Looks like a pretty standard workflow to open Notepad and type "Hello world! nut.js ❤️ you!" on a Windows machine, right?

But I'm on a Mac, sitting in the first floor of my house? And the only Windows machines I own are located in my office in the basement?

Let me show you how I solved this problem without taking the stairs down to my office:

import "@nut-tree/nl-matcher"

import { Key, imageResource, keyboard, screen, sleep } from "@nut-tree/nut-js"

import { connect } from "@nut-tree/remote-plugin";

const { useRemoteScreen, useRemoteKeyboard, isConnected, disconnect } = await connect({
	hostname: process.env.REMOTE_HOSTNAME,
	port: parseInt(process.env.REMOTE_PORT),
	username: process.env.REMOTE_USERNAME,
	password: process.env.REMOTE_PASSWORD,
});

console.log(`Connected: ${isConnected()}`);

useRemoteScreen();
useRemoteKeyboard();

await sleep(3000);

await keyboard.type(Key.LeftWin);
await screen.waitFor(imageResource("start.png"), 10000, 1000);
await keyboard.type("notepad");
await keyboard.type(Key.Enter);
await screen.waitFor(imageResource("notepad.png"), 10000, 1000);
await keyboard.type("Hello world! nut.js ❤️ you!");

disconnect();

Similar to how the Playwright bridge works, the remote plugin is able to provide an abstract screen and keyboard, mouse and clipboard by connecting to a remote system, which allows you to easily scale out your tests to multiple machines. You simply connect to a remote machine and set up your providers via useRemoteScreen() and useRemoteKeyboard() and your tests are no longer running on your local machine. All hidden behind the stable public APIs of nut.js you are now running on-screen image search or OCR on a remote machine.

The best thing here is the fact that this is once again based on existing technologies. Remote plugins are built on top of VNC or RDP, so there's no setup on the remote machine required.

Remote plugins can turn nut.js into a scriptable VNC or RDP client, which means if you're able to log in to a remote machine, you're able to use nut.js to perform remote testing or automation!

Streamlining Remote Testing with Native Framework Support

Let's talk about the benefits of native remote testing and automation. I have a background in test automation, so I'm no stranger to this topic.

One common approach to perform remote automation was to perform it through a remote desktop application, e.g. a VNC client. But instead of solving a problem, one actually gains another one. By automating an application through another application you get twice the work for twice the amount of fun instability. Additionally, you can't simply run a remote desktop application in the background, because if it's not visible on your screen, your automation script is not able to process it's content. So you either have your screen blocked by the remote application or you have to be creative to work around this.

One approach would be to run the automation script in a Docker container that provides a headless VNC session. But these Docker images tend to become rather big and cumbersome to maintain.

Having native remote support in the framework solves both of these issues. You don't need to run an additional application, you can simply connect to a remote machine and start automating it's content, directly accessing it's screen content. This eliminates a whole bunch of hassles and makes remote testing a more stable and reliable experience.

You also don't need a Docker image providing a full desktop environment, a lightweight node image is more than enough.

This in turn reduces the amount of resources that need to be allocated and makes remote testing more performant. Especially in scenarios where you want to monitor desktop applications on multiple machines, this is a huge win!

Conclusion

Native remote support is a huge milestone for nut.js! It'll bring a whole new level of stability and flexibility to your remote testing setup.

The plugin is not yet released, but if it already caught your attention, feel free to reach out to info@dry.software to learn more about it!