Skip to main content

New Years News

Simon Hofmann

Simon Hofmann

Maintainer @ nut.js

nut.js had a great start into the new year with its long awaited 2.0.0 release!

Ready for the future?#

First and foremost, this release is a huge invest in the future of nut.js by introducing a plugin system that allows users to provide their own implementation details for high-level functionality. Desktop automation is a really complex topic, and I came to the conclusion that establishing such a system is the only way to keep nut.js future-proof.

While nut.js is equipped with lots of useful functionality you might be missing some feature for your particular job at hand. The most recent changes enable you to tweak implementation details to your specific needs, giving you full control.

Additionally, having a plugin system allowed me to separate image matching code into its own module. This separation, grounded on user feedback, solves any compatibility issues between the nut.js core package and current / future releases of node and / or Electron.

The limiting factor with regard to newer versions of node / Electron has always been opencv4nodejs-prebuilt, the native addon used to perform image matching. With this module moved out of the core package, the default nut.js installation no longer suffers from this limitation! If you do not require on-screen image search, you're all set to use any current node version!

The cherry on top of all this is the fact that without the image matching module, the core package is usable on Apple Silicon. You can use nut.js on your precious M1 chips, no Rosetta required!

Where's Waldo?#

All this previous lamenting about compatibility with recent node and Electron version, Apple M1 compatibility and so on does not answer one particular question:

What about image matching using recent node versions or an M1 machine?

And I'm proud to say that there's an answer to this question: @nut-tree/nl-matcher.

Like I mentioned in an earlier blog post, it always bothered me that every new node or Electron release requires new builds for opencv4nodejs-prebuilt. Eventually I reached the point where I decided to tackle this issue once and for all and started re-implementing the image matching logic from scratch.

I set up a new build pipeline to easily pre-compile and use the most recent OpenCV version and wrote a new node addon to perform image matching using the Node-API and async workers. This way I could solve the version compatibility problem.

But I even went one step further. Having this new and shiny addon and a streamlined build process I actually rented an M1 machine to tackle the Apple Silicon problem as well. It required some digging and going through CMake docs, but after several debugging runs I'm now able to build a single, universal binary for macOS that is compatible with both Intel and Apple Silicon chips!

Time for a change!#

At this point you might ask yourself why I'm doing all the stuff you read about earlier.

The simple answer is that this is my understanding of how qualitative software development should look like. nut.js has a thorough set of tests, for both the core package and plugins, automated builds for both releases and snapshots, auto-generated API docs and a nice API. It powers several open source projects but also gives a lot of freelancers an advantage over their competitors and even startups and established companies are using nut.js in their products.

But this also comes with a cost. Having all these people rely on software I'm building puts pressure on me to keep things going. Also, maintaining this project for three platforms is hard at times, and with release 2.0.0 I ultimately crossed a line when I not just dedicated a huge amount of my time, but also my own money to the project to bring on-screen image search to M1 machines.

That's why I updated two of my GitHub sponsoring tiers to include the following:

Access to non-public nut.js packages

I decided to keep the above-mentioned @nut-tree/nl-matcher package exclusive to project sponsors as a sponsoring perk.

A few months ago I did a survey among nut.js users where I asked them what they are using nut.js for and whether they would be willing to support the project either financially or via community contributions to keep it going. An astonishing amount of people told me that they were using nut.js in commercial projects, be it in freelance / consulting projects or product development, but on the other hand, they are not willing to support the project in any form.

That's a BIG issue and really had an impact on me that ultimately contributed to this decision.

I know that I'm really fortunate to already have some sponsors, and I'm really thankful for their support, but overall it's also a thing about valuing someone else's work.

Now before anyone gets mad about the sponsoring tiers I chose:

  • I'm confident that nut.js is worth it
  • I'm taking maintaining nut.js serious and hard work should be valued
  • It easily gives you an advantage over possible competitors

From time to time I will also gift access to private packages to people that are actively participating in our community, because this should be valued as well! Existing sponsors will of course also get access!

Alright, quite a lot to digest. Feel free to let it sink and if you feel like it, get in touch with me on Discord.

Here's to a great 2022!

All the best

Simon

Going on - devlog

Simon Hofmann

Simon Hofmann

Maintainer @ nut.js

Those of you who are keeping an eye on the nut.js repository might have noticed something — quite a big change is on the horizon!

As I already mentioned in my previous post, I'm trying to reduce the amount of work required to keep nut.js compatible with upcoming node and Electron version. This became even more pressing since the Electron team switched to an eight weeks release cycle, so nut.js would require a new release every 8 weeks as well to keep up. The single reason for this is the opencv4nodejs-prebuilt dependency required to provide all the image matching functionality. Even though I'm making progress on building a replacement that does not suffer this version dependence, it'll take some more time to get it released. So, while working on solving this issue, I also decided to tackle two things mentioned in the nut.js user survey I posted a while ago. Turns out, nut.js users are undecided when it comes to image processing. Half of you are big fans, the others are unhappy about the drawbacks like node version dependence or package size. That, plus the wish for a plug-in system in nut.js, lead to the following strategy towards the next major release:

  1. Establish a simple plug-in system to make underlying provider packages configurable
  2. Pull existing image processing code and dependencies out of nut.js into a dedicated package that utilises the newly created plug-in system, making image matching an explicit opt-in feature
  3. Keep working on a proper replacement package for image matching that will seamlessly replace the existing plug-in package

What does this mean for nut.js users?#

Unfortunately, there will be breaking changes. At least for those of you using image matching functionality. The good thing is, it should be relatively simple to fix these.

Instead of only installing nut.js via e.g. npm i @nut-tree/nut-js, you'll have to install an additional image matching plug-in package.

Let's look at an example:#

  1. Create a new project and install nut.js
npm init -y
npm i @nut-tree/nut-js@next
  1. Create an index.js file with the following content:
const { screen, imageResource } = require("@nut-tree/nut-js");
(async () => {
try {
await screen.find(imageResource("img.png"));
} catch (e) {
console.error(e);
}
})();
  1. Run the example and check its output
node index.js
Searching for img.png failed. Reason: 'Error: No ImageFinder registered'

The error output tells you that nut.js cannot search for your template image since no suitable ImageFinder has been registered.

A similar error appears when trying to save a screenshot to disk:

const { screen, imageResource } = require("@nut-tree/nut-js");
(async () => {
try {
await screen.capture(imageResource("foo"));
} catch (e) {
console.error(e);
}
})();
Error: No ImageWriter registered

This again tells us that a plug-in suitable for writing image files to disk is missing.

How to we solve this?#

As a first step, let's install the newly created plug-in containing the image processing code extracted from nut.js.

npm i @nut-tree/template-matcher

Once installed, all we have to do is import it in our code:

const { screen, imageResource } = require("@nut-tree/nut-js");
require("@nut-tree/template-matcher"); // THIS IS NEW
(async () => {
try {
await screen.find(imageResource("img.png"));
} catch (e) {
console.error(e);
}
})();

That's all it takes to put the previous image matching code back in place:

Searching for img.png failed. Reason: 'Error: No match with required confidence 0.99. Best match: 0.9249920099973679 at (384, 26, 409.5, 62)'

I'm not using any image matching functionality, do I have to do anything?#

Short answer: No! If you haven't used any image matching functionality in your code, you won't encounter any problems. Actually, you'll even profit from it!

By extracting the image matching code into a separate plug-in the nut.js base package itself:

  • Lost around 70 MB of required disk space
  • Is no longer dependent on a certain node version, so you're able to use it with e.g. node 17 right away

Awesome, can I start using it?#

Currently, everything I just showed you is available in the latest @next release.

But please be warned! This is still under active development and since it's a snapshot release, things might change/break at any time until it gets released as 2.0.0!

The main reason for this blog post is to inform users early on, so hopefully there won't be too many surprises once it's stable!

With that in mind, feel free to test it and let me know what you think about it!

Best regards,

Simon

Incremental steps - devlog

Simon Hofmann

Simon Hofmann

Maintainer @ nut.js

First steps have been made! My endeavour to get rid of nan-type bindings in nut.js, opencv4nodejs, to be precise, is making progress. Slow, but steady.

So far I redesigned how OpenCV itself is built and distributed for further use, solving one of the issues which are bothering me with the existing setup. This redesign eliminates a whole codebase I had to think through and follows a pretty lean approach for providing precompiled libs.

The setup is already automated for all three platforms and I started a new project which uses these new artefacts. The process required some tweaking but right now everything fits together really smooth.

As I already said, first steps have been made and I'm looking forward to take the next ones!

Best regards

Simon

Does it spark joy?

Simon Hofmann

Simon Hofmann

Maintainer @ nut.js

Sometimes you've got to ask this simple question. Does something spark joy for you? In my case, something that definitely does not spark joy is updating opencv4nodejs-prebuilt. The reason being is the fact that opencv4nodejs relies on  Native Abstractions for Node.js , which in turn means that on every new node or Electron release, I'd have to provide additional prebuilds. In theory, this sound rather straightforward. Add a new node / Electron version to your CI config and off we go, but reality looks a bit different. Hands down, opencv4nodejs is great. It's configurable, compatible with recent OpenCV versions and async. However, it also has its peculiarities.

It relies on a custom npm package which provides a wrapper around the whole OpenCV CMake build. This way it’s possible to configure and build OpenCV without leaving the well known npm ecosystem. While this approach definitely has its benefits, it’s an additional burden for me. When I started building nut.js I just used opencv4nodejs. But since I envisioned the installation of nut.js as straight forward as possible the whole „we build everything for you on install" just didn’t suite me. That’s how I ultimately ended up adopting two codebases by forking both the npm OpenCV build and opencv4nodejs itself to provide prebuilt binaries.

As time went by, I noticed a few things I definitely wanted to change:

  • node ABI dependence: Every new node release requires new prebuilds which is the sole reason nut.js is running a bit late with support latest node versions
  • node-gyp: I don’t know why this build system is still so popular. I’m not a fan
  • Way too much functionality: opencv4nodejs aims to provide general purpose bindings for OpenCV while I’m only requiring a very small set of features
  • Mental load: I’m not particularly interested in deep diving another two codebases
  • Complexity: Even tough the setup works I’m the only one who knows why

To put it in a nutshell (haha): I’m constantly looking for alternative solutions which solve the above points. I recently started digging into something which might be promising, I’ll let you know once I know!

Best regards

Simon

Welcome

Simon Hofmann

Simon Hofmann

Maintainer @ nut.js

Huh? A dedicated nut.js page? Including a blog? Tell me more!

Well, it’s finally happening! After more than two years of active development nut.js receives the home it deserves. It’s all rough an sketchy so far, but I recently thought about the progress I made with nut.js and decided to step things up a little bit.

I won’t go into much detail now, but rest assured that I have great plans for the future!

Feel free to look around, there’s not a lot to see thus far. But also make sure to regularly check this page for updates!

All the best

Simon