Screen

A screen is the most important source of information when using a computer. No matter whether it's a smartphone or a desktop computer.

Thus, for desktop automation, the screen is also THE central source of information.

It starts simple, screen dimensions allow you to move in relative coordinates. screen.capture allows you to save screenshots, while screen.grab returns a buffer holding the current screen content, so you can further process it in your code. screen.highlight allows you to highlight screen areas so you can visually debug results from on-screen search.

And that's the actual highlight: On-screen search!

On-screen search

You want to locate a template image on screen? screen.find with an Image will do the job for you!

You are looking for a certain word or text on screen? Use screen.find with either a singleWord or textLine query!

You're interested in a particular window? screen.find using a windowWithTitle query to the rescue!

The one pixel with a very special color? Once again, screen.find and a pixelWithColor query are your friends!

Notice the pattern? screen.find is your one-stop solution if you are searching something on screen!

Searching in a dynamic environment

GUIs are dynamic. You're either manually moving windows around, or have some kind of animation on certain parts of the UI. This becomes cumbersome to deal with when searching for a particular thing on your screen, no matter whether it's text, an image or something totally different.

screen.waitFor aims to lessen this burden by allowing you to specify a timeout and a search interval to regularly check your screen for your desired piece. This allows us to deal with dynamic behaviours like pop-ups, animations and so on.

Searching for multiple occurrences

screen.find is great when searching for a single item. But what if you're trying to locate multiple occurrences of e.g. the same image?

screen.findAll has you covered! It comes with the exact same signature as screen.find, but instead of just a single instance, it will return a list of all detected occurrences of your query!

What else?

This short intro should have given you a good first impression of what's possible with the powerful on-screen search.

For additional info, please take a look at the Screen API documentation