Sikuli (SikuliX) + Beginning with RPA in Ruby (I stumbled many times, but I was able to do it properly)

I heard that you can do it with Ruby

There is a free RPA tool called SikuliX. (Official, Documentation) It seems to be the successor to Sikuli, but in the above formula

SikuliX supports as scripting languages Python language level 2.7 (supported by Jython) running RobotFramework text-scripts is supported (see docs) Ruby language level 1.9 and 2.0 (supported by JRuby) JavaScript (supported by the Java Scripting Engine)

It says something like "You can write commands in Ruby!", So I thought I should play with this for a moment. [^ 1]

However, there are few introductory articles about Ruby + SikuliX, so when I tried it, I got stuck in the initial setting stage. So in this article, for those who want to do something similar, this is the way to do it! I will report that.

(Especially, there are few articles in Japanese. It seems that there are more people in this area than Python, so there are more articles!)

The author's environment is Windows 10.

First, download SikuliX

https://raiman.github.io/SikuliX1/downloads.html Go to and download the two .jar files from the links in red below. [^ 2] Create your favorite folder for SikuliX and save the two in the same place.

Download a version of Java (Open JDK) that runs SikuliX

If you already have Java installed, you'll want to run sikulixide-2.0.4.jar, but please be patient __. As of October 2020, it seems that the latest Java causes an error and only JRuby Interpreter is moved to another location (probably, moving itself is not a problem and you can just continue the following work I think, but if you're worried, please download JRuby Interpreter again).

Now let's download a little older Java. https://jdk.java.net/archive/ It seems that version 11.0.2 is recommended for SikuliX, so look for it and download it.

Unzip the zip and keep the resulting folder in a place you know. I put it directly under the C drive.

I will start it for the time being

At the command prompt, go to the folder where you downloaded __SikuliX __ and enter the following command. * However, this is the case when the OpenJDK folder is placed directly under the C drive like me. If you put it somewhere else, please specify the java path accordingly.

C:\openjdk-11.0.2_windows-x64_bin\jdk-11.0.2\bin\java -jar sikulixide-2.0.4.jar

If you do this, the JRuby Interpreter will move somewhere as before, but this time the IDE will start up without any error.

But __ this still doesn't work for Ruby scripts. __ To be precise, puts "Hello" works, but since the essential gem for Sikulix has not been installed in JRuby, RPA-like things cannot be done as it is.

Install gem "sikuli" on JRuby

So, I want to install the gem on the JRuby interpreter, but where did the JRuby interpreter I downloaded earlier go in the first place? When I searched for it, it was moved to the following.

C: \ Users \ (username) \ AppData \ Roaming \ Sikulix \ Extensions

You can find jruby-complete-9.2.0.0.jar (version number may be slightly different)

All you have to do is install the gem on it. I'm not sure about JRuby gem, but referring to Q & A here, Did.

--First, create an appropriate folder in the same location as the JRuby jar (I named it biojruby, which is the same as the above reference, but I don't mind any name). --And, at the command prompt, enter the same folder as the jruby jar and execute the following. * Please read the jar version and the created folder name as appropriate.

java -jar jruby-complete-9.2.0.0.jar -S gem install -i ./biojruby sikuli
jar uf jruby-complete-9.2.0.0.jar -C biojruby .

gem install gives me some warnings, but don't worry about it.

Let's check.

$ java -jar jruby-complete-9.2.9.9.jar -S gem list sikuli

This also gives some warnings, but it's okay if sikuli is displayed along with the version number.

SIKULI_HOME in the environment variable

The final touch! In the Windows environment variable settings, add SIKULI_HOME. (Apparently it's necessary for the gem sikuli side to recognize SIKULI's .jar? This is also a Stack Overflow sale ... sorry)

RPA execution

Sorry I made you wait! It's finally done. Reopen the command prompt and reopen the .jar on the SikuliX side with OpenJDK.

C:\openjdk-11.0.2_windows-x64_bin\jdk-11.0.2\bin\java -jar sikulixide-2.0.4.jar

I'm sorry for the image below, but here is an example script. in this case,

--Double-click the Edge browser icon on your desktop --Key in "abc" when the same image as "New Tab" is displayed

It is an instruction.

Icons etc. can be easily inserted by selecting "Take screenshot" from the menu. Please try with your favorite icon.

When completed, save it in any location (a folder labeled .sikuli will be created), and then click the Run button ...

How is it? Congratulations to those who have worked properly!

later

https://sikulix-2014.readthedocs.io/en/latest/toc.html I think that I will investigate how to read it in Ruby as appropriate while referring to the surroundings ... but after all there is little information ... So, if there is a topic that seems to be useful, I will continue to cover it in Qiita.

Impressions

This time, I took up SikuliX as a free RPA. I was able to do it with Ruby, but it took a lot of time just to build the environment.

To be honest, this may be a bottleneck in RPA for work efficiency, even if there are few articles that can be referred to. This time I was just playing at home, so it's okay, but just installing it on dozens of PCs seems to be quite a hassle. Isn't that the case with Jython?

For RPA that can be done with Ruby, there seems to be something like Roboware for a fee (no, I'm not a turner), and if you expect solid and stable operation, think about support. It may be better to consider such a place.

Rather, SikuliX may come in handy if you think about what you can do with a small team. So I hope that more people will choose JRuby as an option, and that the officials will also focus on documents etc., so I ended up there.

[^ 1]: To be exact, I am using JRuby, an environment that converts Ruby scripts to Java and operates. SikuliX itself runs on Java.

[^ 2]: It says that you can download both Jython interpreter and Jython interpreter, but when I did this, Jython was given priority and JRuby couldn't work. In this case, I had to uninstall it and try again. Please refer to https://auto-worker.com/blog/?p=473 for uninstalling.