We are building and operating a test automation system for in-house system developers. (SikuliX + Testablish) Write down the technical issues and things you want to do on a daily basis.
Sikuli is fun!
Sikuli is an OSS RPA tool. When it comes to this, I am very worried about recognition failure due to slight misalignment of images due to environmental differences that are common in RPA tools. In fact, even if the screen resolution and tool settings are the same in Win10 & Edge, there are many cases where it fails immediately due to the presence or absence of a monitor connection and small version differences. The main enemy of test automation is that it is difficult to secure idempotency. On the other hand, in the case of a manual test (person), even if there are some differences, the ambiguity can be interpreted in the head and the work can proceed. (So it's sly. However, it may deteriorate the quality.) Sikuli also wants some forgiveness.
Sikuli's image recognition (OpenCV) has an option to set ambiguity. So why not start by searching for accuracy and then gradually lower it to increase ambiguity? Also, the accuracy seems to be 0.7 by default, but I wonder if that is also strictly speaking. (Sometimes you mistakenly recognize another similar element on the same screen.) I think it is logically good to start from around 0.99.
Thought logic Start with a recognition rate of 0.99, and if not found, decrease the recognition rate by 0.1 to make a judgment. (0.4 is Min)
According to RaiMan, "Sikuli is God's eye" realizes image recognition with god-like forgiveness and tolerable image misalignment.
Verify with the following implementation.
----------------------- import sys dir = os.getcwd() acc = 0.99 saveFileName = "filename" saveFilePath = os.path.join(dir, saveFileName) while acc > 0.4: if exists(Pattern(saveFilePath).similar(acc)): find(Pattern(saveFilePath).similar(acc)).highlight() wait(1) highlightAllOff() find(Pattern(saveFilePath).similar(acc)).click() break else: acc = acc - 0.1 wait(1) -----------------------
It is now possible to recognize even a slight deviation !! (It can be identified even if the target image is reduced to 80%) I'm not good at judging characters (because they are drawn with lines and the ratio is small in terms of area? Characters are better with the OCR function) Image recognition is much faster than OCR
May be essential for image recognition operations with RPA
Recommended Posts