Overview

To do RPA in python When using pyautogui There are times when you want to perform tolerances, grayscale recognition, etc. At that time, it will be necessary to insert opencv.

However, I couldn't use it after all. I will post the error details and workarounds.

error contents

pos = pyautogui.locateCenterOnScreen(GetImage(imgname), region=inputregion) This code worked, but suddenly stopped working. pip opencv-python and pos = pyautogui.locateCenterOnScreen(GetImage(imgname), region=inputregion, grayscale=True) I tried to make it correspond to grayscale by rewriting.

`output`


============================= test session starts =============================
platform win32 -- Python 3.8.5, pytest-6.0.2, py-1.9.0, pluggy-0.13.1
================================== FAILURES ===================================
    pos = pyautogui.locateCenterOnScreen(GetImage(imgname), region=inputregion,grayscale=True)
C:\tools\miniconda3\envs\rpa2\lib\site-packages\pyautogui\__init__.py:175: in wrapper
    return wrappedFunction(*args, **kwargs)
C:\tools\miniconda3\envs\rpa2\lib\site-packages\pyautogui\__init__.py:207: in locateCenterOnScreen
    return pyscreeze.locateCenterOnScreen(*args, **kwargs)
C:\tools\miniconda3\envs\rpa2\lib\site-packages\pyscreeze\__init__.py:400: in locateCenterOnScreen
    coords = locateOnScreen(image, **kwargs)
C:\tools\miniconda3\envs\rpa2\lib\site-packages\pyscreeze\__init__.py:360: in locateOnScreen
    retVal = locate(image, screenshotIm, **kwargs)
C:\tools\miniconda3\envs\rpa2\lib\site-packages\pyscreeze\__init__.py:340: in locate
    points = tuple(locateAll(needleImage, haystackImage, **kwargs))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
needleImage = array([[31, 31, 31, ..., 31, 31, 31],
       [31, 31, 31, ..., 31, 31, 31],
       [31, 31, 31, ..., 31, 31, 31],
    ...31, 31, 31, ..., 31, 31, 31],
       [31, 31, 31, ..., 31, 31, 31],
       [31, 31, 31, ..., 31, 31, 31]], dtype=uint8)
haystackImage = array([[60, 60, 60, ..., 60, 60, 60],
       [60, 60, 60, ..., 60, 60, 60],
       [60, 60, 60, ..., 60, 60, 60],
    ...16, 16, 16, ..., 16, 16, 16],
       [16, 16, 16, ..., 16, 16, 16],
       [16, 16, 16, ..., 16, 16, 16]], dtype=uint8)
grayscale = True, limit = 1, region = (0, 945.0, 240.0, 135.0), step = 1
confidence = 0.999
    def _locateAll_opencv(needleImage, haystackImage, grayscale=None, limit=10000, region=None, step=1,
                          confidence=0.999):
        """
        TODO - rewrite this
            faster but more memory-intensive than pure python
            step 2 skips every other row and column = ~3x faster but prone to miss;
                to compensate, the algorithm automatically reduces the confidence
                threshold by 5% (which helps but will not avoid all misses).
            limitations:
              - OpenCV 3.x & python 3.x not tested
              - RGBA images are treated as RBG (ignores alpha channel)
        """
        if grayscale is None:
            grayscale = GRAYSCALE_DEFAULT
    
        confidence = float(confidence)
    
        needleImage = _load_cv2(needleImage, grayscale)
        needleHeight, needleWidth = needleImage.shape[:2]
        haystackImage = _load_cv2(haystackImage, grayscale)
    
        if region:
>           haystackImage = haystackImage[region[1]:region[1]+region[3],
                                          region[0]:region[0]+region[2]]
E           TypeError: slice indices must be integers or None or have an __index__ method
C:\tools\miniconda3\envs\rpa2\lib\site-packages\pyscreeze\__init__.py:202: TypeError
=========================== short test summary info ===========================
FAILED test_iqctrl.py::test_iqcreate - TypeError: slice indices must be integ...
============================== 1 failed in 2.03s ==============================

environment

A list of python modules you are using.

conda install python
pip install pyautogui
pip install rope yapf black autopep8 pylint pytest
pip install tqdm
pip install pyperclip
pip install pyodbc
pip install cx_Oracle

I added opencv-python here.

The OS is Windows 10.

Conclusion

The _locateAll_opencv function has not been tested in python3.x and is still developing. I get an error such as "The index that slices the array must be an integer". I couldn't get involved because it's in the pyautogui module.

Workaround

Currently, either put up with 2.X or give up opencv. If neither is acceptable, you may need to give up grayscale and ambiguous image search.

If you can rewrite the _locateAll_opencv function itself, it would be helpful if you could pull request to Github.

[PYTHON] I get an error when I put opencv in pyautoGUI