Previous article: Making pyautogui [super-appropriate] compatible with multi-display environment Part1

Purpose

I want to pass the coordinates obtained by pyautogui.locateOnScreen () to pyautogui.click () as it is.

environment

python 3.8.5 pyautogui 0.9.50 pyscreez 0.1.26 OS windows only win32api The above cannot be installed from pip, so you need to download and install the one that suits your environment from the link.

Investigation

Since the functions of locateOnXXX () type generally call locateOnScreen (), modify this function.


def locateOnScreen(image, minSearchTime=0, **kwargs):
    """TODO - rewrite this
    minSearchTime - amount of time in seconds to repeat taking
    screenshots and trying to locate a match.  The default of 0 performs
    a single search.
    """
    start = time.time()
    while True:
        try:
            screenshotIm = screenshot(region=None) # the locateAll() function must handle cropping to return accurate coordinates, so don't pass a region here.
            retVal = locate(image, screenshotIm, **kwargs)
            try:
                screenshotIm.fp.close()
            except AttributeError:
                # Screenshots on Windows won't have an fp since they came from
                # ImageGrab, not a file. Screenshots on Linux will have fp set
                # to None since the file has been unlinked
                pass
            if retVal or time.time() - start > minSearchTime:
                return retVal
        except ImageNotFoundException:
            if time.time() - start > minSearchTime:
                if USE_IMAGE_NOT_FOUND_EXCEPTION:
                    raise
                else:
                    return None

Implementation details

Import win32api

Any place where the platform is determined to be win32 at the import location is fine. I wrote it in the first 25th line and after.

    from PIL import Image
    from PIL import ImageOps
    from PIL import ImageDraw
    if sys.platform == 'win32': # TODO - Pillow now supports ImageGrab on macOS.
 Added import of # win32api
        import win32api
        from PIL import ImageGrab
    _PILLOW_UNAVAILABLE = False

Fix locateOnScreen ()

Since the coordinates and values of left, top, width, and height that match the image recognition are stored in retVal, the contents of retVal are rewritten a little before returning as a return value.


def locateOnScreen(image, minSearchTime=0, **kwargs):
    """TODO - rewrite this
    minSearchTime - amount of time in seconds to repeat taking
    screenshots and trying to locate a match.  The default of 0 performs
    a single search.
    """
    start = time.time()
    while True:
        try:
            screenshotIm = screenshot(region=None) # the locateAll() function must handle cropping to return accurate coordinates, so don't pass a region here.
            retVal = locate(image, screenshotIm, **kwargs)
 # from here
            if not(retVal == None) and sys.platform == 'win32':
                displays = win32api.EnumDisplayMonitors()
                left_min = min([display[2][0] for display in displays])
                top_min = min([display[2][1] for display in displays])
                retVal = Box(
                    left = retVal[0] + left_min,
                    top = retVal[1] + top_min,
                    width = retVal[2],
                    height = retVal[3]
                )
 #Additions up to here
            try:
                screenshotIm.fp.close()
            except AttributeError:
                # Screenshots on Windows won't have an fp since they came from
                # ImageGrab, not a file. Screenshots on Linux will have fp set
                # to None since the file has been unlinked
                pass
            if retVal or time.time() - start > minSearchTime:
                return retVal
        except ImageNotFoundException:
            if time.time() - start > minSearchTime:
                if USE_IMAGE_NOT_FOUND_EXCEPTION:
                    raise
                else:
                    return None

Confirmation of modification contents of _screenshot_win32 ()

Also check _screenshot_win32 () modified in previous article just in case.

def _screenshot_win32(imageFilename=None, region=None):
    """
    TODO
    """

    # TODO - Use the winapi to get a screenshot, and compare performance with ImageGrab.grab()
    # https://stackoverflow.com/a/3586280/1893164
    #im = ImageGrab.grab()
    im = ImageGrab.grab(all_screens=True)

    if region is not None:
        assert len(region) == 4, 'region argument must be a tuple of four ints'
        region = [int(x) for x in region]
        im = im.crop((region[0], region[1], region[2] + region[0], region[3] + region[1]))
    if imageFilename is not None:
        im.save(imageFilename)
    return im

result

The coordinates obtained by pyautogui.locateOnScreen () can now be used by pyautogui.click ().

problem

Image recognition speed slows down

Of course, it is natural, but if there are two, the image size will be doubled, and if there are three, the image size will be tripled, so the image recognition speed will drop proportionally. My personal impression is that pyautogui does not officially support multiple displays because of this speed problem.

Affects other packages that reference pyscreez

I don't know if there are other packages that I'm referencing, but I can't say that there is no impact on the current environment because it is directly rewritten. Let's just use it in a play environment, even if you make a mistake, it is not something to do in the important system environment currently in operation. I don't think there is such a person.

Finally

Since the problem of slowdown is big, the conclusion is that it is more realistic for operation to modify it so that a specific display can be specified rather than allscreens.

That's why it was compatible with super-appropriate multi-display. If I have free time, I would like to try that one as well.

[PYTHON] Make pyautogui [super-appropriate] compatible with multi-display environment Part2