[PYTHON] Async / await with Kivy and tkinter

2018 was a shock to me. I used to play with GUI Framekwork's Kivy kivy, but I found that using a generator makes ugly code full of callback functions surprisingly easy to read. After trying various things, I came to understand the asynchronous processing by async / await, which was a magic that I didn't know until then, and I was able to create a little asynchronous processing library. This article is new

――The process until I realized the wonderfulness of the generator and the native coroutine that was born from it --The process of realizing async / await with Kivy and tkinter using it

I want to spell it out.

(To reduce the amount of text, generator is abbreviated as gen and coroutine is abbreviated as coro.)

The hidden power of gen

Gen as a device that produces value

I think many introductory books introduce gen as a value-producing device. (To avoid confusion with the meaning of return, "return", yield is expressed as "give")

def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, b+a

for i in fibonacci():
    print(i, end=' ')
    import time; time.sleep(.1)

0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181 ...

Given the name "generator", it may have been that purpose originally. In fact, until 2018, I only saw that kind of thing, and I think it was difficult for me to get out of that concept because I used to retrieve the value with for-in as above.

Retrieving values without using for-in

Some introductory books also include an example of using send () to retrieve a value from gen.

gen = fibonacci()
print(gen.send(None))
print(gen.send(None))
print('A short break')
import time;time.sleep(1)
print('End of break')
print(gen.send(None))
print(gen.send(None))

0
1
A short break
End of break
1
2

I think this gives a glimpse of gen's "hidden power", but I didn't notice it at this stage either. But what about the next example?

Nothing is born gen

def sub_task():
    print('sub:Process 1')
    yield
    print('sub:Process 2')
    yield
    print('sub:Process 3')

def main_task():
    gen = sub_task()
    try:
        print('main:Process 1')
        gen.send(None)
        print('main:Process 2')
        gen.send(None)
        print('main:Process 3')
        gen.send(None)
    except StopIteration:
        pass

main_task()

main:Process 1
sub:Process 1
main:Process 2
sub:Process 2
main:Process 3
sub:Process 3

You can see that both tasks are progressing little by little with gen.send () and yield as switching points. Isn't this ** parallel processing ** itself!

Hidden power

This was gen's "hidden power". gen is

--It can be paused ... (Stops at the yield and starts moving with gen.send ()) --Communication with the user side is possible at the stop point ... (The value can be sent from the gen side with yield and from the user side with gen.send ())

It was like a function, and because of its stoptable feature, ** parallel processing could be done without relying on multi threading **.

Expectations for gen

This made me hope that I was suffering from an ugly code full of callback functions. Because, for example, when you want to do the following with Kivy

def some_task():
    print('Process 1')
Wait until the button is pressed
    print('Process 2')
Wait 1 second
    print('Process 3')

The actual code is

from kivy.clock import Clock

def some_task(button):
    print('Process 1')
    def callback2(button):
        button.unbind(on_press=callback2)
        print('Process 2')
        Clock.schedule_once(callback3, 1)
    def callback3(__):
        print('Process 3')
    button.bind(on_press=callback2)

It becomes ugliness that can not be seen. You have to wait for something =>You need to stop processing until something happens=> The following processing needs to be separated into another function. But I want you to remember the sub_task () that came out earlier.

def sub_task():
    print('sub:Process 1')
    yield
    print('sub:Process 2')
    yield
    print('sub:Process 3')

Similarly, the callback function does not appear anywhere even though it stopped in the middle. So I started to wonder if Kivy could get rid of the callback function with gen.

Eliminate the callback function (Kivy edition)

I'm going to think about how to do that, but first I would like to mention that the current Kivy master branch can already perform full-scale asynchronous programming using [asyncio] asyncio_doc and [trio] trio_doc. So what we do here is [reinventing the wheel] wheel. However, at that time, it wasn't the case, and I was simply interested in gen, so I chose to do something on my own.

Stop for a certain period of time

I thought about realizing a function that stops gen for that amount of time when a numerical value is sent from gen, leaving the button aside. That's because I happened to see [BeeWare's video] video_beeware and thought it was cool to do such a thing.

def some_task():
    print('Process 1')
    yield 2  #Wait 2 seconds
    print('Process 2')
    yield 1  #Wait 1 second
    print('Process 3')

Think of a way to get the above gen to work as expected. With the knowledge so far

--You need to pass a function to Clock.schedule_once () to reserve the process you want to do after a certain time on Kivy. --You need to call gen.send () to restart gen

I know that. Then, "Why don't we pass the function to restart gen toClock.schedule_once ()?"

from kivy.clock import Clock
from kivy.app import App
from kivy.uix.widget import Widget


def start_gen(gen):
    def step_gen(dt):
        try:
            Clock.schedule_once(step_gen, gen.send(None))  # C
        except StopIteration:
            pass
    step_gen(None)  # B


def some_task():
    print('Process 1')
    yield 1  # D
    print('Process 2')
    yield 2  # E
    print('Process 3')


class SampleApp(App):
    def build(self):
        return Widget()
    def on_start(self):
        start_gen(some_task())  # A

if __name__ == '__main__':
    SampleApp().run()

It was the correct answer. This code works as follows.

1. A gen is created when the app starts and it is passed to start_gen () (line A)
2. start_gen () immediately callsstep_gen ()(line B)
3. step_gen () callsgen.send (), so gen starts working (line C)
4. gen stops at the first yield expression and sends 1 (line D)
5. Since the evaluation result of gen.send (None) is 1, step_gen () will reserve itself to be called again after 1 second (line C).
6. Since there is nothing more to do, the process returns to the kivy event loop.
7. One second later, step_gen () is called and gen.send () is called, so gen starts moving from the position where it stopped last time. (Line C)
8. gen stops at the point where it reaches the second yield expression and sends 2 (line E)
9. (Omitted below)

It was shocking to be able to wait for a while without using the callback function with just 7 lines of function (start_gen ()). Motivated, I will continue to improve this.

Use the value passed to the callback function

The actual elapsed time is passed to the callback function passed to Clock.schedule_once (). Since it's a big deal, I made it so that the some_task () side can receive it. All you have to do is change the gen.send (None) part of the start_gen () to gen.send (dt). Now the some_task () side can get the actual elapsed time as follows (whole code).

def some_task():
    print('Process 1')
    s = yield 1
    print(f"When I asked for a stop for 1 second, it actually{s:.03f}Stopped for seconds")
    print('Process 2')
    s = yield 2
    print(f"When I asked for a stop for 2 seconds, it actually{s:.03f}Stopped for seconds")
    print('Process 3')

Process 1
When I asked for a stop for 1 second, it was actually 1.089 seconds stopped
Process 2
When I asked for a stop for 2 seconds, it was actually 2.Stopped for 003 seconds
Process 3

Waiting for event

Next is the wait for the event, ideally if the gen side writes as follows.

def some_task(button):
    print('Process 1')
    yield event(button, 'on_press')  #Wait until the button is pressed
    print('Process 2')

In the case of event, it is a little complicated because it requires solution </ rb> </ rt> </ ruby> work to connect the callback function, but the procedure is the same. It was realized by passing a function that restarts gen as a callback function for event.

def start_gen(gen):
    def step_gen(*args, **kwargs):
        try:
            gen.send((args, kwargs, ))(step_gen)
        except StopIteration:
            pass
    try:
        gen.send(None)(step_gen)
    except StopIteration:
        pass

def event(ed, name):
    bind_id = None
    step_gen = None

    def bind(step_gen_):
        nonlocal bind_id, step_gen
        bind_id = ed.fbind(name, callback)  #Attach callback function
        assert bind_id > 0  #Check if binding was successful
        step_gen = step_gen_

    def callback(*args, **kwargs):
        ed.unbind_uid(name, bind_id)  #Solve the callback function
        step_gen(*args, **kwargs)  #Resume gen

    return bind

Overall code

The big difference from the time stop is that all the processing related to event can be hidden in ʻevent (). Thanks to that, start_gen () does not depend on kivy at all, and it is as simple as passing step_gen` to the callable sent from gen.

Generalization

The above design feels very good, so I removed the kivy-related processing from start_gen () and hid it in another function, following the event wait for the time stop.

def sleep(duration):
    return lambda step_gen: Clock.schedule_once(step_gen, duration)

Now you can mix sleep () and ʻevent ()`.

def some_task(button):
    yield event(button, 'on_press')  #Wait until the button is pressed
    button.text = 'Pressed'
    yield sleep(1)  #Wait 1 second
    button.text = 'Bye'

Overall code

How gen resumes is completely up to the callable sent by gen, so if you send something like the following, for example

def sleep_forever():
    return lambda step_gen: None  

def some_task():
    yield sleep_forever()  #Wait forever

It is also possible not to restart.

Wait for thread

In order to confirm its versatility, I also dealt with things that have nothing to do with Kivy, thread.

def thread(func, *args, **kwargs):
    from threading import Thread
    return_value = None
    is_finished = False
    def wrapper(*args, **kwargs):
        nonlocal return_value, is_finished
        return_value = func(*args, **kwargs)
        is_finished = True
    Thread(target=wrapper, args=args, kwargs=kwargs).start()
    while not is_finished:
        yield sleep(3)
    return return_value

It's a dull way of looking around to see if it's finished on a regular basis, but now the gen side can wait for it to finish by executing the passed function on another thread.

class SampleApp(App):
    def on_start(self):
        start_gen(self.some_task())
    def some_task(self):
        def heavy_task():
            import time
            for i in range(5):
                time.sleep(1)
                print(i)
        button = self.root
        button.text = 'start heavy task'
        yield event(button, 'on_press')  #Wait until the button is pressed
        button.text = 'running...'
        yield from thread(heavy_task)  #Heavy on another thread_task()And wait for its end
        button.text = 'done'

Overall code

Which is yield or yield from?

It seems to be going well here, but some problems have become apparent. One is that you have to use yield and yield from properly depending on what you are waiting for. (sleep () and ʻevent ()are yield,thread ()is yield from). Moreover, this depends on the implementation, and ifthreading.Thread had a mechanism to notify the end of thread with the callback function, thread () `could also be implemented so that it could wait in yield. It is not good that the usage is different like this, so I decided to unify it to either one.

I think the only option is yield from, depending on which one to unify. Because it's easy to make yield from wait for what you can wait for yield, but not always the other way around. For example

def some_gen():
    yield 1

1 is

def some_gen():
    yield from one()

def one():
    yield 1

By doing so, you can wait at yield from

def some_gen():
    yield from another_gen()

def another_gen():
    yield 1
    yield 4

ʻAnother_gen ()probably can't be made to wait withyield another_gen ()`.

Unified to yield from

So I rewrote sleep () and ʻevent ()` to wait for yield from.

def sleep(duration):
    return (yield lambda step_coro: Clock.schedule_once(step_gen, duration))

def event(ed, name):
    #Abbreviation
    return (yield bind)

With this, the user side does not have to use yield and yield from properly.

#Always yield from
def some_task():
    yield from sleep(2)
    yield from event(button, 'on_press')
    yield from thread(heavy_task)

Overall code

Problems with using arguments passed to callback function

Another problem is that the gen side was originally using the arguments passed to the callback function as follows.

def some_task():
    s = yield 1
    print(f"When I asked for a stop for 1 second, it actually{s:.03f}Stopped for seconds")

What's going on now is actually

def some_task():
    args, kwargs = yield from sleep(1)
    s = args[0]
    print(f"When I asked for a stop for 1 second, it actually{s:.03f}Stopped for seconds")

It is troublesome to get the required value. This is because the formal argument is def step_gen (* args, ** kwargs): so that step_gen () can accept any argument. Fortunately, however, thanks to the unification to yield from, such processing can be done on the sleep () side.

def sleep(duration):
    args, kwargs = yield lambda step_coro: Clock.schedule_once(step_coro, duration)
    return args[0]

With this, the user side

def some_task():
    s = yield from sleep(1)
    print(f"When I asked for a stop for 1 second, it actually{s:.03f}Stopped for seconds")

It came to be enough.

Introduction of async / await syntax

Next, I decided to convert what I have made so far so that it can be handled with the async / await syntax introduced in Python 3.5. This is because it was written in various documents that this is a replacement for gen as coro. For me, I tried to make ʻawaitwith one word shorter and easier to read thanyield from` with two words, but it seems that it actually has more advantages, so please refer to the details. See [Official] pep492.

Conversion procedure

First, gen functions that include yields without from, such as sleep () and ʻevent (), have been given @ types.coroutine. This is because I didn't know how to rewrite the gen function containing the return statement like these as an async function. (If the async function has a yield expression and returns a value in the return statement, a syntax error will occur: SyntaxError:'return' with value in async generator`)

import types

@types.coroutine
def sleep(duration):
    #Abbreviation

@types.coroutine
def event(ed, name):
    #Abbreviation

On the other hand, thread () and some_task () could be rewritten as pure async functions. In particular

--From yield from to ʻawait --fromdef to ʻasync def

Replaced.

async def thread(func, *args, **kwargs):
    #Abbreviation
    while not is_finished:
        await sleep(3)
    #Abbreviation

class SampleApp(App):
    async def some_task(self):
        #Abbreviation
        await event(button, 'on_press')
        button.text = 'running...'
        await thread(heavy_task)
        button.text = 'done'

Finally, replace the character string gen included in the identifier with coro to complete.

Overall code

This is the end of the Kivy edition. As I forgot to say, coro can be executed at the same time as many as start_coro (another_task ()). ʻAwait another_task ()if you want to wait for completion,start_coro (another_task ())` if you want to run in parallel without waiting.

Eliminate the callback function (tkinter edition)

Then I tried the same thing with tkinter, but the procedure was exactly the same as kivy (passing the function to restart gen / coro as a callback function), so it went smoothly.

Wait for time (sleep)

The difference from Kivy is that Kivy uses the only object kivy.clock.Clock, while tkinter uses the method.after ()of each widget. So you need to specify which widget's .after () to call in addition to the time you want to stop.

@types.coroutine
def sleep(widget, duration):
    yield lambda step_coro: widget.after(duration, step_coro)

This means that you need to pass the widget to thread (), which uses sleep () internally.

async def thread(func, *, watcher):
    #Abbreviation
    while not is_finished:
        await sleep(watcher, 3000)  #3000ms stop
    return return_value

Waiting for event

Next is event. Before implementation, tkinter's ʻunbind ()` seems to have [defect] tkinter_issue, so I modified it as follows, relying on the linked information.

def _new_unbind(self, sequence, funcid=None):
    if not funcid:
        self.tk.call('bind', self._w, sequence, '')
        return
    func_callbacks = self.tk.call('bind', self._w, sequence, None).split('\n')
    new_callbacks = [l for l in func_callbacks if l[6:6 + len(funcid)] != funcid]
    self.tk.call('bind', self._w, sequence, '\n'.join(new_callbacks))
    self.deletecommand(funcid)

def patch_unbind():
    from tkinter import Misc
    Misc.unbind = _new_unbind

It is replaced with the modified ʻunbind ()by callingpatch_unbind (). The problem is when to call it, but I think it's better not to do it without permission because tkinter itself is modified. So I decided to have the user call me explicitly. And the implementation of ʻevent ()

@types.coroutine
def event(widget, name):
    bind_id = None
    step_coro = None

    def bind(step_coro_):
        nonlocal bind_id, step_coro
        bind_id = widget.bind(name, callback, '+')
        step_coro = step_coro_

    def callback(*args, **kwargs):
        widget.unbind(name, bind_id)
        step_coro(*args, **kwargs)

    return (yield bind)[0][0]

It became.

How to use

#Install asynctkinter in advance
# pip install git+https://github.com/gottadiveintopython/asynctkinter#egg=asynctkinter

from tkinter import Tk, Label
import asynctkinter as at
at.patch_unbind()  # unbind()Fixed bug

def heavy_task():
    import time
    for i in range(5):
        time.sleep(1)
        print('heavy task:', i)

root = Tk()
label = Label(root, text='Hello', font=('', 60))
label.pack()

async def some_task(label):
    label['text'] = 'start heavy task'
    event = await at.event(label, '<Button>')  #Wait for the label to be pressed
    print(event.x, event.y)
    label['text'] = 'running...'
    await at.thread(heavy_task, watcher=label)  #Heavy on another thread_task()And wait for its end
    label['text'] = 'done'
    await at.sleep(label, 2000)  #Wait 2 seconds
    label['text'] = 'close the window'

at.start(some_task(label))
root.mainloop()

in conclusion

Many people may have known the idea of using gen for concurrency about 10 years ago [already existed] video_curious_course, but for me it was a fresh knowledge a year or two ago and it was a shock. That's why I came up with an article like this. Probably, if the library side implements the event loop like tkinter, use the method like this time, and if the user side entrusts the implementation of the event loop like pygame, set the event loop to ʻasyncioor If you implement it as one task ontrio`, you can introduce async / await to anything basically. Goodbye ugly callback function.

link collection

-asynckivy asynckivy ... The latest version of the one I made this time -asynctkinter asynctkinter ... The latest version of the one I made this time -[So you want to be a Python expert?] Video_powell ... The video that inspired me to realize the greatness of gen -Make You An Async For Great Good! ... ʻAsyncio` A video of making your own mock -Fluent Python ... The most detailed Japanese literature I know about gen / coro. The low version of python I'm using is scratching the ball.