As the end of the year is approaching this year, I will use [Python C / API] in vain. Reversal of purpose and means. This time, I'm aiming for Awaitable, something like Coroutine obtained by executing the following coroutine function spam.
import asyncio
import sys
async def spam():
print('do something')
ret = await asyncio.sleep(1, 'RETURN VALUE')
return ret.lower()
async def main():
ret = await spam()
print(ret)
if __name__ == '__main__':
if sys.version_info < (3, 7):
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()
else:
asyncio.run(main())
Print something, do await sleep to get the result, and call its lower method to return the result. 3 operation. The send, throw, and close methods that Coroutine and Generator do not have Awaitable will not be reproduced this time.
Before mimicking the code written as async def, await. Let's review what this is in the first place. The text when these language features were additionally proposed to Python is PEP 492. After reading this, I sloppyly remembered that await was actually a yield from. I think the reason why this was prepared was as follows. "The Generator specification was originally designed with a view to being used as a coroutine. The ability to pause and resume was the Iterator itself, and the ability to send values PEP 342. I added it in. And I have been doing well before Python 3.4. However, problems due to the indistinguishability of Generator and Iterator and Coroutine have begun to be reported, so the Python 3.5 language specification can be distinguished. Create another method called \ _ \ _ await \ _ \ _ instead of \ _ \ _ iter \ _ \ _. Do not divert the yield from statement for the Generator Function, but instead of the Coroutine Function. Another statement await statement for use inside is newly added to distinguish it. " There are different names \ _ \ _ await \ _ \ _ and \ _ \ _ iter \ _ \ _, but the behavior is not different. Tp_iter and tp_as_sync. Am_await are also provided on the PyTypeObject structure of CPython and are treated as separate members.
Now that I have a better understanding of Awaitable. Immediately write down the coroutine function spam with a class statement. The difference with Iteratable is that it has \ _ \ _ await \ _ \ _ instead of \ _ \ _ iter \ _ \ _. However, the only difference is that it returns an Iterator from here.
class Spam:
def __await__(self):
... # TODO:What Itarator can be returned to mimic a spam coroutine?
Continue disassembling. Iterator returns itself with \ _ \ _ iter \ _ \ _, resumes processing with \ _ \ _next \ _ \ _, creates the next value, and returns the value while interrupting the processing.
class _Spam:
def __iter__(self):
return self
def __next__(self):
... # TODO:How can I imitate a spam coroutine?
class Spam:
def __await__(self):
return _Spam()
So what do we need to implement a "coroutine-like Awaitable obtained by running a coroutine function spam"? It keeps the state of how much of the three actions of "print something, do await sleep to get the result, call the lower method of it and return the result". Makes the object have the \ _state attribute. This time, I used 0, 1, 2 int for the time being, but if you want to write it neatly, you should use Enum. Then implement \ _ \ _next \ _ \ _ which processes by state. The problem here is await or yield from. It delegates processing to another Iterator than yield from. Iterators continue to iterate until another Iterator stops, and the value obtained is returned as is. You need to keep another Iterator running to implement the equivalent. For this reason, we have added the \ _it attribute. When this Iterator stops, a StopIteration exception will be sent, so look at the value attribute. This corresponds to the value of the await, yield from expression in the return statement of the coroutine function or generator function. Pay attention to the position of the lower method. Conversely, when returning a value, give the StopIteration exception a value. Iterator once stopped has a restriction that it must keep returning StopIteration exception, so take measures against it. .. \ _ \ _ Next \ _ \ _ ends with raise StopIteration. The state-holding attribute should start with \ _ to indicate that you don't want it to be rewritten externally. This \ _state initialization is done in \ _ \ _ new \ _ \ _ to make it resistant to multiple calls to \ _ \ _ init \ _ \ _.
class _Spam:
def __new__(cls):
obj = super().__new__(cls)
obj._state = 0
obj._it = None
return obj
def __iter__(self):
return self
def __next__(self):
if self._state == 0:
print('do something')
self._it = asyncio.sleep(1, 'RETURN VALUE').__await__()
self._state = 1
if self._state == 1:
try:
v = next(self._it)
except StopIteration as e:
ret = e.value
self._it = None
self._state = 2
raise StopIteration(ret.lower())
else:
return v
raise StopIteration
class Spam:
def __await__(self):
return _Spam()
Because the disassembly is over. Write this in Python C / API. From here C language. To create a class, define Write PyTypeObject structure directly or PyType_Spec and call PyType_FromSpec. This time, in the direction of using PyType_FromSpec. Start with Spam, which is easier than \ _Spam. To create \ _ \ _await \ _ \ _, register the function in tp_as_sync. am_await. Since this method needs to return an instance of \ _Spam, we'll make sure that the Spam class's attributes have the \ _Spam class. Class attribute addition processing is performed after class creation with PyType_FromSpec in the initialization processing of the module registered as Py_mod_exec.
typedef struct {
PyObject_HEAD
} SpamObject;
static PyObject *
advent2019_Spam_await(SpamObject *self)
{
PyObject *_Spam_Type = PyObject_GetAttrString((PyObject *)Py_TYPE(self), "_Spam");
if (_Spam_Type == NULL) { return NULL; }
PyObject *it = PyObject_CallFunction(_Spam_Type, "");
Py_DECREF(_Spam_Type);
return it;
}
static PyType_Slot advent2019_Spam_slots[] = {
{Py_am_await, (unaryfunc)advent2019_Spam_await},
{0, 0},
};
static PyType_Spec advent2019_Spam_spec = {
.name = "advent2019.Spam",
.basicsize = sizeof(SpamObject),
.flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,
.slots = advent2019_Spam_slots,
};
static int advent2019_exec(PyObject *module) {
int ret = -1;
PyObject *_Spam_Type = NULL; // TODO
PyObject *Spam_Type = NULL;
if (!(Spam_Type = PyType_FromSpec(&advent2019_Spam_spec))) { goto cleanup; }
// Spam._Spam = _Spam
if (PyObject_SetAttrString(Spam_Type, "_Spam", _Spam_Type)) { goto cleanup; }
if (PyObject_SetAttrString(module, "Spam", Spam_Type)) { goto cleanup; }
if (PyObject_SetAttrString(module, "_Spam", _Spam_Type)) { goto cleanup; }
ret = 0;
cleanup:
Py_XDECREF(_Spam_Type);
Py_XDECREF(Spam_Type);
if (ret) { Py_XDECREF(module); }
return ret;
}
Well, it's a postponed implementation of Iterator \ _Spam. First from the \ _SpamObject structure and \ _ \ _ new \ _ \ _. Prepare some value state to hold the state and it to hold the asyncio.sleep (). \ _ \ _ Await \ _ \ _ iterator. Since it will hold other PyObject *, make sure to allocate memory with PyObject_GC_New corresponding to the garbage collection mechanism. Also, after initialization, call PyObject_GC_Track and register yourself in the garbage collector.
typedef struct {
PyObject_HEAD
unsigned char state;
PyObject *it;
} _SpamObject;
static PyObject *
advent2019__Spam_new(PyTypeObject *type, PyObject *args, PyObject *kwargs)
{
static char *kwlist[] = {NULL};
if (!PyArg_ParseTupleAndKeywords(args, kwargs, "", kwlist)) {
return NULL;
}
_SpamObject *obj = PyObject_GC_New(_SpamObject, type);
if (!obj) { return NULL; }
obj->state = 0;
obj->it = NULL;
PyObject_GC_Track(obj);
return (PyObject *)obj;
}
You'll need a function for when garbage collection works. There are three types: traverse, which allows you to pick up the object you are holding, clear, which destroys the reference, and dealloc, which destroys itself. The Py_VISIT macro is useful for writing traverse. PyObject_GC_UnTrack, which is paired with PyObject_GC_Track, should be called at the start of dealloc.
static int
advent2019__Spam_traverse(_SpamObject *self, visitproc visit, void *arg)
{
Py_VISIT(self->it);
return 0;
}
static int
advent2019__Spam_clear(_SpamObject *self)
{
Py_CLEAR(self->it);
return 0;
}
static void
advent2019__Spam_dealloc(_SpamObject *self)
{
PyObject_GC_UnTrack(self);
advent2019__Spam_clear(self);
PyObject_GC_Del(self);
}
\ _ \ _ Iter \ _ \ _ just returns itself, so it's easy. Don't forget to operate the reference count.
static PyObject *
advent2019__Spam_iter(_SpamObject *self)
{
Py_INCREF(self);
return (PyObject *)self;
}
Finally \ _ \ _ next \ _ \ _. It is named iternext on the Python C / API. The built-in functions print and sleep of the asyncio module should be included in the class attributes with the names \ _print and \ _sleep when creating the class, in the same way as Spam._Spam. After that, the code written in Python is PyObject_GetAttrString, [PyObject_CallFunction](https://docs.python .org / ja / 3 / c-api / object.html # c.PyObject_CallFunction) and others are used to steadily port. Check if the return value is NULL each time you call. The process of reducing the reference count of objects that are no longer needed is tedious. PyErr_Fetch, PyErr_GivenExceptionMatches for porting try statements /ja/3/c-api/exceptions.html#c.PyErr_GivenExceptionMatches), PyErr_Restore I will.
static PyObject *
advent2019__Spam_iternext(_SpamObject *self)
{
if (self->state == 0) {
// print('do something')
PyObject *printfunc = PyObject_GetAttrString((PyObject *)Py_TYPE(self), "_print");
if (!printfunc) { return NULL; }
PyObject *ret = PyObject_CallFunction(printfunc, "s", "do something");
Py_DECREF(printfunc);
if (!ret) { return NULL; }
Py_DECREF(ret);
// self._it = asyncio.sleep(1, 'RETURN VALUE').__await__()
PyObject *sleep_cofunc = PyObject_GetAttrString((PyObject *)Py_TYPE(self), "_sleep");
if (!sleep_cofunc) { return NULL; }
PyObject *sleep_co = PyObject_CallFunction(sleep_cofunc, "is", 1, "RETURN VALUE");
Py_DECREF(sleep_cofunc);
if (!sleep_co) { return NULL; }
if (!(Py_TYPE(sleep_co)->tp_as_async)) { Py_DECREF(sleep_co); return NULL; }
if (!(Py_TYPE(sleep_co)->tp_as_async->am_await)) { Py_DECREF(sleep_co); return NULL; }
PyObject *temp = self->it;
self->it = Py_TYPE(sleep_co)->tp_as_async->am_await(sleep_co);
Py_DECREF(sleep_co);
Py_XDECREF(temp);
if (self->it == NULL) { return NULL; }
self->state = 1;
}
if (self->state == 1) {
// next(self.it)
if (Py_TYPE(self->it)->tp_iternext == NULL) { PyErr_SetString(PyExc_TypeError, "no iternext"); return NULL; }
PyObject *ret = Py_TYPE(self->it)->tp_iternext(self->it);
if (!ret) {
// except StopIteration as e
PyObject *type, *value, *traceback;
PyErr_Fetch(&type, &value, &traceback);
if (PyErr_GivenExceptionMatches(type, PyExc_StopIteration)) {
Py_XDECREF(type);
Py_XDECREF(traceback);
if (!value) { PyErr_SetString(PyExc_ValueError, "no StopIteration value"); return NULL; }
// ret = e.value.lower()
PyObject *value2 = PyObject_CallMethod(value, "lower", NULL);
Py_DECREF(value);
if (!value2) { return NULL; }
// raise StopIteration(ret)
PyErr_SetObject(PyExc_StopIteration, value2);
Py_DECREF(value2);
Py_CLEAR(self->it);
self->state = 2;
} else {
// except:
// raise
PyErr_Restore(type, value, traceback);
}
}
return ret;
}
// raise StopIteration(None)
PyErr_SetNone(PyExc_StopIteration);
return NULL;
}
Now that we have the methods of the \ _Spam class, we define PyType_Spec. Make sure to set the flag Py_TPFLAGS_HAVE_GC to indicate that the class should be managed by garbage collection.
static PyType_Slot advent2019__Spam_slots[] = {
{Py_tp_new, advent2019__Spam_new},
{Py_tp_iter, advent2019__Spam_iter},
{Py_tp_iternext, advent2019__Spam_iternext},
{Py_tp_traverse, advent2019__Spam_traverse},
{Py_tp_clear, advent2019__Spam_clear},
{Py_tp_dealloc, advent2019__Spam_dealloc},
{0, 0},
};
static PyType_Spec advent2019__Spam_spec = {
.name = "advent2019._Spam",
.basicsize = sizeof(_SpamObject),
.flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC,
.slots = advent2019__Spam_slots,
};
I implemented Awaitable in Python C / API. We were also able to collect links to the official documentation that provided the information needed to do this. by the way. Is this useful? The code for just this is so long that it might help you get a better understanding of CPython, just like you've just realized the convenience of the async def and await syntax ...
setup.cfg
[metadata]
name = advent2019
version = 0.0.0
[options]
python_requires = >=3.5.0
setup.py
from setuptools import Extension, setup
extensions = [Extension('advent2019', sources=['advent2019.c'])]
setup(ext_modules=extensions)
advent2019.c
#define PY_SSIZE_T_CLEAN
#include <Python.h>
typedef struct {
PyObject_HEAD
unsigned char state;
PyObject *it;
} _SpamObject;
static PyObject *
advent2019__Spam_new(PyTypeObject *type, PyObject *args, PyObject *kwargs)
{
static char *kwlist[] = {NULL};
if (!PyArg_ParseTupleAndKeywords(args, kwargs, "", kwlist)) {
return NULL;
}
_SpamObject *obj = PyObject_GC_New(_SpamObject, type);
if (!obj) { return NULL; }
obj->state = 0;
obj->it = NULL;
PyObject_GC_Track(obj);
return (PyObject *)obj;
}
static PyObject *
advent2019__Spam_iter(_SpamObject *self)
{
Py_INCREF(self);
return (PyObject *)self;
}
static PyObject *
advent2019__Spam_iternext(_SpamObject *self)
{
if (self->state == 0) {
// print('do something')
PyObject *printfunc = PyObject_GetAttrString((PyObject *)Py_TYPE(self), "_print");
if (!printfunc) { return NULL; }
PyObject *ret = PyObject_CallFunction(printfunc, "s", "do something");
Py_DECREF(printfunc);
if (!ret) { return NULL; }
Py_DECREF(ret);
// self._it = asyncio.sleep(1, 'RETURN VALUE').__await__()
PyObject *sleep_cofunc = PyObject_GetAttrString((PyObject *)Py_TYPE(self), "_sleep");
if (!sleep_cofunc) { return NULL; }
PyObject *sleep_co = PyObject_CallFunction(sleep_cofunc, "is", 1, "RETURN VALUE");
Py_DECREF(sleep_cofunc);
if (!sleep_co) { return NULL; }
if (!(Py_TYPE(sleep_co)->tp_as_async)) { Py_DECREF(sleep_co); return NULL; }
if (!(Py_TYPE(sleep_co)->tp_as_async->am_await)) { Py_DECREF(sleep_co); return NULL; }
PyObject *temp = self->it;
self->it = Py_TYPE(sleep_co)->tp_as_async->am_await(sleep_co);
Py_DECREF(sleep_co);
Py_XDECREF(temp);
if (self->it == NULL) { return NULL; }
self->state = 1;
}
if (self->state == 1) {
// next(self.it)
if (Py_TYPE(self->it)->tp_iternext == NULL) { PyErr_SetString(PyExc_TypeError, "no iternext"); return NULL; }
PyObject *ret = Py_TYPE(self->it)->tp_iternext(self->it);
if (!ret) {
// except StopIteration as e
PyObject *type, *value, *traceback;
PyErr_Fetch(&type, &value, &traceback);
if (PyErr_GivenExceptionMatches(type, PyExc_StopIteration)) {
Py_XDECREF(type);
Py_XDECREF(traceback);
if (!value) { PyErr_SetString(PyExc_ValueError, "no StopIteration value"); return NULL; }
// ret = e.value.lower()
PyObject *value2 = PyObject_CallMethod(value, "lower", NULL);
Py_DECREF(value);
if (!value2) { return NULL; }
// raise StopIteration(ret)
PyErr_SetObject(PyExc_StopIteration, value2);
Py_DECREF(value2);
Py_CLEAR(self->it);
self->state = 2;
} else {
// except:
// raise
PyErr_Restore(type, value, traceback);
}
}
return ret;
}
// raise StopIteration(None)
PyErr_SetNone(PyExc_StopIteration);
return NULL;
}
static int
advent2019__Spam_traverse(_SpamObject *self, visitproc visit, void *arg)
{
Py_VISIT(self->it);
return 0;
}
static int
advent2019__Spam_clear(_SpamObject *self)
{
Py_CLEAR(self->it);
return 0;
}
static void
advent2019__Spam_dealloc(_SpamObject *self)
{
PyObject_GC_UnTrack(self);
advent2019__Spam_clear(self);
PyObject_GC_Del(self);
}
static PyType_Slot advent2019__Spam_slots[] = {
{Py_tp_new, advent2019__Spam_new},
{Py_tp_iter, advent2019__Spam_iter},
{Py_tp_iternext, advent2019__Spam_iternext},
{Py_tp_traverse, advent2019__Spam_traverse},
{Py_tp_clear, advent2019__Spam_clear},
{Py_tp_dealloc, advent2019__Spam_dealloc},
{0, 0},
};
static PyType_Spec advent2019__Spam_spec = {
.name = "advent2019._Spam",
.basicsize = sizeof(_SpamObject),
.flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC,
.slots = advent2019__Spam_slots,
};
typedef struct {
PyObject_HEAD
} SpamObject;
static PyObject *
advent2019_Spam_await(SpamObject *self)
{
PyObject *_Spam_Type = PyObject_GetAttrString((PyObject *)Py_TYPE(self), "_Spam");
if (_Spam_Type == NULL) { return NULL; }
PyObject *it = PyObject_CallFunction(_Spam_Type, "");
Py_DECREF(_Spam_Type);
return it;
}
static PyType_Slot advent2019_Spam_slots[] = {
{Py_am_await, (unaryfunc)advent2019_Spam_await},
{0, 0},
};
static PyType_Spec advent2019_Spam_spec = {
.name = "advent2019.Spam",
.basicsize = sizeof(SpamObject),
.flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,
.slots = advent2019_Spam_slots,
};
static int advent2019_exec(PyObject *module) {
int ret = -1;
PyObject *builtins = NULL;
PyObject *printfunc = NULL;
PyObject *asyncio_module = NULL;
PyObject *sleep = NULL;
PyObject *_Spam_Type = NULL;
PyObject *Spam_Type = NULL;
if (!(builtins = PyEval_GetBuiltins())) { goto cleanup; } /* borrowed */
// fetch the builtin function print
if (!(printfunc = PyMapping_GetItemString(builtins, "print"))) { goto cleanup; }
// import asyncio
if (!(asyncio_module = PyImport_ImportModule("asyncio"))) { goto cleanup; }
if (!(sleep = PyObject_GetAttrString(asyncio_module, "sleep"))) { goto cleanup; };
if (!(_Spam_Type = PyType_FromSpec(&advent2019__Spam_spec))) { goto cleanup; }
// _Spam._print = print
if (PyObject_SetAttrString(_Spam_Type, "_print", printfunc)) { goto cleanup; }
// _Spam._sleep = asyncio.sleep
if (PyObject_SetAttrString(_Spam_Type, "_sleep", sleep)) { goto cleanup; }
if (!(Spam_Type = PyType_FromSpec(&advent2019_Spam_spec))) { goto cleanup; }
// Spam._Spam = _Spam
if (PyObject_SetAttrString(Spam_Type, "_Spam", _Spam_Type)) { goto cleanup; }
if (PyObject_SetAttrString(module, "Spam", Spam_Type)) { goto cleanup; }
if (PyObject_SetAttrString(module, "_Spam", _Spam_Type)) { goto cleanup; }
ret = 0;
cleanup:
Py_XDECREF(printfunc);
Py_XDECREF(asyncio_module);
Py_XDECREF(sleep);
Py_XDECREF(_Spam_Type);
Py_XDECREF(Spam_Type);
if (ret) { Py_XDECREF(module); }
return ret;
}
static PyModuleDef_Slot advent2019_slots[] = {
{Py_mod_exec, advent2019_exec},
{0, NULL}
};
static struct PyModuleDef advent2019_moduledef = {
PyModuleDef_HEAD_INIT,
.m_name = "advent2019",
.m_slots = advent2019_slots,
};
PyMODINIT_FUNC PyInit_advent2019(void) {
return PyModuleDef_Init(&advent2019_moduledef);
}
Code example to use this
import sys
import asyncio
import advent2019
async def main():
v = await advent2019.Spam()
print(v)
if __name__ == '__main__':
if sys.version_info < (3, 7):
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()
else:
asyncio.run(main())
Recommended Posts