Quantcast
Viewing latest article 23
Browse Latest Browse All 49

Creating Python modules from an extension with extra C data structures

I'm working on a custom Python loader that creates Python modules from a particular kind of non-Python file, let's call it a "cheese file". I'm writing my project as a C extension module because these "cheese files" need to be processed by a fairly complex C library (and secondarily, as a way to learn about the Python/C API).

When processing a cheese file, the C library allocates some data structures which will need to be deallocated after the Python module object is deleted. My question is, how can/should I store those C data structures along with the Python module object?

Some ideas I had:

  • What I would have thought was the cleanest option would be to subclass Python's ModuleType from C and adding a field into the subclass to store the data structures. The extension types tutorial is apparently quite clear about how to subclass a built-in Python type in C, so I expected to be able to do something like this:

    typedef struct {    PyModuleObject module;    struct cheese_data* data;} PyCheeseModule;static PyTypeObject PyCheeseModule_Type = {    PyVarObject_HEAD_INIT(NULL, 0)    .tp_basicsize = sizeof(PyCheeseModule),    .tp_flags = Py_TPFLAGS_DEFAULT,    .tp_base = &PyModule_Type,    /* other fields */};

    But PyModuleObject is not exposed as part of the Python/C API, so I can't use it in this way.

  • I also considered dynamically allocating a PyModuleDef for each cheese module, and using PyModule_FromDefAndSpec() along with PyModule_ExecDef() to create and execute the actual module. But I'm not really sure if PyModuleDef is supposed to be used this way, since the documentation only demonstrates it with statically-defined C extension modules, and anyway I would have to deallocate the PyModuleDef object itself which kind of brings me back to the same problem.

  • Yet another approach is to wrap the C data structure in a Python object and just add it to the module's dict. But then there's a risk that Python code would change or unset that attribute. I suppose I can find a way to work around that, but it seems pretty inelegant.

  • Maybe I should just be using Cython for this project. But even so, if this problem is solvable in Cython, it should also be possible to do it using the plain old Python/C API, and at least for educational value I'd like to know how.

If necessary, I can expand on these attempts with some additional code.


Viewing latest article 23
Browse Latest Browse All 49

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>