Embedding PyPy in a C application

Written by Emil Loer on May 15, 2016 |

At the PyGrunn 2016 conference last Friday I gave a talk about the Python cffi package and how to leverage it to embed PyPy into a C application. A video of this talk will be published later when it becomes available.

With the release of cffi 1.5.0 and its subsequent inclusion in PyPy 5 it becomes possible to embed PyPy. This is done by compiling the Python code into a dynamic library which can then be used in any other language. In this article I will show you how to do this.

The embedded API

The first step is to define some kind of interface that defines how the C application should call our Python code. We have to specify this using C function prototypes. For our example we will expose a function that performs some kind of computation, but this can of course be anything you want.

float compute(float first, float second);

Now we have to actually implement this computation in Python:

from my_library import ffi, lib

def compute(first, second):
    """ Compute the absolute distance between two numbers. """
    return abs(first - second)

This implementation snippet contains a few special things to make it embed properly. The first line imports the ffi and lib objects from the dynamic library. By doing this the implementation gets access to functions provided by cffi and these can be used for more complicated tasks, such as memory allocation. The name my_library is defined below and corresponds to the name of our dynamic library.

The second thing we notice in the snippet is the @ffi.def_extern decorator. This tells cffi that the decorated function is to be exposed in the public API of the generated C library. The decorated functions will be mapped to the prototypes given in the API definition and their arguments and return values will be converted automatically.

Library generator script

Now that we have our API and its implementation to embed we actually have to embed it somewhere. For this we are using a script that generates the dynamic library. The embedding script requires the two snippets given above to be in the files api.h and implementation.py.

import cffi
ffi = cffi.FFI()


ffi.set_source("my_library", "")

The embedding script is pretty straightforward. We have to specify our external API and provide an implementation. Both are read from disk and correspond to the two snippets given in the previous section.

After specifying the source we have to tell cffi the name of our library, which is my_library in our example. Additionally this is the place to add extra C source to provide types for the API header file, e.g. by including the proper header files (which is not allowed in embedding_api). All that remains is compiling the sources to create our library file.

Running the script produces some output and generates our library:

$ pypy embed.py
generating ./my_library.c
running build_ext
building 'my_library' extension

$ ls my_library.dylib
-rwxr-xr-x  1 djinn  staff  9856 May 15 14:46 my_library.dylib

All that remains is to use it somewhere!

Host application

Using the embedded Python code is actually really simple. It can be done with just the following code:

#include <stdio.h>
#include "api.h"

int main(void) {
    float result = compute(12.34f, 10.0f);
    printf("The result: %f\n", result);

    return 0;

As you can see there is hardly any work required to call our Python code. With the CPython embedding API you would have had to fire up an interpreter and do lots of parameter and return value conversions. But not with cffi! The generated library takes care of all that so you can focus on actually building stuff.

The final thing I have to show you is how to compile and run it:

$ clang -o test test.c my_library.dylib

$ ./test
The result: 2.340000

And there you have it! With only a few lines of code we have written a C program that fires up a PyPy interpreter and runs our Python code as if it was a native C function. Of course I've only shown you the basics here, but the technology is really powerful. For more information please have a look at the cffi documentation.

If you've enjoyed this post please follow me on Twitter or Facebook. I'd really appreciate it!