Cgo and Python
Embedding Python in Go lets applications gradually migrate from Python, reuse existing libraries, and load scripts dynamically without recompiling. Datadog uses this approach in its Go-based Agent so checks can remain in Python while the core application moves to Go. The key is combining cgo with a Go-friendly wrapper around CPython’s C API. ## Why Embed Python in Go? - Supports incremental migration from an existing Python codebase. - Reuses mature Python libraries without reimplementing them in Go. - Enables runtime loading and execution of custom or updated Python scripts. - This dynamic extensibility is especially important for Datadog checks. ## Introducing cgo - CPython exposes a C API, while Go requires a Foreign Function Interface to call C code. - cgo provides that integration while preserving the normal `go build` workflow. - A C preamble placed immediately before `import "C"` can include headers and C code. - The pseudo-package `C` exposes C constants, functions, and types to Go. - `go build -x` reveals how cgo generates intermediate C and Go files, compiles them, and links the final binary. ## Initializing the CPython Interpreter - A Go program must initialize Python with `Py_Initialize()` before executing Python code. - It should shut down the interpreter with `Py_Finalize()` when finished. - `Py_GetVersion()` demonstrates retrieving Python information through the C API. - `#cgo` directives can use `pkg-config` to locate Python development headers and libraries, such as `python-2.7`. - The examples use Python 2, but the same approach largely applies to Python 3. ## Using a Go Wrapper - Direct cgo interaction is mostly boilerplate, so Datadog uses the `go-python` library. - The wrapper exposes operations such as: - `python.Initialize()` - `python.PyRun_SimpleString(...)` - `python.Finalize()` - This hides cgo details and makes embedded Python code look more idiomatic from Go. ## Importing and Calling Python Code - A Python module can be imported with `PyImport_ImportModule`. - Go retrieves a function using `GetAttrString`. - The function is invoked through the Python API, passing empty tuple and dictionary objects even when it accepts no arguments. - The Go code must check for failures when importing modules or locating functions. - A simple `foo.py` module containing a `hello()` function can therefore be loaded and executed from disk. Embedding CPython through cgo provides a practical bridge between Go and Python. A wrapper such as `go-python` makes the integration easier to maintain, while allowing applications like the Datadog Agent to combine a Go core with dynamically executed Python components.
Read original(opens in new tab)