This page looks best with JavaScript enabled

Compiling Python Code to Executable with nuitka

 ·   ·  β˜• 7 min read

From the official documentation of Nuitka it says

Nuitka is the Python compiler. It is written in Python. It is a seamless replacement or extension to the Python interpreter and compiles every construct that CPython 2.6, 2.7, 3.3, 3.4, 3.5, 3.6, 3.7, and 3.8 have, when itself run with that Python version.

How to get started?

  1. Install Nuitka

    pip install nuitka

  2. Make a folder named HelloWorld

    mkdir HelloWorld

  3. CD to HelloWorld

    cd HelloWorld

  4. Make a file ‘hello.py’

    def talk(message):
    return "Talk " + message
    
    def main():
        print( talk("Hello World"))
    
    if __name__ == "__main__":
        main()
    
  5. Run the file with python

    python hello.py

  6. Build it using Nuitka

    python -m nuitka –mingw64 hello.py

Now you get hello.exe. Run it. That’s it.

For in-depth tutorial go to the official documentation.

Use Cases

Use Case 1 - Program compilation with all modules embedded

If you want to compile a whole program recursively, and not only the single file that is the main program, do it like this:

python -m nuitka –follow-imports program.py

Note

There are more fine grained controls than --follow-imports available. Consider the output of nuitka --help.

In case you have a plugin directory, i.e. one which cannot be found by recursing after normal import statements via the PYTHONPATH (which would be the recommended way), you can always require that a given directory shall also be included in the executable:

python -m nuitka –follow-imports –include-plugin-directory=plugin_dir program.py

Note

If you don’t do any dynamic imports, simply setting your PYTHONPATH at compilation time will be sufficient for all your needs normally.

Use --include-plugin-directory only if you make __import__() calls that Nuitka cannot predict, because they e.g. depend on command line parameters. Nuitka also warns about these, and point to the option.

Note

The resulting filename will be program.exe on Windows, program.bin on other platforms.

Note

The resulting binary still depends on CPython and used C extension modules being installed.

If you want to be able to copy it to another machine, use --standalone and copy the created program.dist directory and execute the program.exe (Windows) or program (other platforms) put inside.

Use Case 2 - Extension Module compilation

If you want to compile a single extension module, all you have to do is this:

python -m nuitka –module some_module.py

The resulting file some_module.so can then be used instead of some_module.py.

Note

It’s left as an exercise to the reader, to find out what happens if both are present.

Note

The option --follow-imports and other variants work as well, but the included modules will only become importable after you imported the some_module name.

Use Case 3 - Package compilation

If you need to compile a whole package and embed all modules, that is also feasible, use Nuitka like this:

python -m nuitka –module some_package –include-package=some_package

Note

The recursion into the package directory needs to be provided manually, otherwise, the package is empty. Data files located inside the package will not be embedded yet.

Typical Problems

Dynamic sys.path

If your script modifies sys.path to e.g. insert directories with source code relative to it, Nuitka will currently not be able to see those. However, if you set the PYTHONPATH to the resulting value, you will be able to compile it.

Tips

Python command line flags

For passing things like -O or -S to Python, to your compiled program, there is a command line option name --python-flag= which makes Nuitka emulate these options.

The most important ones are supported, more can certainly be added.

Caching compilation results

The C compiler, when invoked with the same input files, will take a long time and much CPU to compile over and over. Make sure you are having ccache installed and configured when using gcc (even on Windows). It will make repeated compilations much faster, even if things are not yet not perfect, i.e. changes to the program can cause many C files to change, requiring a new compilation instead of using the cached result.

On Windows, with gcc Nuitka supports using ccache.exe which it will offer to download from an official source and it automatically. This is the recommended way of using it on Windows, as other versions can e.g. hang.

Nuitka will pick up ccache if it’s in found in system PATH, and it will also be possible to provide if by setting NUITKA_CCACHE_BINARY to the full path of the binary, this is for use in CI systems.

For the Visual Studio compilers, you are just one pip install clcache command away. To make Nuitka use those, set NUITKA_CLCACHE_BINARY to the full path of clcache.exe, which will be in the scripts folder of the Python, you installed it into.

Runners

Avoid running the nuitka binary, doing python -m nuitka will make a 100% sure you are using what you think you are. Using the wrong Python will make it give you SyntaxError for good code or ImportError for installed modules. That is happening, when you run Nuitka with Python2 on Python3 code and vice versa. By explicitly calling the same Python interpreter binary, you avoid that issue entirely.

Fastest C Compilers

The fastest binaries of pystone.exe on Windows with 64 bits Python proved to be significantly faster with MinGW64, roughly 20% better score. So it is recommended for use over MSVC. Using clang-cl.exe of Clang7 was faster than MSVC, but still significantly slower than MinGW64, and it will be harder to use, so it is not recommended.

On Linux for pystone.bin the binary produced by clang6 was faster than gcc-6.3, but not by a significant margin. Since gcc is more often already installed, that is recommended to use for now.

Differences in C compilation times have not yet been examined.

Unexpected Slowdowns

Using the Python DLL, like standard CPython does can lead to unexpected slowdowns, e.g. in uncompiled code that works with Unicode strings. This is because calling to the DLL rather than residing in the DLL causes overhead, and this even happens to the DLL with itself, being slower, than a Python all contained in one binary.

So if feasible, aim at static linking, which is currently only possible with Anaconda Python on non-Windows.

Windows Standalone executables and dependencies

The process of making standalone executables for Windows traditionally involves using an external dependency walker in order to copy necessary libraries along with the compiled executables to the distribution folder.

Using the external dependency walker is quite a time consuming, and may copy some unnecessary libraries along the way (better have too much than missing).

There’s also an experimental alternative internal dependency walker that relies on pefile which analyses PE imports of executables and libraries.

This implementation shall create smaller Standalone distributions since it won’t include Windows' equivalent of the standard library, and will speed-up first Nuitka compilations by an order of magnitude.

In order to use it, you may enable the internal dependency walker by using the following switch:

python -m nuitka –standalone –windows-dependency-tool=pefile myprogram.py

Note

The pefile dependency walker will test all dependencies of the distribution folder.

Optionally, it is also possible to check all recursive dependencies of included libraries using the following switch along with the above one:

python -m nuitka –standalone –windows-dependency-tool=pefile –experimental=use_pefile_recurse myprogram.py

Note

Some modules may have hidden dependencies outside of their directory. In order for the pefile dependency walker to find them, you may also scan the whole site-packages directory for missing dependencies using the following switch along with the two above:

python -m nuitka –standalone –windows-dependency-tool=pefile –experimental=use_pefile_recurse –experimental=use_pefile_fullrecurse myprogram.py

Note

Be aware that using this switch will increase compilation time a lot.

Windows errors with resources

On Windows, the Windows Defender tool and the Windows Indexing Service both scan the freshly created binaries, while Nuitka wants to work with it, e.g. adding more resources, and then preventing operations randomly due to holding locks. Make sure to exclude your compilation stage from these services.

Windows standalone program redistribuation

Whether compiling with MingW or MSVC, the standalone programs have external dependencies to Visual C Runtime libraries. Nuitka tries to ship those dependent DLLs by copying them from your system.

Beginning with Microsoft Windows 10, Microsoft ships ucrt.dll (Universal C Runtime libraries) which rehook calls to api-ms-crt-*.dll.

With earlier Windows platforms (and wine/ReactOS), you should consider installing Visual C Runtime libraries before executing a Nuitka standalone compiled program.


Ohidur Rahman Bappy
WRITTEN BY
Ohidur Rahman Bappy
πŸ“šLearner 🐍 Developer