Nuitka Home

Bug Fixes

• Python3.5+: Fix, coroutines and asyncgen could continue iteration of awaited functions, even after their return, leading to wrong behaviour.

• Python3.5+: Fix, absolute imports of names might also refer to modules and need to be handled for module loading as well.

• Fix, the fromlist of imports could loose references, potentially leading to corruption of contained strings.

• Python3.8: Fix, positional only arguments were not enforced to actually be that way.

• Python3.8: Fix, complex calls with star arguments that yielded the same value twice, were not yet caught.

• Python3.8: Fix, evaluation order for nested dictionary contractions was not followed yet.

• Windows: Use short paths, these work much better to load extension modules and TCL parts of TkInter cannot handle unicode paths at all. This makes Nuitka work in locations, where normal Python cannot.

• Windows: Fixup dependency walker in unicode input directories.

• Standalone: Use frozen module loader only at libpython initialisation and switch to built-in bytecode loader that is more compatible afterwards, increasing compatibility.

• Standalone: Fix for pydanctic support.

• Standalone: Added missing hidden dependency of uvicorn.

• Fix, the parser for .pyi files couldn't handle multiline imports.

• Windows: Derive linker arch of Python from running binary, since it can happen that the Python binary is actually a script.

• Fixup static linking with libpython.a that contains main.o by making our colliding symbols for Py_GetArgcArgv weak.

• Python3.7: Fix misdetection as asyncgen for a normal generator, if the iterated value is async.

• Distutils: Fix build_nuitka for modules under nested namespaces.

• OpenBSD: Follow usage of clang and other corrections to make accelerated mode work.

• macOS: Fixup for standalone mode library scan.

• Fix, the logging of --show-modules was broken.

• Windows: Enable /bigobj mode for MSVC for large compilations to work.

• Windows: Fixup crash in warning with pefile dependency manager.

• Windows: Fixup win32com standalone detection of other Python version win32com is in system PATH.

• Fix, the python flag for static hashes didn't have the intended effect.

• Fix, generators may be resurrected in the cause of their destruction, and then must not be released.

• Fix, method objects didn't implement the methods __reduce__ and __reduce_ex__ necessary for pickling them.

• Windows: Fix, using a Python installation through a symlink was not working.

• Windows: Fix, icon paths that were relative were not working anymore.

• Python3.8: Detect duplicate keywords yielded from star arguments.

• Fix, methods could not be pickled.

• Fix, generators, coroutines and asyncgen might be resurrected during their release, allow for that.

• Fix, frames need to traverse their attached locals to be released in some cases.

New Features

• Plugin command line handling now allows for proper optparse options to be used, doing away with special parameter code for plugins. The arguments now also become automatically passed to the instantiations of plugins.

  Loading and creation of plugins are now two separate phases. They are loaded when they appear on the command line and can add options in their own group, even required ones, but also with default values.

• Started using logging with name-spaces. Applying logging per plugin to make it easier to recognize which plugin said what. Warnings are now colored in red.

• Python3.5+: Added support for two step module loading, making Nuitka loading even more compatible.

• Enhanced import tracing to work on standalone binaries in a useful manner, allow to compare with normal binaries.

• Fix, the setattr built-in was leaking a reference to the None value.

Optimization

• Proper loop SSA capable of detecting shapes with an incremental initial phase and a final result of alternatives for variables written in the loop. This detects shapes of manual integer incrementing loops correctly now, it doesn't see through iterators yet, but this will come too.

• Added type shapes for all operations and all important built-in types to allow more compile time optimization and better target type selection.

• Target type code generation was expanded from manual usage with conditions to all operations allowing to get at bool target values more directly.

• For in-place operations, there is the infrastructure to generate them for improved performance, but so far it's only used for Python2 int, and not for the many types normal operations are supported.

• Force usage of C boolean type for all indicator variables from the re-formulation. In some cases, we are not yet there with detections, and this gives instant benefit.

• Complex constants didn't annotate their type shape, preventing compile time optimization for them.

• Python3.8: Also support vectorcall for compiled method objects. These are rarely used in new Python, but can make a difference.

• Remove loops that have only a final break. This happens in static optimization in some cases, and allows more optimization to be done.

• Avoid using a preparing a constant tuple value for calls with only constant arguments.

• Avoid using PyErr_Format where it's not necessary by adding specialized helpers for common cases.

• Detect del statements that will raise an exception and replace with that.

• Exception matching is boolean shape, allowing for faster code generation.

• Disable recursion checks outside of full compat mode.

• Avoid large blocks for conditional statements that only need to enclose the condition evaluation.

• Added shortcuts for interactions between compiled generator variants, to avoid calls to their C methods with argument passing, etc.

Organisational

• Updated developer manual with changes that happened, remvoing the obsolete language choice section.

• Added 3.8 support mentions is even more places.

• The mailing list has been deleted. We now prefer Gitter chat and Github issues for discussions.

• Visual Code recommended extensions are now defined as such in the project configuration and you will be prompted to install them.

• Visual Code environents for Py38 and Py27 were added for easier switch.

• Catch usage of Python from the Microsoft App Store, it is not supported and seems to limit access to the Python installation for security reasons that make support impossible.

• Make it clear that --full-compat should not be used in help output.

• Added instructions for MSVC runtimes and standalone compilation to support Windows 7.

• More complete listing of copyright holders for Debian.

• Updated to newer black and PyLint.

• Enhanced gcc version check, properly works with gcc 10 and higher.

Tests

• Pylint cleanups for some of the tests.

• Added test for loading of user plugins.

• Removed useless outputs for search mode skipping non-matches.

Cleanups

• Limit command line handling for multiprocessing module to when the plugin is actually used, avoiding useless code of Windows binaries.

• Pylint cleanup also foreign code like oset and odict.

• In preparation of deprecating the alternative, --plugin-enable has become the only form used in documentation and tests.

• Avoid numeric pylint symbols more often.

• Distutils: Cleanup module name for distutils commands, these are not actually enforced by distutils, but very ugly in our coding conventions.

• The "cannot get here" code to mark unreachable code has been improved and no longer needs an identifier passed, but uses the standard C mechanism for that.

• Removed accessors for lookup sources from nodes, allowing for faster usage and making sure, lookups are only done where needed.

Summary

This release is huge in terms of bugs fixed, but also extremely important, because the new loop SSA and type tracing, allows for many more specialized code usages. We now can trace the type for some loops to be specifically an integer or long value only, and will become able to generate code that avoids using Python objects, in these cases.

Once that happens, the performance will make a big jump. Future releases will have to consolidate the current state, but it is expected that at least an experimental addition of C type float or C long can be added, add to that iterator type shape and value analsis, and an actual jump in performance can be expected.

Bug Fixes

• Compatibility: The value of __module__ for extension modules was not dependent into which package the module was loaded, it now is.

• Anaconda: Enhanced detection of Anaconda for Python 3.6 and higher.

• CentOS6: Detect gcc version to allow saving on macro memory usage, very old gcc didn't have that.

• Include Python3 for all Fedora versions where it works as well as for openSUSE versions 15 and higher.

• Windows: Using short path names to interact with Scons avoids problems with unicode paths in all cases.

• macOS: The usage of install_name_tool could sometimes fail due to length limits, we now increase it at link time.

• macOS: Do not link against libpython for module mode. This prevented extension modules from actually being usable.

• Python3.6: Follow coroutine fixes in our asyncgen implementation as well.

• Fix, our version number handling could overflow with minor versions past 10, so we limited it for now.

New Features

• Added support for Python 3.8, the experimental was already there and pretty good, but now added the last obscure features too.

• Plugins can now provide C code to be included in the compilation.

• Distutils: Added targets build_nuitka and install_nuitka to complement bdist_nuitka, so we support software other than wheels, e.g. RPM packaging that compiles with Nuitka.

• Added support for lldb the Clang debugger with the --debugger mode.

Optimization

• Make the file prefix map actually work for gcc and clang, and compile files inside the build folder, unless we are running in debugger mode, so we use ccache caching across different compilations for at least the static parts.

• Avoid compilation of __frozen.c in accelerated mode, it's not used.

• Prefer using the inline copy of scons over systems scons. The later will only be slower. Use the fallback to external scons only from the Debian packages, since there we consider it forbidden to include software as a duplicate.

Organisational

• Added recommended plugins for Visual Code, replacing the list in the Developer Manual.

• Added repository for Fedora 30 for download.

• Added repository for CentOS 8 for download.

• Updated inline copy of Scons used for Python3 to 3.1.2, which is said to be faster for large compilations.

• Removed Eclipse setup from the manual, it's only infererior at this point and we do not use it ourselves.

• Debian: Stop recommending PyQt5 in the package, we no longer use it for built-in GUI that was removed.

• Debian: Bumped the standards version and modernized the packaging, solving a few warnings during the build.

Cleanups

• Scons: Avoid to add Unix only include paths on Windows.

• Scons: Have the static source code in a dedicated folder for clarity.

Tests

• Added tests to Github Actions, for the supported Python versions for all of Linux, macOS and Windows, covering the later publicly for the first time. We use Anaconda on macOS for the tests now, rather than Homebrew.

• Enable IO encoding to make sure we use UTF8 for more test suites that actually need it in case of problems.

• Comparing module outputs now handles segfaults by running in the debugger too.

Summary

This release adds full support for Python 3.8 finally, which took us a while, and it cleans up a lot on the packaging side. There aren't that many important bug fixes, but it's still nice to this cleaned up.

We have important actual optimization in the pipeline that will apply specialization to target types and for comparison operations. We expect to see actual performance improvements in the next release again.

Bug Fixes

• Fix, the top level module must not be bytecode. Otherwise we end up violating the requirement for an entry point on the C level.

• Fix, avoid optimizing calls with default values used. This is not yet working and needed to be disabled for now.

• Python3: Fix, missing keyword only arguments were not enforced to be provided keyword only, and were not giving the compatible error message when missing.

• Windows: Find win32com DLLs too, even if they live in sub folders of site-packages, and otherwise not found. They are used by other DLLs that are found.

• Standalone: Fixup for problem with standard library module in most recent AnaConda versions.

• Scons: Fix, was using CXXFLAGS and CPPFLAGS even for the C compiler, which is wrong, and could lead to compilation errors.

• Windows: Make --clang limited to clang-cl.exe as using it inside a MinGW64 is not currently supported.

• Standalone: Added support for using lib2to2.pgen.

• Standalone: Added paths used by openSUSE to the Tcl/Tk plugin.

• Python3.6+: Fix, the __main__ package was None, but should be "" which allows relative imports from itself.

• Python2: Fix, compile time optimization of floor division was using normal division.

• Python3: Fix, some run time operations with known type shapes, were falsely reporting error message with unicode or long, which is of course not compatible.

• Fix, was caching parent package, but these could be replaced e.g. due to bytecode demotion later, causing crashes during their optimization.

• Fix, the value of __compiled__ could be corrupted when being deleted, which some modules wrappers do.

• Fix, the value of __package__ could be corrupted when being deleted.

• Scons: Make sure we can always output the compiler output, even if it has a broken encoding. This should resolve MSVC issues on non-English systems, e.g. German or Chinese.

• Standalone: Support for newest sklearn was added.

• macOS: Added resolver for run time variables in otool output, that gets PyQt5 to work on it again.

• Fix, floor division of run time calculations with float values should not result in int, but float values instead.

• Standalone: Enhanced support for boto3 data files.

• Standalone: Added support for osgeo and gdal.

• Windows: Fix, there were issues with spurious errors attaching the constants blob to the binary due to incorrect C types provided.

• Distutils: Fix, need to allow / as separator for package names too.

• Python3.6+: Fix reference losses in asyncgen when throwing exceptions into them.

• Standalone: Added support for dill.

• Standalone: Added support for scikit-image and skimage.

• Standalone: Added support for weasyprint.

• Standalone: Added support for dask.

• Standalone: Added support for pendulum.

• Standalone: Added support for pytz and pytzdata.

• Fix, --python-flags=no_docstrings no longer implies disabling the assertions.

New Features

• Added experimental support for Python 3.8, there is only very few things missing for full support.

• Distutils: Added support for packages that are in a namespace and not just top level.

• Distutils: Added support for single modules, not only packages, by supporting py_modules as well.

• Distutils: Added support for distinct namespaces.

• Windows: Compare Python and C compiler architecture for MSVC too, and catch the most common user error of mixing 32 and 64 bits.

• Scons: Output variables used from the outside, so the debugging is easier.

• Windows: Detect if clang installed inside MSVC automatically and use it if requested via --clang option. This is only the 32 bits variant, but currently the easy way to use it on Windows with Nuitka.

Optimization

• Loop variables were analysed, but results were only available on the inside of the loop, preventing many optimization in these cases.

• Added optimization for the abs built-in, which is also a numerical operator.

• Added optimization for the all built-in, adding a new concept of iteration handle, for efficient checking that avoids looking at very large sequences, of which properties can still be known.

  all(range(1,100000)) # no need to look at all of them
  

• Added support for optimizing ImportError construction with keyword-only arguments. Previously only used without these were optimized.

  raise aise ImportError(path="lala", name="lele") # now optimized
  

• Added manual specialization for single argument calls, sovling a TODO, as these will be very frequent.

• Memory: Use single child form of node class where possible, the general class now raises an error if used with used with only one child name, this will use less memory at compile time.

• Memory: Avoid list for non-local declarations in every function, these are very rare, only have it if absolutely necessary.

• Generate more compact code for potential NameError exceptions being raised. These are very frequent, so this improves scalability with large files.

• Python2: Annotate comparison of None with int and str types as not raising an exception.

• Shared empty body functions and generators.

  One shared implementation for all empty functions removes that burden from the C compiler, and from the CPU instruction cache. All the shared C code does is to release its arguments, or to return an empty generator function in case of generator.

• Memory: Added support for automatic releases of parameter variables from the node tree. These are normally released in a try finally block, however, this is now handled during code generation for much more compact C code generated.

• Added specialization for int and long operations %, <<, >>, |, &, ^, **, @.

• Added dedicated nodes for representing and optimizing based on shapes for all binary operations.

• Disable gcc macro tracing unless in debug mode, to save memory during the C compilation.

• Restored Python2 fast path for int with unknown object types, restoring performance for these.

Cleanups

• Use dedicated ModuleName type that makes the tests that check if a given module name is inside a namespace as methods. This was hard to get right and as a result, adopting this fixed a few bugs and or inconsistent results.

• Expand the use of nuitka.PostProcessing to cover all actions needed to get a runnable binary. This includes using install_name_tool on macOS standalone, as well copying the Python DLL for acceleration mode, cleaning the x bit for module mode. Previously only a part of these lived there.

• Avoid including the definitions of dynamically created helper functions in the C code, instead just statically declare the ones expected to be there. This resolves Visual Code complaining about it, and should make life also easier for the compiler and caches like ccache.

• Create more helper code in closer form to what clang-format does, so they are easier to compare to the static forms. We often create hard coded variants for few arguments of call functions, and generate them for many argument variations.

• Moved setter/getter methods for Nuitka nodes consistently to the start of the node class definitions.

• Generate C code much closer to what clang-format would change it to be.

• Unified calling install_name_tool on macOS into one function that takes care of all the things, including e.g. making the file writable.

• Debug output from scons should be more consistent and complete now.

• Sort files for compilation in scons for better reproducible results.

• Create code objects version independent, avoiding python version checks by pre-processor, hiding new stuff behind macros, that ignore things on older Python versions.

Tests

• Added many more built-in tests for increased coverage of the newly covered ones, some of them being generic tests that allow to test all built-ins with typical uses.

• Many tests have become more PyLint clean as a result of work with Visual Code and it complaining about them.

• Added test to check PyPI health of top 50 packages. This is a major GSoC 2019 result.

• Output the standalone directory contents for Windows too in case of a failure.

• Added generated tests to fully cover operations on different type shapes and their errors as well as results for typical values.

• Added support for testing against installed version of Nuitka.

• Cleanup up tests, merging those for only Python 3.2 with 3.3 as we no longer support that version anyway.

• Execute the Python3 tests for macOS on Travis too.

Organisational

• The donation sponsored machine called donatix had to be replaced due to hardware breakage. It was replaced with a Raspberry-Pi 4.

• Enhanced plugin documentation.

• Added description of the git workflow to the Developer Manual.

• Added checker script check-nuitka-with-codespell that reports typos in the source code for easier use of codespell with Nuitka.

• Use newest PyLint and clang-format.

• Also check plugin documentation files for ReST errors.

• Much enhanced support for Visual Code configuration.

• Trigger module code is now written into the build directory in debug mode, to aid debugging.

• Added deep check function that descends into tuples to check their elements too.

Summary

This release comes after a long time of 4 months without a release, and has accumulated massive amounts of changes. The work on CPython 3.8 is not yet complete, and the performance work has yet to show actual fruit, but has also progressed on all fronts. Connecting the dots and pieces seems not far away.

Built-ins Optimizations

• "any": PR #246

  • nuitka.nodes.BuiltinAnyNodes node added to optimize the "any" built-in.

  • Developed an algorithm to predict the "any" for arguments having repetitive values at compile time. For example:

    any([0]*255) -> False

    any([False, False, True]) -> True

  • Extended support for range, set and dict built-ins.

  • Added the optimized C side support too

  • Added a method getMetaClassBase to make Python 2 and Python 3 compatible while working with metaclasses.

• Issue reported and closed Issue #349

  • Created a new module nuitka.nodes.IterationHandles to work with iterables.

  • Added support of Iteration for non-mutable types.

• "all": PR #407

  • Added nuitka.nodes.BuiltinAllNodes to optimize the "all" built-ins.

  • Developed an algorithm similar to "any" to predict the "all" arguments. For example:

    all([0, 0, 1]) -> False

    all([True]*100) -> True

  • Other similar optimizations are done like "any" built-in.

  • Additionally, added a new testing module tests.optimizations.CommonOptimizations to test the built-ins optimizations at the same place.

• "abs": PR #419

  • Added new operation node ExpressionOperationAbs to optimize the abs built-in.

  • Manually added shapeSlotAbs to different shapes.

  • Finally pre-computed the compile time constant abs

• "max" and "min": PR #442

  • This PR is work in progress and is half complete.

  • This is the first optimizations in which I used reformulations instead of added in a new node.

  • Pseudo-code of "min" reformulation:

    def _min(a, b, c, ...):
     tmp_arg1 = a
     tmp_arg2 = b
     tmp_arg3 = c
     ...
     result = tmp_arg1
     if keyfunc is None: # can be decided during re-formulation
         tmp_key_result = keyfunc(result)
         tmp_key_candidate = keyfunc(tmp_arg2)
         if tmp_key_candidate < tmp_key_result:
             result = tmp_arg2
             tmp_key_result = tmp_key_candidate
         tmp_key_candidate = keyfunc(tmp_arg3)
         if tmp_key_candidate < tmp_key_result:
             result = tmp_arg3
             tmp_key_result = tmp_key_candidate
         ...
     else:
         if tmp_arg2 < result:
             result = tmp_arg2
         if tmp_arg3 < result:
             result = tmp_arg3
         ...
      return result
    

  • Adding support for keyfunc is pending

• "zip": PR #462

  • This built-in uses both types of optimizations that the previous built-ins optimizations used.

  • zip for Python 2 uses the reformulations.

  • Pseudo-code of "zip" reformulation:

  def _zip(a, b, c, ... ):
  # First assign, to preserve order of execution,
  # the arguments might be complex expressions.
  tmp_arg1 = a
  tmp_arg2 = b
  tmp_arg3 = c
  ...
  tmp_iter_1 = iter(tmp_arg1)
  tmp_iter_2 = iter(tmp_arg2)
  tmp_iter_3 = iter(tmp_arg3)
  ...
  # could be more
  tmp_result = []
  try:
      while 1:
          tmp_result.append(
              (
                   next(tmp_iter_1),
                   next(tmp_iter_2),
                   next(tmp_iter_3),
                   ...
              )
           )
     except StopIteration:
         pass
   return tmp_result
  

  • zip for Python 3 needs a new node that calls the zip because unlike zip in Python 2, zip in Python 3 returns a zipobject.

Test suite

• Search mode "All": PR #378

  • In the test suite, I added a new search mode "all" that will test all the modules and return all the results at once. For example:

  $ ./tests/basics/run_all.py all
     Using concrete python 2.7.12 on x86_64
     Comparing output of 'Asserts.py' using '/usr/bin/python' with flags silent, expect_success, remove_output,     recurse_all, original_file, cpython_cache, plugin_enable:pylint-warnings ...
     .
     .
     .
     .
     Total 0 error(s) found.
  

• Search mode "Only": PR #333

  • Added "only" search mode to test only a single module. For example:

  $ ./tests/basics/run_all.py only BuiltinsTest.py
     Using concrete python 2.7.12 on x86_64
     Skipping Asserts.py
     Skipping Assignments.py
     Skipping BigConstants.py
     Skipping Branching.py
     Skipping BuiltinOverload.py
     Skipping BuiltinSuper.py
     Comparing output of 'BuiltinsTest.py' using '/usr/bin/python' with flags silent, expect_success, remove_output, recurse_all, original_file, cpython_cache, plugin_enable:pylint-warnings ...
  

• Reported and closed Issue #334: PR #336

  • Fixed the autoformat bug by reading and writing the files in bytes instead of string.

Documentation

• Nuitka git work flow: PR #485

And other minor doc fixes are added with their respective pull requests.

What I learned

• Learned the software engineering principles and how to keep my work clean.

• I also learned how to effectively use software designing principles like DRY and KISS.

• Got exposed to Nuitka internals which helped me to better understand how compilers in general work.

• Explored how CPython works internally.

• Got some great advice from Mentor Hayen about starting my professional career in Software engineering.

Overall, it was a great experience to be a part of Nuitka :)

Intro

As Google Summer of Code (GSoC) is coming to an end, I am writing this blog post as a final summary describing all the work I have done as well as my experiences in this program.

Summary of My Work

• #314 run_all.py new special-comment mechanism & Urllib3Using.py

  • Before GSoC started, I looked around for whatever work I could help with.

  • In this pull request, I added a checkRequirements function for the Nuitka standalone test suite.

  • This function checks for special-comments at the top of standalone tests in the format of # nuitka-skip-unless-expression: expression to be evaluated OR # nuitka-skip-unless-imports: module1,module2,... and will decide whether to skip a test depending on if its specified requirements are met.

  • In addition, standalone test Urllib3Using.py was created.

  • This pull request was soon merged and allowed me the lucky opportunity of GSoC 2019 with Nuitka :)

• #339 Standalone tests for botocore & boto3 + fix to Urllib3Using.py

  • This PR was also created before the start of GSoC.

  • Standalone test Boto3Using.py was created using moto to mock AWS calls which did not turn out well.

  • Changed Urllib3Using.py with the addition of python version checks as a fix to Issue #373.

• Urllib3 Wheel with Nuitka Pytest Results and Python-Dateutil Wheel with Nuitka Pytest Results

  • At the start of GSoC, I performed manual pytest comparison for PyPI packages urllib3 and dateutil.

  • The findings of my testing were documented in these postings.

  • Manual testing compares the pytest results of an installed nuitka wheel built using python setup.py bdist_nuitka to the regular pytest results of each package.

  • Testing is done to ensure that nuitka is building the wheel correctly.

  • If the pytests pass/fail in the same way, that means Nuitka built the wheel properly.

  • Else if the tests differ, then something is wrong.

  • Virtualenv is used to create a clean environment with no outside pollution.

  • Over the course of performing manual testing, I became familiar with the use of virtualenv, wheel, and pytest.

  • A bug was found with the package urllib3 bdist and I created Issue #413 to document the bug.

• #440 Automating PyPI Wheel Pytest

  • After familiarizing myself with how virtualenv, wheel, and pytest work, I started to work on a script which would automate the pytest comparison for top PyPI packages.

  • The script first uses git to update each package if it is already existing in the local cache, else it will git clone that package into the local cache.

  • The script then uses calls to os.system to automate the creation of a virtualenv which is then used to install pytest and pip install the package's requirements (if any) for running pytest.

  • The script then handles each package depending on different needs before building a regular wheel with python setup.py bdist_wheel.

  • This wheel is then installed into the virtualenv, after which subprocess.Popen is used to run and capture the output of python -m pytest --disable-warnings into a string.

  • The script then resets the package to its original state and builds a nuitka-compiled wheel using python setup.py bdist_nuitka.

  • This compiled wheel is then installed into the virtualenv, after which subprocess.Popen is used to run and capture the output of python -m pytest --disable-warnings into another string.

  • The two strings containing pytest outputs are then compared to find differences.

  • If no differences are found, this means bdist_nuitka worked properly. Else Nuitka compilation did something wrong.

  • The above process is repeated for each suitable PyPI package from the PyPI top 50. (Some packages are left out if they do not contain a test suite or if they do not need to be tested)

  • At the end, a colored summary is given for all the packages tested.

  • This automation script is meant to be run regularly to inform developers of Nuitka regressions.

• Issue #477 Unable to compile modules listed under unworthy_namespaces

  • Raised due to package pycparser failing in the automated test suite.

  • This issue will be addressed in the future.

• Issue #479 bdist_nuitka fails for packages containing py_modules only

  • While I worked on #440, I found a bug with bdist_nuitka failing on PyPI packages containing py_modules only.

  • This bug occurs due to Nuitka making the assumption that a main package always exists for all packages. However, some packages contain only a main module and not a main package.

  • Applies to PyPI packages decorator, ipaddress, and pyparsing.

• #483 Add support for py_modules_only compilation

  • This pull request changes bdist_nuitka.py and various other files to fix Issue #479.

  • Checks are added for the bdist_nuitka command to see if a main package exists. If there is not a main package, it will set its compile target to the main module instead.

  • This also addressed the case of a package with both a main package and a main module, in which case both are included inside the resulting wheel.

  • In addition, distutils examples py_modules_only and package_and_module were created and added for future testing.

  • During this PR, I found an import bug in Nuitka and hotfixed it with #487 Fixup_import_module.

• #484 PyPI Standalone Tests

  • This pull request adds more standalone tests for each top PyPI package.

• #495 Improve pypi automation

  • Improves the PyPI test suite created in #440 with functional improvements, readability improvements, and added documentation.

Things I Learned

Before GSoC, I was very uncomfortable with working inside a terminal. I was unfamiliar with many basic bash commands because I simply did not have any prior professional industrial experiences. I was also very unfamiliar with the Git flow, which is evident in the messy commit histories of my earliest pull requests.

As I continued throughout my GSoC journey, however, I became much more comfortable with working inside the terminal as well as using git as a version-control system (shoutout to my mentor Kay Hayen for helping me through all the annoying conflicts).

Although I am still no expert, I can confidently say that I am now far more proficient working with git and inside the terminal.

In addition, I became much more familiar with many of the most popular PyPI packages as well as the inner workings of python, which I believe will help me go very far in my career as a software developer.

Overall, the GSoC experience was truly astounding and I am more than thankful to my mentor Kay Hayen as well as Google for making this amazing program possible.

Yours, Tommy

Bug Fixes

• Python3.4+: Fixed issues with modules that exited with an exception, that could lead to a crash, dealing with their __spec__ value.

• Python3.4+: The __loader__ method is_package had the wrong signature.

• Python3.6+: Fix for async with being broken with uncompiled generators.

• Python3.5+: Fix for coroutines that got their awaited object closed behind their back, they were complaining with RuntimeError should they be closed themselves.

• Fix, constant values None in a bool target that could not be optimized away, lead to failure during code generation.

  if x() and None:
    ...
  

• Standalone: Added support for sha224, sha384, sha512 in crypto package.

• Windows: The icon wasn't properly attached with MinGW64 anymore, this was a regression.

• Windows: For compiler outputs, also attempt preferred locale to interpret outputs, so we have a better chance to not crash over MSVC error messages that are not UTF-8 compatible.

• macOS: Handle filename collisions for generated code too, Nuitka now treats all filesystems for all OS as case insensitive for this purpose.

• Compatibility: Added support for tolerant del in class exception handlers.

  class C:
  
    try:
      ...
    except Exception as e:
      del e
  
      # At exception handler exit, "e" is deleted if still assigned
  

  We already were compatible for functions and modules here, but due to the special nature of class variables really living in dictionaries, this was delayed. But after some other changes, it was now possible to solve this TODO.

• Standalone: Added support for Python3 variant of Pmw.

• Fix, the NumPy plugin now handles more installation types.

• Fix, the qt plugin now handles multiple library paths.

• Fix, need libm for some Anaconda variants too.

• Fix, left over bytecode from plugins could crash the plugin loader.

• Fix, pkgutil.iter_packages is now working for loaded packages.

New Features

• Python3.8: Followed some of the changes and works with beta2 as a Python 3.7, but none of the new features are implemented yet.

• Added support for Torch, Tensorflow, Gevent, Sklearn, with a new Nuitka plugin.

• Added support for "hinted" compilation, where the used modules are determined through a test run.

• Added support for including TCL on Linux too.

Optimization

• Added support for the any built-in. This handles a wide range of type shapes and constant values at compile time, while also having optimized C code.

• Generate code for some CLONG operations in preparation of eventual per expression C type selection, it then will allow to avoid objects in many instances.

• Windows: Avoid creating link libraries for MinGW64 as these have become unnecessary is the mean time.

• Packages: Do not export entry points for all included packages, only for the main package name it is importable as.

Organisational

• Added support for Visual Studio 2019 as a C compiler backend.

• Improved plugin documentation describing how to create plugins for Nuitka even better.

• The is now a mode for running the tests called all which will execute all the tests and report their errors, and only fail at the very end. This doesn't avoid wasting CPU cycles to report that e.g. all tests are broken, but it allows to know all errors before fixing some.

• Added repository for Fedora 30 for download.

• Added repository for openSUSE 15.1 for download.

• Ask people to compile hello world program in the Github issue template, because many times, they have setup problems only.

• Visual Studio Code is now the recommended IDE and has integrated configuration to make it immediately useful.

• Updated internal copy of Scons to 3.1.0 as it incorporates many of our patches.

• Changed wordings for optimization to use "lowering" as the only term to describe an optimization that simplifies.

Cleanups

• Plugins: Major refactoring of Nuitka plugin API.

• Plugins: To locate module kind, use core Nuitka code that handles more cases.

• The test suite runners are also now autoformatted and checked with PyLint.

• The Scons file is now PyLint clean too.

• Avoid build_definitions.h to be included everywhere, in that it's only used in the main program part. This makes C linter hate us much less for using a non-existent file.

Tests

• Run the tests using Travis on macOS for Python2 too.

• More standalone tests have been properly whitelisting to cover openSSL usage from local system.

• Disabled PySide2 test, it's not useful to fail and ignore it.

• Tests: Fixups for coverage testing mode.

• Tests: Temporarily disable some checks for constants code in reflected tests as it only exposes marshal not being deterministic.

Summary

This release is huge again. Main points are compatibility fixes, esp. on the coroutine side. These have become apparently very compatible now and we might eventually focus on making them better.

Again, GSoC 2019 is also showing effects, and will definitely continue to do soin the next release.

Many use cases have been improved, and on an organizational level, the adoption of Visual Studio Code seems an huge improvement to have a well configured IDE out of the box too.

In upcoming releases, more built-ins will be optimized, and hopefully the specialization of operations will hit more and more code with more of the

Intro

This post compares the pytest results of dateutil to its nuitka-built .whl counterpart.

Dateutil standalone test have already been covered. Manual testing is now done to compare the pytest results of a nuitka wheel built using python setup.py bdist_nuitka to the regular pytest of the dateutil package. Testing is done to ensure that nuitka is building the wheel correctly. If the pytests pass/fail in the same way, that means Nuitka built the wheel properly. Else if the tests differ, then something is wrong. Virtualenv is used to create a clean environment with no outside pollution.

The pytest results were very similar:

Regular pytests: ============= 1977 passed, 76 skipped, 21 xfailed in 7.99 seconds =============
Nuitka wheel pytests: ============= 1976 passed, 76 skipped, 21 xfailed in 7.89 seconds =============

Steps to Reproduce

1. Clone dateutil and nuitka into a new folder

2. Inside the dateutil folder, issue python -m pip install -r requirements-dev.txt to install its requirements.

3. Issue python -m pytest --disable-warnings, this runs the regular pytest for dateutil.

4. Change into the nuitka folder and issue python setup.py develop.

5. Change back into dateutil and issue python setup.py bdist_nuitka to build the dateutil wheel using nuitka. The newly built wheel should be found in the dist folder.

6. Use pip to uninstall the existing dateutil, then issue python -m pip install followed by the newly built .whl filename.

7. Issue python -m pytest --disable-warnings, this runs the nuitka-built wheel pytest for dateutil.

Uncompile Python

dateutil regular pytest:

============================= test session starts =============================
platform win32 -- Python 3.7.0, pytest-4.6.3, py-1.8.0, pluggy-0.12.0
rootdir: C:\Users\Tommy\pipenv-testing\dateutil-testing\dateutil, inifile: setup.cfg
plugins: hypothesis-4.24.3, cov-2.7.1
collected 2074 items

dateutil\test\test_easter.py ........................................... [  2%]
........................................................................ [  5%]
................................................                         [  7%]
dateutil\test\test_import_star.py .                                      [  7%]
dateutil\test\test_imports.py .......................                    [  9%]
dateutil\test\test_internals.py ....                                     [  9%]
dateutil\test\test_isoparser.py ........................................ [ 11%]
........................................................................ [ 14%]
........................................................................ [ 18%]
........................................................................ [ 21%]
........................................................................ [ 25%]
.......x...x............................................................ [ 28%]
........................................................................ [ 31%]
........................................................................ [ 35%]
.....................xx                                                  [ 36%]
dateutil\test\test_parser.py ........................................... [ 38%]
........................................................................ [ 42%]
........................................................................ [ 45%]
.................................xxxxxxxxxxxxxsss......                  [ 48%]
dateutil\test\test_relativedelta.py .................................... [ 49%]
.............................................                            [ 52%]
dateutil\test\test_rrule.py ............................................ [ 54%]
........................................................................ [ 57%]
........................................................................ [ 61%]
........................................................................ [ 64%]
........................................................................ [ 68%]
........................................................................ [ 71%]
........................................................................ [ 75%]
................................................................x....... [ 78%]
..............                                                           [ 79%]
dateutil\test\test_tz.py ............................s...........sssssss [ 81%]
sssssssssssssssssssssssssssssssssssssssss..s............................ [ 84%]
xxx..s......................................s........................... [ 88%]
s....................................................................... [ 91%]
..........s....................................s.s....................ss [ 95%]
sssssssssssss....s..........s........................................... [ 98%]
.............                                                            [ 99%]
dateutil\test\test_utils.py .......                                      [ 99%]
dateutil\test\property\test_isoparse_prop.py .                           [ 99%]
dateutil\test\property\test_parser_prop.py ..                            [ 99%]
docs\exercises\solutions\mlk_day_rrule_solution.py .                     [100%]

============= 1977 passed, 76 skipped, 21 xfailed in 7.99 seconds =============

Compiled with Nuitka

nuitka wheel pytest:

============================= test session starts =============================
platform win32 -- Python 3.7.0, pytest-4.6.3, py-1.8.0, pluggy-0.12.0
rootdir: C:\Users\Tommy\pipenv-testing\dateutil-testing\dateutil, inifile: setup.cfg
plugins: hypothesis-4.24.3, cov-2.7.1
collected 2073 items

test\test_easter.py .................................................... [  2%]
........................................................................ [  5%]
.......................................                                  [  7%]
test\test_import_star.py .                                               [  7%]
test\test_imports.py .......................                             [  9%]
test\test_internals.py ....                                              [  9%]
test\test_isoparser.py ................................................. [ 11%]
........................................................................ [ 15%]
........................................................................ [ 18%]
........................................................................ [ 21%]
......................................................................x. [ 25%]
..x..................................................................... [ 28%]
........................................................................ [ 32%]
........................................................................ [ 35%]
............xx                                                           [ 36%]
test\test_parser.py .................................................... [ 39%]
........................................................................ [ 42%]
........................................................................ [ 46%]
........................xxxxxxxxxxxxxsss......                           [ 48%]
test\test_relativedelta.py ............................................. [ 50%]
....................................                                     [ 52%]
test\test_rrule.py ..................................................... [ 54%]
........................................................................ [ 58%]
........................................................................ [ 61%]
........................................................................ [ 65%]
........................................................................ [ 68%]
........................................................................ [ 72%]
........................................................................ [ 75%]
.......................................................x................ [ 79%]
.....                                                                    [ 79%]
test\test_tz.py ............................s...........ssssssssssssssss [ 81%]
ssssssssssssssssssssssssssssssss..s............................xxx..s... [ 85%]
...................................s...........................s........ [ 88%]
........................................................................ [ 92%]
.s....................................s.s....................sssssssssss [ 95%]
ssss....s..........s.................................................... [ 99%]
....                                                                     [ 99%]
test\test_utils.py .......                                               [ 99%]
test\property\test_isoparse_prop.py .                                    [ 99%]
test\property\test_parser_prop.py ..                                     [100%]

============= 1976 passed, 76 skipped, 21 xfailed in 7.89 seconds =============

Intro

This post compares the pytest results of urllib3 to its nuitka-built .whl counterpart.

Urllib3 standalone test have already been covered. Manual testing is now done to compare the pytest results of a nuitka wheel built using python setup.py bdist_nuitka to the regular pytest of the urllib3 package. Testing is done to ensure that nuitka is building the wheel correctly. If the pytests pass/fail in the same way, that means Nuitka built the wheel properly. Else if the tests differ, then something is wrong. Virtualenv is used to create a clean environment with no outside pollution.

At first, the urllib3 nuitka-wheel pytest was crashing because of the unsafe assumption that imports will always exist (which is not the case if exceptions are thrown). Issue 413 was filed to record and fix this bug.

After the fixes, the pytests were ran again and the results were very similar:

Regular pytests: ====== 3 failed, 836 passed, 456 skipped, 113 warnings in 47.54 seconds =======
Nuitka wheel pytests: ====== 1 failed, 838 passed, 456 skipped, 113 warnings in 47.59 seconds =======

The extra passes are suspicious and require more investigation into why they happen. To make that easy, we are going to fully automate the process and compare outputs with verbose pytest modes.

Steps to Reproduce

1. Clone urllib3 and nuitka into a new folder

2. Inside the urllib3 folder, issue python -m pip install -r dev-requirements.txt to install its requirements.

3. Issue python -m pytest --disable-warnings, this runs the regular pytest for urllib3.

4. Change into the nuitka folder and issue python setup.py develop.

5. Change back into urllib3 and issue python setup.py bdist_nuitka to build the urllib3 wheel using nuitka. The newly built wheel should be found in the dist folder.

6. Use pip to uninstall the existing urllib3, then issue python -m pip install followed by the newly built .whl filename.

7. Issue python -m pytest --disable-warnings, this runs the nuitka-built wheel pytest for urllib3.

Uncompile Python

urllib3 regular pytest:

$ python -m pytest --disable-warnings
============================= test session starts =============================
platform win32 -- Python 3.7.0, pytest-4.0.0, py-1.8.0, pluggy-0.11.0
rootdir: C:\Users\Tommy\pipenv-testing\urllib3-testing\urllib3, inifile: setup.cfg
plugins: timeout-1.3.1
collected 1295 items

test\test_collections.py ....................................s           [  2%]
test\test_compatibility.py ...                                           [  3%]
test\test_connection.py .....                                            [  3%]
test\test_connectionpool.py ............................................ [  6%]
...........................                                              [  8%]
test\test_exceptions.py .............                                    [  9%]
test\test_fields.py ...............                                      [ 11%]
test\test_filepost.py ...........                                        [ 11%]
test\test_no_ssl.py ..                                                   [ 12%]
test\test_poolmanager.py .........................                       [ 14%]
test\test_proxymanager.py ...                                            [ 14%]
test\test_queue_monkeypatch.py .                                         [ 14%]
test\test_response.py ..................sss............................. [ 18%]
...........                                                              [ 19%]
test\test_retry.py ..............................F.F.F..                 [ 21%]
test\test_ssl.py ...............................                         [ 24%]
test\test_util.py ...................................................... [ 28%]
........................................................................ [ 34%]
....................ss.s...s............................................ [ 39%]
............                                                             [ 40%]
test\test_wait.py ...ssssss                                              [ 41%]
test\contrib\test_pyopenssl.py sssssssssssssssssssssssssssssssssssssssss [ 44%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 49%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 55%]
ssssssssssssssssssss                                                     [ 57%]
test\contrib\test_pyopenssl_dependencies.py ss                           [ 57%]
test\contrib\test_securetransport.py sssssssssssssssssssssssssssssssssss [ 59%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 65%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 71%]
sssssssssssssssssss                                                      [ 72%]
test\contrib\test_socks.py ..................                            [ 73%]
test\with_dummyserver\test_chunked_transfer.py ........                  [ 74%]
test\with_dummyserver\test_connectionpool.py ........................... [ 76%]
.......................................                                  [ 79%]
test\with_dummyserver\test_https.py .....................s....s......... [ 82%]
.................................................Uncaught exception, closing connection.
........................................................................ [ 87%]
................................sssssssssssssssssssssssssssssssssss....  [ 93%]
test\with_dummyserver\test_no_ssl.py ..                                  [ 93%]
test\with_dummyserver\test_poolmanager.py ...............                [ 94%]
test\with_dummyserver\test_proxy_poolmanager.py ................         [ 95%]
test\with_dummyserver\test_socketlevel.py .............................. [ 98%]
......................                                                   [100%]
====== 3 failed, 836 passed, 456 skipped, 113 warnings in 47.54 seconds =======

Compiled with Nuitka

nuitka wheel pytest:

$ python -m pytest --disable-warnings
============================= test session starts =============================
platform win32 -- Python 3.7.0, pytest-4.0.0, py-1.8.0, pluggy-0.11.0
rootdir: C:\Users\Tommy\pipenv-testing\urllib3-testing\urllib3, inifile: setup.cfg
plugins: timeout-1.3.1
collected 1295 items

test\test_collections.py ....................................s           [  2%]
test\test_compatibility.py ...                                           [  3%]
test\test_connection.py .....                                            [  3%]
test\test_connectionpool.py ............................................ [  6%]
...........................                                              [  8%]
test\test_exceptions.py .............                                    [  9%]
test\test_fields.py ...............                                      [ 11%]
test\test_filepost.py ...........                                        [ 11%]
test\test_no_ssl.py .F                                                   [ 12%]
test\test_poolmanager.py .........................                       [ 14%]
test\test_proxymanager.py ...                                            [ 14%]
test\test_queue_monkeypatch.py .                                         [ 14%]
test\test_response.py ..................sss............................. [ 18%]
...........                                                              [ 19%]
test\test_retry.py .....................................                 [ 21%]
test\test_ssl.py ...............................                         [ 24%]
test\test_util.py ...................................................... [ 28%]
........................................................................ [ 34%]
....................ss.s...s............................................ [ 39%]
............                                                             [ 40%]
test\test_wait.py ...ssssss                                              [ 41%]
test\contrib\test_pyopenssl.py sssssssssssssssssssssssssssssssssssssssss [ 44%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 49%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 55%]
ssssssssssssssssssss                                                     [ 57%]
test\contrib\test_pyopenssl_dependencies.py ss                           [ 57%]
test\contrib\test_securetransport.py sssssssssssssssssssssssssssssssssss [ 59%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 65%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 71%]
sssssssssssssssssss                                                      [ 72%]
test\contrib\test_socks.py ..................                            [ 73%]
test\with_dummyserver\test_chunked_transfer.py ........                  [ 74%]
test\with_dummyserver\test_connectionpool.py ........................... [ 76%]
.......................................                                  [ 79%]
test\with_dummyserver\test_https.py .....................s....s......... [ 82%]
........................................................................ [ 87%]
................................sssssssssssssssssssssssssssssssssss....  [ 93%]
test\with_dummyserver\test_no_ssl.py ..                                  [ 93%]
test\with_dummyserver\test_poolmanager.py ...............                [ 94%]
test\with_dummyserver\test_proxy_poolmanager.py ................         [ 95%]
test\with_dummyserver\test_socketlevel.py .............................. [ 98%]
......................                                                   [100%]
====== 1 failed, 838 passed, 456 skipped, 113 warnings in 47.59 seconds =======

Bug Fixes

• When linking very large programs or packages, with gcc compiler, Scons can produce commands that are too large for the OS. This happens sooner on the Windows OS, but also on Linux. We now have a workaround that avoids long command lines by using @sources.tmp syntax.

• Standalone: Remove temporary module after its use, instead of keeping it in sys.modules where e.g. Quart code tripped over its __file__ value that is illegal on Windows.

• Fixed non-usage of our enhanced detection of gcc version for compilers if given as a full path.

• Fixed non-detection of gnu-cc as a form of gcc compiler.

• Python3.4: The __spec__ value corrections for compiled modules was not taking into account that there was a __spec__ value, which can happen if something is wrapping imported modules.

• Standalone: Added implicit dependencies for passlib.

• Windows: Added workaround for OS command line length limit in compilation with MinGW64.

• Python2: Revive the enum plugin, there are backports of the buggy code it tries to patch up.

• Windows: Fixup handling of SxS with non zero language id, these occur e.g. in Anaconda.

• Plugins: Handle multiple PyQt plugin paths, e.g. on openSUSE this is done, also enhanced finding that path with Anaconda on Windows.

• Plugins: For multiprocessing on Windows, allow the .exe suffix to not be present, which can happen when ran from command line.

• Windows: Better version checks for DLLs on Python3, the ctypes helper code needs more definitions to work properly.

• Standalone: Added support for both pycryptodome and pycryptodomex.

• Fix, the chr built-in was not giving fully compatible error on non number input.

• Fix, the id built-in doesn't raise an exception, but said otherwise.

• Python3: Proper C identifiers for names that fit into latin-1, but are not ascii encodings.

New Features

• Windows: Catch most common user error of using compiler from one architecture against Python from another. We now check those and compare it, and if they do not match, inform the user directly. Previously the compilation could fail, or the linking, with cryptic errors.

• Distutils: Using setuptools and its runners works now too, not merely only pure distutils.

• Distutils: Added more ways to pass Nuitka specific options via distutils.

• Python3.8: Initial compatibility changes to get basic tests to work.

Organisational

• Nuitka is participating in the GSoC 2019 with 2 students, Batakrishna and Tommy.

• Point people creating PRs to using the pre-commit hook in the template. Due to making the style issues automatic, we can hope to encounter less noise and resulting merge problems.

• Many improvements to the pre-commit hook were done, hopefully completing its development.

• Updated to latest pylint, black, and isort versions, also added codespell to check for typos in the source code, but that is not automated yet.

• Added description of how to use experimental flags for your PRs.

• Removed mirroring from Bitbucket and Gitlab, as we increasingly use the Github organisation features.

• Added support for Ubuntu Disco, removed support for Ubuntu Artful packages.

Optimization

• Windows: Attach data blobs as Windows resource files directly for programs and avoid using C data files for modules or MinGW64, which can be slow.

• Specialization of helper codes for + is being done for more types and more thoroughly and fully automatic with Jinja2 templating code. This does replace previously manual code.

• Added specialization of helper codes for * operation which is entirely new.

• Added specialization of helper codes for - operation which is entirely new.

• Dedicated nodes for specialized operations now allow to save memory and all use type shape based analysis to predict result types and exception control flow.

• Better code generation for boolean type values, removing error checks when possible.

• Better static analysis for even more type operations.

Cleanups

• Fixed many kinds of typos in the code base with codespell.

• Apply automatic formatting to more test runner code, these were previously not done.

• Avoid using shutil.copytree which fails to work when directory already exists, instead provide nuitka.util.FileOperations.copyTree and use that exclusively.

Tests

• Added new mode of operation to test runners, only that executes just one test and stops, useful during development.

• Added new mechanism for standalone tests to expression modules that need to be importable, or else to skip the test by a special comment in the file, instead of by coded checks in the test runner.

• Added also for more complex cases, another form of special comment, that can be any expression, that decides if the test makes sense.

• Cover also setuptools in our distutils tests and made the execution more robust against variable behavior of distutils and setuptools.

• Added standalone test for Urllib3.

• Added standalone test for rsa.

• Added standalone test for Pmw.

• Added standalone test for passlib.

Summary

Again this release is a sign of increasing adoption of Nuitka. The GSoC 2019 is also showing effects, definitely will in the next release.

This release has a lot of new optimization, called specialization, but for it to really used, in many instances, we need to get away from working on C types for variables only, and get to them beig used for expressions more often. Otherwise much of the new special code is not used for most code.

The focus of this release has been again to open up development further and to incorporate findings from users. The number of fixes or new use cases working is astounding.

In upcoming releases, new built-ins will be optimized, and specialization of operations will hit more and more code now that the infrastructure for it is in place.

Short Biography

• Born 1951 as Harald Lieder near the city of Leipzig

• Studied Mathematics and Physics at the Frankfurt/M Johann-Wolfgang-Goethe University from 10/1970 through 06/1976 (Diploma in Mathematics)

• Professional career overview

  • insurance company (4 years): application and systems programming (IBM mainframe, Cobol, Assembler), data center lead

  • book wholesaler (5 years): systems programming and data center lead (IBM mainframe)

  • consulting company (until 2009): leading of or participating in numerous large and very large projects mostly around banking applications, banking mergers, data center optimization and quality assurance.

  • since 2009: retirement, living on Isla Margarita, Venezuela. Enjoying my beautiful house, pursuing my hobbies: mathematics, science theory, biological evolution theory, science fiction (my pseudonym comes from novels by one of my favorite authors Frank Herbert) and of course programming (maintainer of PyMuPDF, and Nuitka).

Read more…