Python is now far more then simply glue or scripting language. For those who think otherwise just check the couple of Python success stories:

I recommend My-Favorite-Python-Things presentation if you are looking for a quick intro to the beauty of Python.

High level languages are mainstream

Nowadays high level languages allow to write simple code with greater flexibility. Creating applications faster makes people choosing dynamic languages, where you don’t need to specify types and waste time to fighting with them (all the boilerplate code about interfaces just to satisfy compilation process). Someone can argue that this behaviour produces buggy code. For those I would say, after Guido van Rossum: “who produces code without test”? Static languages can handle some errors during compile time. But they can’t detect all of them. So in the end you need to write tests, which in the same time are sufficient to test typing and easier to write using dynamic languages. Moreover people still can’t invent good (far to say perfect) type system. Jim Treavor wrote some summary about this.

New techniques and knowledge allows us to produce highly efficient runtimes for dynamic languages (JavaScript V8, LuaJIT, Racket, Common Lisp...), which successfully compete with tremendous software frameworks (JVM, .NET, ...)

All of this makes high level languages increasingly popular in enterprises and daily use.

Is Python continue the success story? Now, Python is very popular. But it’s position is threatened by a competition. Python has a great ecosystem with huge amount of software and community. But it lacks efficient and state of the art runtime, which competition already have.

Python as a glue language. As I stated at the beginning one of the features which contribute to Python success is its ability to easily connect to compiled libraries, which makes it perfect as a glue language. But this made Python popular 20 years ago. The tools, which are still popular, are old. The problem with them is you need to make quite a lot of effort to use them: ctypes

c extensions are evil. They are bound to specific version of Python and are not reusable. What is worse, c extensions API is different in CPython2 and CPython3. Now imagine porting your library to Python3.

Cython - It’s designed to easily writing c extensions and it’s good for this task. But I admint that using c extensions is a last thing you want to do. There are better ways (more elaboration below). Cython is an external tool which requires compilation process. It produces non dynamic behaviour in a final code and another syntax to learn. In the end Cython doesn’t support type inference. The problem with Cython is that when you use it you are done - you have no option than compile it. Cython is not a standard. It can’t be used as an interpreted code. Great summary was done by Kay Hayen, the _nuitka_ creator, in Static Compilation - That is the point post.

swig, boost - those are quiet easy, usually requires some changes in C/C++ code, or write some schema file. In contrast there are new tools which handle this task far more better preserving performance (even outperforming it): cffi - package which allows easily handling your C libraries. You often need to do it while touching a hardware or supporting some other software (like database clients/drivers). Just check how easy is to use it with Python. You don’t need to write any wrapper, typed code. Moreover has CPython and PyPy support.

bitey Python as a brain of your code - other aspect of a glue language There is also other aspect of a glue language. Let’s think about process of low level high performance programming. It probably looks similar to the following process: idea

lot of complicated low level and organization code. Probably in C++ with a bunch of obscure template code (for re usability)

writing glue code

compilation

running

probably a lot of debugging and return back to writing, because of the amount of low level code. Thanks Python’s portability, scripting nature and big amount of tools, it is used as a template and brain engine for your code. That means you only write the smallest amount of required low level code, and python do the rest: generating the organization code, and the context required for your low level code. This takes back to Lisp idea, where code is a data, and makes sense of code about other code that is executed (code as a data to process). So the machine can think about what is executing in runtime, then optimizing it, reasoning about full data information in true generic way without templating you known from C++. This is what C++ and other popular languages lost. At the end we have lower abstraction level but stronger runtime knowledge which allows compiler for: Specialize for unknown hardware (at code writing time), the problem with supported types, and available optimizations.

Automated tuning (e.g. your data for libraries like ATLAS...)

Allow so put other information to compiler which far more reasoning.

People don’t struggle with data types (Runtime allows to makes it fast, appropriate and... already done) So the process looks like: idea

little of Python code (the pretty one) - for organization, and low level code - also much nicer because without the rubbish template and context code. Actually low level code can be generated from python code.

running

debugging, but much less times then in previous process. In terms of performance it gives a better potential then previous approach. Examples which already use it: PyPy, cffi, PyOpenCL, PyCUDA, numba, theano...