Is Python really slow? A practical comparison with C++

The common perception is that Python's implementation is slow, but you can often write fast Python if you know how to profile your code effectively.

I have tried it. I have compared a hight cpu intensive algorithm, the training of a simple one hidden neural network. To do so, I have used my old C++ NeuralNetwork library (flayers) and an implementation in python with Numpy. I have wrote a simple neural net in python and optimize all loops with numpy as suggested in a profiling presentation saw in Pycon2009.

I have compare the training time of a simple fully connected NeuralNetwork will 100 hidden neurones for 10 iteration on letters dataset (cost function = mean square error).

Here is the time to do 10 iteration with flayers (c++):

./fexp / -h 100 -l 0.01 --oh -e 10

...
Optimization: Standard
Creating Connector [16|100] [inputs | hiddens]
Creating Connector [100|26] [hiddens | outputs]
...

real 0m11.187s
user 0m10.837s
sys 0m0.012s

Here is the time to do 10 iteration on the full letters dataset with python:

time ./bpnn.py -e 10 --h 100 -f letters.dat -n

Creation of an NN <16:100:26>

...
real 85m48.646s
user 85m9.163s
sys 0m1.632s

Here is the time to do 10 iteration on the full letters dataset with python and numpy:time

./bpnn.py -e 10 --h 100 -f letters.dat

Creation of an NN <16:100:26>

...

real 1m37.066s

user 1m36.026s

sys 0m0.100s

So if you do the math:

• The numpy implementation is 60 time faster then a basic python implementation.
• My C++ implementation is a little more then 10 time faster then my simply python numpy implementation.
  

Numpy implementation definitly worth it because it reduce the code and has a significant performance impact, the C++ might be required for extreme performance but the trade off of code complexity and time my not work it. Now that I have the choice, I will still use my C++ lib.

Short Essay: Engineering vs Scientist

Engineer vs scientist difference is obscure for many people, me included at the beginning.
Already, during a chat with a research professor in 2001 about the competence war between the engineering and the science department of university of Montreal, I got an initial hint about it, he told me that part of computer science department tension with software engineers was about placement rate. Engineers was much higher than computer scientists. I should have request an explanation. What is this war about, the computer science department request engineer to do some normalization courses even if they have great grades and vis-versa.

In my short carrier, I have observe a major difference that I will try to expose with examples.

After my degree in engineering, I have done most of my graduates courses with scientists. In my first final exam, after some discussions with classmates, I realized that I was the only one that have used an estimation to solve a bottleneck calculus in a problem. Most of the other have lost more than half an hour on solving it exactly and couldn't finish the full exam.

Several times, I add to step in projects to set schedule in order to force people to cut some corners. Perfection is an endless path. If you have to fill a vase with rocks, sand and water, you will most likely start with the rocks then the sand and then with the water if the effort is inverse proportional to the size no? Yes it might be more interesting to optimize cool things and advances features but most likely they aren't part of the basic blocks required to allow your project to move to the next step.

What is the point of having half a perfect solution if you could have done a imperfect full solution. I am not saying that all scientists need of perfection leads them to lose the big picture and engineer aren't perfectionist but simply that bias engineer system vision leads them to gage better when it is time to look for perfection. Time kill projects and over-perfection can drag too much of it...but perfection should remain the goal.

Yes I am generalizing and simplifying but you get the big picture. Define your priorities and assign resource accordingly. So, am I a engineer or a scientist or a little of both? since my manager told my I was more a scientist, I should be a little of both...

Montreal Python6 & Peer Programming

Montreal Python 6 demo packaging and UI major improvements are the results of a peer programming with apt-get install ygingras. In this world where time is the biggest constraint, I could not finalized some of the demo improvements and packaging without help. bitbucket, the mercurial host projects has been very useful to store demo parts (i.e.: flayers, mlboost and digipy). I am not working often off-line, but been able to commit in the plane or in the bus, exchange dundles and create branches easily has very useful.
You can play with the demo, just type:

easy_install digipy (on your favorite linux distribution)

Simply type digipy and tab to see all digipy programs used in the live demo (i..e: use --help to get more details). I have been impress by the powerful setup.py related distutil tools of python to create packages (sdist, register, upload to push code on the Python Package Index is just GREAT!)
Virtual Environment, a tool to create isolated Python environments, of Ian Biking has been really usefull to test the packages.

Peer programing is the type of approach that can bring the productivity of your team from the sum of the productivity to the multiplication of its individual or a best to the team member count exponential relationship. Peer programming has allow me to get out fast of many extreme time consuming techical show stoppers that were burning out my too little available time.

Highlights Pycon Chicago 2009

My short trip to pycon 2009 just confirmed my choice. This community is living and has a powerful momentum. Guido, the python father, mentioned that the 3 most important things are: community, community and community. What he started 19 years ago seems to take over.
The simple python syntax lead everyone to converge to it: pypy, jpython,cpython and IronPython. Alex Martelli talk on abstraction as a leverage was great: abstraction is inevible; try to understand at least 2 lower layers and create hook instead of hacks! (I am already applying it for the demo of python-montreal).
The frameworks for the web development is getting quite amazing (Django, Whirldwind, pylons etc.)
The talk " A Whirldwind Excursion through C Extensions" of Ned Batchelder was great to get a quick start on own to create C extension; yes python is slow and optimization is sometime inevitable.
The panel on Object Relational Mapper and the talk on Drop ACID and think about data were quite interesting. The keynote presentation by reddit founders is a good example that python provides amazing tools to spine out web application companies.
The concept of the evening lightning talks and open discussions on topic of interest was showing the dynamism of the python community. I went to an open discussion on parallel computing and people are moving to python. Now that the process lib has been integrate in 2.6, Twisted, Thrift, PyMPI, Numpy, Scipy power computing with python will expend.
I am glad that Yannick convinced me to attend. I think that I will attend to the tutorial next time and some Sprint to ensure the maximum knowledge assimilation.
During this small trip, I tried the mini laptop eee ha but I brought it back, you should buy the HE: the right shift is at the right place and the battery last longer. Pycon was inspiring !

Research, information management, contrastive divergence, no free lunch theorem, non-parametric...all related

During my master studies, I had to take a course on Research Methodology which introduced me to an interesting concept of information management. With our limited brain capacity, more you read papers about others ideas, less space is available for your own ideas and enforce other assumptions, vision and models.

In practice, great researcher are aware of this consequence of the no free lunch theorem and try to keep of good balance of papers reading and research exploration. By simply applying the contrastive divergence concept of your approach, you can gage your distance to the trend and get a estimation of the impact of a possible discovery.
The Research Machine Learning community, as most other community, has the tendency to recruit top grades students that are use to follow exactly the line of thoughts of their teachers. This long training process is, according to me, extremely damaging to the training of the researcher capacity (i.e.: suboptimal cost function). This explain why most of the master student are researcher cheap labor driving force because they can only experiment others ideas with minor contributions.
Top researchers allow their formal students to follow their own line of thoughts or if they have no specific ideas, suggest ideas. I won't have done a research master without this freedom, thanks Yoshua!
So, if you want to impact the most your community, limit the number of papers your are reading, make your own ideas and play with your concepts to train your own intuitions of the unknown guiding rules you are looking for.

You might say, what's your contributions, I haven't heard about it. My contribution is that I have build experimental proof of back-propagation optimization fundamental problems and build the skeleton of top level explanations. Usually, we don't publish this type of results until you find a solution to the problem which, unfortunately, I haven't reach but, it is coming slowly; it is a long process and I learned to be patient.

So, if you want to impact the most your community, limit the number of papers your are reading to ensure you don't constrain yourself to others models. Why using a parametric model that limits your solution space?

You can trust the collective research discovery learning process that ensure the evolution of the human kind because someone will find it or, use it to increase the likelihood your will make a important discovery (i.e.: use it as a contrastive divergence cost function). If everyone was applying this strategy or cost function, I am pretty sure we will evolve faster. In order to move to this step, we will need to encourage failure strategy publications to ensure other don't wast time reproducing the same ideas but this could be elaborated in another thread post that involve a society evolution.

Traders knows about this simple strategy, buy low, sell high, don't follow the trend, take risks.

Pycon 2009 Chicago

It is about time the see the real face of this python community. I will attend to pycon 2009.
In 3 days, I expect to learn more things then I have learned in the last 6 months and met pationated peoples. Here are some of the talks I will attend:
Designing Applications with Non-Relational Databases (#16)
How Python is Developed (#116)
Twisted, AMQP and Thrift: Bridging messaging and RPC for building scalable distributed applications (#40)
Introduction to Multiprocessing in Python (#6)
The State of the Python Community: Leading the Python tribe (#118)
Google App Engine: How to survive in Google's Ecosystem (#53)
A Whirlwind Excursion through Writing a C Extension (#68)
Abstraction as Leverage (#110)
A winning combination: Plone as a CMS, your favorite Python web framework as a frontend (#100)

Greedy agile, waterfall and local minima

Everyone in the agile world can't contain their words about waterfall inadaptation to real world software projects. I have to admit, I am a fan of XP, scrum and most of agile approaches in general but I feel that people are loosing the big picture provided by the waterfall framework. Agile is kind of a greedy approach that lead you to local minimum where you are stuck too often. I would like to see someone creating value and velocity cost trade off of short term decisions. Those decisions are ofter push through agile methodology without questioning. Currently, at pivotalpayments, we are stuck in a huge local minimum created by such an approach and it will take such an effort to get out of it. The velocity created by a design choice taken long time ago was emazing at firt but it is slowing us doing now so much now. A major refactoring is required and we will stop development for some iterations. At least Agile adapts and give you the illusion of the optimal path...but everyone should know that the greedy approach isn't the optimal one and that looking at your feet won't help much going to your destination.