Montreal Python 6: 2009-04-14; Machine Learning empowered by Python

It is now official, I will do the next Montreal-Python presentation. I will be back from holiday the 13, I hope I wont get flight problems.

Our main presenter will be Francis Piéraut on Machine Learning empowered by Python as announced during the flash introduction in Montreal-Python 5.

Machine Learning is a subfield of AI that considers learning patterns from existing data. Related applications are increasing in many fields where adaptive systems are needed, like fraud detection, face recognition, recommendation systems, disambiguation systems, insurance risk estimation, web traffic filtering, voice recognition, and many others.

The first part of this presentation will cover the basics of machine learning; in the second part, we will dive into a real example and see the complete process of using machine learning to create a real-time digit recognition system using Mlboost, a python library. The practical approach should allow the audience to assimilate the most important concepts of machine learning and the critical need for data preprocessing.

After a Software Engineer degree, Francis Piéraut made a research master in Machine Learning at LISA. During his research work, he developed flayers, a powerful C++ neural network library. During the beginning of his career, his spend several years in Montreal startups companies applying Machine Learning and statistical AI related solutions. In 2005, he released the first version of MLboost, a python library that allows him to speedup his Machine Learning projects by simplifying data preprocessing, features selection and data visualization.

Essay on Adaptation, leaky cost function and online Learning...a society analogy

I fall on the Paul Graham article on cities and ambition and it made my think of writing this blog on adaptation, leaky cost function, online learning and an analogy to the sinking French Quebec society.

To make the bridge with the 3 first concepts, I will use a analogy with the Quebec society.

In order the learn, we need adaptive systems as Neural Networks. In online learning, the adaptation capacity should stay constant along the time. Local minimums can screw you up but let's ignore it for the time been.

Quebec society analogy intro

In order to understand my analogy with the society I live in, I want to share some of my reflexions about the puzzle to understand the Quebec Society. I am born in France and I migrated to Quebec at 9 years olds. During the last 5 years, I tried to elucidate my profound incomprehension of the deep ambitions of the French Quebec Society, if they have some;). According to me, it seems to be a leaky cost function assumption that lead them to their stagnation, coming assimilation and their slow extinction.

To understand my point of view, we need to elaborate on key concepts which are adaptation, equality versus inequality and education access.

Adaptation pros and cons, local minimal and ambition

Adaptation is one of the greatest ability of the human kind but also one of the worst. Adaptation to mediocrity can be a survival strategy to get through hard time but getting use to it reflects true low ambitions or incapacity to do online learning. The Quebec nation seems to have this disease.

• Quebec people accept staying in a destructive mode seen 1982, a constitutional status quo that lead to politic instability, economic stagnation and reduced political power by excluding them self from the Canadian power with the Bloc Quebecois for too long.
• Quebec people accept the status of a sub nation (nation inside the Canada).
• Quebec people accept mediocre governments, mediocre public transport systems, way too expensive and inefficient heath system, highest taxes in north america etc.

Quebec people seems to have no real will for improvement, it is doing the job (you should see "Le confort et l'indifférence"). In the machine learning point of view, is that a local minimal? What is the problem with Quebec society cost function? Is that only the lack of ambition? Knowing that close to half of the population have voted for the separation of Canada in the 95 referendum, it might shadow something more rooted to the French culture.

Cost function assumption : Equality versus inequality

From an anthropology point of view, French nuclear family lead to a conception of a world of equality (see Emmanuel Todd). to simplify, everyone should have the same chance, same access to education, same heath services and so on.
The Anglos-Saxon culture lead to the conception of an inequality world. The inequality conception lead people to work harder knowing there is no lower boundary and they can go deeper if they are too lazy.
Knowing we are born unequal, Anglos-axon assumption conception seems to be better adapted to human kind reality. On the other hand equality lead to an education level increase of the society indepedandly of the economy which as lot of pros and cons.
Why equality is a weak assumption? Equality can stand in rich societies because they can afford it. Unfortunately, Quebec society is getting poorer and its population is disadvantaged by its illusion of equality that leaks from everywhere (i.e.: health system, education, public kid garden etc.)

Missing link: Education and production of wealth

Quebec has the most affordable access to education in north America and few take advantage of it. Anyone knows that the more educated is your society, the more productive and healthy she will be and the more accessible the utopia of an equality world can be possible. By using education to get more productive, a society will create wealth and can afford utopia as the equality concept. French society seems to miss this key point.

French Quebec society is dying and the Anglo-Saxon supremacy should take over

The leaky equality concept and the low ambition of the Quebec society seems to lead this society to online learning incapacity, its incapacity to adapt further. This incapacity leads to its extermination by the growing assimilation to the Anglo-Saxon supremacy of its global cost function model. Inequality based cost function seems to be better adapted for a society that want to stay alive.

Wrap up (it is time to conclude)
Leaky cost function assumption can lead online learning to adaptation incapacity like been stuck in a local minimal as a slow death as the French Quebec nation folklorisation current process. It is simply the evolution, a Darwin consequence, who can't adapt simply die. Facing reality makes life easier.
The most important thing is that the cost function should reflect your goals. If you have a supervisor, try to get a good estimation of its cost function because it will simplify your ascention everywhere.

Quebec French culture creates a huge retention for me to stay in montreal but I wish Montreal a better drive for machine learning, startups as you can get in California. Montreal is simply sub-exploited. Don't take my words for granted, it is an essay. Make you own judgement from your own eyes and exploration.

Leaky assumption and Gradient Descent

Theoretical models are based on strong assumptions as software layers (i.e.: leaky abstractions). Layers are created to simplify complex systems and allows work specialization as define in Marx capital approach...and to accomodate limited human brain capacity.
Experienced practitioners know that their value reside in weak assumptions dept comprehension because market will hire new grade students otherwise.

Standard backpropagation gradient descent algorithm assumes that inputs are independent so we can optimize them independently of each other. This assumption or according to me a leaky abstraction, allows you to optimize all parameters at the same time which simplify the life of software engineers and researchers because parameters are theoritically uncorrelated. In mathematical works, we are assuming that the hessien matrix has values only on its diagonal.

My master thesis done under Yoshua Bengio supervision was mostly focusing on understanding huge neural networks training inefficiency. At that time, our goal was to train neural network language models. According to my undertanding and the experimental proof that I have documented, the problem is basically an optimization problem. The uncorrelated assumption simplification doesn't stand when parameters numbers explode.
Unfortunatly, I have failed to find a solution to this problem but the new trend in reaction to Hinton break throught in 2006 will and is already reviving research in this topic.

In my literatude review, I found that several researchers identified some of the reasons who can explain this inefficiency. According to me, they are direct and indirect consequences of the optimization problem introduced by the leaky abstraction of uncorrelated inputs. Those reasons are the moving target problem and the attenuation and dilution of the error signal as it is propagates backward through the layers of the network. We present in my master thesis other reasons who can explain this behavior, the opposite gradients problems, the non-existence of a specialization mechanism and the symmetry problem.

I will treat those concepts in a futur post. This inspiration of this post has been possible because of a brainless Hollywood movie that has allow me to free valuable brain cycles. there is always a good side of the story.

Newton law doesn't stand in Einstein theory as uncorrelated inputs in huge neural networks. Always remember your leaky assumptions/abstractions.

cygwin or mingw to compile C++ swig projects on windows?

The answer is definitely mingw. 4 simple steps:

1. download swig
2. download mingw
3. add python, swig, mingw to environment variable PATH
   
4. do: python setup.py build_ext --inplace --compiler=mingw32

I have been able to compile it easily with cygwin but I had to download shit load of stuff and I haven't figure out how to use python windows packages and cygwin python package all togetter which was a show stopper for me.

Why was I looking to compile my C++ project on windows?

I want to use my C++ machine learning lib on a real-time video and the only package that was working to grab images in python was videocapture but it is only supported on windows. Now I have to compile my machine learning on windows...

If you want to compile swig C++ project, you need a compiler. If you don't have VisualC++, you are stuck with this error:
error: Python was built with Visual Studio 2003;
extensions must be built with a compiler than can generate compatible binaries.
Visual Studio 2003 was not found on this system. If you have Cygwin installed, you can try compiling with MingW32, by passing" -c mingw32" to setup.py.

At this point, you can try to download free visual C++ compiler or try mingw32 with sigwin. I tried a free visual studio 2003 package but I couldn't make it working. Then, I try cygwin with gcc and it was compiling but I couldn't use the compiled package outside cygwin. so I tried mingw32. The -c option doesn't work and "python setup.py build --compiler=mingw32" doesn't allow you to use your package inside python (i.e.: can't import _flayers in my context). Finally I tried python setup.py build_ext --inplace --compiler=mingw32 and it was working.

After a chat with Simon and Tristan that are doing video stuff on linux at the SAT, I discovered that they were grabbing video on Linux. They referred me to opencv which works perfectly. In my experimentation with opencv, I realized that Pygame is a million time faster to display video then matplotlib.

Swig, Mingw, OpenCV make Python so convenient!

Startups and faster Learning

Every one that is familiar to gradient descent and Machine Learning knowns that learning is proportional to the error. Bigger are the errors, bigger are the possibility to learn something. According to me, startups, by nature, are the ideal environments to put yourself in hight gradient learning situations. It force you to try much more things, take more risk and innovate to survive if you don't want to have no choice to look for another job and/or get penny stocks options. Another interesting thing is that true nature of people appear and mask fall rapidly. Friendships are tested to its limits and it allows you to filter out short term life partner.

Don't forget that high gradient put high constraints to your body and frustrations has to be released somehow. In my case, cycling has been used as stress balancing vehicle, you have to find yours. Resigning is too often chosen as the latest resort. You have to keep in mind that everyone is replaceable, even in startups. If you really want to be rich, you should do the Californian technique: cumulate stocks of multiple startups. don't forget that most of them (~90-95%) fail badly, so increase your chances by switching before reaching burnout and build your contacts network by the same way. After the graduation is definitely the best time to try this fast growing experience when you can take it. On my side, I tried it 5-6 years, you have to know your limits. Successful startups have workaholic CEO so it will increase the pressure and the gradient. If he isn't one, find another one, it should be the 19 mistakes of startups (18 mistakes of startups).

If you have the chance to access higher decision position, you learn that there are irrational error cost function and that you might be optimizing the wrong error criteria. For the ones who aren't following me, I recommend that you see the French movie 99 Francs and pay further attention the the high executive meeting.

I apply the same principle with my son this winter, he doesn't want to put his gloves, he learn/change his mind faster at -10 degrees Celsius.

MLboost 0.3 has been released

I am please to announce the release of MLboost 0.3. MLboost 0.3 will be use for the next Montreal-Python presentation ("Machine Learning enpowered by Python"). It will be announced in the coming weeks. The most important new feature is the integration of pyflayers. pyflayers is a simplified python swig interface to flayers, my C++ neural networks library. I am preparing a live real-time demo of a machine learning application, it should be interesting for the audience and it is good motivation to improve my package. I have played with beamer, a latex package to create presentation slides and I have been particularly been pleased, I do recommend it. On the other hand, it is time to prepare the next winter camping trip and to continue eating great food.

The No Asshole Rule Fraka6 review

During Christmas time off, I finished "The No Asshole Book" from Robert I. Sutton. I fall on this book by the time is was looking for the "Art Of War" for a gift. At first, I taught it was a mistake and that it will end up been a light or low content reading but it was extremely interesting. The book treats:

• How to identify a real asshole
• The damage of keeping Assholes
  
• How to implement the No Asshole Rule
• How to stop inner jerk from getting out (yes it get out sometimes)
  
• How to survive when assholes reign
• Yes I am a powerful Asshole (I was checking if you were really reading;)
  

Having spend 5 years in startups environment, I should have read this book before because high stress environment leads people to act as asshole and applying some of this ideas can help you surviving this jungle. Out of 4 startups, I had to deal with 3 places where asshole behavior was destroying my productivity. Only one place had a certified asshole (not a temporary one; no it isn't Eric). What I like about high educated environment is that people are more likely to stand up and make changes happening. I saw one of my supervisor been fired after I stand up and another been fired on the side after I left which is overall very encouraging. What I learned from this book is that once, I have waited way too much time before changing jobs because I didn't realized soon enough that I was simply wasting my time. There is times where fighting is useless and it simply drains you down and lead you to act sometimes as an asshole yourself to keep you breathing (don't forget small wins first). Good faith is sometime useless, as Robert enforce: Hope for the Best; Expect the Worst and Change the way you seen things (Reframing). Reframing refer to developing indifference and emotional detachment. As highlighted, people tend to recruit high passioned people but it isn't a so good idea because it leads them to irrational decisions that can have critical impacts.

The book doesn't treat one point that high technical people forget. For most company, market drives the show, not the technology, so you have less power then you think. This market lead to more politics and assholes are good at this game. Politics is often use to shadow increasing incompetency because they don't focus on what they should be doing. You have to better understand politics games but according to me, the more politics the more incompetency by square foot is expected.

This make me remember Joel On Sotware book, most years top companies don't last, only a few survive in the top n after many years. I am pretty sure those company apply the No Asshole Rule because if they don't do so, long term they get back what they desire because they are infected by reproduced assholes that blame each other by the time all top people leave.

People forget that power pass by empowering others. This power isn't a sum but can't be exponential with teams that have a great synergy. Open source is a great example.

This book is highly recommended. I will suggest it to my manager, one of the few manager that understands what a manager should be doing.

But what is an Asshole, the standard definition is something like that: insulting terms of address for people who are stupid or irritating or ridiculous. According to the book Asshole follow 2 rules:

1. After spending time with them you feel de-energize, humiliated etc.
2. Asshole are more polite with more powerful people and are worst with lower level people.
   

Don't forget, be slow to brand certified asshole. To get the certified brand, you simply have to be constant.