Fraka6 Blog - No Free Lunch

Strategie Québecoise de l'entreprenariat: what's in it for startups

2012-01-02T22:11:00.000-08:00

I have finally took some time to look at the details of the announcement of the November 15 2012.

http://www.mdeie.gouv.qc.ca/fileadmin/contenu/publications/administratives/strategies/strategie_entrepreneuriat.pdf (page 14 &43)

If you look properly, its isn't very exiting...lot of noise for not much.

access to line of credit of max 30K without interest nor warranty
ability to delay payments of "Fond Local d'investisment"
Investment with Anges-Quebec which means they might take a little more risk

Lot of the new money is invested in the entrepreneur support programs and to simplify access to money to help entrepreneur buying existing corporations (the boomer generation are starting to sell but few can buy).

Conclusion, lot of new money in the system but at the end, not much for entrepreneurs. Most of it is to sustain the existing system so don't wast to much time of this and focus and launching technology and sales.

Why startups shouldn't wast time with Canadian Innovation Commercialization Program (CICP)

2012-01-02T21:21:00.000-08:00

Canadian Innovation Commercialization Program (CICP) is an attractive program because as it is mentioned in its name, its goal is to help companies start commercialisation of their innovations. The idea is interesting, if your innovation can be beneficial to the Canadian government, they might take the risk to be the first client and will pay for it. If the government is a client, your commercialisation will most likely be way easier by eliminating the egg and chicken problem of getting the first real big client and real revenues faster.

Why this program?

The government is realizing that most product fail at the commercialisation phase. SR&ED is great support to develop innovations but their is little to support commercialisation (i.e: precarn program was closed).

But seriously why this program has been put in place?

There is a rumour that it is related to the Bombardier Aerospace and Embraer government subsidy controversy. Canadian was illegally subsidizing bombardier according to the WTO anti-subsidy policy.

CICP is a legal way to subsidize companies. If you look at the pre-qualified innovations of call 001, you will find bombardier and some other mature companies who might need less commercialisation help then other ones as startups as an example:

https://buyandsell.gc.ca/initiatives-and-programs/canadian-innovation-commercialization-program-cicp/pre-qualified-innovations

Why startups shouldn't was their time with this program?

According to the selected companies, its seems that Startups shouldn't wast energy on this program and even less if they are from Québec and or have nothing to do with aerospace and military related projects and are startups. If they do so they will most likely be rejected by something like "The bidder’s company does not have the appropriate management team required to move the proposed innovation into commercial markets" which will bring you back the the chicken and egg problem. Freemium is definitely a way better alternative.

MTL Salon BI quick summary

2011-11-13T20:38:00.000-08:00

I went to MTL salon BI on November 8th.

Here is the my headlines:

Elie Elia: 45% of people are overloaded with data; need actionable information
Freddy Midi: BI = old decision support systems; reports shouldn't be the end of the process; it is a flow; The future of BI looks like www.netvibes.com;
Before graphic was the end but now it is the begining (drill down).
Philippe Nieuwbourg: Since big players have done acquisitions their is no more innovations and price have simply increases; it is a good timing for startups. Visualization is weak...we want minority report interface and it is coming
Online Analytic Maturity Model: 6 levels by Stephane Hamel : http://immeria.net/oamm/; BI and Webanalytics are merging; diff, web analytics is real-time and accept higher error margin; future is agile & predictive
For now BI mostly look at the past but the future of BI is finally moving slowly to predictive analysis
BI clients priorities: Simplicity, cost, quality
Best BI solutions: QLikView; PowerPivot, SPSS (IBM), Tableau Soft
People want to bypass IT for BI answers
Michelle Blanc: disruptive truth about internet marketing
Wayne Knaggs (Adastra): cloud BI: we are moving from SaaS to IaaS and PaaS (Soft->Infrastructure-> plateform)
Semantic web: linkedata.org; triple store (RDF) are becoming standards

IP strategy for tech startups

2011-10-15T20:10:00.000-07:00

Most VC & angels want IP in order to consider investing in your startups.

Why: seems a security illusion. If they were really thinking about it, they will ask for industrial secret most of the time. Let me explain:

First, look at some facts:

95% of patents are useless (lawyer expensive fees & precious time)
Patents are most of the time weapons of litigation which result in licences exchanges agreements (why google bought Motorola mobility)
50% of patents are invalidated in litigation (thanks David)...so you need a portfolio of patents if you really want to play in that yard
It takes on average 6 years after patent deposit before it is validated....which means your competitors can copy you easily for many years because your invention will be public ;)..
You need deep pockets to enforce your patent rights, only big player can play that game

So, if you aren't thinking of been acquired by a very big player or founded by a very deep pocket VC, you are wasting your money, time and are helping your competitors by giving them all your secret recipes and even better, you are paying for it. Isn't that the most inefficient thing you can do?

Btw, you should know that filing is making a deal with government so don't expect efficiency. Last, lawyers are relevant in ambiguity which might explain the patent process.

Speaking at confoo 2012: Empowering IT with Real Users Analytics

2011-10-06T05:46:00.000-07:00

I have been selected to speak at the next confoo web conference about "Empowering IT with Real Users Analytic"

Webapps monitoring landscape is changing. As pointed out by Garnter, IT needs simplified monitoring and Google Analytics can now measure page load times.
As more and more businesses rely on them, techniques such as real time user monitoring, incident detection and root cause analysis have become critical. While been the Holy Grail to users, webapps are a nightmare for engineers as ensuring quality of service becomes more difficult.
More and more companies are starting to realize that they are losing significant revenue, growth and productivity due to down time, high latency and scaling issues with their web applications. Quality of Service guarantees with built-in penalties are becoming the norm, bringing a downside into the equation.
We will present these new problems in detail, followed by a short history of techniques used to measure and estimate the quality of web-based applications. We will review the most popular monitoring technologies, pointing out their pros and cons.

Startupfest best presentations July 2011

2011-08-02T08:20:00.000-07:00

Here are the 5 best presentations that I saw at startupfestival:

Jeff Clavier - Founder and Managing Partner - SoftTech VC Starting by the numbers: (The 3 asses rules slides 14: Team/Product/Market)
Your first startup: slide 5 & 21: most startups fail because the founders give up
SUPER ANGELS, SUPER STARS, AND THE SUPER-SIZED STARTUP MYTH Chris Shipley Cofounder & CEO GuideWired Group
Succeeding-with-freemium (how to succeed in converting free subscribers to paying customers) Sean Ellis - Founder & CEO - CatchFree
Why not do a startup (basically, do you have the profile): Dave McClure - Investor - 500startups.com

http://www.startupfestival.com/en/speakers/

appengine unittesting trick

2011-06-25T20:51:00.000-07:00

I was looking for the ideal way to run all my unittests on my appengine projects.

Let's define what is ideal:

http://project.appspot.com/test to run all unittests
create unittest directly in file it is testing/no need to create unittest file for each file

Solution:

install gaeunit
create or move unittest classes directly in file that it is testing
create /test/project_unittest.py
import all file containing unittest class in project_unittest.py

What's great with python is that unittest.main() run all unittests that are in the scope which means you simply have to import them and no need to create multiple test files if unittests classes are directly integrated in each files.

Governments programs are destructive for new entrepreneurs

2011-05-14T19:31:00.000-07:00

I reached my limits of bureaucracy. Enough is enough. My conclusion is that governments programs are destructive to new entrepreneurs. Let's face it, it is simply incompatible, entrepreneurs are looking for efficiency and government can only produce bureaucracy and inefficient process that are killing wealth creation.

Many entrepreneurs are trapped by government programs where they expect help and finally end up losing precious time, getting ridiculous support for massive paperwork if not excluded by incoherent rules. Entrepreneurs are here to create wealth not justifying bureaucracy existence.

I am tire of hearing bullshit like:

we are here to help entrepreneurs

Stop saying it and do it.

but you have to be profitable, have clients, make more then 200K, been incorporated since, have a product, have N employees, have to been able to pay your new employee to get the support, not this, not that, this, but only in 3 months, sorry in 6 months, it needs to go through the committee of the committee of the board and can't tell you when they might decide but it will required another meeting to decide, you have to come to our office, we need this form and this proof and letter of this and details of this....like if entrepreneur time is free.

If we were respecting your criteria, we wouldn't need your support. Again we aren't looking for advices but only financial supports and efficient process, we know what we have to do. The current system seems to subsidising establish corporations, corporation without financial constraints and most important, for a second time, justifying bureaucracy existence.

Here is in a nutshell what entrepreneurs need:

Real founding support, light and fast decision process...not advices nor endless decisions and paperwork process of months and please stop pretending you can help if you know you can't.

Worst thing is that it is sending a so bad signal to the new entrepreneurs, you should be an bureaucrat official, you will make more money, you will have a pension plan, less stress, no performance evaluation from people you are suppose to help...

Don't worry entrepreneurs can't go that way but can kill this system slowly. Without entrepreneurs, this system cannot sustain. I hope the new majority Conservative Party of Canada will make drastic cuts in this bureaucracy (their 10 billions cuts hasn't been unveiled). Officials aren't untouchable, don't shit in your plate rules applies to everyone. The effect might take longer but it will come and new entrepreneurs will help the momentum but what is bad is that founds might been cuts but no untouchable government jobs. Stop wasting time on their salaries and invest in entrepreneurs directly.

Government programs are making more damaged then good for new entrepreneurs by making them loose too much time and energy and encourage indirectly outsourcing which overall is extremely bad for the economy. Hopefully Canada has natural resources....but it isn't an excuse to tolerate destructive economy programs.

http://www.mdeie.gouv.qc.ca/fileadmin/contenu/documents_soutien/apropos/strategies/entrepreneuriat/renouvellement_entrepreneuriat.pdf (see page 24)

http://www.radio-canada.ca/nouvelles/Economie/2011/05/04/006-indice-entrepreneurial-quebec2011.shtml

Letters order isn't important for our mind: a simple appengine service to play with this idea

2011-04-07T20:53:00.000-07:00

I got several time an email about cool visual illusions and enjoyed a lot the one on the fact that our mind doesn't care about letter order, the only important thing is that the first and last letter be in the right place (refer as a Cambridge study). You might have already seen this:

Olny srmat poelpe can raed tihs. I cdnuolt blveiee taht I cluod aulaclty uesdnatnrd waht I was rdanieg. The phaonmneal pweor of the hmuan mnid, aoccdrnig to a rscheearch at Cmabrigde Uinervtisy, it deosn't mttaer in waht oredr the ltteers in a wrod are, the olny iprmoatnt tihng is taht the frist and lsat ltteer be in the rghit pclae. The rset can be a taotl mses and you can sitll raed it wouthit a porbelm. Tihs is bcuseae the huamn mnid deos not raed ervey lteter by istlef, but the wrod as a wlohe. Amzanig huh? yaeh and I awlyas tghuhot slpeling was ipmorantt! if you can raed tihs psas it on!!

I decided to create a service to play with the idea on google appengine.

You can try it now with your own text:

http://nonhumanreadable.appspot.com/

How to create standalone python apps?

2011-01-10T21:37:00.000-08:00

You might have to run your applications in your customer infrastructure but you might not want to give your recipes (python source code) so here are the alternatives depending on your OS:

windows = py2exe
mac = py2app
linux = pyinstaller (freeze doesn't work->compile errors*)

On linux, pyinstaller works quite well but you have to generate it on the same distribution.

Here are the steps:

download latest version
python Configure.py
python Makespec.py /path/to/yourscript.py
python Build.py /path/to/yourscript.spec
start app: yourscript/dist/yourscript/yourscript(binary executable)

(*) Freeze instructions:

svn checkout http://svn.python.org/projects/python/trunk/Tools/freeze/
python freeze/freeze.py yourscript.py
make

simple multivariate classifier example using python & numpy

2010-10-13T07:51:00.000-07:00

I was wondering how long it could take to write a multivariate classifier in python.

With python and numpy it isn't long. We simply need to be able to compute the covariance matrix, the determinant and to inverse a matrix (covariance matrix). Even if the matrix is singular, which mean it can't inverse it, you can compute the pseudo-inverse (Moore-Penrose) easily (i.e.: numpy.linalg.pinv).

As expected, assuming too much about the data lead to poor classification.

You can find a simple python program of 75 lines here.

Dimensionality reduction; a simple PCA example using python

2010-10-10T00:00:00.000-07:00

Dimensionality reduction is a powerful approach to reduce inputs size, reduce training time and visualize data.

As an example, you can use PCA(Principal Component Analysis) or ICA (independent component analysis) or LLE (Locally Linear Embedding).

to see class grouping. You can try it on your data easily with python in a couple of lines.

import mdp
pca = mdp.pca(ds.data)
pylab.title("PCA")
pylab.plot(pca[:,0], pca[:,1], '.')

The figure presents the PCA dimensionally reduction applied on a digit dataset. You can find the source code here to see you to do a PCA, ICA or LLE using python. Unfortunately, ICA doesn't work on our dataset because it doesn't converge.

PDF watermarking service using pdfrw on google appengine

2010-10-09T23:27:00.000-07:00

If you are looking to watermark a pdf, you can use this simple appengine service:

http://pdfwatermarker.appspot.com/

This service use pdfrw (a PDF file manipulation library written by Paul Gauvin) and reportlab. pdfrw is much faster then pypdf for watermarking.

Summary of machine learning libs available in python

2010-09-21T19:40:00.000-07:00

Here is a summary of all python related machine learning libraries in python (inspired by Similar or Related Projects of PyMVPA, lisa mailing list and personal notes).

pybrain: PyBrain is short for Python-Based Reinforcement Learning, Artificial Intelligence and Neural Network Library. In fact, we came up with the name first and later reverse-engineered this quite descriptive "Backronym". see features. key feature : ecurrent networks (RNN), including Long Short-Term Memory (LSTM) architectures
mlpy:Machine Learning PYthon (mlpy) is a high-performance Python library for predictive modeling. mlpy makes extensive use of NumPy to provide fast N-dimensional array manipulation and easy integration of C code. The GNU Scientific Library ( GSL) is also required. It provides high level procedures that support, with few lines of code, the design of rich Data Analysis Protocols (DAPs) for preprocessing, clustering, predictive classification, regression and feature selection. Methods are available for feature weighting and ranking, data resampling, error evaluation and experiment landscaping. Key feature: feature selection
scikit.learn: scikits.learn is a Python module integrating classic machine learning algorithms in the tightly-knit world of scientific Python packages (numpy, scipy, matplotlib). Key distinct features: lasso, nearest neighbor, isomap, various metrics, mean shift, cross validation, LDA, HMMs
opencv (machine learning): Normal Bayes Classifier, K Nearest Neighbors, SVM, Decision Trees, Boosting, Random Trees, Expectation-Maximization, Neural Networks
Shogun: A Large Scale Machine Learning Toolbox Comprehensive machine learning toolbox with bindings to various programming languages. PyMVPA can optionally use implementations of Support Vector Machines from Shogun. Large scale kernel learning (mostly svms). this wraps other libraries such as libsvm (well-established) and others that get state of the art performance or are good for extremely large datasets, etc.
PyMVPA (Multivariate Pattern Analysis in Python): PyMVPA is a Python module intended to ease pattern classification analyses of large datasets. In the neuroimaging contexts such analysis techniques are also known asdecoding or MVPA analysis.

pylearn (build on top of theano), under V2 construction. New version of plean (c++).
Theano: (deep learning) Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently.
jml: Jeremy's Machine Learning library (C++), include a python interface: Basic classifiers (perceptrons, decision trees, etc) plus ensemble methods (boosting, bagging). Very highly optimized to work with thousands of features and millions of examples. GPGPU support under development. Code derived from this library is extensively used in a commercial computational linguistics application, so it has gone through its paces.
3dsvm: AFNI plugin to apply support vector machine classifiers to fMRI data.
Elefant: Efficient Learning, Large-scale Inference, and Optimization Toolkit. Multi-purpose open source library for machine learning.
MDP Python data processing framework. MDP provides various algorithms. PyMVPA makes use of MDP’s PCA and ICA implementations. interesting features: ica, LLE
MVPA Toolbox: Matlab-based toolbox to facilitate multi-voxel pattern analysis of fMRI neuroimaging data.
NiPy: Project with growing functionality to analyze brain imaging data. NiPy is heavily connected to SciPy and lots of functionality developed within NiPy becomes part of SciPy.
OpenMEEG: Software package for low-frequency bio-electromagnetism including the EEG/MEG forward and inverse problems. OpenMEEG includes Python bindings.
Orange: Powerful general-purpose data mining software. Orange also has Python bindings.
PyMGH/PyFSIO: Python IO library to for FreeSurfer’s .mgh data format.
PyML: PyML is an interactive object oriented framework for machine learning written in Python. PyML focuses on SVMs and other kernel methods.
PyNIfTI: Read and write NIfTI images from within Python. PyMVPA uses PyNIfTI to access MRI datasets.
milk: k-means, svm's with arbitrary python types for kernel arguments. Pythonic interface to libSVM. Stepwise Discriminant Analysis for feature selection. K-means clustering. odels can be pickled and unpickled.
mlboost: Machine Learning Boost Library (python; includes flayers wrapper); minimal version of sourceforge mlboost project. Specialized on features extraction and visualization.

to watch:

http://torch5.sourceforge.net/ Torch5 is the official successor of Torch3, and is now developed at NEC Laboratories America andGoogle Labs. No python wrapper yet.
pylearn v2; new version of pylearn from LISA lab of udm.

How to add activities to redmine timesheet plugin?

2010-08-25T23:54:00.000-07:00

I am using redmine, a great project management web application, and added littestream timesheet plugin.
If you want to add timesheet activities, you can't do it from the UI. You need to add it directly into mysql db like that:

insert into enumerations values (10,'Experimental Development', 3, 0, "TimeEntryActivity", 1, NULL, NULL)
insert into enumerations values (11,'Scientific Research', 4, 0, "TimeEntryActivity", 1, NULL, NULL)

Don't forget to set property the id and the position.

select * from enumerations;

patching class function in python

2010-07-21T14:31:00.000-07:00

Today we had to patch a class function in production. Monkey patching can become tricky if reference are kept at several place like pointer in C and C++.
Here is a simple example on how to make sure all references will use the new definition.
class Foo:
  def f(self):
      print "default f"

def newf(self):
  print "newf"

Foo.f.im_func.func_code = newf.func_code
This is another reason why interpreter language like python are so powerful.

Before filing Software Patents, wait for Bilski case resolution

2010-06-01T10:41:00.000-07:00

The big upcoming US case regarding software patents is In re Bilski. The Supreme Court has been sitting on it for months now. "Bilski" an ongoing set of patent cases that will change the patentability of software in the USA. The Bilski patent itself is a business method patent, not a software patent.

The 2008 ruling of the Court of Appeals for the Federal Circuit (CAFC) was broad enough to reject Bilski's patent and a certain category of software patents.

The Supreme Court agreed to review the CAFC's ruling (as Bilski v. Kappos), and the judges raised the issue of software during the hearing.

The Supreme Court's ruling could greatly change the patentability of software patents, business method patents, and the middle ground of e-commerce patents.

If you are looking for US patent news, I recommend patentlyo.

Update:
Supreme Court Says No to Exclusivity of Machine-or-Transformation Test, In re Bilski

Canadian Businesses Beware: Blessing of Business Patents in US Will Affect You

Easy Presentation Slides with Latex-Beamer

2010-05-18T14:19:00.000-07:00

If you are looking for an elegant way to create slides like the ones I have presented at montreal-python, I recommend latex-beamer.
Here is a example in 6 simple steps:

mkdir trial; cd trial
sudo aptitude install latex-beamer
wget http://mlboost.svn.sourceforge.net/viewvc/mlboost/doc/ml-python-mtl-april2009.tgz
tar -zxvf ml-python-mtl-april2009.tgz
pdflatex ml-python-mtl-april2009.tex
xpdf ml-python-mtl-april2009.pdf

IF you don't remember latex syntax, you can use:

When you start using latex, you never want to go back. Content and visualization should be well isolated/separated.

Future of Quebec Software Engineer

2010-05-15T09:58:00.000-07:00

Currently, new Software Engineer have an easy live. With a shortage of resources, you don't have to be good to find a job. When employers are desperately looking for people, they take what they can and don't complain too much about it (in this context, they feel lucky). As an example, ETS Software engineer students can choose between 17 intern offers. Will the situation change in the future?

For a long time, Canadian dollar was low which makes Quebec and Canadian Software engineer pretty cheap, unfortunately it is changing and it is already not true anymore. They are starting to be expensive. Many software jobs in Québec are related to US company having an office in Canada and many others rely on US clients.
Easy tax credit for experimental development and scientific research will be harder to get since Canadian governmental is back to deficit and Quebec is facing aging population and reduced number of tax payers. For more info, check this post.
Outsourcing, a cheaper and more accessible alternative. As a example, when more Indians will be able to speak comprehensible English, they will become a real danger. Many companies start using cheap remote resource or use outsourcing service like rentacoder to lower down their cost.
Due to globalization and near US market, many companies will prefer perfectly fluent English average engineer as oppose to great engineer with lack of English skills.
Due to globalization and near US market, many companies will promote perfectly fluent English average engineer as oppose to great engineer with lack of English skills.

Unlike Doctors with powerful Professional College, software engineer don't have reserved acts that reduce competition and provide job security, they don't have the luxury to not be competitive.

What should future Francophone engineer need to success and be more competitive in this changing landscape?

They need to be fully bilingual (btw, many company don't consider Francophone university because they know that most of their software engineers aren't bilingual)
They need management skills. Ability to manage technical people is rare and critical to the success of projects, company are more aware of it and are desperately looking for it. Increase success rate of project is critical.
They need leader and decision maker, not only a work force.
They need writing and communication skills. Technical people who neglect this will pay the price in the long run. Without those skills, it is hard to evolve in the upper level of a company.
They need conflict, negotiation skills. Engineers forget that their job is on average 20% technical and 80% HR related. Resolving human related problems is important part of software engineer work.

But the most important is that companies need decision makers who see the big picture, not only throw-able/easy replaceable work force.
Outsourcing is out the door and software engineer have to better understand the idea of comparative advantages because their cheaper advantage is getting away.
Too expensive non perfectly bilingual engineers with lack of management and communication skills jobs will face increase outsourcing pressure.

Disruptive business model: Lew Cirne, serial entrepreneur on the future of Enterprise Software etc.

2010-05-14T20:48:00.000-07:00

I really like New Relic founder (Lew Cirne) point of view about the evolution of business model with the web. Basically, massive sale overhead is not require anymore which lead to reduce drastically software cost and free resource to do better software. Most of the time, software satisfaction is low because monetary resource are sucked by so many overhead layers and isn't use to make better software. You should read his posts:

Software transition: recipe for a disaster

2010-05-14T17:53:00.000-07:00

Everyone knows that software has a life cycle and dies at some point. If your software is part of your day to day operations, you can't fail in managing its transition. If you want to reduce the probability of success of the transition, here are some rules to follow:

Don't plan transition
Don't communicate transition plan if you have one
Promote non-skilled people to decision positions
Kill initiative/ignore message (ex: kill messenger and think the problem is gone)
Avoid long term planning (ex: use argument like business moving too fast)
Don't put a technical team lead
Don't do follow-up of team requests
Don't involve HR to smooth transition
Don't hire pro-actively
Don't prepare a B plan
Don't involve and/or update your team on the transition plan
Don't give overall project responsibility to anyone
Stay in a reactive mode
Don't communicate info in daily meeting
Don't ensure you have a good pulse of your team
Give too much power to people who don't understand software process
Like Greenspan, dream the magic hand will fix everything magically (people might not compensate for bad decisions eternally)
Expect people to stay
Don't make difference between maintenance & development cost
Let non technical people take technical decisions
Ignore problems
Don't recognize people work in crisis
Allow managers to not be able to evaluate technical people
Consider Indian outsourcing can solve everything
Think people are easily replaceable
Don't talk about carrier evolution to your crew

If you are lucky, you will have to replace your team one by one but they could resign all in a short time and put your business in a major crisis situation. Don't forget, there is no free lunch.

Tax credits (RS&DE): the new reality = good news for startups

2010-05-10T19:59:00.000-07:00

Last month, I went to the CEIM (centre d'entreprises et d'innovation de Montréal) to see a presentation on the new reality of tax credit program in Canada starting in 2009. RS&DE tax credit for scientific research & experimental development has changed drastically. Initiate in 1986, the program claims reached 1.5 billion in 2000 and ~4 billions in 2008 at the federal level. Due to the increase of claims (20K claims 2008), non standard claims and unwritten need to reduce tax credits, an important change happened. Many companies can't send anymore non standard endless novel to lost government officials. Corporation need to send 1400 words to explain the scientific advancements, the problems/technical incertitudes/risk and the technology and scientific content. With 15% more staffs and 1400 words limit per claim, way more people can validate your claims. Knowing we are back in deficit, there will be pressure to reduce accepted claims. It will be way harder for cheaters to try to loose the reader because endless novels is no more possible (romans fleuve). Btw, CEIM offers interesting services.

It will be an harder sell for lawyers firms specialized in R&D tax credit paperwork who can't use their government bodies as easily as before to ensure their clients get their tax credits. This is a good news for startups and bad news for the ones who are making things up because it means less useless work for us, more time and more money to create real wealth.

VC founding & StartupCamp 6

2010-05-08T20:43:00.000-07:00

Montréal StartupCamp 6 was quite interesting. As raised during the unconf discussion, French language technology entrepreneurs seems to prefer bootstrapping their companies then looking for VC foundings (ex: FeelingSoftware/Carré Technologies/DokDok/Vizmatic/QMining/Aptats/Nexyka etc.). keynote Dave McClure presentation on how to create good pitch to investors was very good:

Start with problem not solutions
Look for high reaction signal (good and bad)
Stop adding features
Focus on customer reactions, real-time as possible
Volume->Cost->Conversion | acquisition/activation/retention/referral/revenu...
etc.

VCs are basically non-risk taker and followers as sheep and should be treated accordingly.
But what's the point of getting VC founding for software startups? They don't need much more then computers & time. With VC founding you could get 6-12 months where most of it will be re-reimbursed with R&D tax credits. Basically, they give you cash advance for an important share of your company for a ridicule risk. VC mains arguments are:

It will allow you to be the first in the market, its bullshit, everyone knows it is the timing that is the most important.
10% of 15 millions is better then 100% of 1 millions...but you still will do most of the work with more stress from investors and you might get 10% of 2 millions. Most entrepreneurs start their company to take control and not getting back to a slavery mode.

Yes it could allow you to build your product faster to meet a customer needs but better, they could make clients connections which has real value. I think that Steve Dekorte point out exaclty what I profoundly dislike about VCs and haven't presented clearly in a previous post, VC like professional CEO are short term peoples as oppose to founders. With a difference cost function, you will automatically get clashes which lead to frustrations, politics....basically inefficiency due to uncommon vision.

Andy Nulman Keynote presentation was interesting in the perspective of importance of a partner and the need to adapt but the mercantile conclusion was a pathetic anglo-saxon point of view: you could make more cash by doing the chicken dance then doing something interesting.

Phil Telio announcement about the new startup dedicate house notman was great for Montréal. I will definitely apply many of the stuff learned there. As an example, if you are a founder and will move to the CEO position, you better start delegating what you are better at because it will allow you to improve others skills and will make other supervision very efficient.

Ubuntu remix & Asus Eee PC 1005PE 10''/250G 14hr review

2010-03-16T19:54:00.000-07:00

Ubuntu remix is awesome. I am simply amazed. I have installed it from a usb key on my new ASUS Eee PC 1005PC and everything works perfectly, yes everything...with a small fix for the wireless interruptions:

Wireless (see link to fix flaky + interruption)
Bluethooth
Sound
flash (ex on youtube)
hibernate
second monitor short key
lower intensity screen short key
ubuntu one (storage)
video

I don't need to carry my AC plug anymore. The battery doesn't last 14hrs but easily 10 hrs, way enough for an entire day. The Atom450(1.66 GHz 512 cache kb) processor performance are acceptable. If I compare heavy cpu neural network training with numpy, it takes less then 5 times the time compare to my AMD athlon 64 (2.8 Ghz + 512 cache). If I normalize the comparison on cpu frequency, it is on 3 times slower which is acceptable. I don't need windows anymore on my laptop, my life will be much pleasant.

Buy US made by Google.com: search protectionism?

2010-03-16T19:17:00.000-07:00

At confoo conference, I attend to "Comment faire une bonne campagne Search + Social ?" session. I have been close to fall from my chair when I learn that google.com doesn't provide the same results in the United States compare to all other countries.

Everywhere except US: google.com = search around the world
US only: google.com = search in the US

Yes, if you search on google.com everywhere except in the US, you will search from the same index all around the world. If you are in the US, the search is limited to United States only. Isn't that protectionism?
In many fields, it is great that it is ranking closest business higher (ex: closest pizza place) but not filtering non US competitors when service doesn't need to be close by (ex: consulting services).
Knowing that the United states is the biggest promoter of the free market/open market etc., I am totally amaze to learn such a rule. Do as I Say, Not as I Do. As an example, if you don't have an address is the states and/or host our web site there, SEO (seach engine optimization) is useless to get new customers from the states (a insignificant market;), they won't find you if there are using the biggest search engine...google.
Do Oubama administration really need a buy America plan?
The first role of free trading in unrestricted access to information (i.e: économie de l'information: Joseph E. Stiglitz).
You can try it:

What everyone outside the US see with google.com : http://www.google.com/ig?hl=all
What US people see with google.com: http://www.google.com/ig?gl=us

As you will see, results aren't the same.

Opportunities ahead to reduce impact of doctor shortage an money constraint: Machine Learning

2010-03-15T20:53:00.000-07:00

The "Collège des médecins du Québec" is hammering us with advertisement to give Québec Government solution to find new founding to pay more our doctors and specialists. In the industry, we make more money by been more efficient, not increasing prices. It is the easy solution when others are paying. "Collège des médecins du Québec" should promote efficiency to reduce their cost, deliver better service and generate more margin to increase nurses and doctors.
As an example, radiologist are earning on average around 700 K/year and Ophthalmologist 600K/year. Yes our generalist doctor could earn more, the average is around 150K/year. As a heavy tax payer, I am suggesting using optician approach in BC, replace heavy paid optometrist by machines to do the exams. Basically, automate what need to automate and use doctor efficiently which could reduce/eliminate the shortage.
When people go to far with their salary expectation, it is time to bring them back on earth. No one accept fees increase for poorer service.
Public system and doctors studies are founded by our taxes and Health System spending represents close to 50% of Québec spending, money doesn't grow in trees. Yes, they are getting less if they were in the states, but US doesn't have a public systems, only wealthy people have access to it and it can't happen in Canada because it is publicly founded.
I have tried to help happy clinic by contacting doctors to offer smart waiting time system to make people wait less but most of them didn't care much: we are busy, people have to way, its a natural filter. They can make us waiting hours even with appointments, treat us like shit because the service offer is low. When most of us are waiting, we aren't earning money to pay them. Shame on you. Everyone is loosing at this game.
Doctor are getting greedy and are starting to see them as untouchable and are forgetting who are paying their salary.
If well packaged, Machine Learning can be use by anyone to do high level screening and provide valuable information. In 2010, doctors remain one of the only profession who doesn't use much machine to make them more efficient and are fighting to stay the bottleneck. With this crisis ahead, it might be a good opportunity for the machine learning community to get into this shielded area in the benefit of everyone, them too.

Why you Shouldn't Accept VC money earlier than you thought: Story about Venture Capital and an Outdated Decision Maker revised

2010-03-12T23:11:00.000-08:00

I met Gary Haran at confoo and stalled on one of his post:
It might be advantageous to accept VC money earlier than you thought.
I have auto-censure one of my post about VC but it need to be republish to ensure entrepreneurs understand the possible deep consequences of such a decision. The post Anatomy of a failed software project initiate my thought of un-censure it because we don't talk enough about bad experiences to not offend people or been seen as too negative but its reality and we should have realistic expectation. With Amazon EC2, google app engine, rackspace hosting and so on, it is becoming way less costly to bring new technologies to the market. Most software company don't need big founding anymore and have just more reason to avoid VCs.
I strongly recommend boostrapping your company with consulting and/or R&D credits, INRS programs etc.

So here is the revised version on the post I have pushed Apr 29, 2009 12:32 AM and removed.

Once my boss, which was one of the founders of the company that I was working for, told be: Avoid as much as you can VC, it should stay your last resort. I haven't realized the deep importance of this after some American sharks VCs took control of our company and enforce their ultra capitalism short term vision. GM is a good example of short term vision but, that's another interesting story.

Venture Capital represents high risk investments and their only gold is to get the highest return on investment in the shortess time period. That's fair, you simply have to know the rules.

Entrepreneurs build and invest, VC rape everything that is possible. At that time, they replaced the top management with their Californian super stars or ... remaining turnips (j'exagère un peu). One of their last super star, when I was there which I will refer to "Le Chasseur", came to take the highest position of the technical side of the business. At that time, I was working on new technology and this old cowboy grandpa came to tell me that we weren't doing real research. According to him, real research was what he was doing 25 years ago with his wired and transistors. I simply told him that our group of 3 (i.e: including manager) weren't pretending doing research but simply doing applied research and definitely not pure research as his thousand of coworkers were doing in the old monopolistic US company he was referring to. This guy had made lot of money in the good years of the Internet and was annoying me with his stories, I should have told him to go and say it to his little sons and daughters and simple retire and let us work. His only salary could have let us build a much better team to continue innovating in technology which was our core business. Without any support from him, even working against us, we still have manage to release a complete new technology (i.e.: prototype + knowledge transfer to prod+ prod support) that is bringing quite lot of income to the company. Charlie, most important point is to bring technology to the market, nothing else matter...I have succeed even if he desperately try to make us fail to justify moving advance stuff to California.

Those clowns (i.e: I know, I am over generalizing) were just sucking all financial resources of our company. Been still a share older, I was relieve to hear that "Le Chasseur" has finally been fired in a reorganization. If you need reorganization to do cleanup in a private company, your level of politic game should be quite high... and politics leads to corruption and mediocrity in such an organization. We could have become a great company but it might remains an average one, I don't think I deserve that after such investments but money leads. According to old colleagues, the latest reorganization was efficient.

I think that the concentration of smart people is higher in CA then in Montreal, but we definitely fall on some bad apples.

Key decisions can't stand in incompetent hands, soon or later the guillotine will come. If you made money and reach your level of incompetency, please let the others take the lead. If you haven't make money, just reorient your carrier. We don't care about your ego, the fact that your are the initial founder etc. what is important is the company success and everyone will benefit from it.

You might have better story and or points of view about VC but what I saw wasn't great:

They stop/restrain investment in innovation
They drop some incompetents clowns (over generalized to make the point)
They suck all money to pays their clowns
They made your shares less valuable
They drawn massive amount of money on building products that had no real value and on non core technology
They create a real wall between decision makers and the workforces
They invest a lot in fake partnerships
They love yes mans
They feel way smarter (especially CA vs Mtl)

American VC are like the FMI for Argentina in the 2001 crisis or Corea in the Asia Financial crisis of Golden Sach for Greece bankruptcy, you are going to be fucked.
I hope that I will never in my live work again for a company that will be taken over by VC. If you need VC, you might already been badly organized to need desesperatly money from them. I would definitely like to hear good stories about VC where they achieve their gold...at least.

Yes I had a painful indirect experience with VC and an outdated decision maker but I knew that startups environment aren't easy. I learned better the rules and the game by playing it. I wish they will find a way to make money because success bring success and Montreal deserve more of them, there is so much talent here. I have emphasis on one bad decision maker but the CEO and the CFO were interesting character...but you need one bad apple drop by your VC to fuck your company and they will get most of the benefits.

As mentioned in Samual Bouchard blog, we need way more private founding alternatives but were aren't yet there.

python try catch cost vs hasattr (overhead = only 2X)

2010-01-31T18:54:00.000-08:00

To support different version of code, you might end up using try catch code bloc. But, in the worst cast, what is the cost overhead of throwing an exception every-time?
You might have the possibility to use in some case the hasattr to avoid throwing an exception but your code will be less elegant and/or readable.
In order to make a rational decision, let's compare the time cost. I was expecting a major cost for throwing exception but it ends up been only 2 times more costly then doing an hasattr.
You can see the simple code I have used to compare it here. From now on, I will be less scare about memory performance when I am using try catch.

The power of python within Tomcat for powerful webapps (jython2.5.1)

2010-01-22T22:51:00.000-08:00

Many companies are looking to speedup their development process. Unfortunately they are restricted to use well established webserver framework like tomcat and, in this example, they fill limited to Java.
Jython is coming to the rescue. Since September 26th 2009, Jython2.5.1 has been released and can be use to create servlet that runs inside apache tomcat application server. Jython allows you to write python that is running on java VM (100%) and let you use lot of python pure libraries and let you use all java packages with a python synthax.
If you try it, you might have issues like:

How can I create a simple jython2.5.1 servlet? (not deprecated jython2.2.1 that doesn't allow you to use most pure python libs)
Why do I get ImportError when I use standard python package? How can I fix it?
Where should I put jython2.5.1.jar?
Where should I put my python code?
Where can I get a basic example that is working?
Where should I put my pure python libraries?

So I have decided to package a simple example (download: HelloWorld.war).
Here are the steps you should follow:

get tomcat http://tomcat.apache.org/ (I used 5.5.28)
get jython2.5.1 http://sourceforge.net/projects/jython/files/jython/jython_installer-2.5.1.jar
cp ~/jython2.5.1/Lib tomcat5.5.28/share/lib (required to used std python libs)
cp ~/jython2.5.1/jython.jar tomcat5.5.28/share/
download example: wget http://mlboost.svn.sourceforge.net/viewvc/mlboost/jython/HelloWorld/HelloWorld.war
cp HelloWorld.war tomcat/webapps
download java jdk
export JAVA_HOME=sun-jdk-1.6.0_02
tomcat5.5.28/bin/startup.sh
try it: http://localhost:8080/HelloWorld/HelloWorld.py

If you want the flexibility of python but you are stuck with java within tomcat, jython is becoming a real alternative since they've release jython2.5.1.

PS: a war file is a zipped file, you can unzip it in tomcat/webapps for testing so you don't need to rezip it and restart the server. When you are done, simply do a jar cvf HelloWorld.war * in the tomcat/webapps folder and ship that single file to the client tomcat server (make sure jython is installed). If you want to add pure python libraries, you can simply add them into your war file, it will work.

Here is the time comparison of the same service:

python: wsgi httpserver
jython: wsgi httpserver
tomcat: java servlet jython2.5.1

It is interesting to see that tomcat servlet provide better perf

matplotlib & python for powerful data visualization

2010-01-08T20:18:00.000-08:00

Here is an example of data that isn't obvious to analyze:
What is the gain and lost effect of percentage of seats in a point of view of proportional representation? Percentage of seats is usually chosen in legislative assemblies. It is the process used in Canadian and Québec elections.

Powerful visualization allow you to see easily the effect. Python & matplotlib is an amazing combination to do so. It took me 20 minutes to allow me to visualize the effect in federal and Quebec election of 2008.
Upper graph (seats vs votes) shows the lost of proportional vote % if you use a seats approach. As an example, liberals gain ~11% and ADQ lost of ~11%.
Lower graph (lost seats vs votes). The real impact of party is the ratio of this lost on their real vote proportion. In this example, it is a gain of ~25% for each Liberals votes (11/(66/125)) and a lost of 66% for the ADQ and ~88% for QS.
Basically:

In Canadian election, PC & BQ gain power but BQ way more in proportion and Greens lost everything
In Quebec election: QS & ADQ lost lot of power and PQ and LIB gain it: it might explain why they aren't talking of changing election formula
Matplot lib and python is an amazing combination to automate data visualization

to get the code do:

svn co https://mlboost.svn.sourceforge.net/svnroot/mlboost/elections
python elections/seats_vs_prop.py

Gerrymandering Explained (youtube;
Gerrymandering - another reason why rep democracy is fundamentally corrupt http://bit.ly/qO4mpH)

gmail, a powerfull target marketing tool

2010-01-03T21:51:00.000-08:00

When I got my account on gmail many years ago, I was wondering why google was spending so much resource/wasting to provide free email service as hotmail, yahoo and so many others. Were they making enough money on gmail advertisement? Does that worth the investment?
I though at one point that their primary end goal was to launch a corporate email portal so company won't need to hire high paid sys admin to provide mail server support and by the same time help world wide employee getting something way better then outlook/exchange server that pollute our live. They are doing it already but I think their real goal was to do target marketing but not traditional one.
What better can you get then user emails to understand his profile and do target marketing. They get the highest quality info from your emails, yes your emails.
According to my experience in more traditional target marketing for Bell and at Microcell-Lab, when people do traditional target marketing, they have few info about users and derive new information from which they try to generate better predictions. As an example, they use your postal code to estimate your family revenue etc, and use that information to generate better prediction that you will buy X or Y.
Traditional target marketing practitioners use a lift approach to get the top N most probable buyers for a given product or service and will try to approach those people with promotion or email etc. With gmail, it it way more simple, you use user profile info like email words (btw, they are parsing your emails, take a close look, you will see), and use a prediction engine to advertise the info you are most likely going to like or buy and show it to you directly because you are using their mail service.
Gmail is an amazing target marketing tool because it get profile info directly at the user fingerprint, can do way better prediction then traditional target marketing technics and has access to the customer directly and scale well to get more users. Our prediction is always as good as your data. What's the point of improving algorithm if you can get better and high quality data or as google do both. Larry Page and Sergey Brin are just visionary target marketers!

Jython, pyPdf, reportlab experimentation & patches proposals

2010-01-01T21:52:00.000-08:00

I am currently experimenting jython in order to do pdf files manipulations. I encountered several problems and I want to share some of the solutions (time to give back).

Intro, only pure python code and library are working on python and jython. All C related python packages aren't compatible. Jython allows python syntax on top of java VM. One great thing is that you can use java classes within python. Jython2.5.1 as been release last September.

1) manual pdf text modification
I thought it was simple to modify a pdf template to change a text but I was wrong. Even if you are able to re-encode new text and change length, you will hit walls. It is more complex then that (xref etc.). Most pdf lib provide encoding helper function but you will get hard time finding decoding one, as an example ascii85. After some time, I decided to try to make reportlab working with jython.

2) reportlab import error with jython
I tried to used reportlab, a powerful lib to create PDF, but it was generating this error when I was importing reportlab.pdfgen: java.lang.ClassFormatError: java.lang.ClassFormatError: Invalid method Code length 66566 in class file reportlab/pdfbase/_fontdata$py. According to this thread on warkmail, there was a simple solution but the patch wasn't working. You can find the working patch that I have created here and proposed to reportlab team.

3) Saving pdf to memory instead of files
In order to do in memory pdf manipulations, I used the pure pyPdf python lib from Mathieu Fenniack. Basically, I tried to save a canvas in memory and couldn't figure it out why it wasn't working. Basically, I was doing outputStream.writelines(c._

doc.GetPDFData(c)). Unfortunately it doesn't work if you don't call c.showpage() before. I also realized that I could create a canvas directly with StringIO as filename argument because pdfdoc.PDFDocument.SaveToFile(...) check if fname as a write function. I have proposed a canvas api improvement you can find here to make it more friendly to use.

4) Simple comparison python/jython
I was wondering how much slower was jython compare to python. As you can see, it is slower and it degrades with some parameter size (ex: n pages). In this example, it also takes 4 to 6 times more memory.

5) Jython out of memory
If you get:
OutOfMemoryError: java.lang.OutOfMemoryError: GC overhead limit exceeded
use -J-Xmx1024m jython option to allow more memory heap size for the java netbeans.

4) Threading optimization
Jython doesn't suffer from the GIL problem. Look at this video to get more information about it "Mindblowing Python GIL". Basically jython can do real multi-threading. In my context, I could easily parallelize part of my code so I tried it by using the Theadpool of Christopher Arndt. Unfortunately, I still haven't been able to make is faster. pyPdf hasn't been designed to be used in a real threading environment (PdfFileReader can't be shared between threads) which introduce limitations.

5) pyPDF profiling

pyprof2calltree is an amazing tool for profiling as you can see in the figure. Guessing what needs optimization is a path we should never go because we are most of the time wrong and are wasting our time doing uncritical optimizations that make code unreadable. I saw a great presentation fromMike C. Fletcher on python profiler at pycon 2009. I might try to optimize readObject function of pyPdf.

It is amazing to see the tremendous effort people are putting to make python syntax available on each platform (java->jython; .Net->ironpython etc.) It is a sign of python great syntax.

Business Contract 2.0 - Found a template

2009-12-13T00:13:00.000-08:00

Knowing that I am flirting with the idea of doing consulting, I was looking for advices about common mistakes and better ways to evaluate client contracts. Contracts can put you in obligations you haven't thought and make you slip your deliveries, force you to make more work than planned and worst, jeopardize your relationship with clients. Root cause is most likely unclear expectations on both sides. With some spaghetti clauses that only juridic lawyers can understand, it is easy to fall in this path and everyone loose at this game.

RIM (Régionale des ingénieurs de Montréal/ordre des ingénieurs du Québec), was organizing a training Workshop called "Business Contracts 2.0". The presentation from Gilles Thibault of edilex was extremely interesting.

Currently, their is no established standard to create contracts. Basically their is as many form as the number of lawyers out there. Most of the time, lawyers are the only one comfortable with them because they've wrote them and it is in their juridic jargon. Unfortunately it doesn't help much who really used them. Most of the time, clients and contractors have hard time understanding them but it is a lucrative process for high hour rate paid lawyers. According to Gilles, lawyer will disappear if they can't provide better services for contracts (preaching for its business).

Edilex is proposing a template for contract to ensure nothing is missing and enforce structure to ensure clear expectation between each party. A contract need a table of content, its like a plan.

Proposed Template blocks:

Identification & location (name + birth->juridic name + rights applicable)
Party identification (Physic/moral-Society/union/coop ->representative liquidator Trustee/power delegation etc. )
Preamble (context used for fall-back defect clauses)
Lexicon (clarification/disambiguation & shorter sentences)
Object (simple/multiple;utility conditions/redaction)
Cost (adjustment/payment method/warranty/phase delivery etc.)
Attestations party A (not obligation/warranty-improve trust-don't want to fight about obligation and duty to disclose information)
Attestations party B
Reciprocal obligations
Obligations party A (Align with Business process/order of execution)
Obligations party B
Special provisions (orphan/specific/bi-directional)
General provisions
End of contract (resolution/termination)
Start of the contract
Duration
Scope
Annexes

So basically, why a template?

Clearly define what it include and isn't (You can't remove sections)
Provide table of content (Don't need to reread all the contract)
Find holes/unclear-possible point of conflicts
Reduce dramatically judge interpretation during conflict
Can help deciding to not get involve in the project (risk/client honesty)
Provide structure/uniform frame
Enforce clarity

I will try to apply some of those ideas and it is making me way less scared and equipped to sign new contracts. I really like this approach.

Leaky assomption and Gradient Descent- part 2/3

2009-11-18T18:59:00.000-08:00

Last February, I post the first part of this post.
Basically, I was pretending that “uncorrelated inputs” was a leaky abstraction and was the root cause of neural networks back-propagation poor results for training huge neural networks or deep networks. According to me, this simplification was fine while the number of parameters remains small. My hypothesis was that optimization problems are growing with the number of parameters which imply an implicit limit of the usage of this abstraction and explain those poor results.

During my research between 2001 and 2003, I was focusing on finding a way to train a neural network faster as presented on the left side of the figure. Unfortunately, I didn’t found that revolutionary algorithm but simply documented various effects of optimizations problems and ways to reduce or eliminate them with experimental results.

In 2008, I went to see Nicolas Leroux Phd defense and close to the end, he brought back that optimization problems could be the problem without presenting solutions which revive my research interest.

It reminded me a last crazy experimentation I have done in 2003, I found an algorithm that had the characteristic of the right side figure but it did not kept much of my attention at that time. Reducing optimization doesn’t imply necessarily faster classification error in time but should do it per iteration (i.e.: epoch).

During a year, I tried to reproduce that experiment in my free time. PA came to the rescue to discuss the underlying assumptions, to brainstorm and help reproducing the experimentation within flayers. Flayers wasn't suiting our needs anymore, the process to get back into the detailed of the implementation was reducing dramatically our experimentation throughput. We finally decided to drop flayers and rewrite it in python (optbprop) to ensure better collaboration and way faster experimentation. We had to make several optimization to make python speed acceptable but it was still slower then flayers (~10; see post). Even if it was slower, ultra fast experimentation become possible and research speed increased exponentially to try to recreate the experimentation. The complexity was residing into the order of the parameters optimization, what was the right recipe??

In June 2009, some times after ICML, Jeremy joined as the third collaborator and I finally reproduced it, results were even better. I have used an output max sensitivity ordering followed by a max hidden sensitivity backprop strategy.

Unfortunately, it was too good to be true and Jeremy found a critical problem in the solution which was leading to extremely poor generalization. At that point, motivation was too low so we’ve decided to stop our research. I was disappointed but a true relief, I could finally move to something else.

Without this collaboration team, I couldn’t have reached that point alone. I am still not 100% convinced that this research path is dead but it is back to a value that is way lower the motivation minimum threshold.

What could explain such bad results? We were expecting lightning learning speed and at worst little improvements compare to standard stochastic backprop. Our explanation hypothesis is that uncorrelated stochastic learning breaks implicit normalization process required for generalization. Basically, for each example, we update parameters to predict its class correctly but its too violent and unlearn previous examples way faster. Unfortunately, we loose the higher level goal which is generalization. Normalization could be re-integrate with bach learning but we haven't experiment it.

If this is true: “When you're ready to quit, you're closer than you think”, I might write part 3 of this post but It will take some time. There is an interplay between machine learning and optimization but people tend to forget it.

Toward better Web Monitoring Solutions

2009-11-16T21:32:00.000-08:00

Confoo, a web techno conference, will take place in Montreal in March 2010. If my proposal is accepted, I will present " Toward better Web Monitoring Solutions". Here is a summary:

Web applications are slowly becoming the new standard. No more installation nor upgrades, they are accessible from any internet connected device. While being the Holy Grail to users, web applications can be a nightmare to engineers, as ensuring quality of service becomes harder.

In fact, web applications create a high level of testing complexity, bringing new challenges to quality and availability of service.

As more and more businesses rely on web applications, techniques such as real-time web monitoring, incidents detection and root causes analysis have become critical.

We will present these new problems in detail, followed by a short history of techniques used to measure and estimate the quality of web-based applications. We will review the most popular monitoring technologies, pointing out their advantages and shortcomings.

This presentation will be done in collaboration with Sebastien Pierre.

Knowledge Workers - Talent is not patient, and it is not faithful

2009-11-16T20:10:00.000-08:00

Knowledge workers are impatient as great programmers. They are hard to replace and train and still some corporations are not proactive about it. It might not be costly enough?
Of course HR, managers nor VPs aren't struggling to compensate critical components when some of them leave but underlying teams have to. Those decision makers have to remember that they aren't free lunch and mismanagement of knowledge workers have some consequences.
Better management of Knowledge Workers lead to much more productive teams and low turn over but the opposite could cause your decline. What's so hard about creating a win/win approach instead of a loose/loose approach that so many companies seems to fall in. Maybe a generation clash? or simply missing competencies.

What's the relationship between Machine Learning and Data-Mining

2009-09-02T20:55:00.000-07:00

Machine Learning and Data-Mining are extremely related but it isn't clear for most people. I'll try to clarify the link in this short blog.

Let's start with definitions:

Data-Mining (DM) is the process of extracting patterns from data. The main goal is to understand relationships, validate models or identify unexpected relationships.
Machine Learing (ML) algorithms allows computer to learn from data. The learning process consist of extracting the patterns but the end goal is to use the knowledge to do prediction on new data.

Both, in ML and DM, we start by extracting patterns. In DM, the process ends there by looking a the patterns. In ML, we reuse learned patterns to do prediction.

One important difference about patterns extraction is that machine learning algorithms don't need to understand the representation of the patterns but data-miners do. As an example, it is hard to understand exactly what a neural network has learned but decisions tree are easy to understand and compare. On the other hand, comprehensive patterns allows machine learning practitionner to identify data problems and by fixing them, improve the prediction accurary of their model.

So basically, the data-mined patterns learned by any machine learning algos are used to do prediction on new data.

Some people might simply say that they are the same, the only difference is how you use the learned patterns: to understand or to predict.

note:

Unsupervised learning can be considered has data-mining because it doesn't involve prediction. In order to understand discovered clusters difference, we can simply use supervised learning on discovered patterns tagged datasets.

digipy 0.1.1 - Hand Digit Real Time Demo is available

2009-07-05T00:31:00.000-07:00

At Montreal-Python6, I have presented a real-time hand digit real-time demo.
This demo allows you to do real-time digit recognition from your digital camera. It allows you to load any trained neural network and apply in real time the same features extraction. This demo allows you to train, extract features, used trained neural networks inside real-time demo, visualize features in 2D and their frequency distribution and get feature discriminant weight.

The packaging 0.1.1 of the demo is now available on pypi:
(unfortunately, some dependency packages aren't supported by easy_install so you have to do 4 steps instead of 1)

install opencv (sudo aptitude install python2.5-opencv)
install PyQt (sudo aptitude install pyqt4-dev-tools)*
instal matplotlib (sudo aptitude install python2.5-matplotlib)*
sudo easy_install digipy

* unfortunatly, this package isn't supported with easy_install

Here is the noise robustness comparison of the trained neural network on the raw pixels vs extracted features (digit surface + image convolution with train digits means (0-9)):

If you aren't convinced that Feature Extraction is absolutely required now, I have failed.

Once installed, you will get access to those command line tools:

digipy: Real-time hand digit recognition demo application (ex: digipy --test)
digipy-features2D : demo of feature 2D visualization to see possible clusters
digipy-train: demo training of a Neural Network using mlboost
digipy-compare: compare noise effect on test error on raw inputs and feature extracted datasets
digipy-freq-analysis: demo feature analysis (frequency distributions)
digipy-extract-features: demo features extraction
digipy-see-data: show dataset train and test samples

If you have any trouble using it, just let me know (fpieraut at gmail).
Source code is available here http://bitbucket.org/fraka6/digipy.

(note: now digipy use mlboost.nn module for its NeuralNetwork instead of mlboost.flayers swig wrapper)

ICML highlights summary

2009-06-17T19:34:00.000-07:00

Let's try to summarized in several sentences what I have learned:

Language acquisition: Children loose their capacity to distinguish some phonemes to reduce the scope of choices in order to learn their environment language. The aquisition of phonemes categories from a buttom up approach isn't sufficient (signal processing+unsupervised clustering), a lexical minimal pair (ex:ngram) seems to be required to ensure the learning.
Trying to learn the best kernel that restrict optimization to a convex problem seems to be a death end. It might be time change paradigm or move to the non-convex dark side.
Boosting is too sensitive to noise but a robust framework has been presented by Yoav Freund
Deep Architecture seems to be the next big thing. Regularisation, auto encoder and RBF can be used to pre-train networks from un-label data. Temporal coherence (similarity of consecutive frames in video) can be used as a regulation unsupervised technique in the embedding space. Unsupervised training is a regularisation technique that enforce better clustering. The more unlabeled unsupervised examples are used, the better will be the generalization.
Training from IID samples isn't optimal, curriculum learning (i.e.: increase examples complexity) seems to smooth the cost function and lead to faster training and better generalization.
GPU is the way to go to make ML algo scalable.
Feature hashing is an efficient strategy for dimensionality reduction and can be used to train classifiers.
Sparse transformation simplify the optimisation process (i.e: same idea used in the Kernel trick in SVN). PCA is doing the opposite.

Research has stopped in Neural Networks because we couldn't estimate boundary error due to its non convex cost function, there was no more theoretical framework. SVN came to the rescue by providing 3 major benefits, a convex cost function, a better generalization process (margin maximization) and less parameters tuning.

Unfortunately, it doesn't scale well, kernel is hard or impossible to choose to reach optimal solution and it doesn't allow deep architecture. For the same capacity, a shallow architecture needs more neurons then a deep architecture and large shallow architecture are much more likely to numeric issue.

Deep architecture came back with convolution deep neural networks applied to objects recognition and them Hiton proposed a breakthrough, a generative approach to initialised the parameters.

Unsupervised learning lead neural networks to much better initialization state and its regularisation provides better generalisation. But now, even if we are doing better initialisation, we still aren't able to better explore the function space which leave, according to me, still open the question: is this an optimization problem?. Local mimima observe might be an illusion created by the effect of gradients cancellation from opposites gradients which is an optimisation problem induce by the leaky assumption of uncorrelated features which lead people to optimize all parameters at the same time. ICML was inspiring.

International Conference on Machine Learning (ICML2009)

2009-06-10T19:51:00.000-07:00

The 26th International Conference on Machine Learning (ICML 2009) will take place in Montreal next week (14-18, 2009).

The 3 invited speakers are quite interesting. I look forward to heard them:

Emmanuel Dupoux, from Ecole Normale Superieure on: How do infants bootstrap into spoken language?
Yoav Freund, University of California on Drifting games, boosting and online learning?
Corinna Cortes, from Google on can learning kernels help performance?

I have do decide to which tutorials I will attend this sunday:

T6 Machine Learning in IR: Recent Successes and New Opportunities [tutorial webpage]Paul Bennett, Misha Bilenko, and Kevyn Collins-Thompson
T8 Large Social and Information Networks: Opportunities for ML [tutorial webpage]Jure Leskovec
T9 Structured Prediction for Natural Language Processing [tutorial webpage]Noah Smith

Here are some interesting papers:

Curriculum Learning [Full paper]
Deep Learning from Temporal Coherence in Video [Full paper]
Good Learners for Evil Teachers [Full paper]
Using Fast Weights to Improve Persistent Contrastive Divergence [Full paper]
Online Dictionary Learning for Sparse Coding [Full paper]
A Novel Lexicalized HMM-based Learning Framework for Web Opinion Mining [Full paper]
A Scalable Framework for Discovering Coherent Co-clusters in Noisy Data [Full paper]
Bayesian Clustering for Email Campaign Detection [Full paper]
Feature Hashing for Large Scale Multitask Learning [Full paper]
Grammatical Inference as a Principal Component Analysis Problem [Full paper]
Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations[Full paper]

I have to decide between thoses workshops on Thursday:

I look forward to meet old collegues, friends and new researchers. Next week will be awesome.

Is Python really slow? A practical comparison with C++

2009-05-29T20:23:00.000-07:00

The common perception is that Python's implementation is slow, but you can often write fast Python if you know how to profile your code effectively.

I have tried it. I have compared a hight cpu intensive algorithm, the training of a simple one hidden neural network. To do so, I have used my old C++ NeuralNetwork library (flayers) and an implementation in python with Numpy. I have wrote a simple neural net in python and optimize all loops with numpy as suggested in a profiling presentation saw in Pycon2009.

I have compare the training time of a simple fully connected NeuralNetwork will 100 hidden neurones for 10 iteration on letters dataset (cost function = mean square error).

Here is the time to do 10 iteration with flayers (c++):

./fexp / -h 100 -l 0.01 --oh -e 10

...
Optimization: Standard
Creating Connector [16|100] [inputs | hiddens]
Creating Connector [100|26] [hiddens | outputs]
...

real 0m11.187s
user 0m10.837s
sys 0m0.012s

Here is the time to do 10 iteration on the full letters dataset with python:

time ./bpnn.py -e 10 --h 100 -f letters.dat -n
Creation of an NN <16:100:26>
...
real 85m48.646s
user 85m9.163s
sys 0m1.632s

Here is the time to do 10 iteration on the full letters dataset with python and numpy:time

./bpnn.py -e 10 --h 100 -f letters.dat
Creation of an NN <16:100:26>
...
real 1m37.066s
user 1m36.026s
sys 0m0.100s

So if you do the math:

The numpy implementation is 60 time faster then a basic python implementation.
My C++ implementation is a little more then 10 time faster then my simply python numpy implementation.

Numpy implementation definitly worth it because it reduce the code and has a significant performance impact, the C++ might be required for extreme performance but the trade off of code complexity and time my not work it. Now that I have the choice, I will still use my C++ lib.

Short Essay: Engineering vs Scientist

2009-05-10T20:40:00.000-07:00

Engineer vs scientist difference is obscure for many people, me included at the beginning.
Already, during a chat with a research professor in 2001 about the competence war between the engineering and the science department of university of Montreal, I got an initial hint about it, he told me that part of computer science department tension with software engineers was about placement rate. Engineers was much higher than computer scientists. I should have request an explanation. What is this war about, the computer science department request engineer to do some normalization courses even if they have great grades and vis-versa.

In my short carrier, I have observe a major difference that I will try to expose with examples.

After my degree in engineering, I have done most of my graduates courses with scientists. In my first final exam, after some discussions with classmates, I realized that I was the only one that have used an estimation to solve a bottleneck calculus in a problem. Most of the other have lost more than half an hour on solving it exactly and couldn't finish the full exam.

Several times, I add to step in projects to set schedule in order to force people to cut some corners. Perfection is an endless path. If you have to fill a vase with rocks, sand and water, you will most likely start with the rocks then the sand and then with the water if the effort is inverse proportional to the size no? Yes it might be more interesting to optimize cool things and advances features but most likely they aren't part of the basic blocks required to allow your project to move to the next step.

What is the point of having half a perfect solution if you could have done a imperfect full solution. I am not saying that all scientists need of perfection leads them to lose the big picture and engineer aren't perfectionist but simply that bias engineer system vision leads them to gage better when it is time to look for perfection. Time kill projects and over-perfection can drag too much of it...but perfection should remain the goal.

Yes I am generalizing and simplifying but you get the big picture. Define your priorities and assign resource accordingly. So, am I a engineer or a scientist or a little of both? since my manager told my I was more a scientist, I should be a little of both...

Montreal Python6 & Peer Programming

2009-04-20T21:01:00.000-07:00

Montreal Python 6 demo packaging and UI major improvements are the results of a peer programming with apt-get install ygingras. In this world where time is the biggest constraint, I could not finalized some of the demo improvements and packaging without help. bitbucket, the mercurial host projects has been very useful to store demo parts (i.e.: flayers, mlboost and digipy). I am not working often off-line, but been able to commit in the plane or in the bus, exchange dundles and create branches easily has very useful.
You can play with the demo, just type:

easy_install digipy (on your favorite linux distribution)

Simply type digipy and tab to see all digipy programs used in the live demo (i..e: use --help to get more details). I have been impress by the powerful setup.py related distutil tools of python to create packages (sdist, register, upload to push code on the Python Package Index is just GREAT!)
Virtual Environment, a tool to create isolated Python environments, of Ian Biking has been really usefull to test the packages.

Peer programing is the type of approach that can bring the productivity of your team from the sum of the productivity to the multiplication of its individual or a best to the team member count exponential relationship. Peer programming has allow me to get out fast of many extreme time consuming techical show stoppers that were burning out my too little available time.

Highlights Pycon Chicago 2009

2009-03-29T20:34:00.000-07:00

My short trip to pycon 2009 just confirmed my choice. This community is living and has a powerful momentum. Guido, the python father, mentioned that the 3 most important things are: community, community and community. What he started 19 years ago seems to take over.
The simple python syntax lead everyone to converge to it: pypy, jpython,cpython and IronPython. Alex Martelli talk on abstraction as a leverage was great: abstraction is inevible; try to understand at least 2 lower layers and create hook instead of hacks! (I am already applying it for the demo of python-montreal).
The frameworks for the web development is getting quite amazing (Django, Whirldwind, pylons etc.)
The talk " A Whirldwind Excursion through C Extensions" of Ned Batchelder was great to get a quick start on own to create C extension; yes python is slow and optimization is sometime inevitable.
The panel on Object Relational Mapper and the talk on Drop ACID and think about data were quite interesting. The keynote presentation by reddit founders is a good example that python provides amazing tools to spine out web application companies.
The concept of the evening lightning talks and open discussions on topic of interest was showing the dynamism of the python community. I went to an open discussion on parallel computing and people are moving to python. Now that the process lib has been integrate in 2.6, Twisted, Thrift, PyMPI, Numpy, Scipy power computing with python will expend.
I am glad that Yannick convinced me to attend. I think that I will attend to the tutorial next time and some Sprint to ensure the maximum knowledge assimilation.
During this small trip, I tried the mini laptop eee ha but I brought it back, you should buy the HE: the right shift is at the right place and the battery last longer. Pycon was inspiring !

Research, information management, contrastive divergence, no free lunch theorem, non-parametric...all related

2009-03-23T18:38:00.000-07:00

During my master studies, I had to take a course on Research Methodology which introduced me to an interesting concept of information management. With our limited brain capacity, more you read papers about others ideas, less space is available for your own ideas and enforce other assumptions, vision and models.

In practice, great researcher are aware of this consequence of the no free lunch theorem and try to keep of good balance of papers reading and research exploration. By simply applying the contrastive divergence concept of your approach, you can gage your distance to the trend and get a estimation of the impact of a possible discovery.
The Research Machine Learning community, as most other community, has the tendency to recruit top grades students that are use to follow exactly the line of thoughts of their teachers. This long training process is, according to me, extremely damaging to the training of the researcher capacity (i.e.: suboptimal cost function). This explain why most of the master student are researcher cheap labor driving force because they can only experiment others ideas with minor contributions.
Top researchers allow their formal students to follow their own line of thoughts or if they have no specific ideas, suggest ideas. I won't have done a research master without this freedom, thanks Yoshua!
So, if you want to impact the most your community, limit the number of papers your are reading, make your own ideas and play with your concepts to train your own intuitions of the unknown guiding rules you are looking for.

You might say, what's your contributions, I haven't heard about it. My contribution is that I have build experimental proof of back-propagation optimization fundamental problems and build the skeleton of top level explanations. Usually, we don't publish this type of results until you find a solution to the problem which, unfortunately, I haven't reach but, it is coming slowly; it is a long process and I learned to be patient.

So, if you want to impact the most your community, limit the number of papers your are reading to ensure you don't constrain yourself to others models. Why using a parametric model that limits your solution space?

You can trust the collective research discovery learning process that ensure the evolution of the human kind because someone will find it or, use it to increase the likelihood your will make a important discovery (i.e.: use it as a contrastive divergence cost function). If everyone was applying this strategy or cost function, I am pretty sure we will evolve faster. In order to move to this step, we will need to encourage failure strategy publications to ensure other don't wast time reproducing the same ideas but this could be elaborated in another thread post that involve a society evolution.

Traders knows about this simple strategy, buy low, sell high, don't follow the trend, take risks.

Pycon 2009 Chicago

2009-03-15T21:13:00.000-07:00

It is about time the see the real face of this python community. I will attend to pycon 2009.
In 3 days, I expect to learn more things then I have learned in the last 6 months and met pationated peoples. Here are some of the talks I will attend:
Designing Applications with Non-Relational Databases (#16)
How Python is Developed (#116)
Twisted, AMQP and Thrift: Bridging messaging and RPC for building scalable distributed applications (#40)
Introduction to Multiprocessing in Python (#6)
The State of the Python Community: Leading the Python tribe (#118)
Google App Engine: How to survive in Google's Ecosystem (#53)
A Whirlwind Excursion through Writing a C Extension (#68)
Abstraction as Leverage (#110)
A winning combination: Plone as a CMS, your favorite Python web framework as a frontend (#100)

Greedy agile, waterfall and local minima

2009-03-15T21:05:00.000-07:00

Everyone in the agile world can't contain their words about waterfall inadaptation to real world software projects. I have to admit, I am a fan of XP, scrum and most of agile approaches in general but I feel that people are loosing the big picture provided by the waterfall framework. Agile is kind of a greedy approach that lead you to local minimum where you are stuck too often. I would like to see someone creating value and velocity cost trade off of short term decisions. Those decisions are ofter push through agile methodology without questioning. Currently, at pivotalpayments, we are stuck in a huge local minimum created by such an approach and it will take such an effort to get out of it. The velocity created by a design choice taken long time ago was emazing at firt but it is slowing us doing now so much now. A major refactoring is required and we will stop development for some iterations. At least Agile adapts and give you the illusion of the optimal path...but everyone should know that the greedy approach isn't the optimal one and that looking at your feet won't help much going to your destination.

Montreal Python 6: 2009-04-14; Machine Learning empowered by Python

2009-03-04T21:55:00.000-08:00

It is now official, I will do the next Montreal-Python presentation. I will be back from holiday the 13, I hope I wont get flight problems.

Our main presenter will be Francis Piéraut on Machine Learning empowered by Python as announced during the flash introduction in Montreal-Python 5.

Machine Learning is a subfield of AI that considers learning patterns from existing data. Related applications are increasing in many fields where adaptive systems are needed, like fraud detection, face recognition, recommendation systems, disambiguation systems, insurance risk estimation, web traffic filtering, voice recognition, and many others.

The first part of this presentation will cover the basics of machine learning; in the second part, we will dive into a real example and see the complete process of using machine learning to create a real-time digit recognition system using Mlboost, a python library. The practical approach should allow the audience to assimilate the most important concepts of machine learning and the critical need for data preprocessing.

After a Software Engineer degree, Francis Piéraut made a research master in Machine Learning at LISA. During his research work, he developed flayers, a powerful C++ neural network library. During the beginning of his career, his spend several years in Montreal startups companies applying Machine Learning and statistical AI related solutions. In 2005, he released the first version of MLboost, a python library that allows him to speedup his Machine Learning projects by simplifying data preprocessing, features selection and data visualization.

Essay on Adaptation, leaky cost function and online Learning...a society analogy

2009-03-04T19:38:00.000-08:00

I fall on the Paul Graham article on cities and ambition and it made my think of writing this blog on adaptation, leaky cost function, online learning and an analogy to the sinking French Quebec society.

To make the bridge with the 3 first concepts, I will use a analogy with the Quebec society.

In order the learn, we need adaptive systems as Neural Networks. In online learning, the adaptation capacity should stay constant along the time. Local minimums can screw you up but let's ignore it for the time been.

Quebec society analogy intro

In order to understand my analogy with the society I live in, I want to share some of my reflexions about the puzzle to understand the Quebec Society. I am born in France and I migrated to Quebec at 9 years olds. During the last 5 years, I tried to elucidate my profound incomprehension of the deep ambitions of the French Quebec Society, if they have some;). According to me, it seems to be a leaky cost function assumption that lead them to their stagnation, coming assimilation and their slow extinction.

To understand my point of view, we need to elaborate on key concepts which are adaptation, equality versus inequality and education access.

Adaptation pros and cons, local minimal and ambition

Adaptation is one of the greatest ability of the human kind but also one of the worst. Adaptation to mediocrity can be a survival strategy to get through hard time but getting use to it reflects true low ambitions or incapacity to do online learning. The Quebec nation seems to have this disease.

Quebec people accept staying in a destructive mode seen 1982, a constitutional status quo that lead to politic instability, economic stagnation and reduced political power by excluding them self from the Canadian power with the Bloc Quebecois for too long.
Quebec people accept the status of a sub nation (nation inside the Canada).
Quebec people accept mediocre governments, mediocre public transport systems, way too expensive and inefficient heath system, highest taxes in north america etc.

Quebec people seems to have no real will for improvement, it is doing the job (you should see "Le confort et l'indifférence"). In the machine learning point of view, is that a local minimal? What is the problem with Quebec society cost function? Is that only the lack of ambition? Knowing that close to half of the population have voted for the separation of Canada in the 95 referendum, it might shadow something more rooted to the French culture.

Cost function assumption : Equality versus inequality

From an anthropology point of view, French nuclear family lead to a conception of a world of equality (see Emmanuel Todd). to simplify, everyone should have the same chance, same access to education, same heath services and so on.
The Anglos-Saxon culture lead to the conception of an inequality world. The inequality conception lead people to work harder knowing there is no lower boundary and they can go deeper if they are too lazy.
Knowing we are born unequal, Anglos-axon assumption conception seems to be better adapted to human kind reality. On the other hand equality lead to an education level increase of the society indepedandly of the economy which as lot of pros and cons.
Why equality is a weak assumption? Equality can stand in rich societies because they can afford it. Unfortunately, Quebec society is getting poorer and its population is disadvantaged by its illusion of equality that leaks from everywhere (i.e.: health system, education, public kid garden etc.)

Missing link: Education and production of wealth

Quebec has the most affordable access to education in north America and few take advantage of it. Anyone knows that the more educated is your society, the more productive and healthy she will be and the more accessible the utopia of an equality world can be possible. By using education to get more productive, a society will create wealth and can afford utopia as the equality concept. French society seems to miss this key point.

French Quebec society is dying and the Anglo-Saxon supremacy should take over

The leaky equality concept and the low ambition of the Quebec society seems to lead this society to online learning incapacity, its incapacity to adapt further. This incapacity leads to its extermination by the growing assimilation to the Anglo-Saxon supremacy of its global cost function model. Inequality based cost function seems to be better adapted for a society that want to stay alive.

Wrap up (it is time to conclude)
Leaky cost function assumption can lead online learning to adaptation incapacity like been stuck in a local minimal as a slow death as the French Quebec nation folklorisation current process. It is simply the evolution, a Darwin consequence, who can't adapt simply die. Facing reality makes life easier.
The most important thing is that the cost function should reflect your goals. If you have a supervisor, try to get a good estimation of its cost function because it will simplify your ascention everywhere.

Quebec French culture creates a huge retention for me to stay in montreal but I wish Montreal a better drive for machine learning, startups as you can get in California. Montreal is simply sub-exploited. Don't take my words for granted, it is an essay. Make you own judgement from your own eyes and exploration.

Leaky assomptions and Gradient Descent

2009-02-24T20:16:00.000-08:00

Theoretical models are based on strong assumptions as software layers (i.e.: leaky abstractions). Layers are created to simplify complex systems and allows work specialization as define in Marx capital approach...and to accomodate limited human brain capacity.
Experienced practitioners know that their value reside in weak assumptions dept comprehension because market will hire new grade students otherwise.

Standard backpropagation gradient descent algorithm assumes that inputs are independent so we can optimize them independently of each other. This assumption or according to me a leaky abstraction, allows you to optimize all parameters at the same time which simplify the life of software engineers and researchers because parameters are theoritically uncorrelated. In mathematical works, we are assuming that the hessien matrix has values only on its diagonal.

My master thesis done under Yoshua Bengio supervision was mostly focusing on understanding huge neural networks training inefficiency. At that time, our goal was to train neural network language models. According to my undertanding and the experimental proof that I have documented, the problem is basically an optimization problem. The uncorrelated assumption simplification doesn't stand when parameters numbers explode.
Unfortunatly, I have failed to find a solution to this problem but the new trend in reaction to Hinton break throught in 2006 will and is already reviving research in this topic.

In my literatude review, I found that several researchers identified some of the reasons who can explain this inefficiency. According to me, they are direct and indirect consequences of the optimization problem introduced by the leaky abstraction of uncorrelated inputs. Those reasons are the moving target problem and the attenuation and dilution of the error signal as it is propagates backward through the layers of the network. We present in my master thesis other reasons who can explain this behavior, the opposite gradients problems, the non-existence of a specialization mechanism and the symmetry problem.

I will treat those concepts in a futur post. This inspiration of this post has been possible because of a brainless Hollywood movie that has allow me to free valuable brain cycles. there is always a good side of the story.

Newton law doesn't stand in Einstein theory as uncorrelated inputs in huge neural networks. Always remember your leaky assumptions/abstractions.

cygwin or mingw to compile C++ swig projects on windows?

2009-02-09T20:45:00.000-08:00

The answer is definitely mingw. 4 simple steps:

download swig
download mingw
add python, swig, mingw to environment variable PATH
do: python setup.py build_ext --inplace --compiler=mingw32

I have been able to compile it easily with cygwin but I had to download shit load of stuff and I haven't figure out how to use python windows packages and cygwin python package all togetter which was a show stopper for me.

Why was I looking to compile my C++ project on windows?

I want to use my C++ machine learning lib on a real-time video and the only package that was working to grab images in python was videocapture but it is only supported on windows. Now I have to compile my machine learning on windows...

If you want to compile swig C++ project, you need a compiler. If you don't have VisualC++, you are stuck with this error:
error: Python was built with Visual Studio 2003;
extensions must be built with a compiler than can generate compatible binaries.
Visual Studio 2003 was not found on this system. If you have Cygwin installed, you can try compiling with MingW32, by passing" -c mingw32" to setup.py.

At this point, you can try to download free visual C++ compiler or try mingw32 with sigwin. I tried a free visual studio 2003 package but I couldn't make it working. Then, I try cygwin with gcc and it was compiling but I couldn't use the compiled package outside cygwin. so I tried mingw32. The -c option doesn't work and "python setup.py build --compiler=mingw32" doesn't allow you to use your package inside python (i.e.: can't import _flayers in my context). Finally I tried python setup.py build_ext --inplace --compiler=mingw32 and it was working.

After a chat with Simon and Tristan that are doing video stuff on linux at the SAT, I discovered that they were grabbing video on Linux. They referred me to opencv which works perfectly. In my experimentation with opencv, I realized that Pygame is a million time faster to display video then matplotlib.

Swig, Mingw, OpenCV make Python so convenient!

Startups and faster Learning

2009-02-06T22:21:00.000-08:00

Every one that is familiar to gradient descent and Machine Learning knowns that learning is proportional to the error. Bigger are the errors, bigger are the possibility to learn something. According to me, startups, by nature, are the ideal environments to put yourself in hight gradient learning situations. It force you to try much more things, take more risk and innovate to survive if you don't want to have no choice to look for another job and/or get penny stocks options. Another interesting thing is that true nature of people appear and mask fall rapidly. Friendships are tested to its limits and it allows you to filter out short term life partner.

Don't forget that high gradient put high constraints to your body and frustrations has to be released somehow. In my case, cycling has been used as stress balancing vehicle, you have to find yours. Resigning is too often chosen as the latest resort. You have to keep in mind that everyone is replaceable, even in startups. If you really want to be rich, you should do the Californian technique: cumulate stocks of multiple startups. don't forget that most of them (~90-95%) fail badly, so increase your chances by switching before reaching burnout and build your contacts network by the same way. After the graduation is definitely the best time to try this fast growing experience when you can take it. On my side, I tried it 5-6 years, you have to know your limits. Successful startups have workaholic CEO so it will increase the pressure and the gradient. If he isn't one, find another one, it should be the 19 mistakes of startups (18 mistakes of startups).

If you have the chance to access higher decision position, you learn that there are irrational error cost function and that you might be optimizing the wrong error criteria. For the ones who aren't following me, I recommend that you see the French movie 99 Francs and pay further attention the the high executive meeting.

I apply the same principle with my son this winter, he doesn't want to put his gloves, he learn/change his mind faster at -10 degrees Celsius.

MLboost 0.3 has been released

2009-02-01T20:10:00.000-08:00

I am please to announce the release of MLboost 0.3. MLboost 0.3 will be use for the next Montreal-Python presentation ("Machine Learning enpowered by Python"). It will be announced in the coming weeks. The most important new feature is the integration of pyflayers. pyflayers is a simplified python swig interface to flayers, my C++ neural networks library. I am preparing a live real-time demo of a machine learning application, it should be interesting for the audience and it is good motivation to improve my package. I have played with beamer, a latex package to create presentation slides and I have been particularly been pleased, I do recommend it. On the other hand, it is time to prepare the next winter camping trip and to continue eating great food.

The No Asshole Rule Fraka6 review

2009-01-05T18:57:00.000-08:00

During Christmas time off, I finished "The No Asshole Book" from Robert I. Sutton. I fall on this book by the time is was looking for the "Art Of War" for a gift. At first, I taught it was a mistake and that it will end up been a light or low content reading but it was extremely interesting. The book treats:

How to identify a real asshole
The damage of keeping Assholes
How to implement the No Asshole Rule
How to stop inner jerk from getting out (yes it get out sometimes)
How to survive when assholes reign
Yes I am a powerful Asshole (I was checking if you were really reading;)

Having spend 5 years in startups environment, I should have read this book before because high stress environment leads people to act as asshole and applying some of this ideas can help you surviving this jungle. Out of 4 startups, I had to deal with 3 places where asshole behavior was destroying my productivity. Only one place had a certified asshole (not a temporary one; no it isn't Eric). What I like about high educated environment is that people are more likely to stand up and make changes happening. I saw one of my supervisor been fired after I stand up and another been fired on the side after I left which is overall very encouraging. What I learned from this book is that once, I have waited way too much time before changing jobs because I didn't realized soon enough that I was simply wasting my time. There is times where fighting is useless and it simply drains you down and lead you to act sometimes as an asshole yourself to keep you breathing (don't forget small wins first). Good faith is sometime useless, as Robert enforce: Hope for the Best; Expect the Worst and Change the way you seen things (Reframing). Reframing refer to developing indifference and emotional detachment. As highlighted, people tend to recruit high passioned people but it isn't a so good idea because it leads them to irrational decisions that can have critical impacts.

The book doesn't treat one point that high technical people forget. For most company, market drives the show, not the technology, so you have less power then you think. This market lead to more politics and assholes are good at this game. Politics is often use to shadow increasing incompetency because they don't focus on what they should be doing. You have to better understand politics games but according to me, the more politics the more incompetency by square foot is expected.

This make me remember Joel On Sotware book, most years top companies don't last, only a few survive in the top n after many years. I am pretty sure those company apply the No Asshole Rule because if they don't do so, long term they get back what they desire because they are infected by reproduced assholes that blame each other by the time all top people leave.

People forget that power pass by empowering others. This power isn't a sum but can't be exponential with teams that have a great synergy. Open source is a great example.

This book is highly recommended. I will suggest it to my manager, one of the few manager that understands what a manager should be doing.

But what is an Asshole, the standard definition is something like that: insulting terms of address for people who are stupid or irritating or ridiculous. According to the book Asshole follow 2 rules:

After spending time with them you feel de-energize, humiliated etc.
Asshole are more polite with more powerful people and are worst with lower level people.

Don't forget, be slow to brand certified asshole. To get the certified brand, you simply have to be constant.

Sip vs Swig?....Swig

2008-12-30T19:57:00.000-08:00

Yannick suggested me to try sip before getting too far with swig. I have followed religiously the doc but I haven't been able to make it working with my lib. I stopped at the same level of complexity of what I have reached with swig (<=1.5 hrs to make it working with flayers). sip -c . pytrainer.sip
sip: Trainer is undefined

Basically, sip can't find my class Trainer and I haven't find any doc about it neither clear mailing list answers.

I have try the configure.py too but I get the error that I don't get with their example (in both case .sdf file doesn't exist).

Error: Unable to open "pytrainer.sbf": [Errno 2] No such file or directory:
'pytrainer.sbf'

Conclusion:
Swig is my choice at this point because it is simple to use, well documented and simply works.

Why I have switch to Python?

2008-12-26T21:14:00.000-08:00

In 2005, by the time I was working as a "Machine Learning Specialist" at Coradiant, I had to do lot of preprocessing. At that time, I was doing more data-mining then machine learning. The fundamental difference is that machine learning use model to do prediction and data-mining use mostly model to understand links between variables (no prediction is required). In order to do my work, I chose Matlab because I love their visualization functionality and it contains a machine learning package that includes a Decision Tree module that was perfectly suits my need to do root causes analysis. Unfortunately, Matlab is a disaster to do preprocessing and this task was representing the biggest part of my work on that project and C++ isn't better. Doug Eck, a Music and Machine Research consultant, introduced me to Python to help me doing the preprocesssing work. What a discovery! A year later, I released the first alpha version of mlboost.
Later, I worked on a real machine learning task, building a bot detector that give a bot probability base on sessions information. Without python and mlboost, our real-time bot detector prototype would had never be a reality within a so tight schedule.

Later, with the discovery of numpy, scipy, matplotlib (with pylab interface, you get the same api as matlab), python-mysql and ipython, I don't need anymore matlab. With pydev eclipse plugging, I have a powerful python development environment.

So basically, I have switch to python because:

I can do extreme prototyping at a speed that I have never seen
I have an amazing community (matplotlib, pydev, scipy etc.)
The language amazing syntax allows me to write code with minimal number of lines in an elegantly and readable way (ex: list comprehension, *argv, **args, mocking etc.)
The simplicity I can do wrapper with non python libraries (ex: swig)

Here is interesting links:
Matematica vs Matlab vs Python (same fct comparison code)
How to Install Python as a Replacement to Matlab
pythonxy package Python(x,y)
I like this figure to represents to community over python.

Swig is just allsum!

2008-12-23T21:13:00.000-08:00

Today, it only took me 2 hrs to integrate load and train function to my new flayers python interface. I am just impress that it is working so easily. I am simply convince that swig guys have done an allsum work! Thanks guys!

svn co https://mlboost.svn.sourceforge.net/svnroot/mlboost/flayers flayers
cd flayers
sudo python setup.py build_ext --inplace
ipython
In [1]: import flayers
In [2]: trainer = flayers.loadTrainer('test.save')
In [3]: trainer.train(10)

python, matplotlib, ipython, scipy and now swig are making my life much easier.

How I have created a python module from a C++ lib with swig

2008-12-21T21:12:00.000-08:00

Today, I have completed the first step to create a python module from my C++ library. The first function I want to call from flayers is the main wrapper (fexp.cpp:main(int argc, char** argv)). From the new flayers python module, I expect to be able to call it that way :

import flayers
flayers.fexp(["fexp","/","-e","10","-h","100","--oh","--lsm","-l","0.01"])

Unfortunately, it is a little bit more complicated then expected. In order to make it working, I have looked at this great documentation (doc). Here are a summary of issues I have encounter.
1) Extension module doesn't work automatically with C++. You have to add "-c++" option to swig call (ex: swig -python -c++ flayers.i).
2) I added the .cxx file generated from swig instead of .i file to the source file in the setup extension instance (variant from my previous blog post).
3) I have added flayers dependancy classes to the extension sources to ensure I didn't get error like : ImportError: ./_flayers.so: undefined symbol: XXXX and ensure the lib is recompiled at the installation.
4) How to pass the list of options without argc and not get error like "in method 'fexp', argument 2 of type 'char **'": see section "30.9.2 Expanding a Python object into multiple arguments" ? look at my setup.py

You can take a look at my files here (flayers.i, setup.py, flayers.h,flayers.cpp)
I am impress, swig and python are just allsum.

If you want to try it, download flayers and do:
svn co https://mlboost.svn.sourceforge.net/svnroot/mlboost/flayers flayers
cd flayers
python setup.py build_ext --inplace (or python setup.py install)
ipython (or python)
import flayers
flayers.fexp(["fexp","/","-e","10","-h","100","--oh","--lsm","-l","0.01"])

swig and python with Extension module

2008-12-14T19:06:00.000-08:00

There is a much simpler way to create interface to C/C++ code. Simply use Extension module from distutils.core (setup.py).
You simply have to type sudo python setup.py install and you can access your module in python. (import example; example.cube(3))

Thanks to PA recommendation.
My next steps are to create a python interface to flayers (my C++ neural network library) and convert arguments. PA suggest boost.python for arguments conversions.

I have also experience setup module from setuptools to create mlboost setup.py file. setuptools use indirectly easy_install that doesn't work as well as aptitude on ubuntu. Unfortunatly, matplotlib dependency doesn't work but it is included in scipy through mlab interface.

Swig and Python first experimentation

2008-12-13T22:17:00.000-08:00

Today I tried swig to interface python to my C++ library.
According to this example it seems very simple but of course it doesn't work.
I have followed exactly the proposed steps and I got 3 problems:

can't find Python.h: solution->sudo aptitude install python-dev
ld -shared -o _example.so example.o example_wrap.old: example.o: relocation R_X86_64_32 against `a local symbol' can not be used when making a shared object; recompile with -fPIC : solution-> do it
ld -bundle -flat_namespace -undefined suppress -o _example.so example.o example_wrap.o
ld: -f may not be used without -shared :->remove -bundle and add -share option

Here is my config:
AMD Athlon X2 5400+ Dual-Core Processor
unbunto 8.4.10
Python 2.5.2

Fast track solution:
svn co https://mlboost.svn.sourceforge.net/svnroot/mlboost/demo/swig swig-example
cd swig-example
source create_example_python_interface.sh

-------------------------------------------------------------------------------------------------------------------
Now here is exactly what you should do to make it working:
-------------------------------------------------------------------------------------------------------------------
1) create example.c and example.i
example.c
example.i

2) create python interface to example module that contains cube function

sudo aptitude install swig
sudo aptitude install python-dev
swig -python example.i
gcc -c example.c example_wrap.c -I/usr/include/python2.5 -I/usr/lib/python2.5 -fPIC
ld -shared -o _example.so example.o example_wrap.o
ld -share -flat_namespace -undefined suppress -o _example.so example.o example_wrap.o

3) try python interface

fpieraut@fraka7:~/swig$ python
Python 2.5.2 (r252:60911, Oct 5 2008, 19:29:17)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import example
>>> example.cube(3)
27