Miscellaneous

Sanity checklist

This is a list of things that you should check to debug some common problems when using machine learning to solve practical problems.

Ideas

• Machine Learning experimental paradigm, do not train on the test set.
• To explain or to predict? Galit Schmueli
• Checking machine learning blog

Steve's thoughts

Workshop goals:

1. common language
2. pointers for life scientists to tutorials
3. where to apply ML

Basics:

1. features and targets
2. model capacity and over fitting (bias/variance)
3. ground truth and data set bias

Andrew Ng's advice

Andrew Ng had a stanford course slides here which is now part of his Coursera course (currently Lecture 10). He covers:

1. Diagnostics for debugging learning algorithms.
2. Error analyses and ablative analysis.
3. How to get started on a machine learning problem

Coded up in an ipython notebook at astroml.

Machine Learning Pilot School

Instead of teaching people how to build machine learning tools, there is an alternative audience of researchers who are users of machine learning. This is analogous to people who design and build airplanes and the pilots who fly them. Both are highly skilled occupations, but with different skill sets.

Fighter pilots

These are highly specialised researchers who need to know a broad range of tools, each of which may be very specific to a particular type of task. The user needs to be agile as there is often an imminent deadline, and can mix and match different tools. The general problem is often not known up front.

Commercial pilots

In many applications, the problem is well specified, and it is the user interface that it is important. It is up to the user of machine learning to present a comfortable and safe environment to the non-experts, for example decision makers.

Freight pilots

Here the problem is also often well specified, but efficiency and cost are paramount.