Linear Algebra Cheat Sheet for Machine Learning

Familiarity with Linear Algebra helps quite a bit while working on Machine Learning algorithms. As Software Programmers, we don’t get to do lot of serious Mathematics on regular basis tend to forget the principles. I would like to list some of the concepts here.

Matrices & Vectors

Starting with basics, A Matrix is a rectangular array of elements. Typical example is:

A Matrix is denoted by Number of Rows * Number of Columns. The above Matrix is normally called a 3 * 2 Matrix.
Similarly Vector is another often used Entity in Machine Learning algorithms. It’s special type of Matrix in the sense it always has only one column. It also called a n * 1 Matrix. A typical example of Vector would be:


Entries of Matrix or Vectors are denoted by A_{ij} where i is number of row and j is number of column. So in examples above, A_{32} (Matrix) will be 4 and a_{31} (Vector) will be 5.

Matrix Addition

We can perform addition operations over 2 matrices. In this, we add element at position i * j in Matrix M1 with element at the same position in Matrix M2. Let’s take an example :

We can perform subtraction in similar manner. One point to mention here is that we can perform these operations only on matrices of the same dimensions.

Scalar Multiplication & Division

We can also perform multiplication and divisions of Matrices and Vectors with real numbers as well. As you might have guessed, we take every number in the matrix and perform the corresponding operation with the provided number. Again an example will help:

Addition and subtraction can also be performed similarly.

Matrix-Vector Multiplication

Things get a bit interesting here. Firstly we can only perform multiplications between a Matrix and Vector when number of columns of Matrix equals number of rows of Vector. In mathematical terms m * n-dimensional Matrix can be multiplied with only n-dimensional Vector. The result of any Matrix – Vector multiplication is always a Vector. Let’s take help of an example to see how this is done.

As you can see, we are multiplying a 3 * 2 matrix with a 2 dimensional Vector and we get a 3 dimensional Vector in result.

Matrix-Matrix Multiplication

A Matrix-Matrix multiplication can be considered as an extended case of Matrix-Vector multiplication:
Continue reading


Machine Learning: Understanding the Nomenclature

I have been working on Machine Learning Algorithms and APIs since past several months. There are several terms which didn’t come quite naturally to me when I started on this domain. In this blog post, I will try to explain some of them and hope this might help people starting on this exciting area.

What is Machine Learning
Lot of people have given their thoughts on the definition of Machine Learning. Out of them, I would like to quote couple of my favorites here :

Field of Study that gives computers the ability to learn without being explicitly programmed — Arhtur Samuel

A more mathematical and widely quoted one is:

A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E. — Tom M. Mitchell

It would be good to explain by an example. Let’s take one of very popular use-cases of Machine Learning: Spam Filter. When we open our Inbox, we choose to mark some of the mails as Spam. Over a period of time, program behind Spam Filter learns about what type of emails we mark as spam and stops similar type of mails even reaching our Inbox. Behind the scenes, all these things are not manually programmed but algorithms learn behaviors of different customers over a period of time and implement Spam filtering accordingly.

In terms of 2nd definition, Task T here would be classifying an email as Spam or not. Experience E would be recording user behavior in terms of which mails (s)he classifies as Spam and finally Performance P would be how good or bad filter is in correctly classifying a given mail message as Spam or not. In general, performance or accuracy of a Machine Learning program increases with experience or time.

What are the types of Machine Learning algorithms currently used:
Continue reading