We Learned the Wrong Way to Matrix Multiply

I took an informal poll of my friends and it turns out we all intially learned a method of matrix multiplication in school which I call row oriented multiplication. However, the friends who had been working with linear algebra for a long time frequently used a more intuitive method which I call column oriented multiplication. Here I’ll explain the difference and try to show why the column oriented point of view should be a useful tool in your toolbox, especially if you are working in a field that is heavy in linear algebra (e.g. robotics).

Row Oriented Multiplication

When I learned how to multiply vectors and matrices, it went something like this.

$\begin{bmatrix} & & \\ \rightarrow & \rightarrow & \rightarrow \\ & & \end{bmatrix} \begin{bmatrix} \downarrow \\ \downarrow \\ \downarrow \end{bmatrix} =\begin{bmatrix} \\ \cdot \\ \\ \end{bmatrix}$

Or if you prefer formulas, it went like this.
$A x = \begin{bmatrix} ... \\ A_{i1} x_1 + A_{i2} x_2 + ... + A_{i n} x_n \\ ... \end{bmatrix}$

I call this row oriented multiplication. It definitely works but it often hides the geometrical content of a matrix behind nasty algebra. Consider this $3D$ rotation matrix.

$R = \begin{bmatrix} \frac{\sqrt 3}2 & -\frac 1 2 & 0 \\ \frac 1 2 & \frac{\sqrt 3}2 & 0 \\ 0 & 0 & 1\end{bmatrix}$

We can verify that the axis of rotation is the z-axis with row oriented multiplication by seeing that a point on the z axis $\begin{bmatrix}0 & 0 & z\end{bmatrix}^T$ is unaffected by multiplication with $R$.

$R \begin{bmatrix}0 \\ 0 \\ z\end{bmatrix} = \begin{bmatrix} \frac{\sqrt 3}2 & -\frac 1 2 & 0 \\ \frac 1 2 & \frac{\sqrt 3}2 & 0 \\ 0 & 0 & 1\end{bmatrix}\begin{bmatrix}0 \\ 0 \\ z\end{bmatrix} = \begin{bmatrix} \frac{\sqrt 3}2\cdot 0 + -\frac 1 2\cdot 0 + 0 \cdot z \\ \frac 1 2 \cdot 0 + \frac{\sqrt 3}2\cdot 0 + 0 \cdot z \\ 0 \cdot 0 + 0\cdot 0 + 1\cdot z\end{bmatrix} = \begin{bmatrix}0 \\ 0 \\ z\end{bmatrix}$

Now I’ll introduce column oriented multiplication and how it not only simplifies the previous calculation but also reveals the geometrical content of the matrix.

Column Oriented Multiplication

First let’s establish the notation that for any matrix $A$, $A_i$ means the $i^{th}$ column of $A$. We obviously have $A = \begin{bmatrix} A_1 | A_2 | ... | A_n\end{bmatrix}$.

Column oriented multiplication of $A x$ works like this. We use the columns of $A$ as vectors.

$A x = \begin{bmatrix} A_1 | A_2 | ... | A_n\end{bmatrix} \begin{bmatrix}x_1 \\ x_2 \\ ... \\ x_n \end{bmatrix} = x_1 A_1 + x_2 A_2 + ... + x_n A_n$

There are multiple ways to verify that this leads to the same result as row oriented multiplication. I’ll show a really easy way in the appendix. But for now let’s see how column oriented multiplication can be applied.

Using Column Oriented Multiplication

First let’s redo our previous calculation involving our rotation matrix $R$ using column oriented multiplication.

$R \begin{bmatrix}0 \\ 0 \\ z\end{bmatrix} = 0 \cdot R_1 + 0 \cdot R_2 + z R_3 = z R_3$

We can completely ignore the first two columns of $R$, and compute the answer simply by reading off the third column of $R$ and multiplying by $z$. We arrive at the same answer we got with row oriented multiplication.

$z R_3 = \begin{bmatrix}0 \\ 0 \\ z\end{bmatrix}$

Now recall the notation for the standard basis vectors.

$e_1 = \begin{bmatrix}1 \\ 0 \\ 0 \end{bmatrix}, \,\,\,\, e_2 = \begin{bmatrix}0 \\ 1 \\ 0 \end{bmatrix}, \,\,\,\, e_3 = \begin{bmatrix}0 \\ 0 \\ 1 \end{bmatrix}$

Visualize $e_1, e_2, e_3$ as points on the x, y, and z axis respectively. Using column oriented multiplication we can quickly see these relationships.

$R e_1 = R_1, \,\,\,\, \ R e_2 = R_2, \,\,\,\, R e_3= R_3$

This means that the geometrical content of $R$ — the way it acts on the coordinate axes — can be entirely read off from its columns!

$e_1 \rightarrow \begin{bmatrix} \frac {\sqrt 3} 2 \\ \frac 1 2 \\ 0\end{bmatrix}, \,\,\,\, \ e_2 \rightarrow \begin{bmatrix} -\frac 1 2 \\ \frac{\sqrt 3} 2 \\ 0\end{bmatrix}, e_3 \rightarrow e_3$ (read the arrows as “rotates to”)

Try carrying out this rotation on your thumb as $e_1$, your index finger as $e_2$, and your middle finger as $e_3$. Also note that $e_2$ is redundant since it must be perpendicular to $e_1$ and $e_3$.

The story doesn’t end here. Once I got into the habit of seeing matrix multiplication this way, I started to have insights into the nature of other types of matrices, including Jacobian matrices which are ubiquitous in robotics algorithms. But I’ll save that for another post.

Appendix

In all matrix multiply expressions that follow, assume that we are using row oriented multiplication.

$A x = A \begin{bmatrix}x_1 \\ x_2 \\ ... \\ x_n \end{bmatrix} = A (\sum_{i=1}^n x_i e_i)$

$= A (\sum_{i=1}^n x_i e_i)$
$= \sum_{i=1}^n x_i A e_i$
$= \sum_{i=1}^n x_i A_i$ (verify, with row oriented multiplication that $A e_i = A_i$)

So we end up with the formula for column oriented multiplication $Ax = \sum_{i=1}^n x_i A_i$.