The Khatri-Rao Product Visualized

The Khatri-Rao product takes two matrices with the same number of columns and pairs their columns up, replacing each pair by its Kronecker product. The result is a tall matrix with the same number of columns as the inputs. As a result it keeps the width fixed while the heights are multiplied.

1. Definition: the column-wise Kronecker product

Write and as lists of columns, with and . On a matrix, a subscript picks a column and a superscript picks a row: is the -th column of , and (section 3) is its -th row. The Khatri-Rao product is defined column by column:

Column of the output depends only on column of each input. Nothing mixes across columns.

The two inputs have columns each. In the figure below, matching columns share a color: the entries of are circles, the columns of are blocks.

The second step is a grid of blocks, not of numbers: each block is one whole column of , written out as a stacked pair and scaled by a single entry of . Column of the answer therefore only ever sees , and column only ever sees , never the whole of . That single difference is what separates from .

2. One column at a time: the Kronecker product of two columns

Zoom in on a single column pair , . Inside this section a subscript indexes an entry of that one column, not a column of a matrix. The Kronecker product stacks one scaled copy of per entry of :

Laying those same three blocks side by side instead of stacking them gives the outer product :

Its columns are , and , the very blocks that the Kronecker product stacks.

Three copies of , each tinted by the entry of that scales it. The output length is the product of the two input lengths.

Both layouts hold the same numbers in the same order, because reads a matrix one column at a time and stacks those columns into a single long vector. That is why the Khatri-Rao product has a natural interpretation as an outer product in disguise:

Each column of is a flattened outer product matrix.

3. The row-wise variant

Swapping rows for columns gives the face-splitting product. For and , the matrix has rows , where is the -th row of . The two are transposes of each other:

Its defining property is that it turns a Kronecker product of inputs into an elementwise product of outputs:

where is the elementwise (Hadamard) product.

The identity holds row by row, so watch a single row , with short for and short for .

Both Kronecker products are indexed by the pair : entry is on one side and on the other:

4. Three properties worth remembering

4.1 The Gram matrix collapses to a Hadamard product

Entry of the left side is the dot product of column with column , and the two families separate:

A answer that never touches the matrix . This is the single most useful fact about .

4.2 It expresses a weighted sum of outer products as a matrix-vector product

Read the left side one column at a time, then replace each Kronecker column by the outer product it flattens:

The last step is the outer-product reading of a matrix product: hands weight to the -th pairing of a column of with a row of .

4.3 The mixed product rule carries over

Column of is itself a Kronecker product, so the mixed product rule for applies to it directly:

The Khatri-Rao product is also associative: .