KAN: Kolmogorov–Arnold Networks: A review

Why review this?

On Apr 30, 2024, [5] appears on ArXiV and by May 7th, I have heard about this paper from multiple students, from whom I do not hear about new papers. It must be special, I thought. I decided to take a look.

If I was professionally reviewing this paper, I would accept the paper with major revisions. The paper has enough contributions to deserve a publication. But some of the claims need to be toned down, interpretations need to be clarified and comparisons with spline based neural networks be made.

Outline

I make 4 major critques of the paper

1. MLPs have learnable activation functions as well

2. The content of the paper does not justify the name, Kolmogorov-Arnold networks (KANs).

3. KANs are MLPs with spline-basis as the activation function.

4. KANs do not beat the curse of dimensionality.

MLPs have learnable activation functions as well

The authors claim in the abstract,

While MLPs have fixed activation functions on nodes (“neurons”), KANs have learnable activation functions on edges (“weights”). KANs have no linear weights at all – every weight parameter is replaced by a univariate function parametrized as a spline.

This is not a helpful description because one can interpret MLPs as having “learnable activation functions” as well; it depends on the definition what you call the “activation function“. Consider a two layer MLP with input $𝐱\in {ℝ}^n$, weights $W_1$, $W_2$ (ignore biases for now) and activation function $\sigma$,

$f(𝐱)$ $=W_2\sigma (W_1𝐱)=W_2{\varphi }_1(𝐱).$ (1)

If I define ${\varphi }_1(𝐱)=\sigma (W_1𝐱)$ and call ${\varphi }_1(.)$ as the activation function, then I have a learnable activation function in an MLP. Same with Figure 0.1, it is a reinterpretation, not redesign of MLPs as claimed.

What’s in the name

How do KAN’s actually use Kolmogorov-Arnold Theorem (KAT)? The theorem is not actually useful in the development of KANs. KANs are only inspired by KAT not based on it.

So what is Kolmogorov-Arnold Theorem? The paper describes it as the decomposition of any smooth function $f:{[0,1]}^n\to ℝ$ in terms of finite basis function ${\varphi }_q^{(2)}:ℝ\to ℝ$ and ${\varphi }_{q,p}:[0,1]\to ℝ$.

$f(𝐱)=f(x_1,\mathrm{\dots },x_n)={\displaystyle \sum _{q=1}^{2n+1}}{\varphi }_q^{(2)}\left({\displaystyle \sum _{p=1}^n}{\varphi }_{p,q}(x_p)\right).$ (2)

If you plan to use Kolmogorov-Arnold Theorem (KAT), you have understand the central claim of the KAT theorem and how is this theorem different the nearest competitor (Universal Approximation Theorem (UAT) ). Universal approximation theorem states that any function can be approximated by a wide enough 2-layer neural network.

$f(𝐱)$ $={\displaystyle \sum _{q=1}^{\mathrm{\infty }}}{w}_q^{(2)}\sigma \left({\displaystyle \sum _{p=1}^n}{w}_{q,p}^{(1)}{x}_p\right)$ $\text{where }{W}_2={[w_q^{(2)}]}_{q=1}^{\mathrm{\infty }}\text{ and }{W}_1={[{[w_{q,p}^{(1)}]}_{q=1}^{\mathrm{\infty }}]}_{p=1}^n$ (3)

I wrote the MLP in terms of summation instead of matrix multiplication to draw parallels between UAT and KAT. There are two main differences between UAT and KAT,

1. UAT deals with linear layers with common activation function (like sigmoid [3], ReLU, tanh) while KAT deals with arbitrary functions, possibly “non-smooth and even fractal”.

2. UAT needs possibly infinite hidden units for exact approximation while KAT only needs $2n+1$ “hidden units”.

I would claim that central point of KAT is about needing only $2n+1$ hidden units, otherwise it is a weaker theorem than UAT. Does the KAN paper make use of the $2n+1$ hidden units consistently? No. But they justify rest of the paper to be based on KAT by saying,

However, we are more optimistic about the usefulness of the Kolmogorov-Arnold theorem for machine learning. First of all, we need not stick to the original Eq. (2.1) which has only two-layer non- linearities and a small number of terms (2n + 1) in the hidden layer: we will generalize the network to arbitrary widths and depths.

Okay. But aren’t we back to Universal approximation theorem then?

There is one aspect of KAT that the authors highlight, “In a sense, they showed that the only true multivariate function is addition, since every other function can be written using univariate functions and sum.” This is a cool interpretation, but this interpretation does not separate KAT from UAT that is already being used in MLPs.

KANs are MLPs with spline-based activation functions

In practice, the authors end up proposing a KAN residual layer whose each scalar function is written as,

$\varphi (x)$ $=w(\text{silu}(x)+\text{spline}(x))$ (4)

$\text{spline}(x)$ $={\displaystyle \sum _{i=1}^G}{c}_i{B}_i(x).$ (5)

What are splines? For the purpose of this section you do not need to know splines. By the way there exists a history of splines in neural networks which is not cited in this paper [2, 1]. For now, assume splines are functions that are a result of a linear combination $c_i{B}_i(x)$ of a particular kind basis functions $B_i(x)$ (B-form splines). To reinterpret this scalar function as a MLP, let’s rewrite this as,

$\varphi (x)$ $={\displaystyle \sum _{i=1}^k}{c}_i{B}_i(x)+\text{selu}(x)$ (6)

(13)

$={𝐰}^{\top }𝐛(x).$ (14)

Here $𝐰$ contains the learnable parameters of the splines and $𝐛(x)$ is deterministic once the spline grid is fixed though it can be made learnable. Let’s put this back into (2),

$f(𝐱)=f(x_1,\mathrm{\dots },x_n)$ $={\displaystyle \sum _{q=1}^{2n+1}}{\varphi }_q^{(2)}\left({\displaystyle \sum _{p=1}^n}{\varphi }_{p,q}^{(1)}(x_p)\right)$ (15)

$={\displaystyle \sum _{q=1}^{2n+1}}{𝐰}_q^{(2)\top }𝐛\left({\displaystyle \sum _{p=1}^n}{𝐰}_{p,q}^{(1)\top }𝐛(x_p)\right).$ (16)

This is very close to an MLP if we consider $𝐰$s linear weights and the basis functions as activation functions, with the following differences,

1. The activation function $𝐛()$ is applied on the input side which is typically not a part of MLPs. However, it has been common to convert input into a set of feature vectors as a pre-processing step rather than providing MLPs the raw input.

2. Unlike $w_{p,q}^{(1)}$ being scalar in (3), ${𝐰}_{p,q}^{(1)}$ is a vector in (16). This is not a problem because it is still a linear combination of processed input values through a basis function $𝐛(x)$. To make this explicit, we write (16) as a matrix vector multiplication followed by activation functions.

To write (16) as a matrix vector product, consider only the first layer term ,

(24)

You can apply this interpretation repeatedly,

$f(𝐱)={𝐖}^{(2)}𝐁\left({𝐖}^{(1)}𝐁(𝐱)\right),$ (25)

where $𝐱\in {ℝ}^n,𝐁:{ℝ}^n\to {ℝ}^{nG},{𝐖}^{(1)}\in {ℝ}^{(2n+1)\times nG},{𝐖}^{(2)}\in {ℝ}^{1\times (2n+1)G}$.

Here $𝐁(𝐱)$ is unlike other activation functions. Instead of producing a scalar from a scalar, it produces $G$ different values for each scalar value in the input.

The claim that KANs beat the curse of dimensionality is wrong

The authors claim that,

KANs with finite grid size can approximate the function well with a residue rate independent of the dimension, hence beating curse of dimensionality!

This is a huge claim and requires huge evidence. As outlined in the previous section, if all KANs can be written as MLPs then either both MLPs and KANs beat the curse of dimensionality or neither does.

My first objection is how ”curse of dimensionality” is interpreted. Typically curse of dimensionality in machine learning is measured by the amount of data needed for training a function to a desired error.

I do not understand the proof of Theorem 2.1, especially the first step. It is not clear from what theorem in [4] the first result follows. If page number or chapter is provided that would be great.

It is also counter intuitive because a singular grid size $G$ is assumed for all the $n$ input dimensions. What would the bound look like if each dimension of $x$ is divided into different grid sizes.