Matrix Methods in ICA: Infomax vs. FastICA

Theory and Concept

Infomax ICA (Infomation maximization independent Component Analysis) is one of the main algorithms for Blind Source Separation(BSS). This algorithm is able to separate independent source signals from EEG data and is widely used for artifact removal such as EOG, EMG, and ECG artifacts as well as neural signal component analysis.

Blind Source Separation

EEG recorded signals linear mixed by several independent source signals:

$X=AS$

which

• $X \in R^{m*n}$: EEG observed signal matrix (m channels, n time points)
• $A \in R^{m*m}$: unmixing matrix (unknown)
• $S \in R^{m*n}$: independent signal sources

Target of ICA is find unmixing matrix, such that

$S=WX$

Thus the independent components are recovered.

Shannon Entropy

Entropy measures the uncertainty of a random variable and is defined as

$H(X) = - \sum p(x)\log p(x)$

Intuitivly, entropy is maximized if all events are equally likely. If the probability of some event is close to 1 (high certainty), entropy is minimal.

Mutual Information and Minimization

I(X;Y) = H(X) + H(Y) - H(X,Y)
Mutual information measure the dependency between two variables.

The goal of Infomax ICA is to find a nonlinear transformation such that the mutual information of the output signal is minimized, such that the components are as independent as possible.

Definition of Mutual Infomation:

$I(S_1, S_2,...,S_m) = \sum \limits{i=1}^m H(S_i) - H(S)$

Which H(S) is the entropy of the signal:

$H(S) = - \int p(S) \log p(S) dS$

In order to maximize the entropy of the independent components, Infomax ICA uses gradient descent optimization ofn neural networks to adjust the weight matrix W, so that the probability distribution of the output signals is close to the maximum entropy.

The Principle of Infomax ICA Method

Infomax ICA tries to find a nonlinear transformation that maximizes the entropy of the separated signals, resulting in the most independent signals.

$W= \argmax H(Y) = \argmax H(g(WX))$

which

• X is observed signal
• W is unmixing matrix
• g(·) is activation function (sigmoid or tanh)

Key of Informax ICA:

1. Select proper nonlinear function g, e.g. sigmoid.
2. Optimization using gradient descent.
3. The entropy of the output signal is maximized.

Algorithm

Infomax ICA

Assuming the observed signal is X, the source signal is S, then:

$X= AS$

which

• A is the mixing matrix

Our target is to find the unmixing matrix W:

$Y= WX$

to make Y as Independent as possible.

Infomax ICA algorithm optimize unmixing matrix W through maxmizing the output signal entropy, which is solved by gradient descent, the update rule is :

$W^{(t+1)} = W^{(t)} + \eta (I + (1-2Y)Y^T)W^{(t)}$

This formular is a variant of Steepest Descent, which

• $I + (1-2Y)Y^T$ is gradient term ( gradients derived from information theory )
• $\eta$ is learning rate, control the update step. Since it is only updated in the gradient direction and does not use second-order information( such as the Hessian matrix), it strictly falls into the category of steepest descent methods.

Steepest Descent

Steepest descent is the most basic gradient descent method, its update formula is:

$W^{(t+1)} = W^{(t)} - \eta \Delta f(W)$

which

• W is the parameter to optimize (unmixing matrix in ICA)
• $\eta$ is learning rate (step size)
• $\Delta f(W)$ is the gradient direction of the objective function, which indicates the direction of the current fastest descent. The core idea of the steepest descent method is to take a small step in the direction of the gradient and gradually approach the optimal solution.

FastICA

Experiment

Computation time

Independent Component Analysis Evaluation

Convergence speed

Conclusion

Related Papers and Books

1. Bell, A. J., & Sejnowski, T. J. (1995). An Information-Maximization Approach to Blind Separation and Blind Deconvolution. (DOI: 10.1162/neco.1995.7.6.1129)

   • This is the fundamental paper of Infomax ICA, proposed by Bell and Sejnowski.
   • This paper introduced how to seperate independent signals through maximize mutual information
   • Proposed weights updating method based on gradient descent, and performed it in blind source seperation

2. Makeig, S., Jung, T. P., Bell, A. J., Ghahremani, D., & Sejnowski, T. J. (1997). Blind separation of auditory event-related brain responses into independent components.(DOI: 10.1073/pnas.94.20.10979)

   • this paper detailly introduced the application of Infomax ICA in EEG analysis
   • Introduced how to seperate EEG with ICA, and remove artifact signals such as EOG, EMG
   • proposed Extended Infomax ICA, to process Sub-Gaussian and Super-Gaussian signals