Brief summary:

While PCA is traditionally employed for dimensionality reduction and denoising before training, this preprocessing can complicate the interpretability of explainable AI (XAI) methods due to the transformation of input features. To mitigate these challenges, the authors propose a novel strategy called Principal Components-based Initialization (PCsInit). Instead of applying PCA as a separate preprocessing step, PCsInit incorporates PCA directly into the neural network by initializing the first layer with principal components. This method preserves the original input features, enhancing the clarity of model explanations. The paper also introduces two variants of this approach:

1. PCsInit-Act: This variant applies an activation function after the principal components, enabling the network to capture nonlinear patterns more effectively.
2. PCsInit-Sub: Designed for large datasets, this variant computes principal components based on a subset of the input data, improving computational efficiency without significant loss of information.

Experimental results demonstrate that these strategies not only simplify the explanation process but also enhance training efficiency through backpropagation. By integrating PCA into the network’s architecture, the proposed methods offer a more interpretable and effective approach to neural network initialization.

Download the paper at https://arxiv.org/html/2501.19114v1

Github: https://github.com/pthnhan/pcsinit

A more throughout explaination:

Neural networks have revolutionized machine learning, but their complexity often makes them difficult to interpret. One common approach to improving efficiency and interpretability is Principal Component Analysis (PCA)—a dimensionality reduction technique used to remove noise and improve learning. However, traditional PCA preprocessing has limitations when combined with neural networks, particularly in explainable AI (XAI).

A new study, “Principal Components for Neural Network Initialization” by Nhan Phan et al. (2025), proposes a groundbreaking method that integrates PCA directly into the neural network initialization process. This approach, called Principal Components-based Initialization (PCsInit), aims to enhance both training efficiency and explainability. Let’s break it down.

The Problem with Traditional PCA in Neural Networks

PCA is widely used to preprocess input data before training. By transforming features into orthogonal principal components, PCA improves data efficiency but alters the feature space, making it harder to trace how input features influence model predictions.

This lack of transparency poses a challenge for XAI techniques, such as feature importance analysis and saliency maps, which rely on original input features for interpretability. Moreover, traditional PCA preprocessing adds an extra step to the workflow, requiring practitioners to apply PCA externally before training begins.

The Solution: Principal Components-based Initialization (PCsInit)

Instead of applying PCA separately, PCsInit integrates PCA into the first layer of the neural network. This means the network is initialized with weights that already capture the principal components of the input data. This preserves the original feature space, making the model easier to interpret while also improving training efficiency.

Key Benefits of PCsInit

? Better Interpretability – Since the original feature space is maintained, explainability techniques like saliency maps remain meaningful.
? Faster Convergence – The network starts with an optimal representation of the data, reducing the number of training epochs needed.
? Reduced Computational Overhead – By eliminating the need for an external PCA step, training becomes more streamlined.

Implementation

class FullNN(nn.Module):
    def __init__(self, input_dim, n_components, other_layers, activation='none', init_type='xavier', variance_retained=None):
        super(FullNN, self).__init__()
        self.flatten = nn.Flatten()
        self.fc1 = LinearWithActivation(input_dim, n_components, bias=False, activation=activation)
        self.other_layers = copy.deepcopy(other_layers)
        self.n_components = n_components
        self.variance_retained = variance_retained
        self._init_fc1(init_type)

    def _init_fc1(self, init_type):
        initializer = self._get_initializer(init_type)
        initializer(self.fc1.weight)

    def _get_initializer(self, init_type):
        init_map = {
            'xavier': nn.init.xavier_uniform_,
            'he': nn.init.kaiming_uniform_,
            'orthogonal': nn.init.orthogonal_,
            'uniform': nn.init.uniform_,
        }
        if init_type not in init_map:
            raise ValueError(f"Unknown init_type: {init_type}")
        return init_map[init_type]

    def forward(self, x):
        x = self.flatten(x)
        x = self.fc1(x)
        x = nn.ReLU()(x)
        return self.other_layers(x)

    def init_pca_weights(self, X_batch):
        pca = PCA(n_components=self.variance_retained if self.n_components is None else self.n_components)
        pca.fit(X_batch.cpu().detach().numpy())
        self.fc1.weight.data = torch.Tensor(pca.components_).to(self.fc1.weight.device)
        print(f'Number of PCA components: {pca.n_components_}')

PCA Weight Initialization (init_pca_weights method):

In this code, init_pca_weights Initializes the weights of the fc1 layer using Principal Component Analysis (PCA). So, it:

1. Fits a PCA model to X_batch (a batch of input data, converted to NumPy).
2. Extracts PCA components and assigns them to fc1 weights.
3. Prints the number of PCA components used.

Training with PCsInit

We first free the first layer and train the remaining ones for a certain number of epochs. Later, we unfreeze it and train the whole neural network. The initial freeze and later fine-tuning help the last layers adjust to the PCA-based feature space, creating stable higher-level representations. Once these layers identify effective feature combinations, unfreezing the first layer allows fine-tuning of the initial PCA weights while keeping the learned feature hierarchy intact. This two-step method avoids disturbing the essential PCA structure too early, yet still allows for the optimization of all parameters later.

    pca_init_nn_kernel = FullNN(input_dim, n_components, other_layers, activation='relu', init_type=init_type)
    pca_init_nn_kernel.init_pca_weights(X_train)  # Initialize weights with PCA components

    # train on everything except the first layer
    optimizer = optim.Adam([{'params': param} for name, param in pca_init_nn_kernel.named_parameters() if not name.startswith('fc1')],
                            lr=learning_rate)
    train_losses_pcinit_ker, test_accuracies_pcinit_ker, training_time_pcinit_ker = train(pca_init_nn_kernel, train_loader, test_loader, criterion, optimizer, epochs=n_frozen_epochs)

    # train the complete network
    optimizer = optim.Adam(pca_init_nn_kernel.parameters(), lr=learning_rate)
    train_losses_pcinit_ker2, test_accuracies_pcinit_ker2, training_time_pcinit_ker2 = train(pca_init_nn_kernel, train_loader, test_loader, criterion, optimizer, epochs=epochs-n_frozen_epochs)

Variants of PCsInit

The authors introduce two versions of this method:

1. PCsInit-Act (Activation-based Initialization)

? In this version, PCA is used to initialize the first layer, followed by a nonlinear activation function (such as ReLU or Sigmoid).
? This allows the network to capture nonlinear patterns in the data while still benefiting from the principal component representation.

2. PCsInit-Sub (Subset-based Initialization)

? When dealing with large datasets, computing PCA on the full dataset is computationally expensive.
? PCsInit-Sub selects a representative subset of the data to compute the principal components, maintaining efficiency without significant information loss.

Experimental Results & Insights

The researchers tested PCsInit on various datasets and neural network architectures. The results showed:

? Faster Convergence – Models initialized with PCsInit required fewer training epochs to reach high accuracy.
? Improved Explainability – Feature importance methods provided clearer insights into model predictions.
? Competitive Performance – Despite the improved efficiency and interpretability, accuracy remained comparable to or better than standard training methods.

? PCsInit-Sub achieves competitive accuracy and stability across all datasets, mirroring full PCsInit trends. It also provides efficiency with reduced computational overhead, making it suitable for large datasets or limited resources.

? PCsInit-Act demonstrates stability and adaptability across varying data structures. Additionally, it enhances training stability and accuracy, particularly with high-dimensional data.

Final Thoughts: A Step Toward More Explainable Neural Networks

PCsInit offers an elegant and practical solution for integrating PCA into neural networks. By preserving the original feature space and optimizing initialization, this method enhances both interpretability and efficiency—two critical challenges in modern deep learning.

For AI practitioners and researchers working with explainable AI, neural network optimization, or high-dimensional data, this approach could be a game-changer.

Read the Full Paper Here

Principal Components for Neural Network Initialization (arXiv)