A PyTorch-powered, scikit-learn compatible feature selection method that uses a trainable attention mechanism to rank and select the most informative features for classification tasks.
pip install pylafsOr in development/editable mode:
pip install -e .from pylafs import LAFSSelector
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import pandas as pd
# Generate sample data
X, y = make_classification(n_samples=1000, n_features=50,
n_informative=10, random_state=42)
X = pd.DataFrame(X, columns=[f"f_{i}" for i in range(50)])
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
# Scale features
scaler = StandardScaler()
X_train = pd.DataFrame(scaler.fit_transform(X_train), columns=X.columns)
X_test = pd.DataFrame(scaler.transform(X_test), columns=X.columns)
# Select top-10 features with LAFS
selector = LAFSSelector(k=10, epochs=50, lr=0.001, lambda_reg=0.01)
selector.fit(X_train, y_train)
X_train_selected = selector.transform(X_train)
X_test_selected = selector.transform(X_test)
print("Selected features:", selector.selected_features_)
print("Feature importances:", selector.feature_importances_)| Parameter | Type | Default | Description |
|---|---|---|---|
k |
int | 10 | Number of features to select |
lr |
float | 0.001 | Adam learning rate |
lambda_reg |
float | 0.01 | Entropy regularisation coefficient |
epochs |
int | 50 | Training epochs |
batch_size |
int | 256 | Mini-batch size |
attn_hidden_size |
int | 128 | Attention network hidden layer width |
attn_dropout_rate |
float | 0.5 | Dropout rate in attention network |
cls_hidden_size |
int | 128 | Classifier hidden layer width |
weight_decay |
float | 1e-4 | L2 regularisation for Adam |
device |
str/None | None | "cuda", "cpu", or auto-detect |
random_state |
int/None | None | Seed for reproducibility |
verbose |
bool | False | Print training progress |
| Method | Description |
|---|---|
fit(X, y) |
Train the LAFS model and compute feature importances |
transform(X) |
Reduce X to the selected features |
fit_transform(X, y) |
Fit and transform in one step |
get_support(indices=False) |
Boolean mask or indices of selected features |
get_feature_names_out() |
Names of the selected features |
| Attribute | Description |
|---|---|
feature_importances_ |
Mean attention weight per feature (numpy array) |
ranking_ |
Feature indices sorted by importance (descending) |
selected_features_ |
List of selected feature names |
selected_indices_ |
Array of selected feature indices |
model_ |
The trained PyTorch LAFSModel |
LAFS jointly trains two neural networks:
- AttnNet — produces per-feature softmax attention weights
- ClassNet — a feedforward classifier that operates on attention-weighted features
The loss function combines cross-entropy classification loss with an entropy regularisation term on the attention weights. This encourages the attention to focus on the most discriminative features.
After training, global feature importance is computed as the mean attention weight across all training samples. The top-k features by importance are selected.
MIT