Unsupervised Behavioral Compression: Learning Low-Dimensional Policy Manifolds through State-Occupancy Matching

This repository contains the experiments and the code that executes them that are displayed in the paper: "Unsupervised Behavioral Compression: Learning Low-Dimensional Policy Manifolds through State-Occupancy Matching", accepted ad RLC 2026.

Prerequisites

Make sure you have uv installed. A GPU supported by PyTorch is heavily recommended.

Setup

To create or update the venv run:

uv sync

then activate the environment with:

.venv\Scripts\activate

or

source .venv/bin/activate

Experiments

The bash/batch files to run the experiments are in the batch_experiments_commands directory. They should be run in sequence from 1_generation to 3_optimization. Expected runtime for all the experiments: ~30 hours.

The latent_comparison script is used specifically for the latent space ablation studies.

⚠️ IMPORTANT: Make sure to change the env parameter in scripts/parameters.yml for the scripts that require it.

Project Layout

.
├── batch_experiments_commands/ # CLI scripts to run the pipeline
│   ├── unix/                   
│   └── win/
├── compression_autoencoder/    # business logic code
│   ├── autoencoders/             # autoencoder classes
│   ├── envs/                     # multi-reward environment wrapper classes
│   ├── optim/                    # custom RL optimization logic
│   ├── pgpe/                     # custom PGPE implementation
│   ├── policies/                 # policy classes
│   └── utils/                    # miscellaneous library code
├── figures/                    # paper's figures
├── scripts/                    # python scripts and config files
├── .gitignore
├── .python-version
├── pyproject.toml
├── README.md
└── uv.lock

Citation

You can cite the paper or the repository as:

@inproceedings{fraschini2026behavioral,
  title={Unsupervised Behavioral Compression: Learning Low-Dimensional Policy Manifolds through State-Occupancy Matching},
  author={Andrea Fraschini and Davide Tenedini and Riccardo Zamboni and Mirco Mutti and Marcello Restelli},
  booktitle={Reinforcement Learning Conference},
  year={2026}
}