FAIMDL Group 5 project: Comprehensive Road Scene Understanding

This is the repository for the FAIMDL (Fundamentals of AI, Machine and Deep Learning) project of Group 5. This project tackles road scene understanding for autonomous driving, focusing on semantic segmentation and anomaly segmentation on the road, using pre-trained ERFNet and EoMT models.

Pipeline

A detailed 5-pages paper that describes the project and analyzes the results is in report.pdf. Here is a quick summary of the pipeline implemented:

1. Evaluation of semantic segmentation & visualization: we evaluate the mIoU of two versions of EoMT: one pre-trained on Cityscapes (for semantic segmentation) and one on COCO (for panoptic segmentation). We also include a visualization to compare their predictions.
2. Fine-tuning: we take the COCO-trained model and fine-tune it to do semantic segmentation on Cityscapes. We perform five experiments: training just the head, unfreezing 2 layers, then 6, doing a full fine-tune, and using LoRA. Each experiment is performed twice, first with 5/10 epochs and then with 10/15 epochs. We calculate the mIoU for those finetuned models as well.
3. Anomaly segmentation: we test how well the models (along with a baseline ERFNet) can spot OoD objects on the road. We evaluate them on the datasets SegmentMeIfYouCan (RoadAnomaly, RoadObsticle) and Fishyscapes (Lost & Found, Static) using the post-hoc anomaly scoring methods Maximum Softmax Probability (MSP) across different temperatures, MaxLogit, MaxEntropy, and RbA.

Content of the repo

Heads up: the COCO and Cityscapes datasets used are massive, so they aren't uploaded in this repo. However, they will be automatically installed when you run the notebook.

In this repo you will find all you need to reproduce the results. This repo comprises the following files and folders:

• report.pdf: the project report.
• project.ipynb: the main Jupyter Notebook containing all the code. It's built specifically to be run on Google Colab associated to a Google Drive folder.
• configs/: YAML configuration files used for fine-tuning the EoMT models.
• data/: contains the SMIYC and Fishyscapes subsets used for the anomaly segmentation part, in the Anomaly Detection Validation folder.

Additionally, the "Releases" section on GitHub contains:

• 3 pre-trained checkpoints: eomt_cityscapes.bin, eomt_coco.bin, and erfnet_pretrained.pth;
• 10 PyTorch Lightning checkpoints, produced from our five fine-tuning experiments. They are stored as coco_ft_[EXPERIMENT]_best_[NUM]epochs.ckpt. The total size amounts to 6 GB. They were stored in "Releases" as GitHub does not allow to upload files over 100 MB.

How to reproduce it on Google Colab

To get the notebook running without breaking the file paths, you need to set up your Google Drive to match what the code expects.

1. Set up your Google Drive

Create a folder in the root of your Google Drive named exactly FAIMDL Project Code. Upload the contents of this repository into that folder so it looks like this:

MyDrive/
└── FAIMDL Project Code/
    ├── project.ipynb
    ├── configs/               <-- include the 5 files in configs folder of this repo
    ├── ckpts/                 <-- include the 3 files in the "Releases" section of this repo (pre-trained)
    ├── ckpts/finetuning_coco  <-- include the 10 files in the "Releases" section of this repo (fine-tuned)
    └── data/                  <-- include the Anomaly Detection Validation dataset folder

NOTE: Actually, in the ckpts/finetuning_coco folder, you only strictly need the files associated to the highest number of epochs for each experiment (i.e., 5 files).

2. Open the project in Colab and add secrets

The notebook automatically downloads the COCO and Cityscapes data, and logs training metrics to Weights & Biases (W&B). To make this work, accounts for both Cityscapes and W&B are needed. Open the notebook project.ipynb on Colab and add the following environment variables as "Secrets" on Colab:

• CITYSCAPES_USERNAME: your Cityscapes account username.
• CITYSCAPES_PASSWORD: your Cityscapes account password.
• WANDB_API_KEY: your Weights & Biases API key.

Make sure to toggle "Notebook access" on for all of them.

NOTE: W&B will not actually be needed if you do not perform fine-tuning all over again (which is unnecessary), hence you could simply remove the cell where WANDB_API_KEY is read and login is performed (i.e., the 7th code cell from the beginning).

3. Run the notebook

You are ready to run the notebook! Running it will automatically install the other two necessary datasets (COCO and Cityscapes) and save them on your drive, in the data folder previously added. The download is approximately 30 GB and will require a while.

All the required packages will automatically be installed as well.

You should not re-execute the fine-tuning, as it could require days to complete, and the fine-tuned models' checkpoints are already available in the "Releases" section of this GitHub repo. For your convenience, the fine-tuning cells have been "disabled" with the use of the line magics %%script false --no-raise-error.