Zau's batch results. Followup to #987

[brunofavs]

This issue comes as a followup to #987, as the due preparations are done, and everything is ready for the batch executions.

I did 5 runs with 5 folds at RNM=0.05 , so 25 calibrations in total. We were unsure whether the cross validation methods were working, but the results are consistent with the ones I presented in #987 during our last meeting, so its fair to assume cross validation is working as intended.

Results

NOTE : See results in the first comment instead, these are valid but there is only 1 set of results per modality.

calibration_errors.csv

Collection           ,Averages           
lidar_body [m]       ,0.011624           
rgb_body_left [px]   ,2.604892           
rgb_body_right [px]  ,2.2358960000000003 
rgbd_hand_color [px] ,2.6649119999999997 
rgbd_hand_depth [m]  ,0.0095     

---

rgb_lidar_evaluation.csv

Collection # ,Averages          
RMS (pix)    ,7.697367999999999 
X err (pix)  ,4.387492          
Y err (pix)  ,3.302896          

---

rgb_rgb_evaluation.csv

Collection # ,Averages           
RMS (pix)    ,5.042992           
X err (pix)  ,2.8674880000000003 
Y err (pix)  ,3.302876           
Trans (mm)   ,18.659332          
Rot (deg)    ,0.978508 

---

rgb_depth_evaluation.csv

Collection # ,Averages 
RMS (pix)    ,5.481612 
X err (pix)  ,3.202924 
Y err (pix)  ,2.532076 

rgb_lidar_evaluation.csv

Collection #  ,Averages           
RMS (pix)     ,5.841672           
X err (pix)   ,3.762896           
Y err (pix)   ,2.073256           
X StDev (pix) ,2.9350959999999997 
Y StDev (pix) ,1.999436           

Data.yml and jinja template

I will put these here just as a reference if anything regarding the batch execution wasn't clear. This section is really detailed so feel free to skip.

data.yml

#
#           █████╗ ████████╗ ██████╗ ███╗   ███╗
#          ██╔══██╗╚══██╔══╝██╔═══██╗████╗ ████║
#          ███████║   ██║   ██║   ██║██╔████╔██║
#          ██╔══██║   ██║   ██║   ██║██║╚██╔╝██║
#   __     ██║  ██║   ██║   ╚██████╔╝██║ ╚═╝ ██║    _
#  / _|    ╚═╝  ╚═╝   ╚═╝    ╚═════╝ ╚═╝     ╚═╝   | |
#  | |_ _ __ __ _ _ __ ___   _____      _____  _ __| | __
#  |  _| '__/ _` | '_ ` _ \ / _ \ \ /\ / / _ \| '__| |/ /
#  | | | | | (_| | | | | | |  __/\ v  v / (_) | |  |   <
#  |_| |_|  \__,_|_| |_| |_|\___| \_/\_/ \___/|_|  |_|\_\
#  https://github.com/lardemua/atom

# this yml file contains variables to be used in conjunction with batch.yml

# Auxiliary variables, to be used to render other fields in the template.yml.j2 file
package_path: "package://zau_calibration"

dataset_path   : "${ATOM_DATASETS}/zau/dataset_filtered/dataset_corrected_with_odometry_and_depth_and_rgb_and_pattern_poses_filtered.json"
# DATASET        : "${ATOM_DATASETS}/zau/dataset_filtered/dataset_corrected_with_odometry_and_depth_and_rgb_and_pattern_poses_filtered.json"
DATASET_TRAIN  : "${ATOM_DATASETS}/zau/dataset_filtered/dataset_corrected_with_odometry_and_depth_and_rgb_and_pattern_poses_filtered_train.json"
DATASET_TRAINED: "${ATOM_DATASETS}/zau/dataset_filtered/atom_calibration.json"
DATASET_TEST   : "${ATOM_DATASETS}/zau/dataset_filtered/dataset_corrected_with_odometry_and_depth_and_rgb_and_pattern_poses_filtered_test.json"
RESULTS_PATH   : "${ATOM_DATASETS}/zau/results"
CSF            : "lambda x: True"
RNM            : 0.05
TOL            : 1e-3


# Runs are repetitions of the experiments for gathering statistically significant results
runs: [1,2,3,4,5]

cross_validation:
  type: "stratified-k-fold" 
  n_splits: 5 # Number of folds
  train_size:  # Percentage of the dataset used for training, only used in StratifiedShuffleSplit

# Experiments are executions with a set of input parameters
experiments:

  - {name: zau_fixed_odometry}

template.yml.j2

#
#           █████╗ ████████╗ ██████╗ ███╗   ███╗
#          ██╔══██╗╚══██╔══╝██╔═══██╗████╗ ████║
#          ███████║   ██║   ██║   ██║██╔████╔██║
#          ██╔══██║   ██║   ██║   ██║██║╚██╔╝██║
#   __     ██║  ██║   ██║   ╚██████╔╝██║ ╚═╝ ██║    _
#  / _|    ╚═╝  ╚═╝   ╚═╝    ╚═════╝ ╚═╝     ╚═╝   | |
#  | |_ _ __ __ _ _ __ ___   _____      _____  _ __| | __
#  |  _| '__/ _` | '_ ` _ \ / _ \ \ /\ / / _ \| '__| |/ /
#  | | | | | (_| | | | | | |  __/\ v  v / (_) | |  |   <
#  |_| |_|  \__,_|_| |_| |_|\___| \_/\_/ \___/|_|  |_|\_\
#  https://github.com/lardemua/atom

# this yml file contains a set of commands to be run in batch.
# use jinja2 syntax for referencing variables

# Preprocessing will run only once before all experiments.
preprocessing:
  cmd: |

# Define batches to run
experiments:
{%- for e in experiments %}
  {% for run in runs %}
    {% set run_index = loop.index %}
    {% for fold in folds %}
      {{ e.name }}_run{{ '%03d' % run_index }}_fold{{ '%03d' % loop.index }}:
        cmd: |
          echo "..." && \
          echo "..." && \
          echo "\n Calibrating\n" && \
          echo "..." && \
          echo "..." && \
          rosrun atom_calibration calibrate \
          -json {{ DATASET_TRAIN }} -v  \
          -sce \
          -csf 'lambda x: int(x) in {{ fold[0] }}' -uic \
          -ssf "lambda x: x in ['rgb_body_left','rgb_body_right','rgbd_hand_color','rgbd_hand_depth','lidar_body']" \
           -ftol {{ TOL }} -xtol {{ TOL }} -gtol {{ TOL }} -rnm {{ RNM }} && \


          ./eval_scripts/rgb_rgb_batch_eval "rgb_body_left" "rgbd_hand_color" "{{ DATASET_TRAINED }}" "{{ DATASET_TEST }}" "{{ fold[1] }}"
          ./eval_scripts/rgb_rgb_batch_eval "rgb_body_right" "rgbd_hand_color" "{{ DATASET_TRAINED }}" "{{ DATASET_TEST }}" "{{ fold[1] }}"

          ./eval_scripts/rgb_lidar_batch_eval "lidar_body" "rgb_body_right" "{{ DATASET_TRAINED }}" "{{ DATASET_TEST }}" "{{ fold[1] }}"
          ./eval_scripts/rgb_lidar_batch_eval "lidar_body" "rgb_body_left" "{{ DATASET_TRAINED }}" "{{ DATASET_TEST }}" "{{ fold[1] }}"
          ./eval_scripts/rgb_lidar_batch_eval "lidar_body" "rgbd_hand_color" "{{ DATASET_TRAINED }}" "{{ DATASET_TEST }}" "{{ fold[1] }}"


          ./eval_scripts/rgb_depth_batch_eval "rgbd_hand_depth" "rgb_body_left" "{{ DATASET_TRAINED }}" "{{ DATASET_TEST }}" "{{ fold[1] }}"
          ./eval_scripts/rgb_depth_batch_eval "rgbd_hand_depth" "rgb_body_right" "{{ DATASET_TRAINED }}" "{{ DATASET_TEST }}" "{{ fold[1] }}"
          ./eval_scripts/rgb_depth_batch_eval "rgbd_hand_depth" "rgbd_hand_color" "{{ DATASET_TRAINED }}" "{{ DATASET_TEST }}" "{{ fold[1] }}"

          ./eval_scripts/lidar_depth_batch_eval "lidar_body" "rgbd_hand_depth" "{{ DATASET_TRAINED }}" "{{ DATASET_TEST }}" "{{ fold[1] }}"

        files_to_collect:    
          - '{{ dataset_dirname }}/atom_calibration.json'
          - '{{ dataset_dirname }}/atom_calibration_params.yml'
          - '{{ dataset_dirname }}/command_line_args.yml'
          - '{{ dataset_dirname }}/calibration_errors.csv'
          - '/tmp/rgb_rgb_evaluation.csv'
          - "/tmp/rgb_lidar_evaluation.csv"
          - "/tmp/rgb_depth_evaluation.csv"
          - "/tmp/lidar_depth_evaluation.csv"
    {%- endfor %}
  {%- endfor %}
{%- endfor %}
# End the loop

Each evaluation script is a adaptation of the scripts developed in the prior testing phases, in #987.

Here is one example, for rgb_rgb_batch_evaluations

#! /bin/bash

# Check if the correct number of arguments are passed
if [ "$#" -ne 5 ]; then
    echo "Usage: $0 <sensor_source> <sensor_target> <dataset_trained> <dataset_test> <fold lambda>"
    exit 1
fi

SENSOR_SOURCE=$1
SENSOR_TARGET=$2
DATASET_TRAINED=$3
DATASET_TEST=$4
FOLD=$5

echo "..."
echo "..."
echo "Evaluating $SENSOR_SOURCE to $SENSOR_TARGET"
echo "..."
echo "..."

rosrun atom_evaluation rgb_to_rgb_evaluation \
-train_json "$DATASET_TRAINED" \
-test_json "$DATASET_TEST" \
-csf "lambda x: int(x) in $FOLD" \
-ss "$SENSOR_SOURCE" \
-st "$SENSOR_TARGET" \
-pn "pattern_1" \
-uic \
-sfr -sfrn "/tmp/rgb_rgb_evaluation.csv"

[brunofavs]

Whilst this results are valid, it seems I overwrote the results for repeated modalities. Gonna fix and comment here.

[brunofavs]

Updated results

The previous results were valid, but I overwrote pairs with repeating modalities. Here's all the results.

rgbd_hand_color_to_rgbd_hand_depth_evaluation.csv
rgbd_hand_color_to_lidar_body_evaluation.csv
rgb_body_right_to_rgbd_hand_depth_evaluation.csv
rgb_body_right_to_rgbd_hand_color_evaluation.csv
rgb_body_right_to_lidar_body_evaluation.csv
rgb_body_left_to_rgbd_hand_depth_evaluation.csv
rgb_body_left_to_rgbd_hand_color_evaluation.csv
rgb_body_left_to_lidar_body_evaluation.csv
lidar_body_to_rgbd_hand_depth_evaluation.csv
calibration_errors.csv