Question about the visualization example for explanation on images.

[htdung167]

I’ve read the paper and really like this research. I’m new to XAI.

I have a question: why does the paper not overlay the prototype concept activation on the original image to show which regions are activated, but instead selects the top-5 images (Figure 13) with the highest response from the training set?

Is overlaying on the original image against the idea of prototype concepts, or is there some issue with it?
Is it incorrect if I overlay the activation map on the input image to see which part of the image the model is focusing on to make a prediction?

[eddie221]

Thank you for your thoughtful question and your interest in our work!

In Figure 13, we aim to showcase how concept strength varies across classes. For instance, Concept # 2 in Layer 1 (blue box) may correspond to the idea of "stripes," showing strong activation for zebra images but weaker responses for lions. We use the top-5 training images that elicit the strongest activations to reveal the common visual patterns that each prototype concept captures.

Overlaying activation maps on input images won't against the idea of the prototypes concept. On the contrary, it provides complementary insights by revealing where in an image a concept is activated. We include such visualizations in Figure 16 of the supplementary material to highlight this spatial correspondence.