Analytics pane

[julianmichael]

We should have an analytics pane where we can put all of our visualizations of debate and feedback data.

Should go under an analytics supertab (#37) and incorporate analyses of the feedback data (#59).

Can start with something small and simple; worth having an issue here just to get it started, at which point we can think about all of the analyses we want to include.

I'll tentatively list some Qs here that we may be interested in:

• Does evidence get used more or less as the debate goes on?

[sm11197]

From draft paper (David):

• Line graph showing the average judge accuracy across turns of the debate (on the same plot, maybe we can also plot average confidence in the correct answer, to show that we start out underconfident and end up ???)
• Offline vs. online accuracy
• How should we show/communicate about the variance between judgements on the same debate?
• Depending on our overall accuracy, if we have enough accuracy bins we can maybe show a calibration plot
• Bar chart showing accuracy when debater A is honest vs. debater B
• Bar chart showing accuracy on the first debate you've judged on a particular story, versus the second debate you've judged on that story
• Total quote length revealed after 1-2 turns is only 500-1000 chars, which is « than the total story length (how long on avg?) and total arguments+quotes is 2000-4000 chars which is only like 1-2 mins of reading. If we have » 50% accuracy, then this is some info about efficiency