Evaluation

This is to remind what configuration needs to be changed for the user study. Visit this wiki for how to run threadlet.

Study 1: Model building

Threadlet interface: http://localhost:8080/demo/threadall/.

Ask two participants to build models with two different but comparable data subsets namely data/threads-300-set1_features.json and threads-300-set2_features.json. Do the following step for each participant.

Participant 1

• Change in the interface. In demo/threadall/index.js, set the value of dataFilePath to ../../data/threads-300-set1_features.json.
• Change in the modelling. In app.py, the main() function, set filename to data/threads-300-set1_features.json.
• Remember to Save the model.

Participant 2

Similar to Participant 1 but change the data to the second subset.

• Change in the interface. In demo/threadall/index.js, set the value of dataFilePath to ../../data/threads-300-set2_features.json.
• Change in the modelling. In app.py, the main() function, set filename to data/threads-300-set2_features.json.
• Remember to Save the model.

Study 2: Model validation

Evaluation interface: http://localhost:8080/demo/evaluation/.

Evaluate the validity of one of the two models built above, say model from Participant 1. Make sure the data for collecting ground truth comes from a different subset, which is data/threads-300-set2_features.json.

• In demo/evaluation/index.js, set modelFilePath to the .json model that Participant 1 saved (could be in Downloads folder).
• In the same file, set dataFilePath to data/threads-300-set2_features.json.
• Remember to click Submit to save the threads that are labelled. Note that you can submit at any time; do not need to label all of the threads.