Investigate Incomplete / Missing Data

[alexsb]

Some recent studies have data flagged as incomplete in reVISit, but participants have completed the study on prolific, though the data is shown as complete. So something is wrong on how complete is set in reVISit.

A few select cases did exhibit total data loss, except for study metadata. These could be "cheaters", but we will have to investigate more closely.

@abhsarma @ZachCutler04 @JackWilb @hilsonshrestha please track the status of these issues with all study URLs and participant IDs that fall into each category in this issue.

[jaykim1213]

I think these are worth checking:

1. Network loss mid-study
   Might be related to the incomplete cases we're seeing

2. wifi -> cellular switching(or celluar ->wifi) mid-study
   I'm wondering if this would create a new participant and leave the existing one as incomplete

3. Page refresh

4. Cheaters
   I think one way to check for cheaters is looking for duplicate IPs in the participant metadata

[npiccolotto]

We also observed a couple of people (5% maybe?), who were flagged as completed in Prolific but Revisit

1. says they didn't pass a comprehension check, or
2. shows incomplete data (e.g., some have a section in the middle of the sequence missing, for another we got only the last couple of trials and the open feedback mentioning "loading issues"), or
3. doesn't show the participant at all (1 case).

We didn't have time to look into it and we also didn't know in which case the culprit is likely to be Prolific or Revisit. The first situation mentioned was definitely the most common.

There's also the fact that many participants are marked as "in progress" in the Participant View even though they completed all trials, but that's maybe another thing.

[alexsb]

FYI, these issues have tentatively been addressed with the latest release; we're waiting for a bigger test to close this issue though.

[jaykim1213]

Some ideas:

1. If cloud storage fails to connect, show a “Storage Disconnected” modal that cannot be closed unless storage reconnects. Add a refresh/reconnect button to the modal. Currently, Storage Disconnected modal has "Continue Study" button which allows to resume study.
2. Only show the study end page (which has Prolific completion URL) after the participant is actually marked completed in the Firebase/Supabase. Currently, if a participant closes the storage disconnected modal and finishes the study, their data may only be saved to local storage, but they can still see the study end page containing the Prolific URL. I believe this may be one of the causes of participant data loss.

Longer-term idea:

If a participant still can’t connect after 3 refresh/reconnect attempts, we could let them continue with local storage. At the end, we’d automatically download their study data as a JSON file, show a modal asking them to send it to the Study Designer (using the contactEmail from uiConfig), and maybe prefill an email using the contact email in the study metadata.
We’d also need an “Add participant data manually” button somewhere in the analysis page so the Study Designer can restore that data later by uploading the file they received through email.
This might not be the best idea because the participant may forget to send their data, and it creates more manual work for both the participant and the Study Designer. However, it does provide a fallback instead of losing the participant data completely.