Use Case: Evidence Synthesis / Method and Source Analysis

[TNRiley]

Analyzing sources and methods as a part of an evidence synthesis project is one of CiteSource's use cases.

This discussion board is open to anyone who would like to contribute ideas, ask questions, etc. on this topic.

[puzzledhedgehog]

Hi folks, I am thinking about using CiteSource to refine a search for an update, by checking how much I can remove (sources or lines) while still retrieving the articles that have been screened in.
[Not sure if maybe this should go in the benchmarking discussion? Hard to draw the line between comparing methods and benchmarking per se]
First I'd do "normal" database searches + screening, then maybe the search summary table or citation record table will suggest that I can skip one or more of the sources I used originally?
With the input data I generally have on hand (results sets from different databases, which I can import with cite_sources tags), it would be easy to consider dropping a few marginally valuable databases. But how often does it happen that a whole database is truly unimportant?
On the other hand, to consider dropping a few marginally valuable search terms -- it would require quite a bit of prep to generate the results set to compare, say, ScopusMethod1, ScopusMethod2, and ScopusMethod3. And if I did, I would probably try it out live in the database, instead of downloading the different results sets and importing to CiteSource....
But maybe CiteSource could make that (trying out different iterations in individual databases) easier?
It would be really convenient if I had an easy-to-copy-and-paste query with unique identifiers of all the relevant records available in source1, source2, source3 etc. The citation record table gets me most, but not all, of the way there.
Just thinking aloud, though -- just because my original search in source1 retrieved Included Articles A, B, and C but not D doesn't actually mean that D isn't available in source1; maybe it's there and my query just missed it. So maybe, such a "target article" list should include identifiers for everything, whether or not it was in my search results from a particular source....

[TNRiley]

This is more in line with benchmarking, but fits here too ;)

What I would recommend would be to use the upset plot. Your question about how often a database is truly unimportant is a good one, it's one reason we built this tool. While a source or method may not be completely unimportant, there are a lot of factors to be considered when choosing to include/exclude a source/method. Through the wide use of CiteSource, we're hoping that sources can be assessed over time within various fields for future decision making.

Check out the benchmarking vignette, which was a slice in time from a real systematic map, and in that work, I was able to iterate a number of search strings to find the impactful strings. This was much faster than trying to search through the results live in the database and also gave me a way to present the information to the subject matter experts and explain how each string variation worked.

You'll need to know which articles are actually indexed in the source that you are testing. Again, check out the benchmarking vignetted and you'll see that not all of our benchmarking articles were indexed in WoS, so all we were gauging against was the true number that could potentially be found.

[puzzledhedgehog]

Re method and source analysis: I just made some progress on a review where there are

• documents we found via the db searches
• documents we found via citation chaining
• a document that was in the original round of db search results, was excluded at the title-abstract stage because the abstract made it seem irrelevant, was in the citation chaining results but was deduplicated out because it had already been excluded, and then eventually was "recovered" because the first author was wondering, "hmm, there's a lot of trials registrations in the [included] set, but most of them have 1+ related publications. Why does this weird one have a registration and no publications?"
  If I wanted to use CiteSource to do some comparisons and understand the impact of deduplicating the citation chaining output against records that have already been screened out (in terms of the saved workload -- obviously, without rescreening all those duplicates again, I wouldn't actually know which, if any, were actually excluded incorrectly the first time), I guess I would need to start with:
• a file with the database search results (deduplicated or not doesn't matter I think?), called A
• a file with the citation chaining results (deduplicated within itself), called B
• a file with the citation chaining results AFTER it had been deduplicated against the already-screened records? Or maybe I would let CiteSource do that for me? Somehow I need to end up with B-A.
  If B-A is a ton of records, then maybe it's worth the risk of (failing to identify and fix mistakes from the title-abstract screening phase). But if B-A is only a small set of records, then maybe we should re-screen them, for more confidence.
  CiteSource experts -- do you think this is a situation where I should use your tool?
  Thanks!