HelpForum

Forum (Archive)

This forum is kept largely for historic reasons and for our latest changes announcements. (It was focused around the older EPPI Reviewer version 4.)

There are many informative posts and answers to common questions, but you may find our videos and other resources more informative if you are an EPPI Reviewer WEB user.

Click here to search the forum. If you do have questions or require support, please email eppisupport@ucl.ac.uk.

<< Back to main Help page

HomeHomeUsing EPPI-Revi...Using EPPI-Revi...Questions about...Questions about...Inter rater reliability and code allocationInter rater reliability and code allocation
Previous
 
Next
New Post
15/06/2016 15:21
 

I have two possibly related questions:

1.) I have three reviewers (myself included) screening articles based on title and abstract. I want reviewer 1 and 2 to screen 95% of articles together, I want reviewer 3 to screen 5% of articles screened by reviewer 1 and 2 and 5% with each individual reviewer (1 or 2). Is there a way to do this? I should add that reviewers 1 and 2 have already coded about 1/3 of the articles. If possible, I'd like the articles that reviewer 3 screens to be randomly generated.

2.) Is there a way to calculate inter-rater reliablity in EPPI for include vs. exclude AND for reasons excluded (we have about 8 reasons to exclude and article and a reviewer can select multiple options)? How would I go about doing this? Is there a way I can be strategic in the article assignment to aid in this process? It is more important to be able to easily calculate interrater reliablity than to allocate the screening assignment in the way I described above.

Thank you for your help!

 

 
New Post
15/06/2016 16:44
 

Hello Cara,

What you have described doesn't quite add up if I understand correctly what you wish to do. There is a mixuture of double and triple screening. If Reviewer 1 and Reviewer 2 double screen 95% of the items (this is actually double screen 90% and triple screen with Reviewer 3 another 5%)  and you want Reviewer 3 to double screen another 5% with just Reviewer 1 and a further 5% double screen with just Reviewer 2 you end up with 105%.

What you could do is have Reviewer 1 and Reviewer 2 double screen 90% of the items (which is actually double screen 85% and triple screen the last 5% with Reviewer 3). You can then have Reviewer 1 and Reviewer 3 double screen 5% and then have Reviewer 2 and Reviewer 3 screen another 5%. This would add up to 100%

If that is what you had in mind please let me know and I will describe how you would set up the coding assignments to do this.

Best regards,

Jeff

 
New Post
15/06/2016 22:35
 

The approach you described is what I would like to do. Sorry for the confusion!

 
New Post
16/06/2016 15:55
 

Hello Cara,

You will need to create 4 allocation codes. I will call these 'Group 1', 'Group 2', etc.
Group 1 - 85% of items for double screen
Group 2 - 5% of items for triple screen
Group 3 - 5% of items for double screen
Group 4 - 5% of items for double screen

You will also need a number of coding assignments:
Reviewer 1 - Group 1
Reviewer 1 - Group 2
Reviewer 1 - Group 3
Reviewer 2 - Group 1
Reviewer 2 - Group 2
Reviewer 2 - Group 4
Reviewer 3 - Group 2
Reviewer 3 - Group 3
Reviewer 3 - Group 4

This should give you the combinations that you have described.

Best regards,

Jeff

 
Previous
 
Next
HomeHomeUsing EPPI-Revi...Using EPPI-Revi...Questions about...Questions about...Inter rater reliability and code allocationInter rater reliability and code allocation


Copyright 2021 by EPPI-Centre :: Privacy Statement :: Terms Of Use :: Site Map :: Login
Home::Help::EPPI-Mapper::RIS Export::About::Account Manager