Hello Jennifer,

The lowest that you would want to set the threshold is dependant on the similarity scores that you see in your duplicate groups. The similarity scores are often dependant on the completeness of your imported items.  If you have missing data in your records then your similarity scores will be quite low. If you find that some of your author names use complete full names while others use initials there will be lower similarity scores.

What I normally do is run it at the default initially (1.0) to complete as many groups as possible. I then look through the incomplete groups to get a feel of the range of similarity scores. Running it again at 0.95 (as you did) is what I would have done. I would then look at a number of incomplete groups to see the similarity scores and make an estimate of how I could go.  If I could find any items that had similarity scores above 0.9 that weren't duplicates then that would give me a good indication. If everything above 0.9 was a duplicate I would run it again at that level to catch as many more groups as possible.

I would continue to lower the score as long as I couldn't find a group that wasn't a duplicate. The lowest I have probably gone is about 0.85 but that was based on the similarity scores I was seeing in my data.

Best regards,

Jeff