 |
 |
You've asked a very complicated question. After mocking up your data in some 2x2 tables and calculating specificity and NPV, it seems to me that the sensitivity and PPV are coincidentally tied together in your example in a way that may not always be the case. This has to do with the amount of overlap in the items your search terms identify and the specificity of each separate term. In this particular situation, there was only moderate overlap in the true positive items identified by each term alone; and all three terms had very high specificities alone (92% to 97%), and very high NPVs (all 97%). Thus, when you added the terms together you picked up a number of new true positives (the nonoverlaps), without adding alot of false positives (high specificity and NPV). With all three terms the sensitivity was raised from 17 to 27% for the terms alone to 59.4% for all three together, while the specificity only dropped to 83.7%. If there had been more overlap, the sensitivity would not have risen as much. If one or more of the terms had been substantially less specific, alot of false positives would have been added, driving down the PPV. This all would seem to be coincidental and highly dependent on the items searched for and the terms used. There is certainly no mathematical rule that guarantees that you will always be able to improve sensitivity by adding terms without losing specificity and PPV. If you plotted this data out on an ROC plot (a graphical way of examining the tradeoff between sensitivity and specificity) you would see that by adding the terms together you were not merely trading more sensitivity for less specificity (moving to a different threshold on the same curve). Instead you actually moved your instrument effectiveness to a higher curve (meaning both sensitivity and specificity were improved). You also changed your threshold somewhat, so that your specificity did drop some, but not as much as it would have if you had stayed on the same curve. Whether the curve moves seems to be entirely situation dependent. In other situations you could end up on a lower curve. In medical terms this is analogous to using the same diagnostic instrument, but looking for a different disease, with a different prevalence, different symptoms, and different overlap in symptoms. David L. Doggett, Ph.D. Senior Medical Research Analyst Technology Assessment Group ECRI, a non-profit health services research organization 5200 Butler Pike Plymouth Meeting, PA 19462-1298, USA Phone: +1 (610) 825-6000 ext.5509 Fax: +1(610) 834-1275 E-mail: [log in to unmask] > -----Original Message----- > From: Reinhard Wentz [SMTP:[log in to unmask]] > Sent: Tuesday, June 13, 2000 6:17 AM > To: [log in to unmask] > Subject: Sensitivity / PPV / Specificity > > While on the subject of sensitivity, specificty and PPV: > > I know that there is a always a trade-off between sensitivity and > specificty in screening tests or, for me as a librarian, literature > searches: If I increase sensitivity by combining terms using the OR > operator I will always reduce specificity. > > This does not seem to be the case for sensitivity and PPV. > > In the exmaple below: When I COMBINE terms to increase sensitivity above, > say 40%, PPV (librarians prefer to call this 'precision') 'stabilizes' at > about 12%. > > Does this make sense or is my math/stats weak? If correct: does knowing > this phenomenon help when designing screening programmes (or search > strategies)? > > Reinhard Wentz > _________________________________ > > > On a database with some 1750 records with 64 known true+ > > > Term > Records found True+ Sensitivity Precision(PPV) > experimental 58 11 17.2 18.9 > before 125 16 25.0 12.8 > impact 159 17 26.6 10.7 > > before > or impact 270 29 45.3 10.7 precision (PPV) > stays the same? > > before > or impact > or experi- > mental 313 38 59.4 12.1 precision (PPV) goes up! > > ______________________________ > > > > > > ************************************************************* > > Reinhard Wentz, Dipl. Bibl. Tel. 020 7242 9789 Ext 2291 > Cochrane Injuries Group Fax. 020 7242 2723 (Attn: R. Wentz) > > Institute of Child Health e-mail [log in to unmask] > 30, Guilford Street > London WC1N 1EH > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%