Suppose you are interested in the level of a state variable (e.g. a disease is present or absent or of a pre-specified level of severity, a failure is recorded or not, etc.) and have a potentially useful but imperfect diagnostic test method, (e.g. a blood test result for this disease, or a quality control check for a manufacturing failure, is either definitely positive or not) . How do you interpret the result of the diagnostic test for the level of the state variable when the some or all of the information underlying the inference is ambiguous (imprecise)?

The first video clip basically walks through how to use and interact with  the graphical reasoning aid for posterior inferences that Phil and I have developed for the Mathematica Demonstration project. To actually use the player file yourself you need  this mathematica notebook (it has a .nbp extension) and  the freely downloadable Mathematica Player . There are 3 steps to getting going here:

1. Go to  Wolfram (makers of Mathematica)  and download the free Mathematica Player . You'll need to install the player, but usually your operating sytem will guide you through this.
2. Download the .nbp file from this page  or go to for the Mathematica Demonstration project and download the file on ambiguity we have published  there any other interesting demonstration. Look for an orange tab at the top left that says "download live version"
3. Open it from the Player and use the sliders to explore.

Of course the real trick here is to understand what it is you are doing - ie concepts concepts concepts! The additional information in this post just below the video clips contains a simple exposition of the logic of the interface. For a readable introduction to ideas of ambiguity have a look at a paper Phil Gunby and I have written.  It's a bit overstated but it does get across the point: namely it's not only being able to calculate inverse probabilities from precise numbers expressing sensitivities, specificities, and base rates, but also to explore how much ambiguity in any one or more of those inputs creates ambiguity in the final output (the inverse probability you are interested in).

The second video clip  is a quickie tutorial style webcast demonstrating  a graphic pencil and paper approach to inverse probability reasoning - ie how you can use graphical methods and robustness analysis for posteriro inferences when the basic inputs into that posteriro analysis are not precise.

First video clip: an introduction to the basic dynamic interface in the Mathematica Player used for exploring the effects of ambiguities on inferences:


Second video clip: back of the envelope type diagrams for examining ambiguity in inverse inference.



Let S be the logical truth value (1 or 0) of a proposition about the state variable (e.g. a disease is present or absent or a pre-specified level of severity, a failure is recorded or not, etc.), and let D be the logical truth value of a proposition about the outcome an imperfect diagnostic test for the state (e.g. the test result for this disease or failure is either definitely positive or not). Then our question is, in the language of statistics: how do (and should) people conceptualize and calculate a posterior inference about S after having observed some D when some or all of the underlying information about D, about S, and about the relationship between D and S is ambiguous (imprecise)?

From a statistical perspective there are 3 precise numerical inputs that feed into a coherent posterior inferences about binary valued S after having observed the result of the binary valued diagnostic signal D: a sensitivity number, a specificity number, and a base rate number. The first two numbers  characterize uncertainty about the results of the diagnostic D under two different information conditions about the state S. The sensitivity number expresses uncertainty about whether the diagnostic test D will be positive, i.e. D=1, assuming that S=1 is true. The specificity number expresses an uncertainty about whether the diagnostic test D for S=1 will be negative, i.e. D=0, assuming that S=0 is true. The third number, the base rate number, characterizes uncertainty about the the binary state variable S in the absence of , or prior to knowing, any diagnostic information D.

These numerically precise inputs can be presented in a variety of logically equivalent frames or formats to someone who wishes to make posterior inferences about the state variable S. Two stylized facts are known: (1) a majority of ordinary yet intelligent people, lay and professional alike, do not perform the posterior inference task well and (2) these same people typically do worse when information on the 3 numbers is presented in the standard probability formats favoured by statisticians compared to natural frequency formats favoured by behavioural psychologists (see references below involving Gigerenzer). Sometimes tabular and graphical means of communicating the relevant numerical information along with conventional descriptive textual methods are helpful decision aids in either format, although they are by no means panaceas for the problem. Yet virtually all research into the problem (statistical innumeracy) and the potential solutions (alternative formatting aids) ignores the problem of ambiguity. Research shows that decision makers recognize the inherent incompleteness underlying the numerical information presented, no matter what the format, even if they don't know exactly how to incorporate these ambiguities into their inferences. This interactive Demonstration is designed to facilitate the "what-if" exploration of the effects of ambiguities (imprecision) in sensitivity, specificity, and base rate information, alone or in combination, on posterior inferences through a linked tabular natural frequency and graphical probability format representation of underlying uncertainties .

Figure: static picture from Mathematica interface
























The table and the graphic in the Demonstration are set up in the following way. The truth table in the Demonstration shows the four logical possibilities for the two propositions S and D being true or false together. Frequencies or counts (hypothetical, although possibly based on some direct observations)  for each of the 4 logical possibilities in the columns (cells) are specified initially so that for 20 out of 100 the proposition S is true, S=1, while for 80 out of 100 the proposition S is false, S=0. This, gives a base rate for the truth of the proposition S of 20 out of 100, or 20%, shown as a triangle on the x-axis at 0.2 in the figure. At the initial values for the sliders, the sensitivity of the diagnostic test P(D|S=1) is set at 16 out of 20 or 80%, shown as a circle on the right hand margin of the graph; i.e. of the 20 cases where S is true, S=1, 16 also show a positive diagnostic result, D=1.  At the initial values for the sliders, the specificity of the test, P(D=0|S=0)=1- P(D|S=0) , is set at 56 out of 80, or 70%, shown as a circle on the left hand margin of the graph at a height of 30% , 100%-70%: i.e. of the 80 cases where S is false, S=0, 56 or 70% don't have a positive diagnostic, but 24 or 30% do. The resulting precise posterior probability for S being true given a positive diagnostic, P(S|D=1), is 40% or 0.4 in a probability format or 16 out of 40 (from 16+24 cases where D=1) in a natural frequency format, shown as a large square box on the x-axis top margin. A corresponding smaller square on the x-axis along the bottom margin finds the level of the other posterior probability, P(S|D=0). This is, the posterior probability for S being true given a diagnostic outcome that is not positive. The graphic deliberately does not focus on this posterior inference, as we are concentrating attention on the question: how should one interpret a positive diagnostic signal, or D=1? Note, there is a third probability , the base rate or the (unconditional probability) for the diagnostic signal, here 16+24=40 out of 100, or 40%, which can also be calculated given the sensitivity, specificity, and base rate numbers.
 

There are many interesting probability assessments in this simple model, but only 3 logically independent ones. The dotted and dashed lines in the figure are two linear constraints on a coherent inference process. There are two "base rates" , P(S) and P(D), one for the state variable S  and one for the diagnostic test D. P(S) is the marginal or unconditional probability of the proposition that S is true, S=1, i.e. that the underlying state variable is at the pre specified level. P(D) is the marginal or unconditional probability for the diagnostic test result being positive, i.e. that D=1 is true. It is important to recognize that these base rates are not logically independent of one another. The chances that S is true,  S=1, written as P(S=1) or in shorthand P(S), is a weighted average of the chances of S being true with a positive diagnostic, P(S|D=1), and the chances of S being true with a nonpositive diagnostic, or P(S|D=0). The corresponding weights are the chances P(D=1) of a positive diagnostic and the chances of a non-positive diagnostic, P(D=0)=1-P(D=1),  that is:
     P(S) = P(S|D=1)*P(D=1) + P(S|D=0)*P(D=0).

The dotted/dashed line between the squares is all combinations of {P(S),P(D)} that satisfy this equation for the given endpoints, which are conditional probabilities. . At the same time, the base rate or marginal probability P(D) for positive diagnostic results must be an appropriate weighted average of positive diagnostic results when S is true (sensitivity) and the positive diagnostic results when S is false (1 minus specificity) : that is,

    P(D) = P(D|S=1)*P(S=1) + P(D|S=0)*P(S=0).

The dotted/ dashed between the two circles plots all pairs {P(S),P(D)} satisfying this relationship. The intersection of the two lines solves for the unique pair of base rates for S and D, {P(S),P(D)}, that satisfies both linear relationships.
 

Changing any of the 3 components of one of the linear relationships means the components of the other relationship change as well. The Demonstration is set up so that the endpoints (capturing the sensitivity and the specificity settings) and base rate along the dotted/ dashed between the two circles can be changed by the sliders, and the endpoints of the corresponding changes in the dotted/ dashed between the two squares trace out the relevant posterior inferences, P(S|D=1) and P(S|D=0), with the emphasis on the former. There are two sets of sliders, one for reference purposes, the other to examine the impacts of changes in the underlying sensitivity, specificity and base rate information, either separately or jointly. For example, starting out with the initial values of sensitivity of 80%, specificity of 70% and base rate of 20%, changing one or all of the top set of 3 sliders alters the posterior inferences - but leaves visible the reference specifications. Of course the reference specification itself can also be changed by changing the sliders in the lower box. A fuller explanation of the coherency relationships involved in Bayes Theorem is available in the Demonstration Project titled "Bayes Theorem and Inverse Probability". Project Bayes Theorem and Inverse Probability.
 


References
Edwards, Adrain and Gigerenzer Gerd, "Simple Tools For Understanding Risks: From Innumeracy To Insight" British Medical Journal , Vol. 327, No. 7417 (Sep. 27, 2003), pp. 741-744.
Gigerenzer, Gerd and Ulrich Hoffrage (1995)  "How to Improve Bayesian Reasoning Without Instruction:
Frequency Formats" Psychological Review, 102 (4), 684704
Han,Paul K. J. ,  William M. P. Klein, Thomas C. Lehman, Holly Massett, Simon C. Lee and Andrew N. Freedman (2009) "Laypersons' Responses to the Communication of Uncertainty Regarding Cancer Risk Estimates "  Medical Decision Making , Vol. 29, No. 3, 391-403
Lad, Frank Operational Subjective Statistical Methods: A Mathematical, Philosophical, and Historical Introduction Wiley-Interscience (1996)
Mukerji, Sujoy "Foundations of Ambiguity and Economic Modelling" Economics and Philosophy, 25 (2009) 297-302
Schapira M, Nattinger A, McHorney C. Frequency or Probability? A Qualitative Study of risk Communication Formats Used in Health Care. Medical Decision Making 2001 ;21:459-67. 
Snapshot 1: A basic starting point
Snapshot 2: Changing the base rate
Snapshot 3: Changing the specificty