More on the Topic of Experience in Psychotherapy
In one of the last posts I summarized three articles that all came to the conclusion that there was no big difference between novice and expert therapists. I came to the conclusion that the tasks used were presenting ready-to-judge material, which was in no way ecological valid. I hypothesised that the true merit of expertise would rather lay in the skill of information gathering and a vast network of knowledge including contra-indications and possible developmental pathways that need to be taken into account. The article by Kuyken, Fothergill, Musa and Chadwick (2005) The reliability and quality of cognitive case information further elaborates on the process that take place in therapy and focuses mainly on the stage of information gathering. The authors had participants of different experience levels make case formulations using the J. Beck Case Conceptualization Diagram (CCD). 115 mental health practitioners were given extensive information about a hypothetical case (Anna), which included an account of the presenting problem, a psychosocial history, observation of the client during the assessment, standardized psychological assessments and a multi-axial diagnosis. Quality of the case formulation was rated with the Cognitive Case Formulation Scale and was defined as a ‘parsimonious, coherent and meaningful account of a client’s presenting problems in cognitive therapy terms. Interrater reliability for the judges who rated the CCD was good, although the rating of dysfunctional assumptions (in subcategories) only reached a kappa of 0,63. The case formulations of the participants were rated against a benchmark formulation provided by J. Beck and ranged from very poor to good. Only 44,2% of the formulations were found to be at least “good enough”. Such benchmarking by an allegedly expert must be interpreted with caution, because the agreement with the systematic approach of the CCD is measured and not validity of the case formulation. Chi-square analysis revealed that the quality of the formulations did differ between the three groups (novice, experienced and expert). Participants who had a professional qualification or a BABCP accreditation provided case formulations of a better quality than the other participants. So for the time being I keep on believing that the true expertise is found in the systematic and comprehensive information gathering and not in the interpretation of (hypothetical) data. I would like to see an experiment in which novices and experts talk to a (real) client and gather information. The information that is produced by that session could then be presented to other novices and experts. Those would have to make several predictions about the patient that could be verified in a longitudinal study. I hypothesis that variance on the second task will mainly be explained by the expertise level of the first therapist with almost no interaction with the expertise level of the second therapist.
Differences between Clinicians with Different Levels of Experience
The study of Hillerbrand and Claiborn is a 2 (level of expertise)x3 (levels of problem structure) design. 17 expert and 15 novice psychologists with good diagnostic abilities read well structured, ill structured or random structured cases. Consistency of the relevance of the information supplied varied between the cases. Relevance was the extend to which information necessary for a particular diagnosis was identified in the case. As dependent measure the accuracy of the diagnose was used, as well as the number of diagnoses, the rationale for the diagnosis and the predictions for future behaviour.
Significant differences between clinicians and novices were on ratings of their knowledge, confidence and perception of the cases’ clarity. The cognitive process however did not seem to differ. Clinical reasoning skills have been found to be present in lay-persons. It might therefore be hard to distinguish an expert from a layperson, as experience is not a sufficient condition for expertise. On the other hand you can wonder is a second or third year student is a lay-person. Certainly the training received so far will have had some impact on the cognitive processes at hand (hopefully).
The task itself is also important to consider: Whereas clinical therapists mostly have a face to face conversation with the client, in experimental studies (such as this one) the interaction is often substituted by a text about a clinical case. This way of data perception might be more appropriate for a undergraduate student, who is used to reading cases, but less used to direct therapeutic contact.
In the study of Brammer participants came from a doctoral program (n=83) or were graduate students of psychology. With the use of a computer program the structure of information exchange between the therapist and the client was simulated. After reading the case the participants could ask questions that were answered by an artificial intelligence that used pregenerated answers. As the participants asked more questions, the answers supplied additional information concerning the case. The diagnosis was sent to a panel of experts who voted the fit a the diagnosis on a 4-pooint Lickert from unlikely to definitive. The possible diagnoses were restricted to major depression, Bipolar II disorder, alcohol abuse, acute stress disorder, borderline personality disorder, nicotine dependence.
The number of diagnostic questions asked were related to correct diagnosis, as well as the level of training and the years of experience. Due to the correlational evidence no causal assumption can be made. For example the number of diagnostic question was also related to the total number of questions. Path analysis revealed there was no direct effect of level training or years of experience, instead the effect was mediated by the numbers of diagnostic questions asked.
Although the task is not a real interaction with full non-verbal communication, the interaction simulated between clinician and client was differed from a pure evaluative task. The clinician was also required to perform a skill.
The third study also compared novice and experts, but introduces intermediate level experienced clinicians as a control group. All 10 cases used as material were prerated by a panel of experts. The DSM diagnosis and the discussion was given for every case. Difficulty was enhanced by cutting back the length of the description by two thirds.
The time it took for the participants to reached a conclusion (indicate the diagnostic classification) was measured. The dependent measurement was the number of correct answers each participant gave.
The pattern that emerged form analysis can best be explained by a third-degree polynomial function. Therapists without any experience start of giving about 40% correct answers. As experience increases a decline in the percentage of correct answers is visible (less then 30% in between the second and third year of inexperience). The curve then mounts again and nears asymptotic level at about 40% correct answers.
Both experts and novice took longer to come to a conclusion than psychologists of intermediate level. This pattern also emerged in the previous study where the number of (diagnostic) questions was related to categorization quality.
Other studies who compared sub experts and experts in their cognitive structure of decision making found that there was only a difference, when time was limited. This might be the case on a face to face interaction, when decisions to pursue a specific topic have to be taken during the interaction. It might well be that the sub experts are equally able to judge the criterions of mental illness in a written form but would fail to achieve this knowledge in a real life interaction, because they do not have the conversational skills necessary.
I believe that the most important factor is to have a clear understanding of the scientific knowledge of a specific illness and be able to get all related information necessary from the client. This might be easier with a well-structured problem than with a difficult case including multiple comorbid illnesses, because the possible causal relations are much not too complex.
This hypothesis is also supported by the findings of the study of Brammer, where the quality of the diagnosis was related to the numbers of diagnostic questions asked. The difference between experts and novices can be found in the process of information gathering, rather than evaluation of the acquired data. Training new therapists might indeed by quite straightforward teaching them to ask more questions and shape the questions in a way so that the answer yields diagnostic value.
I imagine standing in china in front of a coke-vending machine. If I am an expert (and if I know Chinese) I know which button to push to get the coke. The way I see it giving cases to lay-persons is like telling exactly which button to push. Taking out the coke is easy and I do not expect Chinese and Europeans to do that differently. So the most important thing, the general background knowledge, that makes you remember to ask about manic episodes when hearing of depression, is not considered in these studies. The crucial questions to study how clinical psychologists make diagnostic classifications, might not be the ones that are asked, but those that are not asked.
Behavioural Science Blog 