Challenges to Developing Routine Outcomes Evaluation in Different Practice Settings and Cultures: A Naturalistic Enquiry in Spain and the UK

A naturalistic sessional evaluation of routine outcomes of psychotherapy from a range of theoretical orientations including transactional analysis, using standardised measures for depression, anxiety, general distress and working alliance, was conducted across completed therapy interventions by 113 therapists with 263 clients within an academic institution in the UK and across stages of therapy by 10 therapists with 26 clients in three independent clinics in Spain. Outcomes in both countries demonstrated clinical gains but it was found that such evaluation methodology was more easily applied within a training institute than in private practice; it also appeared to better fit the UK professional climate of evaluation. Suggestions are made concerning the introduction of such research in future.


Introduction
Transactional analysis psychotherapy is conducted in a number of practice settings and cultures.This project was funded by EATA (European Association for Transactional Analysis) in 2011 to explore the possibility of developing psychotherapy evaluation within settings that included private practice and academic organisations in different countries.In this case, one site was a large low cost clinic situated within a training institute in the UK where evaluation has already been established (Van Rijn & Wild, 2013), and the others were independent psychotherapy clinics in Spain, where this type of evaluation was new.In both countries, transactional analysis psychotherapy was practiced alongside other approaches to psychotherapy.This paper presents the research and outcomes in these sites and discusses challenges and learning arising from it.
The paper offers a set of conclusions and recommendations for implementing naturalistic evaluation within non-academic psychotherapy practice settings

Literature
Choice of methodology in evaluation of psychotherapy is particularly important in the field, with a great divide between practitioners and researchers and an equally great need to develop a research evidence base and gain statutory recognition.In transactional analysis an international network with agreed standards of certification and evaluation potentially offers an opportunity to develop multi site, multicultural research that could explore and evaluate the application of the approach in different contexts.
Developing a research evidence base for effectiveness and efficacy of psychotherapy is of great importance to the recognition of theoretical approaches.This could be achieved using different methodologies.The current 'gold standard' of research in many health and statutory settings is still a randomised control trial, which gives findings about 'efficacy' of psychotherapy.Theoretical approaches with this type of research evidence (such as CBT), have been accepted as treatments in statutory clinical guidelines within the UK.This type of research has also given findings about the efficacy of psychotherapy in general through different metaanalyses.
Using these findings, (Lambert & Barley, 2002;Lambert & Ogles, 2004;J. C. Norcross & Wampold, 2011;Wampold, 2001), emphasised the so called 'common factors' in psychotherapy rooted in the therapeutic relationship, over the theoretical approach.However, randomised control trials are normally beyond the resources of ordinary practitioners and clinics.They require a highly structured approach to research design in order to demonstrate causality.This relies on develop-5 (2), 28-34 https://doi.org/10.29044/v5i2p28ing a representative clinical sample and requires randomisation and availability of control groups, as well as research specific recruitment, training and evaluation of the therapist's work, normally involving a treatment manual.
A different approach to measuring effectiveness of practice -'practice based evidence' -involves evaluating the flow and outcomes of therapy as it is practiced in ordinary settings. .In the UK, recognition of prevalence of problems such as depression and anxiety in the population by the Department of Health (DoH, 2002) and the establishment of the stepped care model for treatment in the National Health Service (NHS), have emphasised the importance of collecting routine outcome data in order to develop the quality of services (CSIP Choice and Access Team, 2007).Routine evaluation of outcomes in these settings involves sessional evaluation of psychotherapy using standardised questionnaires.Generic counselling has also been evaluated in primary care in individual effectiveness studies (Mellor-Clark, Connell, Barkham & Cummins, 2001;Stiles, Barkham, Twigg, Mellor-Clark & Cooper, 2006, (Connell, Barkham, & Mellor-Clark, 2008) and systematic reviews (Bower, Rowland, & Hardy, 2003;Hill, Brettle, Jenkins, & Hulme, 2008).All demonstrated its effectiveness in primary health care.
However, practice based evaluation faces a lot of challenges.Clients present to therapy with a range of issues.They are not randomly assigned to treatment conditions and although we can measure the changes they achieve, our findings will always be limited.We cannot ascribe their change to therapy, because we are unable to control and measure for other variables.The percentage of clients completing the measures is often very limited and there is frequently no evidence of the approach practiced by therapists.This limits the internal validity of the research design (Clark, Fairburn, & Wessely, 2008).
To an extent this could be counterbalanced by their external validity, or the fact that they are generally representative of clinical practice.(Stirman, DeRubeis, Crits-Christoph, & Brody, 2003) and have a potential to develop it (Rao, Hendry, & Watson, 2010).
Therefore, in choosing methodology useful to ordinary practice settings we need to consider a type of evaluation that would enable comparison to national and international benchmarks and comparisons between different sites.An important aspect of this is also in choosing a methodology that the therapists and clients can engage with and that has a potential to enhance therapy.Naturalistic evaluation has already been in use in student practice within Metanoia Institute in the UK (Van Rijn & Wild, 2013) and in health settings (van Rijn, Wild, & Moran, 2011) and demonstrated good completion rates and engagement with the methods of evaluation.It was therefore of interest to explore whether this methodology could be used again within a different culture and setting.

Research Aims and Methodology
The aim of the research team was to test out the methodology of an open, non-randomised, practice based evaluation model in two different types of settings and find out whether it was successful in measuring outcomes and able to engage the clients and therapists.Both were measured by the levels of data completeness which needed to reach at least 90% to comply with IAPT benchmarks (Clark, Layard, Smithies, Richards, Suckling & Wright , 2009) and repeat the outcomes of the previous studies (Van Rijn & Wild, 2013;Van Rijn, Wild, & Moran, 2012).If this was the case in both settings, it would suggest that this method of evaluation could be appropriate in different cultures and practice settings.Outcomes of therapy were also measured in order to illustrate how this type of evaluation could contribute to developing an evidence base for an approach.

Research Sites
Metanoia Counselling and Psychotherapy Service (MCPS) in the UK The counselling and psychotherapy service where the research took place has been operating since 1995.MCPS provides low cost counselling and psychotherapy to the general public, who self-refer to the service.Treatment can be extended up to one year, depending on the client's need and availability.The wide range in the length of therapy is unusual in the statutory services in the UK, which tend to offer a set number of sessions, rarely extending to 20.
MCPS became a research clinic in 2010 following an evaluation project in primary care (van Rijn et al., 2011).The outcomes within this current project have been measured between September 2011 and April 2013.

Independent Clinics in Spain
Three independent clinics located in different parts of Spain participated in the research clinic.All the Spanish research has been coordinated and managed by the Institute for Psychosocial Studies "Xoan Vicente Viqueira" (University of La Coruña, Spain).Length of therapy was not limited by the setting.

Therapists within Spain
The level of therapists' experience varied.All therapists were working full-or part-time in different private practice clinics and had completed training in either Transactional Analysis or Integrative Psychotherapy. 10 therapists were involved in the project; four of those had more than five years in practice, six had less.There was no data available about their supervision arrangements.

Clients
Clients UK site Clients self-referred to the service.Of the 304 clients during the year, 263 had an assessment session and at least one therapy session.The profile of the clients for the year remained unchanged compared to the years prior to evaluation and reflected the ethnic mix of the area:

Clients Spanish sites
Clients in Spain also self-referred to the service.26 took part in research.Evaluation involved 11.5% of clients who were at the beginning of therapy and 88.5% who were already in therapy, within a range of 0-48 month's duration.
The profile of the clients was:  76.9 % of clients were female.


The average age of the sample was 37, 8 years old, with a range from 16 to 59. 8 % were under 21 and 20 % were over 50 years old.
The number of clients who refused to take part in research was not recorded for either site.

Therapy
Therapy UK Site After the initial contact, clients had an assessment session.The assessment format had previously been developed for the service by the Head of Clinical and Research Services at Metanoia Institute (Bager-Charleson & Van Rijn, 2011) and highlighted presenting issues such as current symptoms and functioning, developmental history and risk.
Assessors referred clients to practitioners for four exploratory sessions.The aim was to decide whether a working relationship and a focus for therapy could be established.A client would be referred on if they decided to change therapist, or if a therapist considered themself unable to meet the needs of a client.
Therapists were taught and instructed to use the outcome measures as an integrated part of relational therapy, as well as for research.These conversations usually took place at the beginning of each session, when clients handed measures back to therapists.Clients' aims were discussed alongside any changes they noticed during the week, whether they were positive or negative.A working alliance measure was used to encourage feedback and attention to the therapeutic relationship.The aim was to encourage responsiveness in therapists, as an essential component of effective psychotherapy (Norcross, 2002).Even though therapists worked within their own theoretical approach, this expectation of careful tracking of clients' responses aimed to develop a pluralistic therapeutic stance (Cooper & McLeod, 2011).
The proportion of each theoretical orientation within the clinic was: Integrative Psychotherapy and Counselling Psychology 46 %, Person Centred Counselling 31%, Gestalt Psychotherapy 11.8%, Transactional Analysis psychotherapy 11.2%.

Therapy Spanish Site
In the case of new clients, therapists undertook a first assessment session which inquired into the presenting problem, developmental history, risk, and previous experiences with the presenting problem.Within this first assessment session, clients were also informed about the research clinic, its objectives and usefulness for the therapeutic process and they were asked for consent.Experienced therapists were instructed to approach all new clients for inclusion into the evaluation, as well as their ongoing clients.
For the Spanish sample, the proportions were: 46.2% Transactional Analysis psychotherapy, 53.8% Integrative Psychotherapy.

Measures
Both sites used the following measures each session:

Ethical Considerations
Clients had a right to withdraw from the project at any time during treatment.Outcomes were discussed transparently between the therapists and the clients.All the data was confidential and anonymised before analysis.
Therapists chose to practice within the research clinic.
The Metanoia Institute Ethics Committee (an independent body approved by Middlesex University) had given an ethical consent to the project.
The project was also approved by the Ethics Committee of the University of La Coruña, which made sure it complied with research ethical codes.

Results
Table 1 shows a percentage of complete data sets for clients who have had one assessment session and at least one therapy session.The only measure completed in under 90% of cases was the relationship measure, WAI, for both samples.

Descriptive statistics UK
The descriptive statistics in Table 2 show that posttherapy scores in the UK were generally low with the exception of the WAI which was high.The standard deviation demonstrates a moderate dispersion of scores from the mean with greater variance observed on the WAI.
Improvement Rates Spanish Sites Improvement rates were calculated to illustrate the outcomes of the therapeutic process within the clinic and in order to understand the outcomes on clients' symptoms and problems, as well as on the working alliance.Criteria for improvement were calculated by the difference between scores at the start and the end of therapy.Table 3 shows the percentages of clients that showed improvement, no change or deteriorated during the course of the evaluation in the Spanish sample.The UK clinic was able to calculate reliable and clinical change for the sample and this is discussed below.Table 5 Percentage Reliable Improvement Rates, UK Site It was found that there were significant differences from pre to post therapy calculated using a T-test with moderate effect sizes using Cohen's d, as shown in Table 6.

Discussion
Contrary to the expectations of the research team, this project has not demonstrated that the same type of naturalistic evaluation could be used effectively in different settings.Although evaluation in each sample suggested that both sites yielded positive outcomes for clients in terms of depression, levels of anxiety and general distress, the results were not comparable due to a number of differences between the samples.The more important learning in this project was in exploring differences and difficulties in establishing credible evaluation within different practice settings and reflecting on increasing levels of engagement.

Outcomes
It was not possible to compare the outcomes in different research sites, because of differences in the sample size and the way in which evaluation was conducted.Therefore, any comparisons serve only as an illustration.
Outcomes within the UK sample demonstrated evaluation of the therapy clients received from the beginning to end.These outcomes were comparable with previously published clinic findings and this suggested their reliability in achieving outcomes in reducing depression, anxiety and general distress.
The sample within the Spanish sites was similar in composition in terms of age, employment rates and gender, even though it was considerably smaller.The evaluation gave a picture of a stage of therapy, rather than psychotherapy outcomes.It was difficult to reach full conclusions about effectiveness because the clients were at different stages of therapy.However, it is clear from the outcomes that the clients have made gains, particularly in the areas of anxiety and general distress.

Project Set Up
The clinic within Metanoia Institute has been established as a research clinic for a number of years.Students who applied to practice within it knew that evaluation was going to be a part of their practice placement.This was often difficult because MCPS was the first placement for the majority of students and their anxiety was often considerable.This was addressed in their one day induction training at the start of the placement.The training day was an opportunity to talk through best practice in using evaluation.
The training day also involved working in triads, in order to practice introducing the measures with clients.The anxiety expressed by students mainly involved fear of evaluation, its possible impact on the clients and the flow of therapy, and additional workload (and costs) of supervision.
Private clinics in Spain had a very different set up.Evaluation has not been a part of their practice and they were geographically diverse.There was no official training by the researcher but information was provided by phone and email contact.
Therapists frequently refused to participate and among the most cited motives were: unpaid additions to the workload, concerns about having to audio record the sessions, and concerns about own work being evaluated by others, even when the confidentiality and anonymity of the data were explained.These anxieties were similar to those expressed by students in other naturalistic research (Rao, et al., 2010).At Metanoia Institute these anxieties could be sufficiently contained by the existing academic culture, enabling the therapists to engage with the project.The need to start a practice placement could be seen as an important motivation and a help in counterbalancing fears of evaluation.

Client Engagement
Clients who took part in evaluation in both settings showed high completion rates.This is not unusual within naturalistic evaluation which shows that clients value an opportunity to monitor how they are progressing in therapy (Miller, Duncan, Brown, Sorrel, & Chalk, 2006) and an opportunity to give feedback to therapists (Lambert & Barley, 2002).However, therapist engagement with evaluation is essential in making this effective for clients.All therapists were instructed to use the outcome measures as an integrated part of the therapy process, and all therapists were also instructed on what each instrument measured.It was interesting that Spanish therapists reported that they discussed changes with clients at the beginning of the evaluation, but that after a while clients reported fatigue with filling in the same questionnaires every week and these the discussions ceased after a few months.This was not the case with the majority of clients within the UK clinic.However, it may have been that the therapists were not themselves interested in the measures, so they were seen as 'administration' and separate from therapy.

Professional Culture
Although psychotherapy within the UK is practiced within a range of settings, professional climate is impacted by the emphasis on evaluation within the health services, and the recognition of research based psychotherapies.Many therapists who work within organisations are required to use routine outcome evaluation in their practice.Although in this research we found that fears and anxieties amongst the therapists were very similar, students within the UK site were more likely to engage with evaluation and engage their clients.They were motivated in a different way, and supported by their supervisors and training environments.
Therapists in Spain do not practice within the same professional climate.Routine outcomes evaluation is not common practice and qualified therapists working for themselves have little external incentive to engage in this additional piece of work, which is frequently where the gap between researchers and practitioners occurs.

Adherence
Adherence to the approach was not monitored in this project, due to the differences in the supervision arrangements within the sites.For this reason, no assertions about the effectiveness of the approach have been made.

Conclusions and Recommendations
In view of so many difficulties in establishing routine outcomes evaluation in this project, the question emerges about the options for transactional analysis in developing a research evidence base.One of these options involves the use of different methodologies, particularly outside academic environments, such as case study research (McLeod, 2010;Widdowson, 2012), and other qualitative and mixed methodologies.
However, in addition to that, outcomes research will remain an important part of the evidence required in establishing effectiveness of any therapeutic approach to psychotherapy.This could continue to be conducted within large clinics in academic and health settings, although that approach would leave a large part of psychotherapy practice in transactional analysis underresearched an under-represented.Our suggestion is that for optimum success in implementing this type of research in different settings, it will be essential to engage the psychotherapists more fully by: Therapists within the UK Therapists were students at Metanoia Institute who were in their first practice placement.They were engaged in different training courses: Transactional Analysis Psychotherapy (MSc); Gestalt Psychotherapy (MSc); Integrative Psychotherapy and Counselling Psychology (based on Gilbert & Orlans, 2010) (MSc in Integrative Psychotherapy and DPsych in Counselling Psychology); Person Centred Counselling (BSc and MSc).Students were entitled to work at the clinic after being observed and assessed by their tutors as being ready to start to work with clients.They had regular clinical supervision at a ratio of one hour of supervision per six hours of clinical practice.There were 22 supervisors for the 113 therapists taking part.
3% of clients were female.

Table 3 :
Percentage Improvement Rates, Spanish Sites Clinical and Reliable Change UK Site Clinically significant change is change that has taken a client from a score typical of high reported distress of a client group to a score typical of a normal population.Reliable Change (RC) measures whether clients have changed sufficiently that the change is unlikely to be due to simple measurement unreliability.This has been calculated using the standard deviation of the difference between pre and post scores and the Cronbach's alpha of the measures (indicated previously under Measures.Table4illustrates the reliable change scores for the UK sample, indicating how much a client will have needed to change in order for that change to be considered reliable.

Table 4
Reliable Change Scores, UK Site

Table 2
Developing a programme of research seminars and information for the therapists  Ensuring that face to face training takes place prior to any research project  Offering supervision that would combine reflection on clinical and research process and support therapists in using the methodology for clinical development as well as research.Dr Biljana van Rijn Teaching and Supervising Transactional Analyst (Psychotherapy) is Head of Clinical and Research Services at Metanoia Institute.She can be contacted on Biljana.vanRijn@metanoia.ac.ukAdina Dumitru an Associate Researcher at University of La Coruña, Spain