Let’s imagine that we are trying to generate a new type of therapy to treat a disorder, for example borderline personality disorder. To do so, the first step to follow is to compile and evaluate the effectiveness of the different treatments and pre-existing methodologies for that disorder: to look for the effectiveness of the treatments in previous studies. But a simple primary study would not be valid for us, given that it may be biased and may even reflect results that contradict those of others.
Thus, we need to resort to studies that have analyzed the behavior of what we want to observe throughout different studies, and for this the most reliable methodology or document is the use of meta-analysis.
What is a meta-analysis? Let’s look at it throughout this article.
What is a meta-analysis?
A meta-analysis can be defined as the result of carrying out a systematic and exhaustive investigation on a subject or question to be answered in a very specific way,** bringing together and evaluating as a whole the results obtained by a large number of studies on that aspect or subject. This is an in-depth review that allows us to see very clearly the way in which the studies it reviews have been selected and assessed, in addition to analysing their results and assessing them quantitatively by means of various statistical procedures, among which the assessment of the size of the effect and the control of possible publication bias stand out.
It has the great advantage that it allows us to work with large amounts of information from a large number of studies, in a highly organized way and that it allows us to work with the scientific method with large samples in a relatively simple way, as well as to assess differential aspects between studies and sources of heterogeneity that the basic studies may not have taken into account.
However, meta-analysis is not the only existing methodology in the systematization and synthesis of the results of research on a specific topic : there is also the concept of systematic review, which like meta-analyses reviews in an objective, systematized way and reflecting its methodology in order to be able to replicate the data existing in the scientific literature existing so far.
It is necessary to take into account that, in fact, technically every meta-analysis is a systematic review to which the statistical analysis of the results is incorporated . And this is fundamental and allows us much greater precision in our research, being able, for example, to assess from the statistical analysis which treatments for a certain disorder or even symptom are more effective in a consistent manner in the various studies assessed.
When should we use them?
The use or performance of meta-analyses can be of great help in a large number of areas, but they are much more relevant when we are faced with the need to investigate aspects in which previous studies show a certain degree of controversy and contradiction between them, or when we want to investigate what degree of relationship different variables have or try to assess what size of effect exists with respect to the effectiveness of a program or treatment. Thus, in fields such as psychology or medicine this type of review is generally used to explore, investigate, prioritize and apply treatments based on evidence.
However, it must be taken into account that its application is only viable when the primary studies chosen have a quantitative perspective and with a homogeneous sample and conditions or very similar to each other, and obviously that they start from at least the same objective and main question. It would also be irrelevant if what we are looking for are cause and consequence relationships.
Main phases in the performance of meta-analyses
Performing a meta-analysis is not simple and requires the systematic monitoring of a series of steps or phases, which are detailed below.
1. Problem formulation/research question
The first step, probably the most important but also the most obvious, is to formulate or pose the problem, theme or aspect that we want to investigate and analyze . It is not a question of asking a question at random, but of defining it as clearly and concisely as possible and taking into account the various variables that we will have to assess.
Based on this first question, the objectives of the research will be constructed and the meta-analysis will begin to be structured, and even to generate criteria for the inclusion and exclusion of studies.
2. Bibliographic search and study selection
The second step, probably one of the longest, involves an active search of the literature that we will analyze together in order to make a good review of our research topic.
It is not enough to just take every study we see: it is first necessary to have generated inclusion and exclusion criteria that allow us to select from those that exist those that meet the requirements to be able to be evaluated in the meta-analysis.
The criteria should take into account the type of design of each study, how they measured the results and the main characteristics of the samples used or of the experimental methodologies. Thus, if we consider, for example, what effects a drug has on the menstrual cycle, we will not take studies in which the sample is male. Or if age is a factor to be taken into account in our research, we will limit ourselves to studies whose sample includes only subjects of the age that is relevant to us, or that have at least separated the groups by age.
In addition, other factors must be taken into account when making the selection: the language or language in which they are written (if this is relevant) or the date of publication (it is recommended not to use data from studies that are more than ten years old), among others.
For the search it is usually advisable to use indexed databases such as SCOPUS or PubMed , at a computer level, although other types of documents or systems can also be used.
3. Study coding
Once the studies to be included in the meta-analysis have been selected, a coding manual must be drawn up to record the reliability of the data and the variables taken into account in the model used: the sample, extrinsic, methodological and contextual variables must be included among them . In general, any moderating variable that may influence the results should be included, as well as indicating how they have been assessed so that other researchers may arrive at the same results if they use the same methods.
4. Statistical analysis and interpretation
The fourth step, in which we are already properly facing a meta-analysis, is the statistical analysis of the results. In this phase , the implications or interpretations of the results are evaluated and then discussed .
As a general rule, this statistical analysis tends to assess the effect sizes of the different treatments or studies compared, to evaluate whether or not there is heterogeneity (if necessary using strategies to reduce the effect this may have on the interpretation of the data) and in the event that this exists to assess which variables may explain this heterogeneity from tests such as ANOVA.
Once the interpretation of the results has been completed, the last step in carrying out the meta-analysis is to write it up and publish it with at least the introductory sections (which include objectives and hypotheses as well as a theoretical framework), method (which should be very clear and detailed so that other authors can replicate it and should include how and where we have looked for the studies, the coding manual and the type of analysis carried out), results and discussion (where it is interpreted based on the data from the results).
The problem of publication bias
A possible problem to take into account when assessing the results of research and even meta-analyses is that there may be publication bias , understood as the deviation in the results that may be caused by the tendency of the scientific literature to seek out and favour studies that show statistically significant relationships between the variables used and ignore those that do not.
That is, studies in which relationships between variables are seen are published, but those in which the results do not demonstrate a significant relationship tend to be less published or ignored. Fortunately, this effect can be visualized and evaluated in meta-analyses, even though it is a difficult problem to solve.
- Marín Martínez, F., Sánchez Meca, J. y López López, J.A. (2009). Meta-analysis in the field of Health Sciences: an essential methodology for the efficient accumulation of knowledge Physiotherapy, 31(3):107 – 114.
- Sánchez-Meca, J. and Ato-García, M. (1989). Meta-analysis: a methodological alternative to traditional research reviews. In: Arnau J, Carpintero H, editors. Treatise on psicologı´a’a general I: history, theory and method. Madrid: Alhambra. 7 – 69.