How should you assess the quality of a wind health study?

Every day or so headlines pop up in various parts of the world claiming that research has shown that wind turbines cause health impacts, or wind turbines don’t cause health problems or that someone has proven something or other negative or positive about wind energy.

How can you figure out which ones to trust relatively easily? How can you judge the quality of the material? And how do some of the commonly referenced studies stack up? Guidance on this has become more necessary recently as evidence is mounting that anti-wind campaigners are bootstrapping poor studies into higher quality journals, muddying the waters. For the cheat sheet, I have done a rough assessment of reliability of 50 commonly cited pieces of evidence in another post based on the criteria outlined below.

The (relatively) easy way to assess the quality of research related to wind energy breaks down into five categories:

  1. Peer-reviewed in a quality journal - If the journal is currently indexed in a reputable biomedical index such as PubMed and has an impact factor you can have greater comfort about the quality of the study.
  2. Quality of evidence based on type of study – From highest to lowest the evidentiary weight ascribed to different study types is systematic review, critical assessment, randomized trial, cohort study, case study or series and finally expert opinion.
  3. Full declaration of conflicts of interest – You should have lower confidence in a study if the authors do not disclose or downplay membership in anti-wind groups such as Society for Wind Vigilance and Waubra Foundation, or employment by or significant funding from the wind industry.
  4. Quality of referenced literature – You should have lower confidence in studies which reference the book Wind Turbine Syndrome, the Bulletin of Science, Technology and Society, vibro-acoustic disease (VAD) and un-peer reviewed literature as if they are solid sources. Similarly, if the study does not reference any of the 19 reviews world-wide of wind and health which found no health impacts outside of noise annoyance, or the studies on psychology of annoyance, nocebo effect and psychogenic illnesses, confidence should be lower.
  5. Methodology and Structure – Confidence should be lower if there are overly broad hypotheses, weak data, irreproducible methodology, poor statistical correlation or overstated conclusions.

Note: Some nice people in Ontario made a very nicely formatted PDF version of this for use with communities and journalists. For those interested, please feel free to use this: FoW-blog-e-v3.

Peer-review

The first filter should be whether the evidence is published in a high-quality, peer-reviewed journal or not. Journals have a variety of academics and professionals with appropriate credentials on tap who review reports that are submitted to them.  These reviewers assess the report and typically provide feedback to the authors on elements which require improvement before publication.

Things checked for in typical peer reviews include the quality of methodology of the study, reproducibility of the study, whether the references appear to support the study’s aims and whether the conclusions are reasonably stated given the quality of evidence.

Quality of journals can be assessed by checking whether they are indexed in one or more of the major indices and have an impact factor (IF).  The major indexes are Index Medicus, MedLine, PubMed, EMBASE, SCOPUS, EBSCO Publishing’s Electronic Database or SCIRUS. A journal’s impact factor, based on Thomson Reuters Journal Citation Reports, is relatively useful. It’s a measure of a journal’s influence within its field,  based on the ratio of report citations to the number of published reports in the journal.

If a journal is indexed and has an impact factor, they usually will tell you about it themselves, so you don’t need access to anything other than Google. Searching with “<journal name> index” and “<journal name> impact factor” will quickly give you information to assess this.

As an example, the journal The Bulletin of Science, Technology & Society (BSTS) where several reports arguing that wind had strong negative impacts were published in August of 2011 sounds impressive. However, it is not indexed and it has no impact factor. Any reports published in this journal have an automatic red flag on quality. This doesn’t mean there isn’t a gem published in there, but it makes it much less likely, as solid researchers do not submit reports to unindexed journals without impact factors.

A difficulty in terms of assessing wind and health research based upon journal impact factors is that wind health impacts just haven’t merited inclusion in higher impact journals such as the American Medical Journal to-date; they have been too trivial, too diffuse and too non-specific to pass the filter. As such, simply assessing whether the journal has an impact factor or not is the most reasonable validating step, as per the BSTS above.

Material which has not undergone peer-review should be automatically considered of lower evidentiary value than material which is peer-reviewed.  Online publication in escholarship sites is of value as a pre-cursor step to formal review and publication, and it should be weighted lower but not dismissed, especially if it’s associated with a major university with a good academic track record, for example the University of Sydney.

Obviously, newspaper and magazine reports should be taken as a much lower level of evidence, along with the anecdotal stories of health impacts available on YouTube and in anti-wind sites.

Quality of Evidence

Evidence-based medicine (EBM) is a discipline or approach which depends upon evaluating the sources of information for diagnosis and treatment and acting from the strongest possible evidence, not the weakest. For this purpose, we will use the EBM model rather than the hierarchy of evidence for more generic studies due to the specific health implications alleged for wind energy.

At the bottom of the pyramid is general knowledge or an expert’s opinion unsupported by other evidence. This is better than nothing, but is only a basis for care or intervention in situations where there is no other evidence.

Case studies and case series provide insights into an individual example or small groups of individual examples of something considered to be representative of the concern, but below the level of statistical significance.  Sometimes that’s easy: three people with tuberculosis can be diagnosed with a simple skin test and their relative situations compared. Sometimes it’s hard, as when new diseases present themselves: is this person actually a case of SARS or do they just have a bad cold?

A cohort study actively tries to identify all of the people believed to be suffering from or exposed to the same condition, then compares and contrasts within them.  The key is that there is no control group or randomized exposure in a cohort study, so comparisons are limited.

A randomized controlled trial determines a method to achieve a statistically significant sample of individuals in both study groups — those believed to be suffering from or exposed to the causative agent — and control groups — those believed not to be suffering from or exposed to the causative agent.  This allows a comparison of the actual symptoms, histories and pertinent factors of the two groups to draw conclusions.

The bottom four levels of evidence are well represented in the wind and health space. There are many expert opinions, case studies, cohort studies and randomized controlled trials that have been written up. Only a subset of these have been accepted for publication in peer-reviewed journals, or even submitted however. It’s important when reviewing a piece of evidence to once again assess whether it is peer-reviewed in a high-quality journal or not.

Note that case studies/series, cohort studies and randomized trials typically require medical ethics oversight to ensure that the study does (much) less harm than any expected benefits of knowledge, especially where direct interaction with individuals or their medical records is undertaken. Sarah Laurie is under investigation for breaches of medical ethics in Australia due to her broadly self-reported, unsupervised and likely damaging direct interaction with people claiming harm from wind turbines under the guise of research.

These levels of evidence are superseded by more weighty levels. The first of these are critically appraised articles and topics. There are independent organizations which assess the studies published in the top journals related to health, assess them critically based on standard methodologies, and publish synopses and weightings. The level above that are formal systematic reviews and meta-analyses.  Once again, there are organizations which perform these and publish the results. Similar to the impact factor issue, alleged wind health impacts just haven’t rated having formal critical appraisals and systematic reviews by these organizations.

However, the 19 reviews world wide of the evidence related to wind and health have been informed by evidence-based medicine, and the majority have based their assessment methodology on its principles and practices. These have been performed by public health organizations and researchers world wide using deeply accredited and professional resources at the request of governments because of the fears raised by disinformation related to health. None of these reviews has found any evidence of health impacts or any mechanism for them outside of noise annoyance. These reviews have a high weight as a result, typically higher than any individual study while not quite as much as would critical appraisals or systematic reviews of a topic via the formal process. They are the assessments at the highest level of evidence available now and must be taken very seriously.

The first two assessment criteria — peer reviewed in a quality journal and quality of evidence — do not require analysis of the contents of the study. In the wind and health space, peer-reviewed randomized trials in quality journals are the strongest individual study available, and non-peer-reviewed expert opinion and case studies in unindexed, no impact factor journals are the weakest. It’s fairly quick to do these checks.

The next assessments require you to do more work related to the published study itself.

Biases and Conflicts of Interest

There are remarkable numbers of people with axes to grind working to publish in the area of wind and health, perhaps more so than in other domains. People with very weak medical and academic credentials are striving to create bodies of published reports in lower quality journals that support court actions against wind energy firms, governmental agencies and land-owners leasing to turbine operators. They are also working to leverage the publications in very weak journals to manipulate peer review in higher-quality, generalist journals such as Canadian Family Physician.

However, there’s a simple assessment for this.  If the authors are clear and frank in their conflicts of interest, and do not misrepresent those conflicts, then you can lend them more credence. If however, they do not publish their conflicts or assert a degree of neutrality about their adherences which isn’t present in reality, you can reasonably have concerns about the quality of their report as well.

As a primary example, the Society for Wind Vigilance has been an anti-wind advocacy group for several years, and all of its founding members and most of its current members have actively been fighting wind farms near their rural properties, mostly second homes. They have consistently disregarded the weight of evidence to promote cherry-picked elements which support their biased hypothesis that wind energy harms health. They have a 2 km setback policy which is not evidence-based.

The members of this organization are striving to create a body of peer-reviewed literature to support their positions in court. In some cases, such as a Nissenbaum et al report in the journal Noise and Health, the authors did not state that they were long time members of the Society for Wind Vigilance, or that two of their three thanked reviewers were also members.  In another case, a commentary published in the journal Canadian Family Physician, the authors acknowledged that they were members of Wind Vigilance, but claimed it was a neutral organization devoted to assessing wind energy and health.

A quick Google of the authors’ names will rapidly establish whether they are members of Wind Vigilance or are actively fighting wind farms near their homes. For ease of reference, the full list of members of the Society who are publishing or attempting to publish is:

  • Michael A. Nissenbaum,
  • Roy D. Jeffery,
  • Christopher Hanning,
  • Carmen Krogh,
  • Richard R. James,
  • John Harrison,
  • Brett Horner,
  • Jeff Aramini,
  • Carl V. Phillips,
  • Alec N. Salt,
  • Daniel Shepherd
  • Robert Thorne.

Robert McMurtry, a founding member of Wind Vigilance, is a special case.  He is attempting to get around the reality of conflict by resigning from Wind Vigilance and dropping his lawsuit against a wind energy firm in order to allow him the appearance of a neutrality he does not have. This also allows him recommend extension of the unsupported two kilometre setback to five kilometres to give him more ammunition in his fight against a wind farm 2.8 kilometres from his vacation property in Ontario according to recent testimony in Ontario.

Once again, if these individuals publish, fully disclose their ongoing anti-wind campaigning, their membership in Wind Vigilance and the full nature of Wind Vigilance, then this is not necessarily a red flag in terms of their report. However, this has not happened to date in any of their publications. Further, all of their published work to-date related to wind energy and health has had significant and glaring problems.

Exclusion of the credentials of these individuals, listed in their bios in the Wind Vigilance site, is not out of disrespect, but due to the reality that virtually none of their credentials have any relevance to wind energy, noise health impacts, epidemiology or psychology. One of them is a CMA, another a former surgeon, another a former pharmacist and yet another a radiologist. They use their credentials as a mechanism for asserting authority which in this area they do not have.

Another individual and organization who do not publish, but are often quoted and make submissions to wind siting reviews worldwide are Sarah Laurie and the Waubra Foundation. Anything with either name associated with it should be considered highly suspect unless they clearly lay out the ongoing and harshly negative anti-wind campaigning that they have been undertaking as context. They have published no peer-reviewed material of any sort to date, but Ms. Laurie often does media interviews in which she asserts harm to human health from wind turbines and also testifies in siting reviews for wind farms.

In my experience, reports by those engaged in the wind industry are usually transparent about their funding and work alliances by comparison. That said, apply the same metric to them. If you discover someone publishing a peer-reviewed paper on wind and health who does not disclose that they work for a wind energy company, that would be a reason to be concerned. However, do not believe anti-wind campaigners and websites on this point, as they have been publishing inaccurate defamations of the motivations of individuals who are for wind energy in many forums, and appear immune to attempts to correct their inaccuracies.

One simple hint that authorial bias is at play is if they use the term “industrial wind turbines”. This term was focus-tested by Koch Brother backed anti-renewables groups a decade ago in their efforts to find the most effective negative term to use for wind energy.

Quality of Referenced Literature

There is a large amount of published literature of high quality and a remarkable amount of literature of poor quality that is constantly referenced in the wind and health field.  A companion piece will rank the most commonly referenced wind and health material, but for now here are a few red flags to watch out for.

Exclusion Red Flags

  • The report does not refer to any of the 19 reviews of wind energy and health conducted world wide by independent, professional and respected groups and researchers.
  • The report does not reference any of the literature on the psychology of noise annoyance, nocebo effects and psychogenic illnesses, especially the reports from the Universities of NottinghamAuckland and Sydney by Lawrence et al, Crichton et al and Chapman et al respectively.
    Note the timeframe of the publication of these studies versus the length of time for peer review; it is conceivable that the process of publication for articles appearing in mid-2013 started before Crichton and Chapman’s reports were published.
  • The report deals with acoustics but excludes references to the work of Geoff Leventhall, who is unequivocal in his statements on the very low level of health impacts possible from the levels of sound that wind turbines emit at all frequencies. Worse, if the report makes claims that Leventhall asserts that wind turbines cause health impacts by cherry-picking past reports of his, this should be a very large red flag as his position on this is widely available and clearly stated.

Inclusion Red Flags

  • The report refers uncritically to the book Wind Turbine Syndrome, the very poor, unpeer-reviewed, case series that Dr. Nina Pierpont published in her vanity press, and uses it as valid evidence of wind energy health impacts.
  • The report takes uncritically the Noise and Health report by Nissenbaum et al without referencing the two published critiques in the same journal. This is especially problematic with the members of Wind Vigilance such as Krogh et al, who as fellow members of that group must be expected to be aware of serious critiques.
  • The report refers to articles from the August 2011 issue of The Bulletin of Science, Technology & Society. This issue of the unindexed, no impact factor, occasionally published journal was devoted to articles critical of wind energy which would not have survived peer review in a rigorous journal, as they display very significant lapses of methodology and vastly overstate findings.
  • The report refers to un-peer reviewed material from acousticians such as Rick James, Robert Rand and Stephen Ambrose. There is significant peer-reviewed acoustics material in wind energy and related fields to support arguments about wind energy noise and health; un-peer-reviewed reports are not required and should be viewed with suspicion.
  • The report refers to work on “vibro-acoustic disease” or VAD except to dismiss it. This work has been proven to be baseless by Norwegian researchers who created much more robust studies than the original and found exactly zero evidence of any acute or chronic physiological changes attributable to infrasound. They also point out that the one piece of “evidence” — pericardial thickening — that the original researcher, Castelo-Branco, had he misinterpreted, as he thought the pericardium was supposed to be three-four times thinner than it in actuality is. Further, work by Prof Simon Chapman shows that 34 of 35 papers on the subject were authored by the same Portuguese research group, 74% of citations to these papers were self-citations by the group (versus 7% more typically) and no linkage to wind turbines exists within any of the papers. The full data set between wind turbines and this invented illness is a single case study presented at a conference of a single boy that has not been published in a peer-reviewed journal. At best this is ongoing, long-running incompetence and ignorance; at worst, a workers’ compensation ploy.

As an example, the commentary by Krogh et al in the May 2013 issue of Canadian Family Physician hit every single one of these exclusion and inclusion red flags, indicating that they are basing their conclusions on very weak supporting evidence, and further that they are cherry picking their supporting material to ignore very strong material which disagrees with them.  Numerous Rapid Responses in the journal have pointed out their duplicity in this regard.

Methodology and Structure

If you wish to dive more deeply into assessing wind and health studies than the material on red flags and relative weighting allows you to, and to more closely scrutinize the contents, then I recommend this material on critical appraisal of journal articles. As a warning, good critiques usually require deeper understanding of the domain, statistics and effective reporting. Looking for published critiques may be a more effective use of your time.

That said, there are five categories you can consider that are relatively amenable to lay person analysis:

  • Hypotheses – Case studies, case series and many cohort studies are not testing hypotheses, but merely recording observations. The absence of hypotheses does not make a study incorrect, but it automatically flags it as of lower evidentiary value on the evidence pyramid.  One key thing to assess is whether the hypotheses are overly broad.  Research can’t boil the ocean, and hypotheses must be testable within the confines of a usually resource- and time-constrained study, so if the hypothesis was “Wind farms make people sick” this would be an indication of a problem.
  • Weak data – In general, more data for a longer period of time is better than less data. Thousands of data points will generally allow more robust conclusions than a dozen data points.  Data can be subject to selection bias, where apparently random data is skewed by people with complaints opting in or being selected; if this is duly noted in the report the authors get points but if it is ignored they get demerits. This is one of the many serious flaws in the study that the book Wind Turbine Syndrome is based upon; the study group was found by advertising via anti-wind groups for those who blamed wind farms for negative health impacts, a guarantee of selection bias.  Data may be selected from certain time periods that support a hypothesis. This is one of the key differentiators between the first four levels of evidence: expert opinion has no data, case studies and series have one to a small number of data points, cohort studies attempt to capture all data points around a narrow point and randomized trials attempt to arrive at statistically significant study and control groups. Rigorous reviews of dozens or hundreds of papers in a domain carry more weight than any individual study.
  • Methodology – This section should explain the approach and procedures related to executing the study in sufficient detail that it can be reproduced. If the section is missing, vague or has steps which appear to constitute magic, then red flags should go up.  This can be very difficult to do for many domains. Acoustics sampling for wind noise at homes, for example, is much less controversial than wind farm opponents would have us believe, but there is still room for expert differences of opinion that are more esoteric than laypeople can be expected to follow.
  • Statistical correlation – Correlation is straightforward; if one thing changes and the change in another thing can be predicted, they are correlated. Pearson’s Coefficient of Correlation is the most common method of expressing this. It ranges between -1.00 and +1.00. A high positive or negative correlation means that when one value changes, you can predict the change in the other value with high certainty.  As values approach zero, the correlation drops.  A positive correlation means that as the first value increases, so does the second value. A negative correlation means that as the first value increases, the second value decreases. What’s important to note here is that assertion of high correlation should be backed up by the numbers, typically in the format r=0.70, p<0.01.  The p is an indicator of how statistically significant the change is, and it’s looked up on statistics tables.  An explanation of how they correlated the data is important as well.That said, this rapidly becomes esoteric in many fields because to understand the relevance often requires deep domain knowledge you won’t have. As a rule of thumb, higher correlation is good, showing a p value is good, explaining the approach used is good, and if there are no other red flags related to peer review, authorial unstated conflicts, etc you can feel comfortable that something odd isn’t going on.
    What is also important is that correlation does not necessarily equal causation. Just because the two numbers change together in highly predictable ways doesn’t mean that changing one causes the change in another. It could equally mean that there are other factors which influence both.  Moderate red wine consumption is correlated with lower heart problems, but this likely has more to do with other things associated with red wine consumption such as higher income, higher education and greater likelihood of regular physical exercise. It’s important that confounding factors be considered, as they may have a much larger impact than the specific data points under consideration. For example, a questionnaire assessing blood pressure, wind speed and wind turbine operation may show a correlation, but if it ignores medication compliance, diet, weight and age it may draw false conclusions.
  • Conclusions - Given the pyramid of evidence described above, it should be clear that no single study and report can be expected to have an unequivocal and bold conclusion. If a report says “A causes B” with no equivocation, calls for further study or hedging as vibro-acoustic case studies presented in conferences have done, that’s a red flag.
    This causes all sorts of problems with communicating scientific results, because extremely strong studies that show clearly what is going on have limp conclusions by the standards of non-scientists, who have a much lower standard of evidence. Counter-intuitively, it’s the reports whose conclusions appear most equivocal that are most to be trusted as accurately portraying what is going on.That all said, there’s significant range for dissembling within this general guideline. For example. Nissenbaum et al’s paper in Noise and Health only studies sleep, yet concludes that wind turbines impact health.  Further, they don’t show dose response data which supports their conclusion that wind turbine noise impacts sleep and even state that in the body of their report, yet make that conclusion nonetheless.

Assessing the quality of medical literature is challenging. Assessing the quality of wind energy and health related literature is easier, because the number of people publishing in the field is smaller, and their often very strong biases and very significant conflicts of interest are well known and easy to discover.  The weaknesses of the base material of those concluding wind farms cause health problems is also easy to assess and confirm.

2 comments

  1. Will · · Reply

    I was interested to see Alex Salt on the list of members of Wind Vigilance. I had assumed his research position on physiological effects had preceded his interest in wind turbine noise. It might be interesting to follow the funding trail for his work. While pretty much everything else in WTS is not defensible Salt’s work, while only suggestive, still offers some window of opportunity for highlighting uncertainty and calls for further research.

    1. This is an excellent resume of how to understand the validity of research findins.

      I also find Google Scholar very useful for primary searching.

Be nice, be respectful, be relevant.

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 1,045 other followers

%d bloggers like this: