Larry Sarner,
To be called a crank by larry sarner is a little like being told that you have a head cold by someone who is terminal with brain cancer. From you larry, being called a crank is a compliment.
Now how did larry and i start communicating? Well early on I did the simple thing of asking Stephen and Larry about their research. Stephen referred me to larry and larry referred me back to stephen’s web site for the data. Of course, as anybody with half a brain (9 years of age or older) knows, there are discrepancies between the protocol the flub four describe and the data that appear on the web site. So I asked about the fact that the data didn’t make sense to me and about the appropriateness of not telling the subjects who they were – you know – the basic grammar school ethics issues. At that point, Larry (a real crank if ever there was one) sent me the forwarded email at the bottom. Now of course scientists respond to requests for data with data – not threaten to sue people who question what has happened. Cranks, of course, threaten to sue people. Now someone once told me in a phone conversation that this wasn’t uncommon for larry – something about little emily’s school being sued?????
Anyway larry, as i said above, being called a crank by the likes of larry sarner is like a tribute – i’ll wear it proudly. Thanks.
Now here are the questions posed on this list a long, long time ago. As one of the authors of the travesty of april fool’s day, 1998 maybe you can now share with the world your answers?
For Linda Rosa, Emily Rosa, Larry Sarner and Stephen Barrett.
You continually cite poor research practices from TT practitioners. Let’s look at yours:
1. Where is your independent measurement of HEF presented to subjects? You don’t believe HEFs exist so you didn’t think you needed to make sure that a detectable level of HEF was presented. Rotten science to ignore such details.
2. Again, because you didn’t believe in or understand the issues of HEF you placed subjects in a position in which no objective researcher would – because in the position you/emily chose their hands would be facing slightly toward the presented hand. Rotten ‘science’ to ignore such details.
3. Again, because you didn’t believe in or understand the issues of HEF you placed subjects in a position in which no objective researcher would – because in the position you/emily chose the subjects would be very uncomfortable. Anyone who cares to replicate the position you chose will realize that it will cause most people to be uncomfortable and focused on their back and neck, a direct violation of the concept of ‘centering’. Rotten ‘science’ to ignore such details.
4. So let’s examine your ‘real’ science? Should a TT practitioner be able to detect any HEF, at any time, and under any conditions? I don’t think so. So we have preliminary work to do if we want to answer questions such as who, where, when and under what conditions.
5. What about your subjects? Who exactly are they? How do we, your readers, know whether there were in fact 21 subjects or no subjects or maybe 100 subjects. If you publish the names of your subjects we will be able to ask them about their training in TT, their practice of TT, verify their knowledge and practice of TT and their participation in Emily’s experiment. If you fail to give this information out we have no way of judging the quality of their skills or verifying that ‘research’ was conducted at all. Also, if you publish the names of your ‘reported’ subjects any subjects who participated and were not reported would be able to come forward.
6. Speaking of ‘science’ – the 95% confidence interval for the number of successes in 10 trials in phase 2 does not include ’5′ successes. This isn’t a question of my re-calculating the confidence interval. This is your PUBLISHED confidence interval. Good and ethical scientists do not choose a statistical test and then when the test doesn’t bear out their desired conclusion – ignore the test result and its implications and present a conclusion contradicted by their data and their own written summary of their analysis. In this case, the conclusion that the data are consistent with random guessing in the trials is false. The conclusion a reputable researcher would have advanced at this stage is that the data from this experiment were inconsistent with random guessing when using an alpha = 0.05 test. Why is this important? Because it demonstrates a disregard for accuracy in reporting and lack of comprehension of science and statistics. At this point in your article – your readers should be very, very wary of your motives, your veracity and your methods.
7. But you know it doesn’t stop there. With no reason whatsoever to introduce issues of which hand is involved you report the results, by hand, for the first set of 150 trials. You use statistical procedures that went out in the dark ages and falsely conclude that there is no hand ‘effect’. The correct, scientifically valid procedure, would have been to do a chi-square 2 x 2 analysis on hand and success. Instead, you used two tests of left hand vs H0 and right hand vs H0. What would have had to be reported if you reported the chi-square results in the article? You would have had to report a significant performance differential by hand. This, even more so than the item in 6 should have your readers either moaning over incompetence or seriously questioning ethical issues. It certainly is not a basis for congratulating an 11 year old on her scientific insight and skill.
8. So a reasonably informed and objective reader has to be asking how could such obviously poor conclusions have been reached when two adults, a former psychiatrist and a consulting statistician were involved in this project. So maybe you can shed some light as to how it is that the statistical analyses were performed or not performed and how the contradicted assertions were formulated based on those analyses. JAMA, on the other hand, can perhaps explain how an article with a stated confidence interval that excluded the hypothesized value ever passed statistical review and why the statistical reviewers didn’t examine the claims about right hand vs left hand more carefully.
9. Now this article has been around for 18 months and the authors have all had their requisite 15 minutes of fame – but now scientific integrity demands that you explain your work, justify your conclusions and show that your work can be replicated or, alternatively, withdraw your support for the article.
10. Let’s consider the replicability issues. There is a question to be addressed about the method used to calculate the point estimate for p-hat – the proportion of successes. Do the subjects who have 20/30 repetitions really provide twice/thrice as much information for our overall estimate as those subjects who have 10 repetitions? Of course not. Anybody would realize that their extra repetitions provide increased accuracy for their contribution to the total estimate but that they are still just one of 21 subjects each of whom contributes an equally valued estimate to the total. Some are more accurate estimates of the individual subjects detection rate because of the increased number of repetitions but none of the subjects could really be described as having a corner on a more valuable, more highly weighted, contribution to make to the point estimate of the success proportion. So what happens if we calculate the proportion of successes using 21 estimates with varying degrees of accuracy? We get a far lower proportion of successes with a probability of occurrence of that low – or lower – a result of approximately 0.0067. That is, in well conducted, fair trials, we would expect to be able to replicate the adverse result obtained by you 67 times in 10,000 runs of the experiment. That isn’t too promising for ‘good science’. Hell, they gave up on replicating cold-fusion long before that.
12. Should a TT practitioner be able to beat random guessing success rates? Well, did you beat random guessing success rates? One might argue that you should have been able to run an experiment with unassailable results on a yes/no basis at least 50% of the time. Of course such reasoning went out in the 1800′s. What you missed, because you are so obsessed with criticizing TT is that you have to consider the utility of such a skill. Even if a TT practitioner could detect a HEF a very, very small proportion of the time, two issues would apply. First, if detected and subsequently manipulated to the advantage of the patient then there is a tremendous advantage to the patient. Second, and more to the point, is that even if the TT practitioner is unable to detect an HEF more than some small percent of the time but the procedures incorporated by TTPs are successful in altering the client’s experiences – and TT clients must, in general, be assumed to be rational consumers and able to make such determinations – without the assistance of the Rosas, Sarner and Barrett – then there is also an advantage to them. Hence, if we factor utility into the issue we see that the assertions advanced, and the conditions imposed, by the researchers are absurd.
If I knew a stock broker who could pick winners 10% of the time I might be willing to let that broker handle all my financial transactions. What does such a decision depend on? It depends on the definition and utility, to me, of ‘success’ and what the losses and gains are when the definition of ‘success’ is not met. If the losses are small as would be the case with TT, and the gains relatively large even when not “successes” I would probably let the broker manage my affairs. But, of course, this experiment and the whole approach of your and Randi’s challenges are merely to disconfirm a hypothesis that you have advanced which is meaningless and incorrect. Simply put, the reasoning behind this experiment is ‘childish’ and scientifically ill-informed. The conclusions advanced are arguably contradicted by the data presented. The lack of attention to significant, potentially controllable confounding factors, is rampant and the ethical issues of reporting conclusions that are contradicted by the data are profound..
13. In any event, since only 8 of the subjects were RN’s, none were ‘certified’ in TT, according to Barrett’s web site, and the experimental conditions were not appropriate, no conclusions about TT as a method can be drawn from the 4th grader’s experiment. To have concluded that TT as a method failed the experiment is a wholly and totally inappropriate stretch when no random selection was involved and no evidence exists to support a claim that the subjects tested had even minimal TT skill and knowledge. Were it not for the fact that the authors are on record as criticizing the scientific skill and ethics of others, these fllaws might not be so glaring. But all the authors are on record as being critical and judgmental about flaws in the work of other people. They should have held themselves to a much higher standard.
14. What about the ethics of deceiving subjects in an experiment? The authors can hide behind the fact that Emily wasn’t attending public school where federal requirements for an institutional review board’s approval for the project would have been required but the standards for ethical research do not change just because there is a loophole to hide behind. The reality is that the authors admit to deceiving the subjects and such deception and the potential harms that subjects in this experiment were exposed to would never pass an independent, human subjects review board. Like it or not. Ignorant of it or not. There is a standard for ethical conduct to which ‘real’ researchers adhere. That the authors chose to ignore the standards for ethical research is not all that surprising. That they admitted, in writing, to such violations speaks strongly to their ignorance of solid scientific method and practice and to the intellect of their intended audience. Basically, the authors seem to believe that they can say and do anything at all and nobody will notice.
15. So what might we actually need to know about the experiment?
a. We, the readers, need to know who the subjects were and if they had any legitimate claim to TT ability, and why they were used if they did not have such legitimate claim to TT skill.
b. We, the readers, need to understand why conclusions advanced by the authors are not supported by the data
c. We, the readers need to know how such a flawed article appeared in JAMA
d. We, the readers, need to know why the authors and JAMA were so intent on a rush to publication that the flaws were either ignored or not detected.
e. We, the readers, need to understand what would motivate people to write an article where their research results were only replicable 67 times out of 10,000
f. We, the readers, need to know why it is that the consulting statistician didn’t perform a chi-square test on the left hand vs right hand data or why it was not reported, if run..
g. We, the readers, need to know if the subjects were really trained in TT.
h. We, the readers, need to know where these 21 people are because they are very strange
in terms of their inability to guess correctly.
i. We, the readers, need to know if the authors have simply made a number of critical errors in judgment because they were led by an ill-informed 9 year old or if there is something more wrong with the article.
j. We, the readers, need to know if the assertion that a TT practitioner, operating under totally contrived circumstances should be able to perform at levels assumed in this experiment and ‘Randi’s Test’ is appropriate.
k. We, the readers, need to know why the researchers didn’t use ‘certified’ TT practitioners – maybe it is because certified TT practitioners weren’t deceived by the motives of the child. In any event, certification is certainly a more credible, though not a completely adequate standard, to employ to assess competency then no certification at all.
l. We, the readers, need to know why the authors reported results where only 1 out of 13 subjects in the second set scored above a 5. This distribution of scores is clearly inconsistent with the null hypothesis and any competent researcher looking at a bar chart for the second phase would be scratching their heads in puzzlement rather than waving their hands for attention.
m. Why JAMA accepted an article for publication where the authors admit to deceiving subjects.
So, Larry, et.al. let’s hear your reasoned defense for your flawed work just as you would demand from TT researchers whose work you felt was lacking.
Bear
From: Larry Sarner [mailto:nttsg@...]
Sent: Friday, October 08, 1999 12:15 AM
To: Green Trees Consulting
Cc: sbinfo@…
Subject: Planned response to Thomas Cox
Mr. Cox:
I cannot disagree with you more on ALL of the points you made. Despite your background and obvious intelligence, I found your statistical criticisms naive and your methodological arguments uninformed. I doubt it would serve either of us for me to invest any more time in educating you or in trying to reconcile our differences.
>At this point there are two issues of ‘ethical conduct’ involved. The
>reporting of a conclusion that is inconsistent with the data and a question
>of whether or not the subjects gave their informed consent.
And at this point, our correspondence takes on an entirely different character. This is not scientific give-and-take, a difference in interpretation, or even “bitter disagreement”. Your “issues” are actually charges of scientific misconduct. The informality of email is an inappropriate venue to discuss matters of such gravity. Do you have experience in investigating or lodging such charges? If not, you should obtain competent advice as to whether you even have a case to build, much less whether your “evidence” can sustain it. I will say this much for the record. Your criticisms and arguments are tinged with a certain malice I have encountered before. Before saying them again to others, especially when coupled with charges of scientific misconduct, you should know that I will hold you accountable, in _all_ appropriate venues, for any false and damaging utterances you make. I will do the same for any costs in defending against frivolous or unsubstantiated charges of misconduct. This is not a game.
Larry Sarner
Chairman, National Therapeutic Touch Study Group
Filed under: bela scheiber, double blind, emily rosa, experiment, experimental design, james lippard, jean mercer, larry sarner, linda rosa, monica pignotti, statistics, stephen barrett | Tagged: bad science, deception, emily rosa, experimental design, jean mercer, junk science, larry sarner, malice, monica pignotti, poor experimental design, pseudoscience, quackery, scientific misconduct, statistics, stephen barrett | Leave a Comment »
