Archive for the 'Methodology' category

The case study protagonist as unreliable narrator.

Feb 18 2013 Published by under Ethics 101, Methodology, Teaching and learning

Even though it seems like my semester just started, I'm already grading the first batch of case study responses from my "Ethics in Science" students. (Students, if you're reading this: I'm quite happy with how the class is doing! You'll get detailed feedback on your response by the end of the week.)

In case you're not familiar with case studies in the context of an ethics class, they usually consist of a brief description of a situation in which a protagonist is trying to make a decision about what to do. I ask my students to look at this description and identify who has a stake in what the protagonist does (or doesn't do); what consequences, good or bad, might flow from the various courses of action available to the protagonist; to whom the protagonist has obligations that will be satisfied or ignored by his or her action; and how the relevant obligations and interests pull the protagonist in different directions as he or she tries to make the best decision. On the basis of these details, I ask my students to choose a course of action for the protagonist and explain why it's an ethical course of action.

But here's something that makes the analysis difficult for the students: Often it's hard to pin down the fact of the case with certainty. The scenario is described from the protagonist's point of view. It seems to the protagonist that there's favoritism in the lab group, or that it's obvious why some of the measurement turned out the way they did, or that a colleague is going to react a particular way if a concern is brought to that colleague's attention. However, my students have been quick to notice in their discussions of the cases, what seems to be true to the protagonist might be false. For any number of reasons, the protagonist may have a skewed perspective on what's going on in other people's minds, on what the issues are with the experiment, even on his or her own competence.

The protagonist, in other words, could be an unreliable narrator.

Making a good ethical decision is easier when you can pin down all the relevant facts (including things like what future events would flow from the protagonist's various courses of action). But, as in real life, the case studies with which we ask our students to grapple have a lot of uncertainty built in. Postponing a decision about what to do until all the facts are in just isn't a practical option. Sometimes you do the best you can with knowledge you recognize is gappy.

Indeed, one of the big reasons I try to get my students to understand discussion as a valuable part of ethical decision-making is that, left to our own devices, each of us can be just as unreliable a narrator as the protagonist of the case study we're thinking through. The protagonist suspects favoritism. We suspect jealousy. Maybe the protagonist is wrong, but maybe the protagonist is right and we're wrong instead. Given the state of our knowledge in the world, we don't won't to lean on ethical decision-making strategies that require us to guess correctly about all of the unknowns.

The moral of the story is assuredly not the "there are no wrong answers" crap that humanities professors get from their naïve undergraduates. Instead, it's that taking account of other people's perspectives may be useful in helping us gain some critical distance on our own (and on the ways it might turn out to be wrong). Also, it's that an ethical course of action might require some active fact-finding to test whether one's perceptions in a situation are reliable before acting rashly on the assumption that they are.

* * * * *
Related posts:

The value of (unrealistic) case studies in ethics education.

Some ethical decisions are not that hard: thoughts on Joe Paterno.

Question for the hivemind: workplace policies and MYOB.

Passion quilt: a meme for teachers.

Share

2 responses so far

GRE scores and other tools to evaluate people for lab positions.

Feb 09 2012 Published by under Academia, Methodology, Teaching and learning

In the last 24 hours there has been an interesting conversation on the Twitters (with contributions from @drugmonkeyblog, @CackleofRad, @mbeisen, @Namnezia, @dr_leigh, @doc_becca, @GertyZ, @superkash, @chemjobber, @DoctorZen, and a bunch of other folks) on the value of standardized tests (like the GRE) in evaluating candidates for a lab position.

The central question at issue seems to be whether GRE scores are meaningful or meaningless in identifying some quality in the candidate that is essential for (or maybe reliably predictive of) success in the environment of an academic lab. And, it's worth noting that the conversation has not been framed in terms of using GRE scores as the only piece of evidence one has about applicants. Rather, it's been about the reliability of GRE scores as a predictor compared to college transcripts, letters of recommendation, personal essays, and the like.

I have thoughts about this issue, thoughts which are informed by:

  • my teaching experiences
  • my own experiences with the SAT and the GRE (I aced them)
  • my own experiences doing research in four different lab settings (three of them while I was an undergraduate)
  • my experiences teaching test preparation courses (for SAT I, SAT II, and MCAT)
  • my experiences as the graduate student representative on a graduate admissions committee (albeit not for a science department)
  • my experiences on hiring committees (where GRE scores weren't an issue but things like letters of recommendation, grades, and personal statements were)
  • broader ongoing conversations with colleagues about the challenges of finding reliable proxies with which to assess the success of our educational efforts.

What I have observed from these:

  1. There are extremely smart, capable people with severe test-anxiety. I'm talking puking-at-the-very-thought-of-sitting-fot-the-test anxiety. The people I've known with this manifest it most strongly when faced with standardized tests; generally they've found ways to deal with the other kinds of exams that are part of their schooling. I doubt that GRE scores would be reliable indicators of the fitness of such people for a position in an academic lab, unless that position involved taking standardized tests on a regular basis.
  2. My own success on standardized tests is mostly a measure of how well I understood the structure of those standardized tests. This is a lesson that was reinforced by my experience teaching others how to do better on standardized tests. I did not make my test prep students smarter about much of anything except strategies for taking the standardized tests. (In a few instances, my work with them may have helped them identify conceptual issues or problem solving skills that they needed to sharpen before test day, but again, I take it the "help" they got was primarily a matter of knowing what material and skills the test was going to assess.) Is understanding the structure of the GRE, or developing a good strategy for taking it, a crucial component of success in an academic lab? Probably not. Is it a reliable proxy for something that is? Maybe, but it would be nice to see an explanation of what that is rather than just putting our faith in the test to tell us about something that matters.
  3. Plenty of people with awesome test scores are hopeless in the lab. Plenty of people with non-awesome test scores are really successful in the lab. What's the level of correlation? I don't know, and you probably don't either. Maybe someone should do an empirical study so we know.
  4. One place that standardized tests seem to be of use (or so I've heard repeatedly over the years from lots of admissions committee folks) is in "calibrating" grades, especially of schools with which one might have less familiarity. What does an A at Podunk U. mean compared to an A at Well-Known Tech? Presumably the GRE scores of the candidates give us some information (so, if they're really low from the Podunk U. student, maybe Podunk U.'s As aren't requiring the same level of mastery as Well-Known Tech's As). But, there's always the possibility that Well-Known Tech has a better developed organization from the point of view of getting its students into grad school, and that part of this might include in-house test prep. Also, what if the lone Podunk U. student who is applying to your program has test-anxiety?
  5. GRE scores are often thought of as an objective counterbalance to letters of recommendation because, as the common wisdom has it, letter writers lie. Or maybe they just put the best possible spin on the candidate's talents. Or maybe they're actually just overestimating the candidate's potential. Or maybe they don't write good enough letters for the students who are not like them in certain relevant respects (including scientific style, socioeconomic background, gender, race, sexuality, etc.). Surely, in many cases there is something like a positive bias in letters of recommendation (and some faculty will advise students to ask someone else for a letter if they themselves are unable to write a glowing recommendation). And, there are instances in which a letter writer will undervalue the talents and potential of students (although one hopes that the other letter writers in such cases will compensate). Still, the letters at least present a space in which actual concrete examples of the student's awesomeness (or shortcomings) can be discussed. Some of these examples may touch on situations or challenges directly relevant to what the applicants may have to face in the academic lab in which they are seeking a position. Plus, at least in fields that are not totally enormous, there is (or could be) a professional cost to lying to a colleague in the profession, even in a letter of recommendation for a student.
  6. If I had to rely on just one proxy, it would be the applicant's personal statement. Again, it strikes me that this is an instrument that creates a space where an applicant can describe past experiences and current interests, challenges overcome and lessons learned from them that might be applied to future challenges. A personal statement can give you a glimpse into what the applicant cares about and why. It can also give you a sense of whether the applicant can think and communicate clearly. However, this is probably another area where someone should do some empirical work to see what kind of correlation there actually is between the quality of the personal statement and the success of the applicant in the position for which the personal statement was part of the application package.
  7. Every single proxy we might look at to select among applicants can fail. It's not clear to me that it could be otherwise, especially given that we're using the proxies to try to predict future success, which you can't do with perfect accuracy unless you have a machine for seeing into the future (and even then ...).
  8. It strikes me that active thinking-on-your-feet interview questions might provide more relevant information. It used to be that you couldn't really use these for things like grad school admission because you couldn't afford to fly all your applicants out to campus. (By the time you saw prospective grad students, they were admits trying to choose between the programs that had accepted them.) But maybe now with tools like Skype those looking to make sensible choices among applicants should do some video interviewing?
  9. Then again, if video interview questions for lab positions become a thing, someone will probably set up a video interview preparation company.

Yeah, I'd say to take GRE scores with a grain of salt. But, I think that's the right attitude to take to all the bits of evidence an applicant presents. Honestly, my attitude toward test scores probably has a lot to do with my knowledge about how easy it can be to do well on them (at least compared to the other pieces of one's application package). It probably also has to do with at least a few gatekeepers who treated GRE scores as definitely more reliable simply because they were quantitative, rather than qualitative.

If you have an applicant-screening item that has never led you astray, please share it in the comments.

Share

10 responses so far

Scientific authorship: guests, courtesy, contributions, and harms.

Nov 05 2011 Published by under Conferences, Ethical research, Methodology, Professional ethics, Tribe of Science

DrugMonkey asks, where's the harm in adding a "courtesy author" (also known as a "guest author") to the author line of a scientific paper?

I think this question has interesting ethical dimensions, but before we get into those, we need to say a little bit about what's going on with authorship of scientific papers.

I suppose there are possible worlds in which who is responsible for what in a scientific paper might not matter. In the world we live in now, however, it's useful to know who designed the experimental apparatus and got the reaction to work (so you can email that person your questions when you want to set up a similar system), who did the data analysis (so you can share your concerns about the methodology), who made the figures (so you can raise concerns about digital fudging of the images), etc. Part of the reason people put their names on scientific papers is so we know who stands behind the research -- who is willing to stake their reputation on it.

The other reason people put their names on scientific papers is to claim credit for their hard work and their insights, their contribution to the larger project of scientific knowledge-building. If you made a contribution, the scientific community ought to know about it so they can give you props (and funding, and tenure, and the occasional Nobel Prize).

But, we aren't in a possition to make accurate assignments of credit or responsibility if we have no good information about what an author's actual involvement in the project may have been. We don't know who's really in a position to vouch for the data, or who really did heavy intellectual lifting in bringing the project to fruition. We may understand, literally, the claim, "Joe Schmoe is second author of this paper," but we don't know what that means, exactly.

I should note that there is not one universally recognized authorship standard for all of the Tribe of Science. Rather, different scientific disciplines (and subdisciplines) have different practices as far as what kind of contribution is recognized as worthy of inclusion as an author on a paper, and as far as what the order in which the authors are listed is supposed to communicate about the magnitude of each contribution. In some fields, authors are always listed alphabetically, no matter what they contributed. In others, being first in the list means you made the biggest contribution, followed by the second author (who made the second-biggest contribution), and so forth. It is usually the case that the principal investigator (PI) is identified as the "corresponding author" (i.e., the person to whom questions about the work should be directed), and often (but not always) the PI takes the last slot in the author line. Sometimes this is an acknowledgement that while the PI is the brains of the lab's scientific empire, particular underlings made more immediately important intellectual contributions to the particular piece of research the paper is communicating. But authorship practices can be surprisingly local. Not only do different fields do it differently, but different research groups in the same field -- at the same university -- do it differently. What this means is it's not obvious at all, from the fact that your name appears as one of the authors of a paper, what your contribution to the project was.

There have been attempts to nail down explicit standards for what kinds of contributions should count for authorship, with the ICMJE definition of authorship being one widely cited effort in this direction. Not everyone in the Tribe of Science, or even in the subset of the tribe that publishes in biomedical journals, thinks this definition draws the lines in the right places, but the fact that journal editors grapple with formulating such standards suggests at least the perception that scientists need a clear way to figure out who is responsible for the scientific work in the literature. We can have a discussion about how to make that clearer, but we have to acknowledge that at the present moment, just noting that someone is an author without some definition of what that entails doesn't do the job.

Here's where the issue of "guest authorship" comes up. A "guest author" is someone whose name appears in a scientific paper's author line even though she has not made a contribution that is enough (under whatever set of standards one recognizes for proper authorship) to qualify her as an author of the paper.

A guest is someone who is visiting. She doesn't really live here, but stays because of the courtesy and forebearance of the host. She eats your food, sleeps under your roof, uses your hot water, watches your TV -- in short, she avails herself of the amenities the host provides. She doesn't pay the rent or the water bill, though; that would transform her from a guest to a tenant.

To my way of thinking, a guest author is someone who is "just visiting" the project being written up. Rather than doing the heavy lifting in that project, she is availing herself of the amenities offered by association (in print) with that project, and doing so because of the courtesy and forebearance of the "host" author.

The people who are actually a part of the project will generally be able to recognize the guest author as a "guest" (as opposed to an actual participant). The people receiving the manuscript will not. In other words, the main amenity the guest author partakes in is credit for the labors of the actual participants. Even if all the participants agreed to this (and didn't feel the least bit put out at the free-rider whose "authorship" might be diluting his or her own share of credit), this makes it impossible for those outside the group to determine what the guest author's actual contribution was (or, in this case, was not). Indeed, if people outside the arrangement could tell that the guest author was a free-rider, there wouldn't be any point in guest authorship.

Science strives to be a fact-based enterprise. Truthful communication is essential, and the ability to connect bits of knowledge to the people who contributed is part of how the community does quality control on that knowledge base. Ambiguity about who made the knowledge may lead to ambiguity about what we know. Also, developing too casual a relationship with the truth seems like a dangerous habit for a scientist to get into.

Coming back to DrugMonkey's question about whether courtesy authorship is a problem, it looks to me like maybe we can draw a line between two kinds of "guests," one that contributes nothing at all to the actual design, execution, evaluation, or communication of the research, and one who contributes something here, just less than what the conventions require for proper authorship. If these characters were listed as authors on a paper, I'd be inclined to call the first one a "guest author" and the second a "courtesy author" in an attempt to keep them straight; the cases with which DrugMonkey seems most concerned are the "courtesy authors" in my taxonomy. In actual usage, however, the two labels seem to be more or less interchangeable. Naturally, this makes it harder to distinguish who actually did what -- but it strikes me that this is just the kind of ambiguity people are counting on when they include a "guest author" or "courtesy author" in the first place.

What's the harm?

Consider a case where the PI of a research group insists on giving authorship of a paper to a postdoc who hasn't gotten his experimental system to work at all and is almost out of funding. The PI gives the justification that "He needs some first-author papers or his time here will have been a total waste." As it happens, giving this postdoc authorship bumps the graduate student who did all the experimental work (and the conceptual work, and data analysis, and drafting of the manuscript) out of first author slot -- maybe even off the paper entirely.

There is real harm here, to multiple parties. In this case, someone got robbed of appropriate credit, and the person identified as most responsible for the published work will be a not-very-useful person to contact with deeper questions about the work (since he didn't do any of it or at best participated on the periphery of the project).

Consider another kind of case, where authorship is given to a well-known scientist with a lot of credibility in his field, but who didn't make a significant intellectual contribution to work (at least, not one that rises to the level of meriting authorship under the recognized standards). This is the kind of courtesy authorship that was extended to Gerald Schatten in a 2005 paper in Science another of whose authors was Hwang Woo Suk. This paper had 25 authors listed, with Schatten identified as the senior author. Ultimately, the paper was revealed to be fraudulent, at which point Schatten claimed mostly to have participated in writing the paper in good English -- a contribution recognized as less than what one would expect from an author (especially the senior author).

Here, including Schatten as an author seemed calculated to give the appearance (to the journal editors while considering the manuscript, and to the larger scientific community consuming the published work)that the work was more important and/or credible, because of the big name associated with it. But this would only work because listing that big name in the author line amounts to claiming the big name was actually involved in the work. When the paper fell apart, Schatten swiftly disavowed responsibility -- but such a disavowal was only necessary because of what was communicated by the author line, and I think it's naïve to imagine that this "ambiguity" or "miscommunication" was accidental.

In cases like this, I think it's fair to say courtesy authorship does harm, undermining the baseline of trust in the scientific community. It's hard to engage in efficient knowledge-building with people you think are trying to put one over on you.

The cases where DrugMonkey suggests courtesy authorship might be innocuous strike me as interestingly different. They are cases where someone has actually made a real contribution of some sort to the work, but where that contribution may be judged (under whatever you take to be the accepted standards of your scientific discipline) as not quite rising to the level of authorship. Here, courtesy authorship could be viewed as inflating the value of the actual contribution (by listing the person who made it in the author line, rather than the acknowledgements), or alternatively as challenging where the accepted standards of your discipline draw the line between a contribution that qualifies you as an author and one that does not. For example, DrugMonkey writes:

First, the exclusion of those who "merely" collect data is stupid to me. I'm not going to go into the chapter and verse but in my lab, anyway, there is a LOT of ongoing trouble shooting and refining of the methods in any study. It is very rare that I would have a paper's worth of data generated by my techs or trainees and that they would have zero intellectual contribution. Given this, the asymmetry in the BMJ position is unfair. In essence it permits a lab head to be an author using data which s/he did not collect and maybe could not collect but excludes the technician who didn't happen to contribute to the drafting of the manuscript. That doesn't make sense to me. The paper wouldn't have happened without both of the contributions.

I agree with DrugMonkey that there's often a serious intellectual contribution involved in conducting the experiments, not just in designing them (and that without the data, all we have are interesting hunches, not actual scientific knowledge, to report). Existing authorship standards like those from ICMJE or BMJ can unfairly exclude those who do the experimental labor from authorship by failing to recognize this as an intellectual contribution. Pushing to have these real contributions recognized with appropriate career credit is important. As well, being explicit about who made these contributions to the research being reported in the paper makes it much easier for other scientists following up on the published work (e.g., comparing it to their own results in related experiments, or trying to use some of the techniques described in the paper to set up new experiments) to actually get in touch with the people most likely to be able to answer their questions.

Changing how might weight experimental prowess is given in the career scorekeeping may be an uphill battle, especially when the folks distributing the rewards for the top scores are administrators (focused on the money the people they're scoring can bring to an institution) and PIs (who frequently have more working hours devoted to conception and design of project for their underlings rather than to the intellectual labor of making those projects work, and to writing the proposals that bring in the grant money and the manuscripts that report the happy conclusion of the projects funded by such grants). That doesn't mean it's not a fight worth having.

But, I worry that using courtesy authorship as a way around this unfair setting of the authorship bar actually amounts to avoiding the fight rather than addressing these issues and changing accepted practices.

DrugMonkey also writes:

Assuming that we are not talking about pushing someone else meaningfully* out of deserved credit, where lies the harm even if it is a total gift?

Who is hurt? How are they damaged?
__
*by pushing them off the paper entirely or out of first-author or last-author position. Adding a 7th in the middle of the authorship list doesn't affect jack squat folks.

Here, I wonder: if dropping in a courtesy author as the seventh author of a paper can't hurt, how either can we expect it to help the person to whom this "courtesy" is extended?

Is it the case that no one actually expects that the seventh author made anything like a significant contribution, so no one is being misled in judging the guest in the number seven slot as having made a comparable contribution to the scientist who earned her seventh-author position in another paper? If listing your seventh-author paper on your CV is automatically viewed as not contributing any points in your career scorekeeping, why even list it? And why doesn't it count for anything? Is it because the seventh author never makes a contribution worth career points ... or is it because, for all we know, the seventh author may be a courtesy author, there for other reasons entirely?

If a seventh-author paper is actually meaningless for career credit, wouldn't it be more help to the person to whom you might extend such a "courtesy" if you actually engaged her in the project in such a way that she could make an intellectual contribution recognized as worthy of career credit?

In other words, maybe the real problem with such courtesy authorship is that it gives the appearance of help without actually being helpful.

(Cross-posted at Doing Good Science)

Share

6 responses so far

Limits of ethical recycling.

Jul 11 2011 Published by under Ethical research, Methodology, Plagiarism, Reader participation, Tribe of Science

In the "Ethics in Science" course I regularly teach, we spend some time discussing case studies to explore some of the situations students may encounter in their scientific training or careers where they will want to be able to make good ethical decisions.

A couple of these cases touch on the question of "recycling" pieces of old grant proposals or journal articles -- say, the background and literature review.

There seem to be cases where the right thing to do is pretty straightforward. For example, helping yourself to the background section someone else had written for her own grant proposal would be wrong. This would amount to misappropriating someone else's words and ideas without her permission and without giving her credit. (Plagiarism anyone?) Plus, it would be weaseling out of one's own duty to actually read the relevant literature, develop a view about what it's saying, and communicate clearly why it matters in motivating the research being proposed.

Similarly, reusing one's own background section seems pretty clearly within the bounds of ethical behavior. You did the intellectual labor yourself, and especially in the case where you are revising and resubmitting your own proposal, there's no compelling reason for you to reinvent that particular wheel (unless, if course, reviewer comments indicate that the background section requires serious revision, the literature cited ought to take account of important recent developments that were missing in the first round, etc.).

Between these two extremes, my students happened upon a situation that seemed less clear-cut. How acceptable is it to recycle the background section (or experimental protocol, for that matter) from an old grant proposal you wrote in collaboration with someone else? Does it make a difference whether that old grant proposal was actually funded? Does it matter whether you are "more powerful" or "less powerful" (however you want to cash that out) within the collaboration? Does it require explicit permission from the person with whom you collaborated on the original proposal? Does it require clear citation of the intellectual contribution of the person with whom you collaborated on the original proposal, even if she is not officially a collaborator on the new proposal?

And, in your experience, does this kind of recycling make more sense than just sitting down and writing something new?

Share

10 responses so far

A question for the trainees: How involved do you want the boss to get with your results?

Jul 08 2011 Published by under Academia, Methodology, Misconduct, Reader participation, Teaching and learning, Tribe of Science

This question follows on the heels of my recent discussion of the Bengü Sezen misconduct investigations, plus a conversation via Twitter that I recapped in the last post.

The background issue is that people -- even scientists, who are supposed always to be following the evidence wherever it might lead -- can run into trouble really scrutinizing the results of someone they trust (however that trust came about). Indeed, in the Sezen case, her graduate advisor at Columbia University, Dalibor Sames, seemed to trust Sezen and her scientific prowess so much that he discounted the results of other graduate students in his lab who could not replicate Sezen's results (which turned out to have been faked).

Really, it's the two faces of the PI's trust here: trusting one trainee so much that her results couldn't be wrong, and using that trust to ignore the empirical evidence presented by other trainees (who apparently didn't get the same level of presumptive trust). As it played out, at least three of those other trainees whose evidence Sames chose not to trust left the graduate program before earning their degrees.

The situation suggests to me that PIs would be prudent to establish environments in their research groups where researchers don't take scrutiny of their results, data, methods, etc., personally -- and where the scrutiny is applied to each member's results, data, methods, etc. (since anyone can make mistakes). But how do things play out when they rubber hits the road?

So, here's the question I'd like to ask the scientific trainees. (PIs: I've posed the complementary question to you in the post that went up right before this one!)

In his or her capacity as PI, your advisor's scientific credibility (and likely his or her name) is tied to all the results that come out of the research group -- whether they are experimental measurements, analyses of measurements, modeling results, or whatever else it is that scientists of your stripe regard as results. Moreover, in his or her capacity as a trainer of new scientists, the boss has something like a responsibility to make sure you know how to generate reliable results -- and that you know how to tell them from results that aren't reliable. What does your PI do to ensure that the results you generate are reliable? Do you feel like it's enough (both in terms of quality control and in terms of training you well)? Do you feel like it's too much?

Commenting note: You may feel more comfortable commenting with a pseudonym for this particular discussion, and that's completely fine with me. However, please pick a unique 'nym and keep it for the duration of this discussion, so we're not in the position of trying to sort out which "Anonymous" is which. Also, if you're a regular commenter who wants to go pseudonymous for this discussion, you'll probably want to enter something other than your regular email address in the commenting form -- otherwise, your Gravatar may give your other identity away!

Share

5 responses so far

A question for the PIs: How involved do you get in your trainees' results?

Jul 08 2011 Published by under Academia, Methodology, Misconduct, Reader participation, Teaching and learning, Tribe of Science

In the wake of this post that touched on recently released documents detailing investigations into Bengü Sezen's scientific misconduct, and that noted that a C & E News article described Sezen as a "master of deception", I had an interesting chat on the Twitters:

@UnstableIsotope (website) tweeted:

@geernst @docfreeride I scoff at the idea that Sezen was a master at deception. She lied a lot but plenty of opportunities to get caught.

@geernst (website) tweeted back:

@UnstableIsotope Maybe evasion is a more accurate word.

@UnstableIsotope:

@geernst I'd agree she was a master of evasion. But she was caught be other group members but sounds like advisor didn't want to believe it.

@docfreeride (that's me!):

@UnstableIsotope @geernst Possible that she was master of deception only in environment where people didn't guard against being deceived?

@UnstableIsotope:

@docfreeride @geernst I agree ppl didn't expect deception, my read suggests she was caught by group members but protected by advisor.

@UnstableIsotope:

@docfreeride @geernst The advisor certainly didn't expect deception and didn't encourage but didn't want to believe evidence

@docfreeride:

@UnstableIsotope @geernst Not wanting to believe the evidence strikes me as a bad fit with "being a scientist".

@UnstableIsotope:

@docfreeride @geernst Yes, but it is human. Not wanting to believe your amazing results are not amazing seems like a normal response to me.

@geernst:

@docfreeride @UnstableIsotope I agree. Difficult to separate scientific objectivity from personal feelings in those circumstances.

@docfreeride:

@geernst @UnstableIsotope But isn't this exactly the argument for not taking scrutiny of your results, data, methods personally?

@UnstableIsotope:

@docfreeride @geernst Definitely YES. I look forward to people repeating my experiments. I'm nervous if I have the only result.

@geernst:

@docfreeride @UnstableIsotope Couldn't agree more.

This conversation prompted a question I'd like to ask the PIs. (Trainees: I'm going to pose the complementary question to you in the very next post!)

In your capacity as PI, your scientific credibility (and likely your name) is tied to all the results that come out of your research group -- whether they are experimental measurements, analyses of measurements, modeling results, or whatever else it is that scientists of your stripe regard as results. What do you do to ensure that the results generated by your trainees are reliable?

Now, it may be the case that what you see as the appropriate level of involvement/quality control/"let me get up in your grill while you repeat that measurement for me" would still not have been enough to deter -- or to detect -- a brazen liar. If you want to talk about that in the comments, feel free.

Commenting note: You may feel more comfortable commenting with a pseudonym for this particular discussion, and that's completely fine with me. However, please pick a unique 'nym and keep it for the duration of this discussion, so we're not in the position of trying to sort out which "Anonymous" is which. Also, if you're a regular commenter who wants to go pseudonymous for this discussion, you'll probably want to enter something other than your regular email address in the commenting form -- otherwise, your Gravatar may give your other identity away!

Share

3 responses so far

What are honest scientists to do about a master of deception?

Jul 07 2011 Published by under Chemistry, Current events, Methodology, Misconduct, Tribe of Science, Uncategorized

A new story posted at Chemical & Engineering News updates us on the fraud case of Bengü Sezen (who we discussed here, here, and here at much earlier stages of the saga).

William G. Schultz notes that documents released (PDF) by the Department of Health and Human Services (which houses the Office of Research Integrity) detail some really brazen misconduct on Sezen's part in her doctoral dissertation at Columbia University and in at least three published papers.

From the article:

The documents—an investigative report from Columbia and HHS’s subsequent oversight findings—show a massive and sustained effort by Sezen over the course of more than a decade to dope experiments, manipulate and falsify NMR and elemental analysis research data, and create fictitious people and organizations to vouch for the reproducibility of her results. ...

A notice in the Nov. 29, 2010, Federal Register states that Sezen falsified, fabricated, and plagiarized research data in three papers and in her doctoral thesis. Some six papers that Sezen had coauthored with Columbia chemistry professor Dalibor Sames have been withdrawn by Sames because Sezen’s results could not be replicated. ...

By the time Sezen received a Ph.D. degree in chemistry in 2005, under the supervision of Sames, her fraudulent activity had reached a crescendo, according to the reports. Specifically, the reports detail how Sezen logged into NMR spectrometry equipment under the name of at least one former Sames group member, then merged NMR data and used correction fluid to create fake spectra showing her desired reaction products.

Apparently, her results were not reproducible because those trying to reproduce them lacked her "hand skills" with Liquid Paper.

Needless to say, this kind of behavior is tremendously detrimental to scientific communities trying to build a body of reliable knowledge about the world. Scientists are at risk of relying on published papers that are based in wishes (and lies) rather than actual empirical evidence, which can lead them down scientific blind alleys and waste their time and money. Journal editors devoted resources to moving her (made-up) papers through peer review, and then had to devote more resources to dealing with their retractions. Columbia University and the U.S. government got to spend a bunch of money investigating Sezen's wrongdoing -- the latter expenditures unlikely to endear scientific communities to an already skeptical public. Even within the research lab where Sezen, as a grad student, was concocting her fraudulent results, her labmates apparently wasted a lot of time trying to reproduce her results, questioning their own abilities when they couldn't.

And to my eye, one of the big problems in this case is that Sezen seems to have been the kind of person who projected confidence while lying her pants off:

The documents paint a picture of Sezen as a master of deception, a woman very much at ease with manipulating colleagues and supervisors alike to hide her fraudulent activity; a practiced liar who would defend the integrity of her research results in the face of all evidence to the contrary. Columbia has moved to revoke her Ph.D.

Worse, the reports document the toll on other young scientists who worked with Sezen: “Members of the [redacted] expended considerable time attempting to reproduce Respondent’s results. The Committee found that the wasted time and effort, and the onus of not being able to reproduce the work, had a severe negative impact on the graduate careers of three (3) of those students, two of whom [redacted] were asked to leave the [redacted] and one of whom decided to leave after her second year.”

In this matter, the reports echo sources from inside the Sames lab who spoke with C&EN under conditions of anonymity when the case first became public in 2006. These sources described Sezen as Sames’ “golden child,” a brilliant student favored by a mentor who believed that her intellect and laboratory acumen provoked the envy of others in his research group. They said it was hard to avoid the conclusion that Sames retaliated when other members of his group questioned the validity of Sezen’s work.

What I find striking here is that Sezen's vigorous defense of her's own personal integrity was sufficient, at least for awhile, to convince her mentor that those questioning the results were in the wrong -- not just incompetent to reproduce the work, but jealous and looking to cause trouble. And, it's deeply disappointing that this judgment may have been connected to the departure of those fellow graduate students who raised questions from their graduate program.

How could this have been avoided?

Maybe a useful strategy would have been to treat questions about the scientific work (including its reproducibility) first and foremost as questions about the scientific work.

Getting results that others cannot reproduce is not prima facie evidence that you're a cheater-pants. It may just mean that there was something weird going on with the equipment, or the reagents, or some other component of the experimental system when you did the experiment that yielded the exciting but hard to replicate results. Or, it may mean that the folks trying to replicate the results haven't quite mastered the technique (which, in the case that they are your colleagues in the lab, could be addressed by working with them on their technique). Or, it may mean that there's some other important variable in the system that you haven't identified as important and so have not worked out (or fully described) how to control.

In this case, of course, it's looking like the main reason that Sezen's results were not reproducible was that she made them up. But casting the failure to replicate presumptively as one scientist's mad skillz and unimpeachable integrity against another's didn't help get to the bottom of the scientific facts. It made the argument personal rather than putting the scientists involved on the same team in figuring out what was really going on with the scientific systems being studied.

Of all of the Mertonian norms imputed to the Tribe of Science, organized skepticism is probably the one nearest and dearest to most scientists' basic understanding of how they get the knowledge-building job done. Figuring out what's going on with particular phenomena in the world can be hard, not least because lining up solid evidence to support your conclusions requires identifying evidence that others trying to repeat your work can reliably obtain themselves. This is more than just a matter of making sure your results are robust. Rather, you want others to be able to reproduce your work so that you know you haven't fooled yourself.

Organized skepticism, in other words, should start at home.

There is a risk of being too skeptical of your own results, and there are chances to overlook something important as noise because it doesn't fit with what you expect to observe. However, the scientist who refuses to entertain the possibility that her work could be wrong -- indeed, who regards questions about the details of her work as a personal affront -- should raise a red flag for the rest of her scientific community, no matter what her career stage or her track record of brilliance to date.

In a world where every scientist's findings are recognized as being susceptible to error, the first response to questions about findings might be to go back to the phenomena together, helping each other to locate potential sources of error and to avoid them. In such a world, the master of deception trying to ride personal reputation (or good initial impressions) to avoid scrutiny of his or her work will have a much harder time getting traction.

Share

16 responses so far

Scientific knowledge and "what everyone knows".

May 01 2011 Published by under Medicine, Methodology, Science and pseudo-science, Scientist/layperson relations

Those of you who read the excellent blog White Coat Underground have probably had occasion to read PalMD's explanation of the Quack Miranda Warning, the disclaimer found on various websites and advertisements that reads, "These statements have not been evaluated by the Food and Drug Administration. This product is not intended to diagnose, treat, cure or prevent any disease." When found on a website that seems actually to be offering diagnosis, treatment, cure, or prevention, PalMD notes, this language seems like a warning that the big change that will be effected is that your wallet will be lightened.

In response to this, Lawrence comments:

This statement may be on every quack website but is on every legitamate website and label as well. Take vitamin C for example. Everyone knows that it can help treat & cure diseases. Vitamin C has been used for centuries to cure disease by eating various foods that are high in it. Even doctors tell you it is good to take when you are sick because it helps your body fight off the disease. So the fact that this statement is required to be on even the most obviously beneficial vitamins pretty much means that the FDA requires a companies to lie to the public and that they have failed in their one duty to encouraging truth in health. Once I realized this, it totally discredits everything the FDA says.

Sure if something is not approved by a big organization whose existance is supposed to safeguard health it makes it easier for the little con artest to step in at every opportunity, but that doesn't mean that the big con artests arn't doing the same thing

PalMD's reply is succinct:

"Everyone knows..."

A phrase deadly to science.

I'm going to add my (less succinct) two cents.

There are plenty of things that people take to be something everyone knows. (The "everyone" is tricky, because there are enough people on the planet that it's usually (always?) possible to find someone who doesn't know X.). And, I'm happy to grant that, for some values of X, there are indeed many people who believe X.

But belief is not the same as knowledge.

What "everyone knows" about celebrities should help us notice the difference. Richard Gere? Jamie Lee Curtis? Even in the event that everyone has heard the same rumors, the extent of what we actually know is that there are rumors. Our propensity to believe rumors is why the team at Snopes will never want for material.

This is not to say that we have to do all of our epistemic labor ourselves. Indeed, we frequently rely on the testimony of others to help us know more than we could all by ourselves, But, this division of labor introduces risks if we accept as authoritative the testimony of someone who is mistaken -- or who is trying to sell us snake-oil. Plus, when we're accepting the testimony of someone who knows X on the basis of someone else's testimony, our connection to the actual coming-to-know of X (through a mode other than someone else's say-so) becomes more attenuated.

At least within the realm of science, the non-testimony route to knowledge involves gathering empirical evidence under conditions that are either controlled or at least well characterized. Ideally, the effects that are observed are both repeatable in relevantly similar conditions and observable by others. Science, in its methodology, strives to ground knowledge claims in observational evidence that anyone could come to know (assuming a standard set of properly functioning sense organs). Part of how we know that we know X is that the evidence in support of X can be inspected by others. At this basic level, we don't have to take anyone else's word for X; the testimony of our senses (and the fact that others who are pointing their sense organs at the same bits of the world and seeing the same things) gives us the support for our beliefs that we need.

Claims without something like empirical support might inspire belief, but they don't pass scientific muster. To the extent that an agency like the FDA is committed to evaluating claims in a scientific framework, this means that they want to evaluate the details of the experiments used to generate the empirical data that are being counted as support for those claims. In other contexts, folks may be expecting, or settling for, other standards of evidence. In scientific contexts, including biomedical ones, scientific rules of evidence are what you get.

Why then, one might ask, might a physician suggest vitamin C to a patient with a cold if there isn't sufficient scientific evidence to say we know vitamin C cures cold?

There are a few possibilities here. One is that the physician judges (on the basis of a reasonable body of empirical evidence) that taking vitamin C is unlikely to do harm to the patient with a cold. If the physician's clinical experience is that cold patients will feel better with some intervention than with no intervention, recommending vitamin C may seem like the most benign therapeutic option.

It's also possible that some of these physicians accept the testimony of someone else who tells the there is good reason to believe that vitamin C cures colds. Being human, physicians sometimes get burned by testimony that turns out to be unreliable.

It's even possible that some physicians are not so clear on scientific rules of evidence, and that they make recommendations on the basis of beliefs that haven't been rigorously tested. The more high profile of these physicians are the kinds of folks about whom PalMD frequently blogs.

Share

2 responses so far

Friday Sprog Blogging: science fair experimental design.

Feb 11 2011 Published by under Critters, Kids and science, Methodology

The elder Free-Ride offspring is thinking about a project studying the behavior of Snowflake Free-Ride, the rabbit in residence at Casa Free-Ride. While finding interesting questions to ask about the bunny is pretty easy, working out reasonable ways to get data that might help answer those questions is somewhat harder:

Elder offspring: I want to see whether Snowflake finds food with her eyes or her nose.

Dr. Free-Ride: What are your thoughts on how to do that?

Elder offspring: Well, we need a room ...

Dr. Free-Ride: ... OK. Tell me more.

Elder offspring: We need a room with a fan up at the top.

Dr. Free-Ride: Why do we need a fan up at the top?

Elder offspring: To blow away the smells.

Dr. Free-Ride: Hmm. So you're looking for some mechanism to mask smells and see if she can still find the food.

Elder offspring: Yes.

Dr. Free-Ride: I guess I'm not totally convinced a fan is the best way to mask a smell. Also, I worry that it might freak her out.

Elder offspring: Oh.

Dr. Free-Ride: Well, your hypothesis is that she's either finding the food by smell or by sight. So how would you tell if sight is what she's using?

Elder offspring: We start the fan and put the food there and if she can find it ... We may also need to use a clothespin, like in those cartoons --

Dr. Free-Ride: We're totally not putting a clothespin on the rabbit's nose, smart aleck!

Elder offspring: (snickering) I know.

Dr. Free-Ride: Let's back up a little bit. We're talking about two possible ways you think the rabbit could locate food -- one is by vision, one is by smell. Masking smell means we have to figure out a way to get the volatile stuff that the nose detects away from her. But my own hunch is that masking sight might be easier. Do you have any thoughts on how to mask --

Elder offspring: Blindfolds.

Dr. Free-Ride: Uh, no. You'll have to be more clever, since you can't blindfold the bunny.

Elder offspring: Put her in a dark room.

Dr. Free-Ride: I don't know how good her night vision is. (Or how good your night vision is if you're in the dark room trying to observe her.)

Elder offspring: If we hear munching ...

Dr. Free-Ride: Isn't she always munching on something?

Elder offspring: We'd use a food where the munching sounds like crunching.

Dr. Free-Ride: Aside from utter darkness, can you think of any other way to mask visual contact with the food?

Elder offspring: What if we surround a carrot by things that are visually distracting?

Dr. Free-Ride: Does that really test whether she's using vision to find the carrot, or whether she can pick it out visually amongst a bunch of visually distracting things? Maybe you need to think about whether there's some way to disguise it looking like a carrot, but it would still be there for her to smell.

Elder offspring: How about we put it behind a curtain or something?

Dr. Free-Ride: Ah, a barrier that keeps her from seeing it. Then, with the carrot out of sight but in smelling range, you'd see if she reacted like, "Where's the carrot. GIMME THE CARROT!"

Elder offspring: Yeah.

Dr. Free-Ride: OK, that seems like a key part of your experimental design: how exactly are you going to mask the carrot's visibility but not its smell?

Elder offspring: Invisibility cloak!

Dr. Free-Ride: You don't get to use things that don't exist in your science fair project. Unless you can successfully invent them, in which case -- if you can successfully invent an invisibility cloak, I submit to you that that would probably be a more impressive science fair project than this information on rabbit behavior that you obtain using the invisibility cloak.

Elder offspring: Yeah, OK.

Dr. Free-Ride: Hey, is it going to be a problem that you have exactly one rabbit to study?

Elder offspring: Nah.

Dr. Free-Ride: What's that going to do to the conclusions you can draw.

Elder offspring: I probably can't say that all rabbits are like this based on the behavior of this one rabbit. But, she's a pretty typical rabbit.

Dr. Free-Ride: How do you know she's pretty typical?

Elder offspring: Because, she's a breed [New Zealand white] that's raised for lab use, and they want typical animals for lab use.

Dr. Free-Ride: Which means you would be surprised if she were very weird, as rabbits go?

Elder offspring: Yes.

Dr. Free-Ride: Of course, she's been living with you for almost a year now. That might be enough to make a rabbit weird.

Elder offspring: Hey!

Dr. Free-Ride: I'm just saying. So, back to your experimental design, since Snowflake is a smart rabbit -- she learns stuff -- what if you make a curtain or some other barrier and she starts associating it with carrots?

Elder offspring: Maybe sometimes we could just put a rock behind it instead of a carrot.

Dr. Free-Ride: Good call -- something that isn't edible and doesn't smell like a treat.

Share

7 responses so far

Dispatch from PSA 2010: Symposium session on ClimateGate.

Nov 05 2010 Published by under Conferences, Doing science for the government, Environment, Methodology, Philosophy, Politics, Professional ethics, Scientist/layperson relations, Tribe of Science

The Philosophy of Science Association Biennial Meeting included a symposium session on the release of hacked e-mails from the Climate Research Unit at the University of East Anglia. Given that we've had occasion to discuss ClimateGate here before, i thought I'd share my notes from this session.

Symposium: The CRU E-mails: Perspectives from Philosophy of Science.

Naomi Oreskes (UC San Diego), gave a talk called "Why We Resist the Results of Climate Science."

She mentioned the attention brought to the discovery of errors in the IPCC report, noting that while mistakes are obviously to be avoided, it would be amazing for there to be a report that ran thousands of pages that did not have some mistakes. (Try to find a bound dissertation -- generally only in the low hundreds of pages -- without at least one typo.) The public's assumption, though, was that these mistakes, once revealed, were smoking guns -- a sign that something improper must have occurred.

Oreskes noted the boundary scientists of all sorts (including climate scientists) have tried to maintain between the policy-relevant and the policy-prescriptive. This is a difficult boundary to police, though, as climate science has an inescapable moral dimension. To the extent that climate change is driven by consumption (especially but not exclusively the burning of fossil fuels), we have a situation where the people reaping the benefits are not the ones who will be paying for that benefit (since people in the developed world will have the means to respond to the effects of climate change and those in the developing world will not). The situation seems to violate our expectations of intergenerational equity (since future generations will have to cope with the consequences of the consumption of past and current generations), as well as of inter-specific equity (since the species likely to go extinct in response to climate change are not the ones contributing the most to climate change).

The moral dimension of climate change, though, doesn't make this a scientific issue about which the public feels a sense of clarity. Rather, the moral issues are such that Americans feel like their way of life is on trial. Those creating the harmful effects have done something wrong, even if it was accidental.

And this is where the collision occurs: Americans believe they are good; climate science seems to be telling them that they are bad. (To the extent that people strongly equate capitalism with democracy and the American way of life, that's an issue too, given that consumption and growth are part of the problem.)

The big question Oreskes left us with, then, is how else to frame the need for changes in behavior, so that such a need would not make Americans so defensive that they would reflexively reject the science. I'm not sure the session ended with a clear answer to that question.

* * * * *

Wendy S. Parker (Ohio University) gave a talk titled "The Context of Climate Science: Norms, Pressures, and Progress." A particular issue she took up was the ideal of transparency and how it came up in the context of climate scientists interactions with each other and with the public.

Parker noted that there had been numerous requests for access to raw data by people climate scientists did not recognize as part of the climate science community. The CRU denied many such requests, and the ClimateGate emails made it clear that the scientists generally didn't want to cooperate with these requests.

Here, Parker observed that while we tend to look favorably on transparency, we probably need to say more about what transparency should amount to. Are we talking about making something available and open to scrutiny (i.e., making "transparency" roughly the opposite of "secrecy")? Are we talking about making something understandable or usable, perhaps by providing fully explained nontechnical accounts of scientific methods and findings for the media (i.e., making "transparency" roughly the opposite of "opacity")?

What exactly do we imagine ought to be made available? Research methods? Raw and/or processed data? Computer code? Lab notebooks? E-mail correspondence?

To whom ought the materials to be made available? Other members of one's scientific community seems like a good bet, but how about members of the public at large? (Or, for that matter, members of industry or of political lobbying groups?)

And, for that matter, why do we value transparency? What makes it important? Is it primarily a matter of ensuring the quality of the shared body of scientific knowledge, and of improving the rate of scientific progress? Or, do we care about transparency as a matter of democratic accountability? As Parker noted, these values might be in conflict. (As well, she mentioned, transparency might conflict with other social values, like the privacy of human subjects.)

Here, if the public imputed nefarious motives to the climate researchers, the scientists themselves viewed some of the requests for access to their raw data as attempts by people with political motivations to obstruct the progress (or acceptance) of their research. It was not that the scientists feared that bad science would be revealed if the data were shared, but rather that they worried that yahoos from outside the scientific community were going to waste their time, or worse to cherry pick the shared data to make allegations that the scientists to which would then have to respond, wasting even more time.

In the numerous investigations that followed on the heels of the leak of stolen CRU e-mails, about the strongest charge against the involved climate scientists that stood was that they failed to display "the proper degree of openness", and that they seemed to have a ethos of minimal compliance (or occasionally non-compliance) with regard to Freedom of Information Act (FOIA) requests. They were chided that the requirements of FOIA must not be seen as impositions, but as part of their social contract with the public (and something likely to make their scientific knowledge better).

Compliance, of course, takes resources (one of the most important of these being time), so it's not free. Indeed, it's hard not to imagine that at least some FOIA requests to climate scientists had "unintended consequences" (in terms of the expenditure of tim and other resources) on climate scientists that were precisely what the requesters intended.

However, as Parker noted, FOIA originated with the intent of giving citizens access to the workings of their government -- imposing it on science and scientists is a relatively new move. It is true that many scientists (although not all) conduct publicly funded research, and thereby incur some obligations to the public. But there's a question of how far this should go -- ought every bit of data generated with the aid of any government grant to be FOIA-able?

Parker discussed the ways that FOIA seems to demand an openness that doesn't quite fit with the career reward structures currently operating within science. Yet ClimateGate and its aftermath, and the heightened public scrutiny of, and demands for openness from, climate scientists in particular, seem to be driving (or at least putting significant pressure upon) the standards for data and code sharing in climate science.

I got to ask one of the questions right after Parker's talk. I wondered whether the level of public scrutiny on climate scientists might be enough to drive them into the arms of the "open science" camp -- which would, of course, require some serious rethinking of the scientific reward structures and the valorization of competition over cooperation. As we've discussed on this blog on many occasions, institutional and cultural change is hard. If openness from climate scientists is important enough to the public, though, could the public decide that it's worthwhile to put up the resources necessary to support this kind of change in climate science?

I guess it would require a public willing to pay for the goodies it demands.

* * * * *

The next talk, by Kristin Shrader-Frechette (University of Notre Dame), was titled "Scientifically Legitimate Ways to Cook and Trim Data: The Hacked and Leaked Climate Emails."

Shrader-Frechette discussed what statisticians (among others) have to say about conditions in which it is acceptable to leave out some of your data (and indeed, arguably misleading to leave it in rather than omitting it). There was maybe not as much unanimity here as one might like.

There's general agreement that data trimming in order to make your results fit some predetermined theory is unacceptable. There's less agreement about how to deal with outliers. Some say that deleting them is probably OK (although you'd want to be open that you have done so). On the other hand, many of the low probability/high consequence events that science would like to get a handle on are themselves outliers.

So when and how to trim data is one of those topics where it looks like scientists are well advised to keep talking to their scientific peers, the better not to mess it up.

Of the details in the leaked CRU e-mails, one that was frequently identified as a smoking gun indicating scientific shenanigans was the discussion of the "trick" to "hide the decline" in the reconstruction of climatic temperatures using proxy data from tree-rings. Shrader-Frechette noted that what was being "hidden" was not a decline in temperatures (as measured instrumentally) but rather in the temperatures reconstructed from one particular proxy -- and that other proxies the climate scientists were using didn't show this decline.

The particular incident raises a more general methodological question: scientifically speaking, is it better to include the data from proxies (once you have reason to believe it's bad data) in your graphs? Is including it (or leaving it out) best seen as scrupulous honesty or as dishonesty?

And, does the answer differ if the graph is intended for use in an academic, bench-science presentation or a policy presentation (where it would be a very bad thing to confuse your non-expert audience)?

As she closed her talk, Shrader-Frechette noted that welfare-affecting science cannot be treated merely as pure science. She also mentioned that while FOIA applies to government-funded science, it does not apply to industry-funded science -- which means that the "transparency" available to the public is pretty asymmetrical (and that industry scientists are unlikely to have to devote their time to responding to requests from yahoos for their raw data).

* * * * *

Finally, James McAllister (University of Leiden) gave a talk titled "Errors, Blunders, and the Construction of Climate Change Facts." He spoke of four epistemic gaps climate scientists have to bridge: between distinct proxy data sources, between proxy and instrumental data, between historical time series (constructed of instrumental and proxy data) and predictive scenarios, and between predictive scenarios and reality. These epistemic gaps can be understood in the context of the two broad projects climate science undertakes: the reconstruction of past climate variation, and the forecast of the future.

As you might expect, various climate scientists have had different views about which kinds of proxy data are most reliable, and about how the different sorts of proxies ought to be used in reconstructions of past climate variation. The leaked CRU e-mails include discussions where climate scientists dedicate themselves to finding the "common denominator" in this diversity of expert opinion -- not just because such a common denominator might be expected to be closer to the objective reality of things, but also because finding common ground in the diversity of opinion could be expected to enhance the core group's credibility. Another effect, of course, is that the common denominator is also denied to outsiders, undermining their credibility (and effectively excluding them as outliers).

McAllister noted that the emails simultaneously revealed signs of internal disagreement, and of a reaching for balance. Some of the scientists argued for "wise use" of proxies and voiced judgments about how to use various types of data.

The data, of course, cannot actually speak for themselves.

As the climate scientists worked to formulate scenario-based forecasts that public policy makers would be able to use, they needed to grapple with the problems of how to handle the link between their reconstructions of past climate trends and their forecasts. They also had to figure out how to handle the link between their forecasts and reality. The e-mails indicate that some of the scientists were pretty resistant to this latter linkage -- one asserted that they were "NOT supposed to be working with the assumption that these scenarios are realistic," rather using them as internally consistent "what if?" storylines.

One thing the e-mails don't seem to contain is any explicit discussion of what would count as an ad hoc hypothesis and why avoiding ad hoc hypotheses would be a good thing. This doesn't mean that the climate scientists didn't avoid them, just that it was not a methodological issue they felt they needed to be discussing with each other.

This was a really interesting set of talks, and I'm still mulling over some of the issues they raised for me. When those ideas are more than half-baked, I'll probably write something about them here.

Share

52 responses so far

Older posts »