NSF Broader Impacts, take eleventy

Apr 04 2012 Published by under [Education&Careers]

We have been through this before. And before that. And before that. Yet still, I keep reading proposals where the PIsclear said "OMG! BI section!" and either vomited up everything that sounds remotely like outreach that they can think of, or wrote down everything they already do that seems like training.

Click on the links above for what a BI section IS, but it is certainly NOT something that should be feared. It is also not:

-Entering information on your organisms/system into some database like Tree of Life. Whereas that is a good thing to do, it does not constitute meaningful BI.

-Creating your own website to host your data and "distribute" software. You might as well yell your results down the hall. You will probably affect the same number of people. Again, software should be accessible, but this is not a BI of note.

-Training grad students. You already get paid to do this.

-Teaching. Ditto.

-Doing all these things in some mixed BI turd sandwich. Just because you write a whole paragraph doesn't mean you have thought about potential BI in your project.

Broader Impacts do not have to be complicated, but you need to show that you've given some thought about engaging people you wouldn't otherwise. This is really not that hard if you find out what your institution is already doing on this front. If you are at a museum or have one associated with your institution, you are surrounded by BI opportunities. If you are in a city, the same. If your institution has ANY program to engage high school or minority students (or both) in science, your work is 80% done. If you have field sites in location where science engagement would be seen as a Good Thing, develop that.

Opportunities to develop interesting and useful BI plans abound, which makes it frustrating when PIs can't even bother to see what is happening at their own institution. While the Web is a useful tool, most websites don't get much traffic, and when they do, it is from people search for that specific thing. In this case, your page about your sub-sub-sub-field is going to be looked at by people you probably already know by first name. Engage them at the bar.

38 responses so far

  • Lab Rockstar says:

    Wow, you've really made some good points. I wish the grant writing class I took last year was this helpful.

  • LD says:

    Sorry, but I disagree. Training graduate students has always been considered a component of Broader Impacts, at least with all the Program Directors at NSF that I have talked to.

  • proflikesubstance says:

    Training postdocs, grad students and undergrads IS very important, but it can't be the extent of your BI section. Period.

  • Isabel says:

    I'd love to see an example of what you think is a great BI. As a grad student just starting to grapple with this, the section does feel very contrived. And I do a lot of science outreach and really enjoy it! Also, do I know in advance the ethnicity of the undergrads I am going to be training, etc?

    The 'how will this help society' feels the most contrived. It is not surprising that things that should be in IM section come out in response to this question. If you aren't in biomed or working directly on climate change or something, what's left except the intellectual merit or some real stretches that sound phony?

    If it counts as much as the science I agree with the commenter who said BI should be much more of a structured part of the proposal, not just a small tacked on section.

  • proflikesubstance says:

    In reality, the BI section does not hold nearly the weight of the science and nor should it. It should also be something that is not too onerous on the PI or lab, which is why hooking up with an on-going campus initiative is a huge advantage.

    The BI section doesn't really cover the "how this will help society" angle, that is more a piece that goes under IM. A good BI section has concrete deliverables, such as a training workshop, internship program, museum exhibit, teaching module, etc. If you can give the preparation of that deliverable a budget, then you are committing to it.

  • ecologist says:

    I'm sorry, but you are wrong on so many levels that it isn't even funny, and the attitude you have is one of the reasons why the BI issue is so f****d up. Let me explain.

    NSF has a definition of broader impacts. It says nothing about excluding training students "because you already get paid to do that". Is training students part of the research? No. Then it is by definition broader and it is one of the things included in the NSF definition.

    (Parenthetically, the upcoming revision of BI criteria, which I don't think has come out yet, will help this problem a lot.)

    Most of what people talk about as broader impact is not an impact of the research at all; in fact, it has nothing to do with the research. It has devolved into a way to do public service work, in the general area of "science", but having, in most cases, nothing to do with the research. For example, I read a favorable account of some chemists who were applauded for doing BI work by helping junior highschool students learn about pH. Now, I would bet money that their NSF-funded research was not about pH at the junior high school level, and while the outreach to young students is a great thing, it is not by any stretch an impact OF the research.

    This attitude that the training of students (and others who might not be grad students) somehow doesn't count as BI because "you are already doing it" needs to be argued against every time it comes up, because it is a fundamental distortion of what NSF wants to do with BI.

  • proflikesubstance says:

    Have you actually read the BI section of the RFP?

    As stated above, training IS important, but if your BI section relies on training people in your lab (unless you can demonstrate that you will be training a large proportion of under-represented minority students), you are not going to score well. This is based on what I have seen as both a panelist and proposal writer.

    Maybe your panels do it differently.

  • Isabel says:

    "The BI section doesn't really cover the "how this will help society" angle, that is more a piece that goes under IM."

    Well, that is confusing, because it is straight from the BI bullet points.
    "What may be the benefits of the proposed activity to society?"

    I still say it all feels either contrived or completely self-evident, except maybe "To what extent will it enhance the infrastructure for research and education, such as facilities, instrumentation, networks and partnerships?"

    To what extent would it hurt an application to not hit all the bullet points. Say, focus on two or three?

  • proflikesubstance says:

    "The BI section doesn't really cover the "how this will help society" angle, that is more a piece that goes under IM."

    Well, that is confusing, because it is straight from the BI bullet points.
    "What may be the benefits of the proposed activity to society?"

    Sorry, I wasn't clear. The benefit to society that is typical in the BI section is tangible - the training and education aspects. The benefit to society that often ends up in the IM section is of the pie-in-the-sky "my data will solve everything" variety that I assumed you were referring to.

    IME, a solid BI plan that focuses on one or two aspects, while at least paying lip service to the other points, usually goes over the strongest.

  • Isabel says:

    Thanks. I was confused - I actually did think "my data will solve everything" belonged in BI, since training, outreach, etc are covered in other bullet points. I didn't know what else was left to impact "society", except perhaps how my research may lead to a cure for cancer.

  • proflikesubstance says:

    Again, different groups may approach this differently, but based on experience with both DEB and IOS panels, I generally see the pattern described above. Sometimes the pie-in-the-sky stuff gets put in the BI section, but it has little impact there.

  • ecologist says:

    Have I actually read the BI section of the RFP? Yes. I have, but I wonder whether you have. Here's what it says:

    "How well does the activity advance discovery and understanding while promoting teaching, training, and learning? How well does the proposed activity broaden the participation of underrepresented groups (e.g., gender, ethnicity, disability, geographic, etc.)? To what extent will it enhance the infrastructure for research and education, such as facilities, instrumentation, networks, and partnerships? Will the results be disseminated broadly to enhance scientific and technological understanding? What may be the benefits of the proposed activity to society?"

    (From NSF current Grant Proposal Guide, downloaded 4/4/2012).

    That means that promoting teaching, training and learning are perfectly good BI activities. As are things that increase participation of underrepresented groups, or that enhance infrastructure (whatever that means). It even means that if you disseminate the results broadly (I have always interpreted this to mean more broadly than just writing journal articles) that's a BI. And if your research will benefit society, that could be a BI also.

    So explain again why teaching, leading workshops, participating in training activities, preparing teaching materials, and disseminating results to students are not contributions to BI?

  • proflikesubstance says:

    So explain again why teaching, leading workshops, participating in training activities, preparing teaching materials, and disseminating results to students are not contributions to BI?

    All I'm saying is that I have watch numerous proposals get hammered for the ole' "I'm gonna train my students, incorporate my research into my classes and publish papers" routine. It. Doesn't. Fly.

    On the contrary, if you have a plan to develop teaching materials that will be disseminated in a coordinated fashion in a measurable way, then that is a good thing. If you get involved in organized training activities (e.g. workshops) that go beyond your lab, great. You need to propose things you can be accountable for, NOT things that can be vapor promises.

  • Isabel says:

    I think the problem that keeps being alluded to with things like workshops is that they don't usually incorporate the latest research (i.e. in the rest of the proposal).

  • The times they are a changin'.

    The revised Broader Impacts Criterion is all about benefits to society:

    "The Broader Impacts criterion encompasses the potential to benefit society and contribute to the achievement of specific, desired societal outcomes."

    We go into more detail here, for those who are interested: http://scienceprogress.org/2012/03/owning-the-national-science-foundation%E2%80%99s-broader-impacts-criterion/.

  • GMP says:

    I have to temporarily break my "no more comments on PlS blog" policy here to say that I sit on panels in physics/materials and engineering divisions and actually what PlS says holds largely true there. If your BI is only "I will train my own grad students/postdocs" you will be toast; yes, panelists do say that's part of what the university is paying you to do and what you do to get anything done anyway. Many people also say "meh" when you say you are introducing a special topics grad course on your sub-sub-subfield, because let's face it, you do it to train your own grad students to be better at their research.

    PlS experience is closely aligned with what I have witnessed. A good BI that does not require a lot of legwork would mean plugging into whatever your institution or your college is doing to recruit women and underrepresented minorities (maybe they have an REU center? Recruit summer students through them. Maybe they have residential programs for women? Connect to the program. And get support letters from everyone to show you are serious. Budgeting for all you plan on doing also shows you are serious, although be aware that REU and RET supplements you can get extra every year) or to reach out to K-12 and the general public. For fuck's sake, ask around -- what is it that your institution has the infrastructure and funds to do already in terms of outreach? Do you want to connect to K-12? There are NSF-sponsored activities that you can contribute to if you don't have the time to go teach middle schoolers-- you can have kids come visit a lab, host a teacher over the summer, contribute to middle school curricula. There are also high-traffic websites that reach out to K-12, connect with them if you have materials you want to share.

    I cannot stress enough how insufficient (and I would say lame) it looks if your BI only says "will train grad students and teach them how to write papers and send them to conferences." Engagement of undergrads in research, recruitment/retention of women and minorities in STEM, outreach to K-12 and the general public, all the while while leveraging institutional resources, other NSF funded activities, and providing evidence that you are serious (letters of commitment from partners, money budgeted for activities, and ASSESSMENT!!!) are what can help create a serious BI.

  • Isis the Scientist says:

    My policy is to only comment on the PLS blog in order to crack wise.

    Consider the wise cracked.

  • proflikesubstance says:

    Isis, I thought that was your General Blog Commenting Policy.

  • Isabel says:

    "For fuck's sake, ask around -- what is it that your institution has the infrastructure and funds to do already in terms of outreach? Do you want to connect to K-12? There are NSF-sponsored activities that you can contribute to if you don't have the time to go teach middle schoolers-- you can have kids come visit a lab, host a teacher over the summer, contribute to middle school curricula. "

    What does any of this have to do with the specific research described in the proposal? It is going straight into a middle school curriculum???

    It sounds like "If you fund my research I will participate in your other activities."

    GMP, do you have a link for a good example? Where the *research proposed* is directly linked to outreach and K-12 curricula?

    Thanks.

  • proflikesubstance says:

    Isabel, a solid BI section does not have to directly involve the research. Usually people incorporate the research, but just as often the commit to doing tangential things that broaden participation.

    It sounds like "If you fund my research I will participate in your other activities."

    Not exactly. Done well, a PI leverages current activities and adds value through the new grant. For instance, if there is a initiative to develop teaching modules for high school students, the PI could propose to fund and participate in the development of one on their subject of interest. In that way, you use available infrastructure while using a modest budget contribution to enhance an on-going project.

  • Isabel says:

    "How well does the activity... How well does the proposed activity... To what extent will it ... Will the results be disseminated broadly to.... What may be the benefits of the proposed activity.....?
    "

    This is from NSF guidelines for the BI section. They are not asking about your general involvement as a scientist in outreach etc but that is what they really want to know, apparently. I can see that with two equal proposals you might look at a special NSF required section on the CV, perhaps, but a whole bogus part of the proposal? Are you really surprised that so many people are confused or want to blow this section off?

    interesting that they say activity instead of research. Is that significant? Maybe the proposed activity is 'doing my day to day science' and is not referring to the specific experiments proposed.

  • proflikesubstance says:

    You are missing the point. This is not your general interest in outreach, nor is it a bogus section. If you want to intertwine teaching and research directly, write a CAREER proposal. However, NSF is asking PIs to engage the public/under-represented minorities/K-12 students in science in a meaningful way, preferably stemming from the funded work. When I used the example of the high school module on a topic of interest above, I didn't mean some random topic the PI happens to be involved in. Can you translate work from the bench directly into teaching high school students? Maybe in some fields, but not all. However, can you use aspects of the work to help shape a module to engage high school students? Almost certainly. These are the types of things that work well and that probably would not be done in the absence of funding for the project.

    It is a balance to put together a meaningful BI section and it takes some thought. We'll see how the new BI definition changes things.

  • Alex says:

    A few things:
    1) PLS and GMP are both well-informed and what they say largely comports with what other well-informed people have observed. However, at the same time, when you look at what's funded and talk to different people, you still come away with a, um, range of views on what is a good BI. NSF needs to do better at communicating the expectations.

    2) There's a fine line between "Intelligent leveraging of existing NSF-funded infrastructure" and "Being so lazy that they'll just plop their stuff into somebody else's stuff rather than do anything new." That line is very much in the eye of the beholder.

    3) There's a spectrum of BI activities that range from very direct connections to the research (e.g. you will have minority students in your lab working on that specific project, or you will be lecturing at a summer school on the specific topic of your work for an audience of grad students and postdocs from under-represented groups) to "good citizenship" that any scientist could potentially do (e.g. go to middle schools and tell them about atoms, because they're nowhere near ready to hear any specifics of what you actually do with atoms). I applaud the good citizens, but (1) it feels somewhat forced to say "I'm going to do [high level stuff] and as a part of that I will help 3rd graders play with magnets and wires and light bulbs" and (2) I'm not convinced that it's an efficient use of resources to have every PI out there doing outreach. There are people who eat, sleep, and breathe that stuff. They can do it better than most of us.

    Yeah, yeah, we have to do it, panelists have to take the criterion seriously. None of this is saying that panelists shouldn't take the criterion seriously and PIs shouldn't write for the criterion. But we should not pretend that this is a sensible state of affairs.

    4) The line between "good citizen" and "doing your job" is also a fine one. I teach 3 courses each term at a university with a lot of low-income and minority students. For me, spending 12 hours/week helping 100+ under-represented students in STEM is called "doing my job." If somebody in a different setting says that they'll spend 1 hour/week doing demos at an inner city middle school, they're doing something above and beyond and they are considered good citizens who get to write a nice Broader Impact statement. Their demos and my freshman classes probably have nothing to do with the research that either of us is proposing, but one of us gets to put it in a Broader Impact and one of us doesn't. Yeah, yeah, I'm whining right now, but still.

  • Isabel says:

    "say "I'm going to do [high level stuff] and as a part of that I will help 3rd graders play with magnets and wires and light bulbs" and (2) I'm not convinced that it's an efficient use of resources to have every PI out there doing outreach. There are people who eat, sleep, and breathe that stuff. They can do it better than most of us."

    I wonder if this pressure on scientists to do science outreach is because NSF is putting less resources directly into K-12 science education? About 10 years ago I helped build a small public observatory and volunteered public observing nights for a while. While there I assisted a retired high school chemistry professor, who also loved physics and astronomy, with a free teach-the-teachers program. It was very simple but very popular with local teachers as those who teach K-8 often have little science background or have a background in a different science field. He told me about all these great NSF funded programs in the 1960's and 70's (I think) for teachers that are now completely non-existent. He was trying to make up for that lack, at least a little.

  • proflikesubstance says:

    Alex, I have seen proposals from people who work at institutions with high minority STEM enrollment do VERY well with a BI section focused entirely on getting current undergraduate students involved in the proposed research. Even if it is "doing your job" in this case, getting underrepresented groups directly involved in research is the Trump Card when it comes to BI sections.

  • proflikesubstance says:

    And FWIW, I have also seen BI sections get hit with the "too ambitious" label. I don't think NSF or the panels want people killing themselves in the BI section and solving all the world's problems.

  • Alex says:

    So what do they write? Is it enough to say "I have supervised X students from group Y (citations to papers published with students from that group, link to web page with photo of a multicultural research group) and the pool from which I will continue to recruit students is such-and-such percent group Y"? What else would you want to see?

  • proflikesubstance says:

    Wait, you have a track record with pubs and you are sweating this? If I were in that situation I would budget summer money for 2-3 students/yr and say they would be reserved for minority students (after demonstrating track record). Then I would pick 2-3 local high schools (especially if some of your students are local and would be willing to serve as role models to their high schools) and budget transportation and expenses to do some demo in the appropriate science class (with letter of support from superintendent, teachers or both) once a year as a science recruitment pitch for your institution. Not only does that tie in minority recruitment and high schools, but has a budget and support from local teachers, all wrapped up with the bow of having local students go back to their high schools and get others interested. Plus, you get money for summer researchers and the effort to you is minimal in terms of "outreach".

    Nice and tidy BI sections that check a lot of boxes are fucking gold.

  • Alex says:

    That sounds like a good plan. Thanks. The one thing I would tweak is that I have stronger contacts at community colleges than at high schools, and the CC's are minority serving institutions that some of my minority student coauthors studied at before transferring to my university. It would be less hassle for me to set up events at one of those schools, and maybe reserve a summer slot in my lab for a student who is either still at that school or is in the summer between finishing there and starting at my place.

  • proflikesubstance says:

    Whatever works for you, but I can't imagine negative feedback on that type of thing. Just because you have a track record of success with something doesn't mean you have to go out and change what you do for a new proposal.

  • the_commentariette says:

    My sense is that focusing strictly on the highest quality scientific research is somehow seen as too elitist

    It's basically a political correctness tax, so you just pay the least that you can. Try to leverage university or department activities, try to pay money instead of time and attention as much as possible, and maximize the value of your least productive students...

    If programs for K-12 education, science outreach and activities for females, under-represented minorities, etc are important (and they are), then they should be taken seriously. That means they should be proposed, funded, and evaluated on their merits, by people who are knowledgeable in these areas.

    It would be much more efficient to take x% of all research funding and spend it on well constructed programs than it is to try to make all PI's spend x% of their efforts on something that's peripheral to their actual competency. That's why I think it's mostly "anti-elitism-theater".

    If you enjoy visiting school science classes and have time for it, that's great. If you volunteer at your local food pantry, that's great, too! But these activities have nothing to do with the quality of the research you're proposing as PI -- it doesn't even have anything to do with whether the research results will eventually be beneficial to humanity in some way.

  • proflikesubstance says:

    And here I was all disappointed when I thought Ecologist was going to be the only NSF-disgruntler to weigh in.

  • DrugMonkey says:

    Whenever someone deploys "political correctness" unironically you know you are dealing with a selfish asshole. True in right wing politics, true on academia.

  • Isabel says:

    Hey, I'm disgruntled! Grad students can be disgruntled. And "science outreach" is my middle name!

    Yes, my BI section was criticized (for stupid reasons) and my grant rejected. Having this section which is so difficult for many of us to pull off smoothly provides convenient reasons for rejecting grants in a time of limited funds, we know that much.

    I agree 100% with commentariette's comment, and find it perfectly reasonable. I think DM's weird response says more about him than it does about anybody else. I'm also slightly disturbed by reviewers (PLS, GMP) undisguised contempt for applicants who don't play this game well.

    According to the link provided upthread, BI has been controversial since its inception, and it is only going to get worse.

    Science education and curriculum design is best done by educators. I worked as an RA on an NSF-funded curriculum for early grades as a science consultant. I was the only scientist working with half a dozen educators including PhD students in education. I was lucky enough to observe the summer trial run. These people know what they're doing!

  • Isabel says:

    I just reread commentariette's comment and not sure I agree 100% about the "anti-elitist" part. I think it may be more because NSF is not fulfilling its other obligations besides funding basic science and is trying to dump them on scientists.

    NSF has already agreed to fund basic research, so individual scientists should not have to prove that their basic science is helping society or fulfilling other goals of NSF. They should only have to convince the reviewers that it is advancing science.

  • profguy says:

    PLS and GMP are giving basically accurate advice about how to play the game - maybe the rules are a little more flexible than they make them out to be, but they are giving reasonable descriptions of the criteria in practice.

    Perhaps what irks some commenters is that the tone in which PLS and GMP give the advice at times makes it sound like they endorse the current BI system. It's a fine line between learning how to play the game and starting to believe in it a bit too much. I am quite in favor of outreach and all that (at least for senior scientists - I think it's crazy that BI is a required element in NSF graduate student fellowship proposals). But like other commenters, I think it's pretty clear that while NSF's implementation of BI is well intentioned, it's a blunt and largely ineffective instrument.

    It reminds me of the first time I was on a review panel for a mission-oriented federal agency (not NSF). This agency supports science, but it has to fit the mission. How well a proposal has to fit varies in practice from token to serious. I was shocked at the degree to which the reviewers (university people, not employees of the agency) would say "well, the science in the proposal is good, but it's not mission-oriented enough". I thought that was the PM's job to decide and the university people should just advocate for the best science. It seemed to me that the university people had way too much internalized the extraneous rules that they themselves had been forced into after getting proposals rejected (or just fearing that they would). I'd like to think I'll still see it that way the next time I'm on a panel for that agency or another like it...

    I don't blame anyone for playing the game, but it's good to recognize that one is playing it and not confuse how it is done for how it should be done.

  • Dan says:

    Do you have any thoughts on BI for NSF Graduate Research Fellowships? I'm at a loss for what to advise my undergraduate students, who don't even know where they are going to go to graduate school when they write the proposal. Have you or other commentors been on a panel reviewing these?

  • BrianM says:

    It seems to me that it would be much more efficient if NSF reduced funding for basic science resesarch by x % and use this to fund Science outreach programs written by people who are great at it and love to do it. But, alas, the reality is far more annoying...

Leave a Reply