Gender Differences in R00 Institutions

(by datahound) Aug 14 2014

Following my post noting the occurrence of differences between men and women K99 awardees in their likelihood of receiving an R01 grant NIH, through Sally Rockey's blog, noted that application rates may play a role:

"Of the 2007 cohort of K99 PIs, 86 percent of the men had applied for R01s by 2013, but only 69 percent of the women had applied."

This point has been taken up over at DrugMonkey.

Although such differences in application rates between genders are common in NIH statistics, I was surprised that the rates were this different since the K99 cohort from a single year is, presumably, relatively uniform in terms of career stage, accomplishment (having successfully competed through the same program), and so on.

In considering factors that could contribute to this difference, I thought of the nature of the institutions at which these individuals get their R00 awards (if they do transition). As one (certainly imperfect) measure of institutional characteristics, I used the FY2013 institutional ranking of NIH funding. For the FY2007 cohort, of the 108 men with R00 awards, the median ranking for their R00 institution is 37 and the mean is 81. In contrast, for the 62 women, the median is 57 and the mean is 113. For the FY2008 K99 cohort, the median for men is 44 and the mean is 71. For women, the median is 45 and the mean is 103.

For the purposes of further analysis, I divided institutions into 5 groups (NIH funding ranking 1-25, 6-50, 51-75, 76-100, and >100. The distributions for men and women for the two cohorts are shown below:

2007-2008-Rank plot

The distributions are relatively similar for the institutions near the top of NIH funding rankings. However, there are differences in the remainder of the distribution, most strikingly for institutions with NIH funding ranking >100. For men, 20-21% of the R00 awardees were at such institutions whereas 31-36% of the women were. This reveals that a larger percentage of women over men with K99 awards are beginning their independent careers at institutions that are relatively less research intensive, by opportunity or choice.

How does this relate to the likelihood of receiving an R01 award? The results for the FY2007 cohort are shown below:

2007 Funding Groups New Plot-2

For the investigators at institutions with rank 1-25, the percentages of investigators who have achieved R01 funding is comparable for men and women. However, this is not true for the other sets when a higher percentage of men than women have received R01 funding. For example, more than 20% of all women in this cohort are at institutions with NIH funding rankings >100 and have not received R01 funding compared with 7% of all men.

The corresponding plot for the FY2008 K99 cohort is shown below:

2008 Funding Group New Plot-2

Again, more than 20% of all women are at institutions with NIH funding rankings >100 and have not received R01 funding. In addition, for this cohort, the fraction of women at institutions with NIH funding rankings from 1-25 who have received R01 funding is substantially lower than that for men at the same set of institutions.

These data provide insights into some factors that may contribute to the disparities in R01 funding for women and men in the K99-R00 program. Of course, as one parses the program into smaller groups, the statistical power decreases. Nonetheless, these analyses should provide guidance to allow a better understanding of the role of different factors in NIH funding outcomes.

8 responses so far

A Pilot Study of Continued Funding after Holding a Single R21 Award

(by datahound) Aug 08 2014

In a recent post, I highlighted the growth in the number of applications and, to a lesser extent, awards for R21s. At the end of the post, I noted that many individuals who held R21 awards in FY2013 had no other R-mechanism funding and noted that one could track outcomes for these individuals over time going back to an earlier year.

As a first step, I have examined a sample of approximately 800 investigators who received an R21 award in FY2009 and held no other NIH awards (including both R and all other mechanisms). I then examined the funding for these investigators in FY2013. Of the sample of 801 investigators, 576 investigators (72%) had no funding in FY2013.

FY2009 was a year in which NIH received additional funds through ARRA. Of the R21 awards in the sample, 367 were supported by ARRA and 434 were not. Of the ARRA-supported investigators, 277 (75%) had no support in FY2013. Of the non-ARRA-supported investigators, 299 (69%) had no support in FY2013. Of the investigators who were funded, it appears that slightly more than half have R01 funding.

This study is a preliminary study with a sample from a single year, but it provides a general sense of the outcomes after having a single R21 awards. Not that this sample includes investigators at a variety of career stages.

9 responses so far

Non-R01 Individual Investigator Mechanisms: The Growth of R21 Applications

(by datahound) Aug 04 2014

In a previous post about R01s, I noted that the fraction of the NIH budget going to R01s decreased over the period from FY2003 to FY2013. This fact is of concern, of course, for a variety of reasons. But first, it is important to understand the observation as completely as we can. One set of factors that has contributed is the growth in the number of R21 awards, driven in large part by a huge increase in the number of R21 applications and the introduction of additional mechanisms.

For the R21 mechanism, the numbers of applications and awards NIH-wide for the period from FY2003 to FY2013 are shown below:

R21 Apps Awards plot

These data reveal that the number of applications increased more than 2.6-fold from 5283 in FY2003 to almost 14000 in FY2012. NIH responded to this "proposal pressure" by increasing the number of awards from 1255 in FY2003 to almost 2000 in FY2012. Nonetheless, the success rate for R21s has remained between 12.9 and 14.9 % over the past five years, generally 2 or more percentile points below the R01 success rate, even for new (as opposed to competing renewal) applications. Thus, the competition for R21 awards is more severe that it is for R01 awards, a fact which is not, in my experience, widely appreciated by the all in the community.

When I was Director of NIGMS, we decided to stop accepting unsolicited R21 applications. This decision was made for two reasons. First, we found the peer review process for R21s quite frustrating. NIGMS was trying to use the R21 mechanism to support "high risk-high potential reward" research, that is, new ideas for which a modest investment could provide a proof of principle that could be used to drive future inquiry. However, despite efforts by CSR and NIGMS staff to orient reviewers, we frequently received scores and summary statements that did not align e.g. 'This is a potentially important and impactful project, but there is no preliminary data' and a bad score or 'This is a solid proposal supported by much preliminary data' with a good score. Because of this, we often struggled to develop sensible paylists. NIH made this problem worse by using the R21 mechanism for many other purposes other than the "high risk-high potential reward" goal. This confused reviewers, applicants, and even NIH staff.

Second, we had misgivings about whether the duration of the R21 and the size of the award would, in general, support substantial research compared to taking the same funds and supporting a smaller number of R01-sized grants. This led to the EUREKA R01 awards, used by NIGMS and a few other institutes.

I do not know of any studies that bear of the success of R21 awards in promoting scientific discovery or in keeping investigators "in the game". One interesting observation is that, in FY2013, more than 2300 (62%) of the R21 awards were held by investigators who had no other R21 or R01 awards (competing or non-competing) in the same year. Looking back to earlier year, one could track subsequent results for such investigators if that would be of interest.

12 responses so far

K99-R00 Evaluation: IC distribution

(by datahound) Jul 22 2014

The K99-R00 program is an NIH-wide program but, as is typical at NIH, each institute and center has considerable flexibility about the details about how the program is administrated. For example, for the two cohorts of K99 awardees that I have been examining, the number of K99 awardees ranges from 1 for the National Institute (then National Center) for Minority Health and Health Disparities (NIMHD, MD) to 54 for the National Cancer Institute (NCI, CA). The number was not simply proportional to budget size. For example, the number of K99 awards from the National Institute of Allergy and Infectious Diseases (NIAID, AI) was 13, smaller than five other institutes despite the fact that NIAID has the second largest budget at NIH. Moreover, as one would expect by chance, the fraction of women and men among K99 awardees varies from IC to IC. This may be relevant to understanding the disparity between women and men in the probability of transitioning from an R00 award to an R01 or similar award.

These data are summarized graphically below:

IC distribution


The size of each circle is proportional to the number of R00 awardees from each IC.

These data may be relevant to understanding the gender disparity. For example, both NIAID and the National Institute for Neurological Diseases and Stroke (NINDS, NS) have percentages of men among R00 awardees that are slightly higher than the NIH average but all of the women from these institutes who have received R00 awards have gone on to obtain R01 or equivalent (DP2) funding through the present.

Understanding the origins of the gender disparity between women and men going from a K99 award to R01 or equivalent funding is important for determining what policy adjustments should be considered.

10 responses so far

K99-R00 Evaluation: A Striking Gender Disparity

(by datahound) Jul 21 2014

In a recent post, I examined the NIH K99-R00 (Pathway to Independence) program. Looking at two cohorts of individuals who received K99 awards in FY2007 and FY2008, this analysis revealed that 91% of these 360 awardees transitioned from the K99 phase to the R00 phase, indicating that they had found suitable academic positions. Further, of those with such positions, 51% had obtained R01 funding through the present and some others had received other types of grants.

Over the process of organizing these data, I noticed that the fraction of women receiving K99 awards appeared to be lower than that for men. In order to examine this point more systematically, I assigned gender to these 360 awardees based on name and web searches. Overall, 142 of the 360 (or 39%) of the K99 awardees in these two cohorts are women. This is consistent with the value reported by NIH for this program.

Of the 218 men with K99 awards, 201 (or 92%) went on to activate the R00 portion. Of the 142 women, 127 (or 89%) went on to these R00 phase. These differences in these percentages are not statistically different.

Of the 201 men with R00 awards, 114 (57%) have gone on to receive at least 1 R01 award to date. In contrast, of the 127 women with R00 awards, only 53 (42%) have received an R01 award. This difference is jarring and is statistically significant (P value=0.009).

These results are summarized graphically below:

Men-Women-K99-R00-R01 percent

To investigate this further, I looked at the two cohorts separately. For the FY2007 cohort, 70 of the 108 men (65%) with R00 awards have received R01 grants whereas only 31 of the 62 women (50%) have (P value = 0.07). For the FY2008 cohort, 44 of the 93 men (47%) with R00 awards have received R01s whereas only 22 of the 65 women (34%) have (P value = 0.10). The lack of statistical significance is due to the smaller sample sizes for the cohorts separately rather than any difference in the trends for the separate cohorts, which are quite similar.

What could account for these disparities? Of course, it is unknown at this point if the differences reflect differences in R01 success rate as this would depend on the number of applications submitted by each applicant. I do not have access to data regarding the numbers of applications submitted by these grantees (although NIH does and could do this analysis).

One possibility is that the disparities is due to differences in the timing of funding for men versus women, that is, women might become R01 funded at more similar rates, but a greater disparity is observed at this point because of the lack of time for follow-up after the K99 award. To examine this point, I looked at the percentage of first R01 grants awarded each year to men and women as shown below:


The curves for men and women for each cohort are quite similar. This does not support the hypothesis that the timing of R01 funding is primarily responsible for the observed gender disparity.

Another factor that I considered is the impact of other R01-like awards, primarily the NIH Director's New Innovator (DP2) awards, on the analysis. Overall, nine K99-R00 awardees have received DP2 awards. Of these, six are men and three are women. Interestingly, three of these six men have subsequently received additional R01 awards whereas none of the three women have. In addition, one male K99 awardee received a Pioneer (DP1) award. The inclusion of these additional grants does not significantly affect the overall conclusion about the occurrence of a gender disparity.

Confirmation of and further analysis of these disparities is certainly warranted. Indeed, the K99-R00 program may be highly appropriate for such analyses as investigators at similar career stages competed for initial support and then obtained relatively comparable resources to launch their independent careers.

22 responses so far

My first pass evaluation of the K99-R00 program

(by datahound) Jul 17 2014

In 2006, NIH initiated a new grant program The Pathway to Independence award program. This transition award was a direct response to a National Academy of Sciences report "Bridges to Independence" that highlighted the long time between degree completion and the initiation of an independent research career. This program using a combination of the K99 and R00 mechanisms (and are often referred to a "Kangaroo" awards).

The first awards were made in FY2007. In 2011, Drugmonkey posted a qualitative evaluation of the K99-R00 program and noted that some K99-R00 recipients had gone on to receive R01 funding. However, it was too early to perform a more quantitative analysis. However, now sufficient time has passed that quantitative data are more interpretable.

I analyzed the grant trajectories of the 182 K99 recipients awarded in FY2007.

Here are the key results:

170 (or 93%) transitioned to the R00 phase, indicating that they had obtained an appropriate faculty position.

Of these 170 individuals, 101 (or 59%) have obtained at least 1 R01 grant through the present. In addition, 4 had obtained New Innovators (DP2s), 1 had obtained a Pioneer (DP1) and 1 had moved to the NIH intramural program.

The timing for obtaining R01 grants is shown below:

2007 plot

The plot shows that R00 awardees received R01 support as early as FY2009. The number of R00 awardees receiving their first R01 peaked in FY2012 and has been falling. The line to FY2014 is dotted as this fiscal year is not yet complete. The cumulative percentages of R00 awardees with R01 support are as follows:

FY2009-4%; FY2010-14%; FY2011-27%; FY2012-43%, FY2013-53%; FY2014-59%


For comparison, I also analyzed the 178 K99 awardees from FY2008.

158 (or 89%) transitioned to the R00 phase.

Of these 158 individuals, 66 (or 42%) have obtained a least 1 R01 grant through the present. In addition, 6 had obtained New Innovators (DP2s) and 1 had moved to the NIH intramural program.

The timing for obtaining R01 grants is shown below:

2007 2008 plot

The FY2008 cohort is obtaining R01 grants with a similar but slightly longer time delay compared to the FY2007 cohort, due at least in part to the impact of the ARRA funds on the FY2007 cohort. For the FY2008 cohort, the determination of the overall fraction obtaining R01 funding and the final trend will depend on the results through the end of FY2014 and, perhaps, beyond.

The cumulative percentages are:

FY2009-0.6%; FY2010-1.2%; FY2011-7%; FY2012-16%, FY2013-37%; FY2014-42%

From these data, there can be no doubt that the the K99-R00 program has been effective in transitioning individuals from postdocs to faculty positions. It is clear that individuals with these awards compete well for such positions through some combination of the skills and qualities that allowed them to successfully compete for the K99 award and the interests of the hiring institutions in having faculty with some extant funding and a track record of obtaining such funding. The role of this program in influencing hiring has been discussed in another Drugmonkey post.

The results in obtaining R01s is also quite impressive to me. Of course, it is not clear what percentage of these individuals would have obtained positions without the "kangaroo hop". One comparison could be to look at F32 recipients who went on to faculty positions during the similar period, but this will take some work. Again, the factors that have led to this success include scientific skills, grant writing skills, relatively good contacts with NIH staff, funds for preliminary data, and others.

It is less clear how much impact this program has had on the timing of the transition to independence if this is defined by obtaining independent R01 funding. There is still a substantial lag between obtaining a faculty position and obtaining R01 funding, comparable to what Drugmonkey posted with my data from NIGMS (although these results were from times where success rates were somewhat higher). Nonetheless, overall it is hard not to see the K99-R0o program as a substantial success.


18 responses so far

R01 Success Rates: New and Renewal Applications

(by datahound) Jul 08 2014

In a recent post, I looked at the distributions of A0, A1, and A2 awards for new (Type 1) versus renewal (Type 2) R01 grants. I noted differences between trends for new and renewal awards over time and commenters Ryan and DrugMonkey noted possible reasons for these differences. As DrugMonkey said "Remember the structural issues here with Type 2s. There is a fixed number of possible Type 2s and an unlimited number of Type 1s that can come in for consideration. As each Type 2 flails out at the A1 (or previously A2) stage, it is gone forever from the pool of potential Type 2s."

NIH does release data regarding the number of applications for new versus renewal R01s (and many other mechanisms. The data for new R01 applications and awards for FY2003 to FY2013 are shown below:


These data show that the number of applications for new (Type 1) R01s grew with an initial burst immediately after the end of the doubling and then another stimulated by the American Recovery and Reinvestment Act (ARRA) implementation. The number of applications in FY2013 was 24% larger than that for FY2003. The number of awards declined relatively steadily over this period of time with a 26% drop from FY2003 to FY2013.

The situation is very different for renewal (Type 2) applications as shown below:


The number of applications for renewal (Type 2) applications grew from FY2003 to FY2006 and then declined relatively steadily through FY2013. There has been a 32% drop in the number of applications for Type 2 applications from the peak to FY2013. The number of renewal awards fell by 47% from FY2003 to FY2013.

The success rates for new and renewal R01 applications are compared below:

R01 Success Rates-2

The success rate curves parallel one another relatively closely. The success rate for renewal applications was 48.3% in FY2003 and it fell to 31.4% in FY2013, a drop of 35%. The success rate for new applications was 24.1% in FY2003 and fell to 14.3%, a drop of 40%. As DrugMonkey noted, the pool of projects eligible to be the subject of renewal applications limited.

9 responses so far

NIH Mechanism Tables

(by datahound) Jun 30 2014

In a comment to my recent post, Dave asked how the fraction of the NIH appropriation going to different areas of NIH activity could be easily quantified. The answer to this question is "yes and no." The key source about the NIH budget is called a "mechanism table". These are developed each year and are available from the NIH Budget Office (and from at least some IC websites). Below are NIH-wide mechanism tables for FY2003 and FY2013:

Mechanism Table 2003-2013

The mechanism table has the budget broken down into major categories including Research Grants, Research Centers, Other Research, Training Awards, Intramural Research, Research Management and Support (administrative costs for running NIH) and a number of other smaller categories. The distribution among these different categories for NIH as a whole for FY2013 is shown below:

Pie Chart-2013


The largest categories for FY2013 are Research Project Grants (51.0%), Intramural Research (11.1%), Contracts (10.0%), and Research Centers (9.3%).

These percentages were not vastly different in FY2003: Research Project Grants (51.4%), Intramural Research (9.6%), Contracts (8.6%), and Research Centers (9.1%). Thus, the fraction of the budget going to Research Project Grants dropped from 51.4 to 51.0% over this decade while Intramural Research grew from 9.6% to 11.1%. These data must be examined carefully before interpretation. For example, the growth in Intramural Research is due, in part, to an accounting change wherein the entire National Library of Medicine budget moved from a separate item to Intramural Research. The Intramural Research plus National Library of Medicine budget was 10.7% of the overall budget in FY2003.

Examination of the NIH-wide mechanism tables over time provides look at changes in NIH investment strategy in very broad strokes over time. In further posts, I will examine these changes, differences in mechanism tables between Institutes and Centers, and more breakdowns of the Research Project Grant pool.

2 responses so far

A0, A1, and A2 R01 Awards from 2003-2013

(by datahound) Jun 25 2014

Few science policy topics have led to as much discussion as the NIH policy with regard to the number of amendments allowable for grant applications. This policy originated from the NIH Enhancing Peer Review draft report. Figure 8 in this report shows that the fraction of R01s that were funded in their original submission (A0) had dropped from more than 60% during the doubling years to less than 30% in 2007 with a concomitant increase in the fraction of A1 and, particularly, A2 applications that were funded. These data, some other analyses, and anecdotes from study sections, suggested that study sections were "queuing" applications, giving better scores to A2 applications since these was these application's "last chances."

The Enhancing Peer Review report proposed "consider(ing) all applications as new" so that there was no reason for study sections to treat applications differently based on their amendment status. Organizations including FASEB did not support this proposal. In order to help decrease the time between the initial submission and the time of funding for applications that would eventually be funded, the NIH countered by eliminating the option of A2 submissions.

From my perspective, this proposal would have been sensible in times of reasonable paylines. However, since this proposal was adopted, paylines have continued to fall (except during the ARRA years) to the point that the variability in scores means that two chances may not be adequate, even for outstanding proposals. I wrote a column about this for ASBMB today entitled "On deck chairs and lifeboats".

After resisting pressures from many in the scientific community for years, the NIH recently reversed course and now will allow an application that did not get funded in A1 form to be resubmitted as a new A0 application without restriction. This policy is a hybrid of the old policy and the initial proposal from the Enhancing Peer Review report. To my knowledge, NIH has not provided much guidance into the analysis that led to this reversal other than statements of concern about the impact of the "No A2" policy on early stage investigators.

As I thought about this policy, I realized that I had never seen data about the differences between new (Type 1) and competing renewal (Type 2) applications. Type 2 applications, in general, score better that Type 1 applications due, in part, to the fact that only applicants that have been relatively successful submit renewal applications. The differences can been seen in an NIGMS Feedback Loop post that revealed that (for the January 2010 Council round in FY2010) 50% of the Type 2 applications scored better than the 20th percentile compared with less than 30% of the Type 1 applications, even those from established investigators.

Below are plots of the number of NIH-wide R01 awards for Type 1 and Type 2 awards as a function of amendment status for FY2003-FY2013:

Type1-2 plot-2

The curves showing the mix of A0, A1, and >A1 (almost all A2) applications for Type 1 and Type 2 awards are remarkably similar, with comparable rises in the fraction of A2 awards from FY2003 to FY2009, followed by a drop to essentially zero due to the "no A2" policy. The results for Type 1 and Type 2 awards are compared directly in the plot below which shows the fraction of A0 awards out of total awards for each type.

A0-Total plot

In FY2003, the fraction of A0 awards among Type 1 grants was slightly larger than 0.5 whereas that for Type 2 was higher at approximately 0.6. As these fractions dropped, rose, and then dropped again, these two fractions are more equal near 0.45 in FY2013.

Closer examination of the first graph reveals another important point.  In FY2003, there were 4564 Type 1 awards and 2618 Type 2 awards. By FY2013, these had dropped to 3403 Type 1 awards and 1390 Type 2 awards. Note that these are new and competing awards, not all R01 awards (which would include non-competing (Type 5) awards). The drop in the number of Type 1 awards is 25% whereas the drop in the Type 2 awards is nearly twice as large (47%). One can speculate that this drop in the number of competing renewal awards since the ARRA bump may be a driving force in concerns expressed in recent years by established investigators about the direness of the funding situation.

What accounts for these decreases in the number of awards? Of course, the NIH appropriation has not grown substantially since FY2003 and has dropped substantially when corrected for inflation. However, the average R01 sizes (total costs in nominal dollars) increased by 24% for Type 1 awards and only 16% for Type 2 awards (although the distributions should be examined to understand these increases more fully, as discussed in an earlier post). The drop in award numbers reflects that fact that the total expenditures on R01s was $9.76 B in FY2003 and was $9.80 B in FY2013 whereas the overall NIH appropriation was $26.74 B in FY2003 and $29.13 B in FY2013, that is, the fraction of the NIH budget going to R01s has decreased over this period.

15 responses so far

Public Health Data-Project Tycho

(by datahound) Jun 19 2014

It is remarkable to me that the US is dealing with localized outbreaks of infectious diseases such as measles and pertussis (whooping cough) given the availability and effectiveness of vaccines. Data that allow assessment of the impact of vaccines have recently become available through a project at the University of Pittsburgh Graduate School of Public Health-Project Tycho. The Project Tycho team obtained and digitized "all weekly surveillance reports of nationally notifiable diseases for U.S. cities and states published between 1888 and 2011." This data set, which includes almost 88 million cases, is freely available to researchers and the public and the first set of results have been published "Contagious Diseases in the United States from 1888 to the Present." A subscription is required for the New England Journal of Medicine article but the information and more is available at the Project Tycho website. Some of the data entry was accomplished through a partnership with Digital Divide Data.

Below are figures available from the Project Tycho website showing trends for measles and pertussis.


Project Tycho is a great resource for research, teaching and advocacy and represents an innovative example of converting information that exists in principle into data that can be analyzed and extended.

2 responses so far

Older posts »