by Jose Cavazos, MD, PhD

Study sections are under a different branch of the NIH, the Center for Scientific Review (CSR).  The purpose of NIH study section is to evaluate scientific merit.   It is a time consuming activity but I have learnt a lot about epilepsy.  We, scientists, are in the business of selling ideas.  A good grant application is a sales pitch that makes a compelling case for a particular idea investigated by a specific group of scientists at a precise location.  The applications are formally graded by the scores on each of the following criteria significance, approach, innovation, investigators and environment, and then, receiving an overall score.   Study sections that review training applications score each application on the following criteria: candidate, mentor, training plan, approach, training potential and environment.   A bad sscore on any category can tank an otherwise good application.  Not describing well the environment or training plan could be interpreted as lack of attention to detail, which might cast doubt about feasibility of the application.  

Let’s talk about the process.  Reviewers typically agree to four year commitments to attend three study sections per year at hotel near Washington, DC or Bethesda, MD.   We used to receive a box with a few reams of paper, but now, it is only a few sheets of paper and a CD that are received about 6 weeks prior to the meeting.  Each reviewer is assigned 5 – 12 applications.  Typically, the applications are assigned a primary reviewer within an area of expertise, but tertiary reviewers for other neurological applications that might have some relationship to epilepsy (i.e.: similar techniques, pharmacology, pathophysiological mechanisms, etc.) may also be assigned.  Reviewers submit their comments electronically blind to other the reviewers’ scores.   The meeting lasts for 1 or 2 days starting at 8 am with two 15 minute breaks and one hour lunch before ending close to 5 pm.  Depending upon the funding mechanism (i.e.: R01, F31, K08, R21), between 20 and 50 % of applications that are in the lower half of the scoring matrix are triaged, and not discussed in the meeting.  That means that only the three reviewers that were assigned to those applications are to provide a critique.  If any there is great disparity between the scores of reviewers, any member of the panel can request a discussion.  The applications are often reviewed alphabetically, sorted by an overall scores of each of the assigned reviewers using an ordinal scale from 1 to 9, with 1 been exceptionally strong with no weaknesses and 9 indicating numerous major weaknesses.   A major weakness is defined as a flaw that severely limits the impact of the proposed research.  

The primary reviewer then presents the strengths and weaknesses of the application with particular emphasis on why their overall score was a given number.  The secondary reviewer then presents additional points to strengthen or weaken the initial reviewers’ discussion, and the tertiary reviewer adds discussion points that have not been touched.  Lastly, the floor is open to questions from the other panel members for clarifications.  Every so often, the panel members’ questions might sway the entire score in a positive or negative manner.  Finally, after the chair of the study section summarizes briefly the discussion leading to the final scores from each reviewer.  The panel members allocate a score between the final scores the assigned reviewers (i.e.:  between 2 and 4 if the scores are primary – 2, secondary – 4, tertiary – 2), or inform the chair that they are voting out of range.  Reviewers are constantly reminded that “these weaknesses do not sound as a score of x”.   

Overall, the review is a fair and deliberate process.  It is clear when an applicant has not spent much time on an application considering each of the criteria.  In general, the approach is the most important aspect on the application.  It is critical to consider whether the experimental approach is feasible and innovative, but it is also as important as to whether the specific aim helps testing the hypothesis.   In addition, a thoughtful discussion of alternative approaches and pitfalls can overcome questionable feasibility.   In training applications, the training plan has equal value than the research plan.  A very individualized training plan for a promising applicant can speak strongly about the candidate’s potential training in a non-elite institution.   Each study section has a culture that provides different weights to each of the criteria.  Significance might be more important for a clinical scientist group, while innovation might be more important for another study section. 

Once the scores of all study section members are averaged, the Scientific Review Administrator (SRA) puts together the comments from all reviewers and the discussion (if any), and submits the recommendations to the Program Directors from the NIH institutes.  Then, each NIH institute has a council which decides the specific priorities to fund.   If the funding payline is hypothetically at 15%, applications better than 14% are going to get most or all the requested funding.  However, a borderline application of high scientific merit (top 14 – 20%) but within a high priority for the program might be chosen for funding by the Council.  However, the overwhelming majority of awarded NIH grants are based exclusively on their scientific merit.

The most important persons that can tell you about these details are the Scientific Review Administrator (SRA) for each study section and the Program Directors at NINDS.  

Many review panels from foundations that fund research grant follow similar procedures to evaluate the scientific merit of applications for funding.

Useful links:

NIH Reviewer orientation: http://grants.nih.gov/grants/peer/reviewer_guidelines.htm

NIH Review criteria at glance: http://grants.nih.gov/grants/peer/guidelines_general/Review_Criteria_at_a_glance.pdf

Center for Scientific Review:  http://cms.csr.nih.gov/AboutCSR/Welcome+to+CSR  (check the video on the bottom of the page).

Jose E. Cavazos, M.D., Ph.D.
Last Reviewed: 4/15/11

Article from the April 2011 Epilepsy.com Spotlight Newsletter. Other articles in this issue inclue:

• On My Mind: Temporal Lobe Epilepsy
• Antiepileptic Drugs Interact with Folate and Vitamin B12
• Detection of Seizure Movements Using a Wrist Device
• EEG Improvement after the Ketogenic Diet
• The Value of Auras
• Building the Momentum Against SUDEP