So you think you can couple systems? Reviewing our research proposals

Killian Farrell ·
17 May 2017
Science Communication | Applying Science |     8 comments

Killian Farrell

Last week’s class marked the final meeting for our course MEES620: Coupled Human and Natural Systems. Our class has gone through many different topics related to coupled systems, from resource assessment to cultural views of nature. This was reflected in the many topics and research approaches in the proposals reviewed on this last day.

In the previous two weeks, each students presented their initial thoughts for their research proposal on a coupled human and natural system (Presentation Week 1 Blog, Presentation Week 2 Blog). Since that class, students wrote and submitted their final written proposals, and in turn also reviewed their classmates' papers by providing ratings on the different sections of the proposal. During class time, the panel of 3 students presented their initial ratings and then discussed and negotiated (or didn’t) a final score for each proposal.

Students participated in 3-person review panels (two students seated on the left, one student over the interactive video network), while the rest of the class observed. One student gave an overview of each proposal, then each reviewer gave initial scores, and then students debated until a final set of scores was agreed upon. Photo Credit: Vanessa Vargas-Nguyen
Students participated in a 3-person review panel (two students seated on the left, one student over the Interactive Video Network), while the rest of the class observed. The lead reviewer gave an overview of the proposal, then each reviewer gave initial scores, and finally debated among themselves to come up with a final set of scores. Photo Credit: Vanessa Vargas-Nguyen

After a quick summary of each proposal was given by the lead reviewer, the panel then gave their initial scores on the 4 main sections of the paper: Description of the Coupled System, Research Approach, Intellectual Merit, and Broader Impacts (based on NSF GRFP guidelines). The scale for scores was: 1 – poor, 2 – fair, 3 – good, 4 – very good, and 5 – excellent. The review panel then debated until a final set of scores was agreed upon. This was occasionally very quick, but often involved some debate with neither side willing to give in to each other’s reasons for differing scores. In some cases, the panelists were unable to reach a consensus on a final score, and the facilitator had to step in to wrap up the discussion. In the end, some students raised their scores and others lowered, but overall the final scores tended to be the average score given by the three panelists.

Three students on a panel discuss scores for another student’s proposal. Photo Credit: Vanessa Vargas
Three students on a panel discuss scores for another student’s proposal. Photo Credit: Vanessa Vargas-Nguyen

Looking at all the scores students gave each other on each section of the proposals, the section with the lowest average score was ‘Research Approach’. This is perhaps unsurprising, since methods weren’t the main focus of this course, and students' relative inexperience in this section led to occasionally vague or incomplete descriptions. However, through this class we were able to explore new topics in coupled human and natural systems, guiding us to possibly take other specific courses such as social network analysis or cultural modeling [pdf] to become better versed in the difference methodologies that can be used. Though this section may have been a little lacking today, in the future, with more practice and classes, the students will be able to improve.

Another common comment was that the reviewers would like to see more information or details to be provided – whether more description of one side of the coupled systems, more details in the research approach, or a better explanation of expected results and benefits to society. Our proposals originally had a limit of 2 pages, which was then extended to 3. All students passed the original page limit, often writing 3 or 4 pages. This highlights the difficulties of wholly describing coupled human and natural systems, along with the research approach, in a proposal with a short page limit. This experience gave us some good practice in identifying and briefly describing what was necessary. During the panel, the reviewers gave suggestions on which sections need more emphasis or trimming down.

Examples of proposal titles. Class proposals covered a wide range of complex topics that could be difficult to thoroughly describe within a limited amount of space.
Examples of proposal titles. Class proposals covered a wide range of complex topics that could be difficult to thoroughly describe within a limited amount of space.

The section of the proposal with the highest average scores for our class was the ‘Broader Impacts’ section. This was the section where you described how the proposed work could benefit society, and could be applied to help reach societal goals. The students presented their case strongly in almost all proposals for their coupled human and natural systems. This also demonstrates how the topics of the Coupled Human and Natural Systems course is so important. The coupled human and natural systems thinking could be applied to the research of pretty much everyone in the class, and understanding these systems together will be critical to help solve many important global issues, such as pollution and climate change.

An important measure of the success of each proposal was how applicable the research is to societal problems around the world.
An important measure of the success of each proposal was how applicable the research is to societal problems around the world. Image Source: aaas.org

Overall our proposal review panels were a success, with students giving fair evaluation of each other's proposals, and everyone honing their skills on writing and working with topics related to coupled human and natural systems. The good work and good discussion throughout the class showed the success of the course in teaching about such broad and complex topics.

And one last note, a final measure of the success of this class and blog: Can you use the word epistemology in a sentence? (hint)

Can you use the word epistemology in a sentence? Image source: Massimo (howtobeastoic.wordpress.com)
Can you use the word epistemology in a sentence? Image source: Massimo (howtobeastoic.wordpress.com)

Some further reading on topics from this course:


Next Post > Tell a compelling story: Talking about science communication to landscape ecologists in Baltimore

Comments

  • Suzi Spitzer 7 years ago

    Nice blog! The panel activity was super awkward and uncomfortable but it was a good experience. I really enjoyed this semester as a whole and this was a nice, academically rigorous closing. Thanks everyone for honest feedback :)

  • W. Cruz 7 years ago

    I think the review panel it was a good experience to go through. Although, I feel that part of being a scientist is to know how to critique people and do it in a respective manner as we expect from our peers and professors. In classes I took previously this activity was done anonymous similar to a real scenario which reduce the conflict between students but taking critics it is also part of being a scientist.

  • Rachel E 7 years ago

    Great blog Killian! I feel like this caught me up on what was happing on the other side of the phone last week when I could only really hear mumbling. The methods definitely seemed like the hardest part to me and in the papers I read. Finding a couples system in general and explaining how it was coupled was quite tricky as well. I like the title. It definitely portrayed this. Thanks for a great semester everyone!

  • Krystal Yhap 7 years ago

    Nice blog Killian. I agree with Wilmelie, this ended up being a really good exercise. Since this is my first semester in grad school I hadn't really had any experience with review panels or what it is like to be apart of one. I know in others classes students have told me that these panels are usually conducted in an anonymous way like Wilmelie described and I could definitely see how that is beneficial. But, even with the added pressure of conducting the review panel in an open way, I think this experience helped us to understand that different dynamics and situations that can take place on a review panel. This exercise also helped us to learn how to digest criticism of our own work as well as how to effectively and constructively criticize others work, which is not always easy. I actually expected that there would be more agreement and consensus on scores than there were. It was definitely interesting to see and participate in debates over scores.

  • Natalie Yee 7 years ago

    I love the last picture! I think the panel review was an interesting way to critique everyone's projects as they related to coupled human and natural systems. It allowed for a culminating project to tie up the concepts from the semester. The Broader Impacts section was one of the most important sections because it was based off the guidelines given by the NSF. This project will ideally give us more preparation should we decide to try for funding through this organization. The discussion and reasons why certain scores were given was extremely valuable because if funding organizations will not give feedback as such; it's either a yes or a no. With this project, we are better equipped to improve our the way we write and present our proposals.

  • Michael Paolisso 7 years ago

    I agree, Killian, great blog! Of course I loved the reference to epistemology at the end. It was a very good review, and you all did a great job reading, critiquing and presenting and defending your points.

    I was very impressed. I hope not to have any of you on review panels for my future proposals, unless you like cultural models and consensus.

    Michael

  • Noelle O. 7 years ago

    I think experiencing a panel review and learning how to provide constructive criticism is really important for scientists, as we navigate the grant proposal process. I do agree with Wilmelie that it might've been better to have some anonymity. For the MEES fish ecology class, we were split into two groups and had a professor facilitate each group through a similar process (one lead reviewer with two secondary reviewers). Group #1 only reviewed papers from Group #2 and comments on your own paper were submitted anonymously. It felt a bit strange having the review panel over the IVN system, but it was still a learning process nonetheless!

  • V Leitold 7 years ago

    I agree with Krystal that critiquing others’ work can be a challenging task, both from the standpoint of the reviewer and the person whose proposal is being reviewed. Regardless, I think that everyone did an excellent job preparing well-thought-out evaluations and making sure that the comments were indeed constructive and the criticism justified with sound arguments. To me, it was especially interesting to see how divergent three reviews could be in some cases, and I certainly learned a lot just by listening to the different perspectives that students took to evaluate the proposed projects. The facilitation by the ‘panel chairs’ was also done in a professional manner and helped resolve conflicts among reviewers when that became necessary.

Post a comment