Instructor Notes
Considerations about this course material
This training was written with a few things in mind:
There is little RR training for those that are not doing computational based research. This course attempts to offer skills that are not coding-focused, so that a wider range of people can use them. While we do offer opportunities to introduce the idea of coding, we wanted to make this course inclusive to all disciplines and all levels of technical literacy.
People are in different stages and maturities on any topic. To make them feel welcome and comfortable, we highlight that this training won’t make them an expert, and we don’t expect them to have background knowledge. This training is moreso about helping them further down the path. We wrote the courses so that for each area, everyone should be able to walk away with an action item to improve their processes.
There are some RR improvements that need to happen in a bigger scale, such as the ‘publish or perish’ affect on reproducibility. We can’t solve the world’s problems today, but we can help the people in front of us, and that matters.
Career stage and how that will affect your talk
As per the culture page in your instructor menu, researchers from different career stages will have different considerations.
Generally, career level of attendees are skewed towards research students and early career researchers. However, depending on your department, you may find more senior staff in your cohort.
You may also consider how your attendees came to be here- did they sign up themselves? Were they encouraged to sign up by a supervisor or senior leader? You will find people who have signed up may have more background knowledge then those who were encouraged to go.
Different terminology and disciplines may respond differently
There’s a discussion to be had here on different terminology. Where a biologist may talk about a protocol, a humanities researcher may talk more of analysis, and a data scientist may talk about their code or pipeline.
The UKRN hosted a webinar on “Reproducibility, Transparency, Positionality? Perspectives From Different Research Fields” that may be a helpful resource. The recording and slides are available.
As a teacher
You don’t need to be an expert
People often mistakenly think that to teach a subject, you should be an expert.
As per orchid00’s article:
Expert Blind Spot
Experts are frequently so familiar with their subject that they can no longer imagine what it’s like to not see the world that way. This is called expert blind spot and can lead to what’s known as the expertise-reversal effect - experts are often less good at teaching a subject to novices than people with less expertise who still remember what it’s like to have to learn the things. This effect can be overcome with training, but it’s part of the reason world-famous researchers are often poor lecturers. The challenge of identifying and working around expert blind spots is one reason why we welcome instructors who still identify as “novices”! Someone who is still in the process of learning can be a more effective instructor because they are speaking from their own recent experience. In these ways and others, the high connectivity of an expert’s mental model poses challenges while teaching novices. However, that’s not to say that experts can’t be good teachers. Experts can be effective as long as they take the time to identify and correct for their own expert blind spots.
orchid00 (last updated Sept 2019) A Brief Introduction to the Carpentries Pedagogic Model. Retrieved on 2024-05-03 from https://orchid00.github.io/The_Carpentries_info/overview-of-carpentries-pedagogic-model.html licenced as CC-BY
People remember stories, not statistics
A study by Stanford professor Chip Heath found that during the recall of speeches, 63 per cent of people remember stories and how they made them feel, but only 5 per cent remember a single statistic.
Stories help people to relate to your lessons, to understand how and why it affects them.
Get a feel for your cohort
It can be helpful to ask your attendees a bit about themselves.
Finding out what area of research they are in can help you highlight resources that are more relatable. (Qualitative vs quantitative, computational vs not.)
Finding out their career level can help guage what they are familiar with already, and what their underlying concerns, risks and opportunities are.
Not sure on a question?
If a student asks you a question you don’t have an answer for, it is completely okay to say “I’m going to do some research into that, and get back to you.”
Contributing to these lessons
We aim to continuously improve these lessons and would love your feedback. To provide feedback or suggest changes, please contact Adam Partridge on a.partridge at sheffield.ac.uk or via a git issue submission here https://github.com/amandamiotto/ReproducibleResearch/issues .
Some other helpful resources are - teaching tips by the carpentries - Data storytelling and getting the impact across to non-experts
What is Reproducible Research?
Why does Reproducibility matter?
Instructor Note
In my own experience (as @amandamiotto ) for a project where I had to contact researchers regarding moving their old data, a good third had left and either moved overseas and changed 2-3 institutes, were no longer in research or had names that were ‘Australian-ised’ or had changed for various reasons, meaning it was hard (and at times impossible) to track and find them.
Lack of open data - this can be really specific to a field. Some health fields can’t make their data open, and this statistic doesn’t talk about what data could be FAIR or not published for commercial reasons.
Instructor Note
This is a good time to stop and ask people about their own knowledge and experiences. It gives you an opportunity to understand how much background your participants already have - if people have signed themselves for the class, they may have already been quite knowledgeable.
Questions you could ask are:
Do you know of other examples of reproduction studies?
Who here has tried to reproduce their own work?
Who here has tried to reproduce someone else’s work?
What can these statistics tell us about these studies?
Instructor Note
The Nature paper has some excellent graphics that you may like reference in your internal training, however due to copyright, we have decided to not use it here.
Introducing 7 Steps towards Reproducible Research
Step 1 - Planning to be organisedWhat is your next step?
Instructor Note
Does your institute have a Data Management policy, procedure or page that can be linked for your attendees?
This is a great place to put it!
Link to your institutional Data Management Planning Page
Instructor Note
This is a good time to talk about what research data storage options you have at your institute. Discuss backing up your data.
You may get a question on recovery verses backup - have a chat to your storage team who can assist you with understanding the differences.
Step 2 - Keeping your files Tidy and OrganisedHow to structure Column Data - Tidy data formatOther useful concepts:How to handle missing data
Instructor Note
Point out the resources segment here at the bottom of the page. There is a really good tutorial in the Data Carpentry for Ecology page that covers a range of spreadsheet things
This is also an opportunity to do a live exercise - you could pull up a bad dataset and ask participants to point out potential improvements.
Have a spreadsheet that has some messy data in it to clean. You can ask everyone to write down two changes they would make, then have an interactive session where you ask people to suggest their changes.
Step 3 - Methodology and ProtocolsWhat is your next step?
Instructor Note
This really did come up time and time again when talking to researchers on how they established trust in a paper. This lesson is a good opportunity to point out where in your institute researchers can get assistance with methodology and statistics, or relevant training.
Step 4 - Documentation and writing it down
Instructor Note
This is a massive lesson - you may want to put a break in the middle or just before this one.
Instructor Note
You could do this as a verbal activity or a written one.
Ask people to pick someone they interact with regularly (supervisor, student, collaborator). Then ask them, if contact to this person was completely cut off, what information would they need to know to continue their own work?
Instructor Note
Talk about that this scenario does happen. It could be negative things (‘bus factor’) but also positive too! They could have been offered a fantastic opportunity with little time to transfer.
Any personal stories here can also be useful if you have them, and are comfortable to share.
Instructor Note
Include the options for documentation platforms or elabnotebooks here from your institute.
Instructor Note
If you’re in a classroom, get them to then stick them on the wall.
In a virtual setting, you can put people in break out rooms and share them there, or put them in a chat or shared google doc etc.
Instructor Note
You can pull a recipe from online to critique.
You can ask participants to put their hand up if they had:
A base in their ingredients
Described what type of base (gf, thick, thin, bought, home made)
Described how to obtain the base (did they make it from scratch, did they talk about a bought base etc)
Watch the hands go up and slowly come down.
If you really want to be a bit dad joke/cheesy ha, you can play “All about the bass” as a ‘pencils down’ alarm after your 3 minutes.
Instructor Note
You can ask for people to suggest different formats
Instructor Note
A good opportunity to share ideas or brainstorm together different solutions.
Instructor Note
Good place to include any Carpentry or data science introduction resources - If you have none, feel free to borrow something from here: https://amandamiotto.github.io/HackyHourBookmarks/
Instructor Note
The following tabs are a good place to link your SoP templates, Data management plan system/templates, talk about research storage solutions etc.
Step 5 - Testing and ControlsProviding Authenticity and ValidityWhat is your next step?
Instructor Note
Can do this next challenge in the class, either pen and paper, on the board or in the chat.
Instructor Note
May be a natural space to ask people what types of data they are working with, where is it coming from?
Instructor Note
A good time for suggestions on where to learn R and Python. You’ve probably already included it elsewhere, just remind people at this point.
Instructor Note
Include links from your institute about where to publish analysis pipelines - your institute may have a preferred place, or your Office of Research or Library may have a good write up on this.
Step 6 - AutomationCan you automate any repetitive tasks?What is your next step?
Instructor Note
This is a good opportunity to ask researchers what they wish they could automate. If you have a research support IT team, local research software developers or eResearch team, this is a good plug for them.
Step 7 - Publishing, Persistent Identifiers and Preparing for ReusePersistent IdentifiersDeposit your final data/analysisOpen vs FAIR vs Can’t shareLicensingWhere to deposit?What is your next step?
Instructor Note
Opportunity to ask people for ideas on what needs to be done, either on whiteboard or in chat.
Instructor Note
Does your institute have a ‘Once your project is done’ checklist?
May be worth checking with your library or research office.
Instructor Note
Include information around who mints DOIs at your organisation.
Instructor Note
Does your institute have a policy, statement, procedure or resources on open access or publishing data and materials openly?
Link to them here.
You can also find examples of who has published openly in your institute.
Have you got an open network?
Instructor Note
Link to your organisation’s IP/Copyright person.
Instructor Note
Link to your organisation’s commercialisation team.
Wrapping up
Instructor Note
If you have spare time, you may want to share the ‘culture’ page that’s in the instructor menu. Consider your audience - this may be less relevant to an audience of students and ECR’s, however your more senior academics and leaders may find the content thought-provoking.
Instructor Note
A place to share what is happening at your own institutes, and where attendees can get support.