Julie Gould

Hello, I’m Julie Gould and this is Working Scientist, a Nature Careers podcast, and in this episode, I’m exploring the PhD thesis.

Now, from what we’ve gathered, the PhD training system is undergoing change or is still undergoing change, but whatever the PhD training ends up looking like, it will, one way or another, need to be assessed. So, I wrote about this in 2017, asking whether or not the PhD thesis needed updating, overhauling and/or scrapping. Now, when I was at the Revisiting Forms and Forces of Doctoral Education conference in Hanover in September this year, I asked Janet Metcalfe from Vitae to give an overview of what the current assessment of PhD training looks like.

Janet Metcalfe

The examination process reviews the research, it looks to see whether or not that researcher has done that research and done it well, and by that deduces whether or not we’ve got a trained researcher. If we step back and look about if we were designing a system from scratch, would this be the way you would want to identify or examine whether or not you have a trained researcher? So, I think we need to be really brave and say, let’s fundamentally look at the way we examine doctoral degrees and look more in terms of how do you assess whether somebody is a trained researcher.

Julie Gould

Now, one way to train researchers is to make sure that they are aware of how their work fits into the bigger picture. At the same doctoral education conference, I asked David Bogle, the vice president of research at University College London about how University College London has made it mandatory for all PhD candidates to include a one-page impact statement in their thesis as a way of making sure that they understand how their work fits into the bigger picture and has an impact in any way, whether on society, on their research or just on themselves.

David Bogle

What we really wanted, part of the institutional mission, the current strategy of institution is three lines: leadership, cross-disciplinarity and impact. We want our research to make an impact, so if we want our research to make an impact, we have to be persuading our researchers that they have to think about the impact that they’re going to make. So, in order to support and reinforce that, we introduced that there should be just a one-page impact statement at the front of the thesis. There’s an abstract and then there’s a statement that says this is the difference that it’s going to make in the world. Now, we have various training things about what that might mean, and it’s very varied because it’s not a narrow view of impact. It’s any impact this might have on the curriculum, on society, on business, on anything – what difference is it going to make in the world? And eventually, I see this impact statements improving. I think some people from outside will look at it and thing what’s all this about, but as we move on, people will get better at articulating just in one page, this is the difference this is going to make to the world, and eventually, we get more towards being a bit more predictive about it. At the end of the day, it hasn’t made an impact but this is why I’ve done it and this is what it’s going to do because I was also a bit tired myself of when I examined even engineering PhDs sometimes they duck that question and you think engineering has to make a difference in the world. But everything, all of this research enterprise, we need to be thinking about… it might not end up making that difference, but they need to at least be thinking about it, certainly from the end of the thesis.

Julie Gould

So, this was part of a discussion about skills development and labour markets, so what sort of skills are you hoping that people will develop by having to write this impact statement as part of their thesis.

David Bogle

Well, skills development is a much broader thing because doing the thesis, thinking about things, thinking about their own personal development, doing courses, doing skills development, the impact statement itself, I would have said it’s about the difference that the work is going to make. So, it’s not a skill in that sense. I guess it’s just like writing an abstract. How do you do that clearly and crisply? How can you communicate that something which is clear and crisp with evidence of some sort or some at least argument and everybody needs to do that. We need to do that in academia, we need to do that when we write significant statements in journal articles and if you go into the private sector, the board wants a one-page what difference, why should I put money into this.

Julie Gould

Another way to assess whether or not someone would make a good researcher is to see if they’ve published any work, and some programmes require their PhD researchers to publish their research during a PhD, others don’t. Some people have a PhD thesis by publication, others don’t. In a commentary that Anne-Marie Coriat from the Wellcome Trust wrote for Nature Human Behaviour which was published in October this year, she argues that having the experience of publishing is a good thing, but making it mandatory might not be.

Anne-Marie Coriat

If asked do I think that students should publish, the answer to that, well do I think it’s a good thing, the short answer is it is not a bad thing and learning writing skills is a hugely important part of PhD training. If asked the question in a slightly different way, do I think it should be a requirement that all students publish in peer-reviewed journals in order to pass a PhD, my answer is absolutely, emphatically, no. To sort of determine that publication in a peer-reviewed journal and often that’s translated into and one with high impact factor or one word summary is then driving pressures on the system which say that you can predict the way that science is going to develop, and I think that’s quite unhealthy in many ways.

Julie Gould

But what if, like Janet Metcalfe suggests, the thesis is completely revamped. Throw the old star out the window and start with a blank canvas. What would this new form of assessment look like? Inger Mewburn is an associate professor and the director of research training at the Australian National University and she’s also the author of the blog Thesis Whisperer. Now, she believes in a portfolio system like the one that is used in architecture, which is what she initially trained in.

Inger Mewburn

The way that architecture is assessed is through a portfolio, and what you do is you spend years studying and you make a range of buildings, you draw them, you make models, you take photos and so on, and then just before you graduate, what you’re expected to do is to take a piece of work, maybe redraw some of them, write a bit about them, put them in context and create sort of like a book, sort of like a fashion look book and that’s got writing, that’s got drawings, that’s got photos and that’s got maybe some certificates that you got along the way that you demonstrated that you learnt about fire hazards or something, and you put it all in the form of a book that an employer can look through and the book represents you. It’s your portfolio. So, the portfolio can contain anything but it’s kind of a collection of objects and a narrative that tells people who you are as a designer. And I think I’m attracted to this idea for the PhD because a portfolio idea is quite flexible and what I don’t like about some of the things that have been suggested to fix it like let’s whack a bit more coursework in there, is that it’s sort of at odds to what the enterprise of the PhD is. The PhD is meant to turn out individual, beautifully crafted, entirely bespoke and unique knowledge creators, and we need people like that in this world where increasingly we need creative people with really different sorts of talents. We don’t want to turn out just cookie cutter researchers. We want to turn out a really wide range of different people and a portfolio sort of gives you the chance to say, well, you can include all sorts of objects in it, maybe you’ve done some coursework, maybe you’ve written a paper, maybe you’ve done journal articles or maybe you’ve done presentations, maybe you’ve done a thesis, maybe there’s some video of you in there, and there’s also a narrative about who you are.

Julie Gould

Would you include a viva voce style assessment to go with it as well?

Inger Mewburn

I think you absolutely would. I think that a portfolio makes the most sense when the person narrates it and that’s the point of the architectural portfolio. So, you’re meant to put it on a desk in front of either an assessor or an employer and you turn the pages and you talk them through everything you’ve done, why you’ve done it, what you learnt from it and what you would have done differently, so it becomes a conversation. How you set a set of standards or criteria for what something like this looks like is challenging, but I would draw everyone’s attention to the fact that we don’t really have a benchmark or set of criteria or a perfected rubric for the PhDs as we have them at the moment.

Julie Gould

Gillian Houston is the Chair of the UK Council for Graduate Education and she thinks this model would be a great step forward, but it would still need to be accompanied by more traditional aspects of the PhD thesis.

Gillian Houston

So, I feel that that’s very appropriate to architecture, but I also think that those elements would have to be accompanied by a review of the literature or a review of design in your field and how it’s relevant to your work, and I mean an analytical review, so you have to say what you think is good and what is not so good, and a general contextualisation of your work in the wider world, and I also think you would have to have certain academic conclusions. So, I think those are very good developments, but I don’t think it’s like a replacement. I think they have to be integrated into a coherent whole, which is true at the time you graduate.

Julie Gould

Now, Janet Metcalfe believes that the future of PhD assessments needs to include an assessment of researcher development as well as of the work that they do.

Janet Metcalfe

If you are assessing competencies, somebody would have to review that. So, certainly in the way we’ve been looking at recognition of researchers’ competencies, we’re using skilled assessors, those that are used to dealing with that sort of qualitative information, doing independent reviews by several assessors who then come together in the same way you get academics to do that initial review of the thesis, then a similar process in terms of reviewing the portfolio of evidence around their skills and competencies, and then they come to a joint conclusion as to whether or not that evidence is robust and sufficient in order to identify somebody as a skilled researcher.

Julie Gould

Thank you to Janet Metcalfe, David Bogle, Anne-Marie Coriat and Gillian Houston for their contributions to this episode an