PBRF

I have been asked by management at UC to make sure that I comply with PBRF requests to check with my co-authors about relative contributions to multi-authored journal articles.
I won’t be writing to my co-authors to ask them to waste their time replying to an operationally meaningless question.

First, this PBRF process is supposed to deliver an evidenced based portfolio. There is no evidence available to anyone on the PBRF panel – or to the co-authors themselves – to either accept or refute quantitative or qualitative claims such as:
“Author A contributed x% while author B contributed y%” , or “Author A’s contribution is a multiple/fraction k of Author B’s contribution and a multiple/fraction j of Author C’s contribution” , “Author A’s contribution was marginal but Author B’s was vitally significant ” (written by B, or perhaps by A who is B’s lover, or grad student, or friend, or????)

A request by a bureaucrat that I provide them unverifiable input claim reports is nonsense in the scientific community and has no place in an evidence based portfolio. Full stop.

Even output claims in the PBRF process have to pass a “quality control test” of some kind – where the “quality” of quality control can range from well reviewed top tier professional journal refereeing processes (and even with a decent process some individual reviews stink) right down to nominal (read non-existent) review of conference acceptances. ie Some elf or gnome…or lord of the conference…. “external” to the output claim being made has to “review” it, and as importantly be able to point to a transparent process whereby such review “might have been done” in a “competent manner”. There is nothing remotely like this for this attempt to request verifiability for “input claims”.

At least statements like …”the four of us had a great discussion over a beer about idea X with visitor Y in a local bar, and then we just got to it and wrote the damn paper – 42 revisions it took till we all agreed it was ok to submit” or “i spent 3 weeks, 50 hours a week, coding recoding testing examining searching exploring…..and came up with a very neat 2-dimensional diagram that occupied 1/2 of one page of our 10 page paper…..” while my co-author spent hundreds of hours in the lab running experiments with mice rats and other feral creatures from which she was able to derive a counter example to SMith’s hypothesis that we ended up not even discussing in the paper although at the outset we had planned to ” have some evidential basis for them – but of course NOT third party verifiable evidential, and certainly prone to psychological and social self protection mechanisms of selective recall . That’s why we don’t promote, or demote, on the basis of unverifiable input claims, isn’t it?? But we surely rely on this sort of “cheap talk” conversation in our joint research endeavours (and “sole authored ones”, just no less acknowledged) and recognise it for the good natured banter that it is….as long as the damn paper eventually gets submitted!!

Without any evidential basis to test such claims , much less any third part verifiable evidential basis, reports which purport to answer a contributions question can be and will be strategically manipulated, gamed. Duh?

Oh yeah, author “X was included in the list of co authors becasue we all realise that we’re playing a publications game in the academic rat race and multi authored outputs aren’t yet discounted very highly so we’re gaming the system : perceived marginal cost is less than perceived marginal benefit in a game of asymmetric information with the employers….” or “my supervisor and I had an implicit agreement that he’d be a co author on all the work I did in the first two years before and after my phd and I was afraid I wouldn’t get my phd or that she would not give me a good reference in the future so…well yes, we worked equally on this paper” or ” joe blog is included even though he is dead now becasue well, he had alzheimer’s in the last few years of his life and we had some great discussions and some of the key ideas were his…he just didn’t knwo it becasue of all the rugs he was taking……” ….you name it. Let your imagination run riot.

And worse, these sorts of subjective reports can become a source of acrimony , intimidation if when they are made public downstream. Disagreements about them can’t be resolved by any possible evidence – it’s just claim vs counter claim – and agreements aren’t worth the paper they are written on. That is – they convey no information whatsoever. Cheap talk. So they are a waste of time, my time, my co author’s time. As long as disagreements about input claims remain private no one cares that they bad mouth a colleague who they have had a falling out with or good mouth a colleague who they are trying to help. There isn’t anyone or anything to challenge that sort of claim. The very knowledge that these reports might be made public would and should give authors incentive to opt out of the game of course, or if that’s not possible simply submit the most conflict avoiding reports possible. . But will these reports ever be made public? It’s not credible. Do the PBRF people really think they will be able to take advantage of a social reputation system where those who do make comparative performance claims on the basis of no evidence whatsoever might in the future have to face an audience of , well, “colleagues”, be they friend or foe, or ?? and ask to justify/rationalise/explain these claims ? Shame and blame? I don’t think so. They, the PBRF bureaucrats, would be in deep doodoo by revealling names and claims about comparative performance …so they won’t …so the people reporting the claims won’t have any reputational issues to worry about….so where is the enforcement mechanism for any kind of semi-informative cheap talk reporting?

At a conceptual level this sort of question “what is the relative contribution of each author to….” reflects simplistic understanding of any production process, much less poorly understood, messy, teamwork research processes.

Don’t get me wrong. If the contribution question was something like: ” What was the financial contribution of each author to the completed project?” I might expect back a set of numbers which are commensurable – unless of course I have to do some exchange rate conversions, or intertemporal payment conversions or? And I’d be asking for receipt too and not accepting unverifiable claims of financial expense. Or the contributions question might be framed in terms of aggregate labor time inputs : but even verifiable labor time inputs are hard to come by. I have no idea for example how much time any of my co-authors actually spent or din’t spend in some activity that was more or less related to the outputs we have, and I have a hard time recallin in any detail my own time input. Its hard to make sensible comparisons without sensible numbers. And those time input numbers just aren’t recorded (thank goodness!). But we all know that it i not labor time that matters, but amounts of high quality effort, diligent and creative application of intelligence and skilled knowledge. I have some very ambiguous private information about myself on those fronts – but even there would ahve a difficulty apportioning this local public good input between different private good published outputs. ANd I certainly do not have any information on corresponding things for my co-authors, nor do they know anything but me on these matters except the cheap talk I provide about my private information. Yet we all manage to get along here and actually produce some stuff!!

This is NOT a rocket science question, about what concepts are involved here. You host a dinner party at your house, potluck, followed by a games evening to celebrate ….something . Mary Jane brings chips, dip and a fine australian wine wine, Billy Bob a couple of six-packs and burger ingredients, Grandma bakes her best-ever chocolate cake, Steph brings an extra table and chairs for the kids and two bottles of bubbly, you supply the house and BBQ and of course your impeccable skill as a host , everyone plays the game except Grandma fell asleep halfway through the night, some stay later and pitch in with the dishes and clean up , Ken takes those who drank too much home in his cab, you clean up in the morning with some help from your kids. A great evening was had by all.

This is a simple description of inputs and outputs. It could be codified in a spreadsheet with a 2×2 table: names in the top row and “activities” in the first column, and a number in each cell that could be simply a binary indicator “yes/no, 1/0” someone did something in that activity cell. (we had to put your ex wife in because she owns half the house….even if she wasn’t actually present and “contributing” on the night).

contributions table — Contributions to the Dinner Party

Now tell me what the relative contributions of Grandma and Steph are? Does grandma’s “one cake” indicate she contributed “less” than Steph, indeed only 25% of Steph’s 4 items of chips and dip? It looks safe to say Billy Bob contributed twice as much beer as you – except , you point out, Billy Bob brings el cheap Canadian Lagers while you have fine imported Irish Red beer – not that you’d mention this in polite company, but, now that you are asking….. Everybody played games, but did they all contribute “equally” because the spreadsheet records that, well most people played 5 games, the latter ones while some were a little out of it (no names mentioned). Besides you say, my contribution to the games was small, 2, only 40% of the 5 others have because I was pouring wine, getting drinks, cooking burgers on the barbie…….

Note that most of the cells in the table are empty – that reflects a lot of things, but mainly specialization. Is that a problem in producing something? Of course not. Is it a problem for people who like to take ratios in an input table? Yep! Something divided by zero is , well, a mighty big relative contribution mathematically speaking, indeed infinite. But of course almost everybody in this table has a relatively infinitely large relative contribution in some input dimension, the one he/she controls exclusively.

You get the point now? It looks like* (see below) it makes a lot of sense to monitor and record who did what, but once you step out of one simple dimension on commensurable quantities to talk about relative contributions of the different inputs is just plain silly. Everybody contributed, everybody did some different things, there was some duplication, but hey….we produced a good night. Oh yes you say, but look at the “value” of supplying a home, a house, compared to the “value” of Billy Bob’s six pack; or the years of blood sweat and tears the Grandma has gone though raising family to have the skills and energy at age 101 to make that chocolate cake. Surely her “relative contribution” via loving labour input is worth more than the chips and dip? Once you start asking these questions you take a big step in to the thicket of subjective evaluations.

But who codifies it on the night? And what would the monitoring codifying operation do to the nature of the evening – and the interaction? Suppose nobody actually codified anything on the night yet some external authority – the investigating police – asked each person, a year later, to write a table to recall who did how much of what on the night, with the threat to impose a punishment if there was some sort of “inconsistency”…as if “agreement” meant anything other than a conspiracy of the blind against the blind! You see now best beloved? This is PBRF input reporting…

Perhaps “contribution” means marginal product , averaged over some range? I seem to recall Euhler’s theorem that for a linear homogenous production function y=f(K,L,X) total product will be “exhausted” (ie output y is divisible private good ) , divided up, apportioned out , when all factors are paid the value of their marginal product. In this case it does make conceptual sense to talk of relative input shares as a proportion of total output….but do the bureaucrats have this kind of model in mind? One might then think they would ask authors to “report your marginal product” please. But they don’t even get this far….but choose a simpler path . My guess is they think in terms of a linear production technology Y= aL1+bL2+cL3 where Y is published output and where labour inputs associated with the research process are nice neat tidy homogenous units a=b=c , so then a new quantity “big L” = L1+L2+L3 can be defined. Then they are asking me to report: what proportion of “big L” did you supply? My reply to this has to be that “big L” doesn’t exist except in their imagination, that I don’t share their hypothesis about the nature of the research production process – ok it could be true but I know for sure that my labour skills are Notperfect substitutes for those of my co-author(s). Hmm, isn’t that a reason why we are co-authors in the first place?

Might I suggest an alternative simpler rough and ready quantitative model that we all can agree on and has a fine empirically verifiable basis : y=Min {L1,L2,L3} where labour inputs are binary indicator variables for the presence or absence of the author”i” = 1,2,3 listed on the submitted manuscript and y is a binary indicator variable of a piece of “authorized, notarized, certified….deep fried” research output ? In this case input ratios at unit output levels are 1:1:1 and, over the range 0,1 all of our marginal products are 1 (thanks Seamus) . By the way, if we all accept this production model, we don’t need a question “field” on the PBRF form “what was your verifiable input contribution” in addition to the coarse – and intellectually offensive – “quality assured” indicator variable that is already embedded in the PBRF process. (Hey, do you think I could publish this very very intellectually deep mathematical model? perhaps in the new journal “PBRF Follies Review”? Don’t laugh..Alan Woodfield has already published something on the incentive non compatibility of this kind of self report your contribution scheme in his NZEP paper “The Underlining Game” )

” Hmm”, says the bureaucrat- “all too complicated for me. I’m not trying to pay you on the basis of these inputs, i just want a “general idea” about “contributions”, sort of, like, you know….a number, something I can put into a spreadsheet, to see, well, that you, and not your coauthors have been “performing”. You know what I mean, don’t you? So….give me the number you think I want to hear – for free – and just get on with it.”

rubbish

John Fountain

(JF insubordination index 1, JF compliance index 0)