[cmath] NSERC Survey concerning the CCA report (and re-allocation exercises)

[Walter Craig]

Dear members of the Canadian mathematics commnuity,

I am writing to encourage you to fill out the NSERC survey that was 
distributed on December 13, with a reminder from NSERC on January 10. It 
is available on the website

http://www.nserc-crsng.gc.ca/NSERC-CRSNG/Reports-Rapports/Assessment-Assessment_eng.asp

for which you should enter the code: 7243623

It will be avaliable until January 25, so we have to be somewhat prompt.

This survey is the form in which NSERC is consulting its constituency 
over decision-making criteria for an upcoming budget re-allocation 
process between the different scientific disciplines. This process is 
going to (again) make decisions as to the distribution of the total 
budget envelopes of the different Evaluation Groups (EGs). In this it is 
extremely important to us, especially as our funding has historically 
been so far behind. Here is the website of the NSERC background material 
for their envisioned reallocation process:

https://ensemble.nserc-crsng.gc.ca/surveys/SC00002/L001/Backgrounder6-e.pdf

Criteria for making such choices is the topic of a recent Council of 
Canadian Academies (CCA) report, available here:

http://www.scienceadvice.ca/en/assessments/completed/science-performance.aspx 


A brief outline of this report is that dry `indicators' or `metrics' 
such as blind publication counts or citation indices, averaged or not, 
are in general not good indicators of research quality, activity, 
capacity or trends, and certainly not for individual researchers, or in 
comparisons between groups of researchers in different scientific areas. 
Indeed they conclude that consultation with expert panels, and in the 
context of the individual disciplines, is the best way to make informed 
decisions. But these type of indicators would possibly be of use as 
reference material in expert panel deliberations which compared more 
broadly a discipline (such as math) with the same discipline in other 
countries.

The NSERC survey is in a way asking us whether we agree with the CCA 
report. It is a long and confusing questionnaire, which spends a lot of 
time asking whether various dry indicators are valid for making 
decisions on various detailed things, such as (1) Research Trends, (2) 
Research Quality and (3) Research Capacity. Little is mentioned of the 
CCA recommendation that expert panels are most important. I would be 
very happy if you could dedicate a small bit of time to filling our this 
questionnaire, at least on behalf of the mathematics community. Because 
it is so confusing, and there are literally hundreds of pages of 
background material to read for a complete story, I am including a 
summary of my own answers below, hoping that they will clarify the 
process of your response.

     Best regards,      Walter


Walter Craig
Chair: Mathematics - NSERC Liaison Committee


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

NSERC Survey on Reallocation Decisions, and CCA Report on Informing 
Research Choices:  [Outline of W. Craig's responses]



Questions Q1 - Q5 are just standard, where one works and one's seniority

Questions Q6 - Q12 and all of their sub-parts consist essentially of a 
tedious passage through the tables given in Background, Appendix A. The 
questions are like `Do you agree or disagree with indicator X being used 
for the factor Y' (where Y = (1) Research Trends, (2) Research Quality 
and (3) Research Capacity as given above).

My own survey answers here included the statements `only if they are in 
the context of a deeper Expert Panel review, and taken internationally 
and within the context of the discipline'.

Q13: Is the present DG allocation scheme overly driven by historical 
awards? A: YES, we have been historically underfunded for decades!

Q14: A long list of questions on the addequacy of DG budget allocations, 
which I responded as essentially all being inadequate (ie my answers are 
either 2 or 1 = strongly disagree).

Q14(c) is particularly important. `Is the overall funding available to 
our EG 1508 fair, in relation to the other EGs. My answer is 1 = 
strongly disagree.

Questions 15 through to the end are regarding hypothetical scenerios in 
which more funds enter into the DG program. Here is an outline of my 
answers:

Q15: [NO]

Q16(a): [7 = strongly agree]
    (b): [7 = strongly agree]

Q18 is important again - by which qualities should we assess allocation 
of any (hypothetical) DG funding increases. My answer to Q18(b) is `to 
rectify historical discrepancies in discipline funding that have 
persisted over time'.

Q19: Should there be periodic reallocation. A: YES

Q20: on zero-sum reallocation. Q20(b) `Funds should be redistributed 
with historical data in mind, to make up the historical underfunding of 
mathematics & statistics'.

Q21: What are acceptable yearly fluctuation bounds in EG budgets? My 
answer = `There should be a built-in method of averaging, so that in a 
given year, and in a given discipline, the grant awards are not held 
unfairly hostage (forced into crisis) by a downward fluctuation in 
returning grant amounts.'

Q22: The CCA report strongly advocates that expert 
judgments/deliberative methods be of dominant importance in budget 
allocations/reallocations, with supporting data given by multiple 
quantitative indicators, such as Average Relative Citations, or Weighted 
Publication Counts. These latter should always be used within the 
context of the specific discipline, as inter-discipline comparisons are 
not meaningful.

Q23 and Q24: At what frequency should there be a reallocation exercise, 
adnd what portion of an EG budget should come into play. I responded: 
`Each 5 years until an approximate steady-state is reached'. And 15% of 
the budget should be allowed to fluctuate (this proportion allows 50% 
change over 3 reallocation exercises, which actually is close to what 
mathematics & statistics needs).



That is all, good luck.



PS By the way, you can apparently take the survey as many times as you 
want, it is just a website and doesn't seem to count. I am not 
encouraging you to submit duplicates, but if you want to abandon a 
scratch copy that you have partially started, you seem to be perfectly 
able to start again.




-- 
Walter Craig  frsc
Professor of Mathematics and Canada Research Chair
Department of Mathematics and Statistics
McMaster University
905-525-9140 ext 23422