Apologies for coming back yet again to this rather parochial topic, but the "modelling" that the Province of Ontario keeps rolling out to justify its wildly-gyrating decisions on the pandemic really is the gift that keeps on giving. I'm in no position to comment on the medical qualifications of the so-called "Ontario Science Table" that keeps coming up with this stuff: for all I know, they are worthy successors to Banting and Best. I can, however, safely assert that as statisticians, they are clearly not worthy successors to Poisson and Bayes.
Here are some extracts from a post I wrote on the subject back on January 13, around the height of Ontario's second wave of COVID:
....it's quite apparent that most of the modelling consists of simply projecting the current trends for infections, hospitalizations and so on into the future in a linear fashion.
This naturally produces some scary-looking numbers -- the Science Table noted that "on the worst days" recently the case count has grown by 7 percent. You only need the "rule of 72" to know that extrapolating that growth rate gives you a doubling of cases every ten days. If you project that rate forward for a full month (i.e. three 10-day doublings), you get a daily case rate of 20,000-plus in mid-February, compared to the 3,000 or so we have been averaging recently.
....this kind of modelling can produce some very weird results. One member of the Panel commented that a faster growth rate would potentially lead to a 40,000 per day case rate, as indeed it would. Then again, it happens that the number of new cases reported this week has actually been slightly below the levels seen a week ago. Is that a new trend? Maybe, maybe not, but simply projecting it forward would lead to a much different outcome from the one the Province is using to justify its decisions.
So, back in January the modellers used the worst single day's data to project a couple of months ahead, a technique which falls rather a long way short of good practice. They ignored the fact that the actual case count was moving lower at the time they did their projections. And they turned out to be dead wrong, as daily cases in the Province slipped close to 1000 in the weeks after the forecast was issued, meaning that the modelling was off by a factor of twenty.
This is not just about smartypants second-guessing of the experts. It's more serious than that. The fall in case counts led the Ford government to ease restrictions in the Province, with the disastrous results we are now seeing in the form of the third COVID wave. Did the fact that the experts had been so wrong help to convince Ford and his team that they knew better? It seems quite possible.
Let's turn now to the Science Table's most recent set of COVID projections, issued on April 16: here is a link to the slide deck. Once again the modelling shows a dire situation that is set to get worse, and this time the data appearing around April 16 were continuing to deteriorate, so the panel has not made the second error it made back in January.
Instead it's found a different error to make. I mentioned in the first paragraph that the Ontario Government's pandemic response has been "wildly gyrating", particularly in the last month or so. On April 1 it implemented an "emergency brake" shutdown across the Province, but the April 16 projections prompted it to introduce a stay-at-home order and tighten restrictions on businesses.
These dates are important in understanding the statistical error that was made. It is generally accepted that any new restrictions take about two weeks to have a measurable effect. This means that the impact of the "emergency brake" was not yet evident in the data by April 16, but unless you think the April 1 measures were entirely ineffective, that impact was just about to show up. The data the Science Table was using as the basis for its extrapolations in effect reflected the situation that had existed before April 1 rather than the situation as it actually was at the time the projections were made public.
You can probably guess where this goes next. The daily case count for Province peaked precisely on April 16 -- a nice touch by the Gods of statistics -- at 4812. It has since moved gradually lower: the count published today (April 27) shows 3265 cases, the lowest in almost a month. The seven-day moving average has also rolled over and is starting to fall. That's good news, of course, and the fact that it was probably going to happen even before the April 16 stay-at-home order was proclaimed means we can look forward to a continuing steady decline in the case count in the weeks ahead.
What message should the Ford government to take from this? First, please listen to the scientists when they talk about the science. If you had done that back in January/February you wouldn't have seen the third wave we're enduring now, and I wouldn't be writing these snarky posts. Second, when you get presented with this modelling, and preferably before you release it to the public, have some non-medical person with statistical experience interpret it for you. You must have a few economists around the place who can do that, and it could save you from some very short-sighted decisions.
No comments:
Post a Comment