The Google flu tracker is in trouble again for overestimating the seriousness of this year’s flu epidemic, just like last year, and the year before…and the year before that.

Google Flu Trends was launched in 2008 to predict how many cases of flu are likely to occur within the following nine weeks from any point in time. The way it does this is by monitoring web searches to find those where the search term is associated with flu such as "cough" or "fever". The analysis of those searches is then used as the basis of the prediction.

The problem, as we previously reported, is that based on this analysis, Google Flu Trends overestimates how many cases, and gets it particularly wrong at the height of the flu season when heightened media attention increases interest in these keywords.

Analysis of the results of the 2012/2013 season by a team from Northeastern University found that Google Flu Trends (GFT) predicted twice as many doctors' visits as were recorded from the ‘real’ data by the US Centers for Disease Control and Prevention (CDC). A similar pattern in 2011/2012 saw the figures overestimated by more than 50 per cent.

In November 2013 GFT udated its model to improve its predictive capability (see Google Updates Flu Model), and while this had some beneficial effect the researchers still point to a pattern of overprediction:

 GFT vs. CDC estimates after latest GFT update
Source: Lazer et al.
 

This graph shows the proportional error in GFT (calculated as (GFT‐CDC)/CDC) from July 2010 to January 2014. Observations to the right of the blue line are after GFT adopted its new algorithm.

The researchers state:

While the update has dampened the size of GFT estimates (by about 12% for those observations in which we have overlap between the old and new model), GFT is still estimating high almost 75% of the time. It also still estimated about 30% higher than the CDC in the 2013‐2014 flu season.

In a report on the study, Professor David Lazer of Northeastern University, who led the study, says:

“Big Data have enormous scientific possibilities, but we have to be aware that most Big Data aren’t designed for scientific purposes.”

Lazer adds that the Google engineers could have significantly improved their results by incorporating lagged data from the Centers for Disease Control and Prevention as well as making a few simple statistical tweaks to the model. It’s possible to get more accurate results than those from Google Flu Trends by just projecting current CDC data three weeks into the future, and a combination of the two gives the most accurate model.

The full report was published in the journal Science and PDF of the paper “Google Flu Trends Still Appears Sick: An Evaluation of the 2013-2014 Flu Season” is available for download from the Social Science Research Network.