Soil and big data

A recent conference in Canberra was titled "Soil and Big, Data and the Future of Agriculture", and was summarised in the Land -
http://soilbigdata.org/blog/blog/the-dirt-on-soil-data
In some ways, the idea is on the right track, and Barnaby Joyce is throwing $1.5 million at the project through CSIRO.
But somehow we have been missing something with this sort of approach. How could our productivity have been improved so successfully until about 25 years ago, then stalled since?
My most recent example has been research into liming. Questions are being asked about why the W.A. wheatbelt has not adopted what was advised decades ago, to hold soil pH at 5.5 (CaCl2 measurement). Yet during this time, there is no credible economic case that it would pay for farmers to adopt this suggestion. The assumption seems apparent that researchers say it should happen, so it will.
The liming example may be an isolated exception, and of course, such contentions would naturally be strongly contested by administrators of our R&D systems. However over the past few years, I have collected enough evidence to support my argument. The debate now should be whether to present and debate such arguments, or instead to implement a way to do the job better.
So good luck to Barnaby with his system, but I will need more than just luck to gizump Barnaby's $1.5 million.