How do you evaluate R&D quality?

Oil IT Journal editor Neil McNaughton asks, ‘just how do we evaluate the quality of research?’ He finds he is in the good company of the RAND Corporation and the Society of Petroleum Engineers. A Geological Society event on digital geology leads him to revisit a ‘letter to the editor’ ...

As companies and governments around the world ‘downsize’ and are looking everywhere to cut costs, one good subject for a big cut is usually R&D. In terms of government funded R&D, the debate is often limited to a battle of statistics, how much of GDP is devoted to research being a popular yardstick. This approach fits neatly into conventional party lines—with the left (at least in Europe) generally arguing that more public money should be devoted to R&D and the right arguing that the market should do more. This is a very good argument to have, as some R&D, like particle physics, is unlikely to progress far if left to market forces.

Oil & gas R&D

Other fields, like oil and gas R&D, would appear to be a more natural fit with industry funding. But even in the US, as our article on page 11 shows, there are considerable public funds available for upstream R&D. The newly-established Research Partnership to Secure Energy for America is to receive some $375 million of US taxpayers’ largesse for distribution to oil and gas research establishments around the country.


The public-private funding debate fits neatly with left-right paradigm of politics and gets a lot of attention. In fact I myself gave this issue some column-inches in my March 2003 editorial. But there is, I think, a much more interesting question, and one that is sometimes drowned out by the above debate and that is, how do you evaluate the quality of research, irrespective of how it is funded?

Peer review

Many university research departments get funding on the basis of their publication record. Individual researcher’s careers, likewise, advance or stagnate according to the number of papers they publish in certain scientific journals. Now I’m not talking about getting something printed in Oil IT Journal here, nor indeed in the Journal of Petroleum Technology. To gain R&D street cred, you have to get published in a heavyweight like Nature, or the Journal of the American Medical Association. To get into these illustrious publications, a paper has to undergo peer review—usually by anonymous reviewers. In the excellent Wikipedia entry on peer review, we learn that Nature only publishes 5% of the papers it receives. In other, more specialized journals, the acceptance rate may be much higher. What of oil and gas-related publications? The Journal of Petroleum technology has the good grace to indicate when a paper has not been peer reviewed—as is usually the case. Other more heavyweight publications from the SPE and SEG are presumably peer reviewed. But I imagine that finding a reviewer for some of the specialized papers in oil and gas R&D must be problematical.


The topicality of this subject is demonstrated by a new report* from the RAND Corporation, ‘Measuring the Benefits from Research,’ which addresses these issues and also enumerates other ways of evaluating R&D including economic rate of return and benchmarking. The SPE is to hold its first ever R&D Symposium in San Antonio this April to ‘discuss the big challenges facing the energy industry and the R&D requirements to meet them.’

Digital outcrops

All of which makes for a rather awkward segue into my next topic—the excellent meeting held under the auspices of the Geological Society in Manchester University this month on digital outcrop geology, or, giving it its official title, ‘From Outcrop to Asset.’ We’ll be reporting from this event in next month’s Journal, but the theme of the event was LIDAR and 3D photogrammetrical mapping of geological outcrops, and the subsequent manipulation of these interesting datasets in the computer or visualization center.


This is an exciting field and I think it’s fair to say that it is revolutionizing field geology. In the old days, geologists were limited to what could be chipped off with a hammer and chisel, mapped with binoculars, or captured on a plane table. Today, LIDAR/GPS along a sea cliff, mountain or quarry face generates a few gigabytes of 3D data that can be manipulated just like a 3D geo-model. Moreover, the dataset can be augmented by conventional measurements like sedimentological descriptions, shallow cores, ground penetrating radar and so on.

New data types

From the IT standpoint, this work introduces new data types to manage. The LIDAR generates ‘point cloud’ data in formats that are more or less manufacturer-specific. End users may be more interested in visualization in VRML or geomodelers like GoCad. But there is a bigger issue here and that is the capture and publication of a geological outcrop ‘model’. One presenter in the Outcrop to Asset meet described scanning and measuring sections from an article in the Journal of the Geological Society and incorporating the results into the geomodel. This is a familiar pattern to those working in the upstream. Digital data, even if it’s just a spreadsheet, is published on paper, then laboriously scanned or re-keyed into digital form, prior to further publication ... on paper! There has to be a better way!

Letter to Editor

In a letter*** to the editor of Geoscience last year I suggested how online geo-publishing could improve on tradition. A Wikipedia-like (with a more transparent peer review), ‘publish and subscribe’ model might leveraging an as yet undefined GeolRSS format. This would greatly improve on the current paper/pdf publication paradigm. Ironically, my letter was accepted for online publication only, somewhat limiting its impact! But after the Manchester event, I am convinced that the web has much more to offer geological publication by publishing and sharing the digital data sets of the original observations. We’ll get there eventually!




Click here to comment on this article

Click here to view this article in context on a desktop

© Oil IT Journal - all rights reserved.