Training the data managers

Oil IT Journal editor Neil McNaughton reports on a joint CDA/ECIM/PPDA initiative of a ‘competency matrix’ for upstream data management. What do data managers really need to know? What should be the relationship with the IT department? And which part of the IT canon do you really need?

Speaking at last month’s ECIM data management conference, Malcolm Fleming of the UK’s Common Data Access*, advocated ‘professionalizing’ data management with a program of training and accreditation. The idea is to develop a universal ‘competency map’ for E&P data managers. The map describes the core skills required and sets minimum levels for different degrees of competency.

The idea is for a uniform reference tool to aid in education, hiring and career development. The intention is to develop the competency map into an accreditation governed by a ‘certifying authority.’ The joint CDA/PPDM/ECIM initiative has established requirements for well and geophysical data management and at last month’s ECIM, kicked off a new ‘map’ for geospatial data management.

All of this is to be available in a ‘portal’—to be rolled out by year end 2012. Future users will be able to sign up and register as ‘a data manager’ and create their own profiles through self assessment—which can then be verified by a third party. Fleming intimated that this might be the UK Geological society or possibly the Science Council—which already awards certificates to ‘chartered scientists.’ A pilot is underway with five oil company sponsors, three service companies and around fifty users.


Recently I started receiving curious messages from LinkedIn saying that so and so had endorsed me for competency in various fields—Petroleum, Geology, Energy and so on. I was then invited—as I am sure many of you have been—to endorse them for similar accreditation. I clicked mindlessly on a few of these, before I realized that LinkedIn would have gone on offering me more and more opportunities to ‘endorse’ until I had run through my ‘500+’ connections. While this was entertaining, it was not as entertaining as playing solitaire. I did wonder if this kind of cross-endorsement might evolve into a future academic qualification—perhaps along the lines of a ‘third party assessment.’ Maybe someday folks will speak with pride of their degree from the University of Facebook.

I have to say that I am not at all sure that the CDA competency map is the right way of going about things. It grinds the subject finer and finer without indicating what teaching occupies each slot. That would be a rather harder thing to do because upstream data management is a mixture of some very dull stuff and some very complicated stuff, all embedded in a rather disorderly way of working.

I’d like to know what a ‘self assessed’ spatial data manager would do when confronted with a large project where half the data has been loaded with the wrong projection—would he or she spot the error? Would he/she be able to fix it? The reality is that such issues require the sort of specialist knowledge that comes from in-house domain experts (oil companies used to employ surveyors) or a third party. Logs and seismics likewise hide a host of non trivial problems for the data manager which are hard to solve without in-depth domain knowledge.

The CDA approach focuses on the geoscience side of the equation which is clearly key. But it is not the whole story. My personal view is that you will not be able to do much good in data management without a pretty good grasp of the fundamentals of IT as well. Even if your company has a strict demarcation line between data management and IT, you still should have a good idea of what IT has to offer so that you are a competent buyer/user of its services.

You need to be aware of the many different ‘solutions’ that IT offers to a particular problem. One is to make simple tasks unbelievably complicated—witness the famous ‘hello world’ program in just about any modern language. But it is not all bad. Back in the 1970s, there was a general realization that IT—then dominated by IBM—was getting far too good at carving out more and more work for itself, making things unnecessarily complicated and obliging folks to spend more and more time on operating system upgrades. Fast forward to October 2012 and we have, not IBM but Microsoft, releasing Windows 8—with yet more upheaval. The process continues unabated.

So what did those guys in the 1970s do to ‘solve’ the unnecessary complication and constant change imposed by the vendors? They observed that most all IT tasks involve common problems that can be solved with a set of standard tools. These involve reading and writing files, looking for and editing strings, piping data around a workflow, sorting, counting and so on. All the stuff that makes up data management in fact!

The work done by these pioneers is the one part of the IT canon that should be in every data manager’s ‘competency matrix.’ If you did not guess, the toolset is the Unix/Linux Shell. This has stood the test of time because it addresses the unchanging fundamentals of IT—rather than its ephemeral manifestations like the shiny new graphical user interface of Windows 8.

Most all of my own data management experience involved programming the Shell which I learned from books and a few courses. While many may be a bit out of date some are still pretty valid. Anyhow I thought that a ‘useful books’ page would be a good addition to the Oil IT website. Check it out on the link above. Feel free to email me ( with your own, maybe more recent, suggestions. You want to be a data manager? Read a book. Oh, and by the way, sixteen years ago, a publication called ‘Petroleum Data Manager’ hit the newsstands. It is now called Oil IT Journal and it includes a significant ‘body of knowledge’ on upstream data management. OK, it could be better organized—another item for the ‘to do’ list I guess.

* Visit CDA’s competency page.

Click here to comment on this article

Click here to view this article in context on a desktop

© Oil IT Journal - all rights reserved.