Total (now Total Energies) moved into the retail energy business with the recent completion of its acquisition of the Direct Energies electricity utility business. As TotalEnergies’ Ridha Nabli explained in a presentation at BigData/AI Paris 2021, this has led to the development of a real-time data platform ready to accommodate a forecast growth of some 500k new clients/year by 2025. The Direct Energies platform was slow. Daily batch runs were too long to run overnight and infrastructure costs were growing at 40% per year. Quality was suffering as was user confidence in the data.
As a first step in ‘Project Opera’, Total deployed a data governance solution from French software house DataGalaxy. The company also decided on a completely re-engineered solution with a shared, cloud based data environment regrouping legacy databases, data lakes and business intelligence solutions. A benchmark proof of concept study looked into solutions from Oracle, Amazon, Google and Microsoft but finally settled on Snowflake’s hosted Data Cloud offering.
Snowflake ‘ticked all the boxes’, with straightforward, scalable, centralized data access and query and has cut processing time by a factor of 10. A full time database administrator is no longer required. Snowflake’s ‘revolutionary’ (this is France!) data sharing covers various data formats including Avro JSON and XML. Security-wise the system leverages the Azure Active Directory and implements column and row security ‘by design’. All data, in rest and in motion, is encrypted and the solution is certified at ISO 27001/SOC Type II levels. Data in the Snowflake cloud feeds applications including Qlik business intelligence and Alteryx’ Databricks data science offering. The system is structured such that data science resources do not affect production.
Keyrus Livingstone was systems integrator for the project which was apparently conducted independently of Total’s in-house ‘Digital Factory’.
More from Big Data Paris.
© Oil IT Journal - all rights reserved.