Leveraging Big (and Small) Data in the Energy Sector

This blog accompanies the titular talk, made by Ronnie Scott, Charter's Chief Technology Officer, at the Global Energy Show, in Calgary, AB, June 14, 2023. It also compliments our Charter Engage: Know IT podcast, "Moving Beyond Spreadsheets." 🎧 Listen Now

The value of data lies in its potential to enhance business in multiple ways, offering substantial benefits to those who can effectively utilize their data assets.

“Data - The New Oil”

It has often been said that “Data is the new Oil,” and while it is a cliché, it does represent a significant truth, being that oil is one of the most valuable assets in the modern world. In some cases, there is a tangible sale value for data, but in most cases, its value is less tangible. That value revolves around how data can be leveraged to improve business in countless ways; with significant return to those who are able to take advantage of the data they own.

One of the more common ways of seeing these returns is through concepts such as “Digitization,” in which typically human-based, analog processes can be replaced with efficient, automated, and insightful solutions. The outcome of digitization is improved business efficiencies, reduced costs, new income opportunities, and enhanced user experience. It is through such implementations that we have seen companies in the Energy sector achieve soaring levels of productivity that are bringing new life to an otherwise mature industry.

Unfortunately, there has been a raft of limitations in the Information Technology (IT) sector that has led many companies to fear the potential negative ramifications of making IT a core part of the Operational Technology (OT) environment. For decades, we have seen oil, gas, electricity, mining, and other resource industries build fantastically resilient, scalable, and efficient OT infrastructure. It is, of course, essential that as we move to a new digital business model, we do not compromise those many benefits of the OT world.

But the simple question remains, “What if…?”

• What if I can increase productivity?

• What if I can improve visibility?

• What if I can reduce downtime?

• What if I can predict failure?

• What if I can keep my staff safer?

• What if I can make my customers happier?

Taking advantage of the capabilities IT offers, while still providing the operational benefits of tried-and-true OT operational models, there is a strong possibility that we can gain the best of both worlds.

Delivering Enhanced OT

Moving from the static and location-bound models of the past to a dynamic network-based infrastructure offers both immediate and future operational benefits to any organization. Regardless of whether the system being managed is a large gas processing plant, an individual mine or well, or just a single machine or vehicle, there are huge operational advantages from introducing IT infrastructure to your OT environment.

This is the starting point for many OT digitization projects and involves the real-time monitoring and management of those environments. The fundamental underlying protocol for this is the Internet Protocol (IP), which provides similar capabilities to the SCADA networks that exist today, but with the added benefit of infinite reach.

From there, a raft of enhanced services can be layered on top of the existing infrastructure to enhance the capabilities of the infrastructure. Simple technologies, such as new IP-based sensors, cameras, and end-user devices provide new ways to gather information and interact with the physical world. In turn, the addition of software applications and computational intelligence provides insights and processes that can provide real-time insights and automated responses that can entirely change business operations.

The only caveat to this is the potential risks that come from bringing the IT and OT worlds together. However, a mix of good security design and clear demarcation and boundaries provides a strong barrier to potential threats, while still offering significant benefits to the organization.

The Data Lifecycle

To leverage the data inside an organization, there are three phases that must be considered, collecting, structuring, and analyzing that data.


Collecting data from the source is the first step in leveraging it for your benefit. To do this, we can use active data sources, legacy data sources, and latent data sources.

Active data sources are those you already have in your environment. They may be existing monitoring tools, ERP and CRM databases, inventory management systems, vendor datasets, and others. These give us real-time visibility of the existing infrastructure.

Legacy data sources are those outside of the digital world. Paper records and offline storage can be ingested and added to the active data to provide deeper insights into the longer operational life of an environment.

Latent data sources are those not currently being tapped or captured. By connecting otherwise standalone systems to capture more data, or by adding low-cost sensors and other data capture tools, the depth of visibility can rapidly increase.


To best leverage the data being collected, it is necessary to centralize and normalize that data into a common data repository. Sometimes referred to as “Data Lakes” or a dozen variations on that theme, having the data in one place allows for the analysis and data mining we need to see the huge potential benefits that we are after.

It should also be noted that there is value in both “structured” and “unstructured” data. Using deep learning and machine learning across unstructured data can provide similar benefits to the analysis of structured data but require significantly more processing capabilities to achieve those outcomes. Either way, the key is to have the data centralized.


Individually, each data source may have significant value, but in Charter’s experience, we have discovered that bringing these disparate sources together quickly results in outcomes that are far greater. This is where the analysis phase kicks in.

As an example, by blending existing information regarding operational and historical statistics, measured against vendor data, and tied to inventory management systems, we can begin to see correlations and relationships that were invisible prior. This might allow us to predict future failures, plan capacity growth, identify anomalies, and improve staff safety.

The potential outcomes are immeasurable, but having the data is the starting point for new innovations.

What Do I Stand to Gain?

As already mentioned, the potential outcomes from a digitization strategy are too great to document in this Blog. But to summarize, there are three broad questions it can help you answer:

1 - What do I know?

Most organizations are capable of building strong processes around the day-to-day operations, in which they exist - identifying when things are broken, responding to changes in demand, and keeping the environment functional. However, many of these processes are time-consuming, time-sensitive, or operationally impacting. Accelerating awareness of things we already understand can give significant immediate benefits to productivity, resilience, and safety.

2 - What don’t I know?

However, the real value of data comes when we look beyond the information that we do know and begin to see what is hidden just below the surface. Looking at relationships and causality between different facets of the operation is one example of how we might be able to improve productivity, by centralizing and analyzing data.

3 - What could you know?

Going even further, adding more data sources, and introducing tools such as artificial intelligence, big data analytics, deep learning, machine learning, and data mining, we can begin to see things no human brain is able to put together. Combining information such as timing, weather, productivity, staff alertness, sources of resources, and more could begin to find ways of improving an environment’s reliability, increasing productivity, or finding entirely new processes for delivering your outcomes.

Where Do I Start?

Looking at all the “What ifs?” can make it feel like there is so much you could do. And figuring out where to start can be a challenge. Charter has worked with several customers as they begin their digitization process, and we have found the “Start Small” approach to be the most effective for many.

Starting small refers to taking on manageable, finite projects that can be bound to the SMART goals, being Specific, Measurable, Attainable, Relevant, and Time-Bound. Such projects are not only much more likely to succeed, but they also provide learning opportunities - which can be leveraged as you grow into more significant and longer-term projects.

Examples of smaller projects, we have seen be effective, include the following:

• Using off-the-shelf, cloud-based data management and analysis tools to quickly obtain insights from existing data, to improve operational processes.

• Leveraging new wireless technologies to provide location and environment awareness - increasing safety, efficiency, and operational visibility.

• Deploying simple video-based applications to users, married with video-based communications, to provide remote helpdesk or remote assistance capabilities to field workers.

• Deploying IP-based sensors and fixed cameras alongside existing legacy-based, wired control systems to enhance real-time notification of issues to operational management teams.

Sharing the Journey

Charter believes that leveraging others’ experience and capabilities can significantly reduce the time and effort required to embark on a digitization process, and we are ready to work with you to deliver on your desired business outcomes.

Charter offers an array of capabilities to assist you in your “Digitization” journey. Here are some of the practices we have to assist you:

• Charter Advisory and Consulting Services

○ Governance, Risk, and Compliance Practice

○ Security Consulting Practice

○ Cloud Adoption Practice

• Charter Business Practice Services

○ Energy, Resources, and Industry Practice

○ Business Architecture Practice

• Charter Professional Services

○ Service Delivery

○ Project Management

○ Charter Managed Services

○ Charter Service Desk

Please contact Charter at info@charter.ca for more information.

About the Author
Ronnie Scott, Charter, Chief Technology Officer

Ronnie Scott has over 35 years of broad IT experience, including programming, network architecture, as well as senior consultative roles for Financial Services, Internet Service Providers, ILEC Carrier Networks, and large enterprise customers across New Zealand, Australia, and Canada.

Ronnie is currently the CTO at Charter Telecom Inc, a Value-Added Reseller specializing in IT service delivery. As CTO, Ronnie brings his extensive technological background with a strong Business and Service Delivery lens to Enterprise IT Infrastructure solutions.

[LinkedIn profile: bit.ly/3E9QdBk ]

About Charter

Founded in 1997, Charter is an award-winning solutions innovator known for providing connectivity solutions that enable innovation and drive business transformation. Connectivity is required for all the smart devices that are now able to securely join the Internet plus it is foundational to the organization as it provides more data for business analysis while creating the opportunity for the workforce to collaborate much more effectively with ecosystem partners.

For more information on this press release or on Charter, please contact:

Dawn van Galen
Marketing Manager