How many times have you looked at the list of data sources coming into your commodity/energy trading and risk management systems (CTRM/ETRM systems) and been shocked at the complexity that has built up over the years? Is the number of sources “shared” by multiple end-points growing? Is there pressure in the organization to get data faster? Are different systems producing a different value for the same product? The list could go on, but these are common issues for anyone in the trading business. The good news is that Data Stores can help reduce all these pain points.

Data Stores capture and store data. As its name implies, you can think of this as a “store” where things are kept and archived or as a place to find and procure items. In addition to relieving some of the problems we mentioned in the introduction, here are some of the ways Data Stores enhance an ecosystem.

Provides a centralized place to source data

There can only be one source of truth. This is a basic concept but often gets lost when in a high-speed high-pressure production environment. If we have multiple systems in the CTRM ecosystem that use the same source of data (this can be ANY type of data) there should be one internal place for that data to be sourced from. This concept reduces maintenance as there is only one externally facing interface and one datastore for historical records. All systems that need the same data fetch it from the Data Store and because of that model always have the same data for the same product/time/ etc.

Serves as one centralized interface for easy access

A Data Store can simplify your interface diagrams and thus make support and troubleshooting of any data issues easier and more efficient. With only one external facing interface logging and error handling is centralized. Support teams can look to ONE place for items related to an issue. It also allows for tracking of the internal requests for data.

Purges data and frees up space

Implementing a Data Store in a CTRM ecosystem could allow the application to purge the market data it uses after a determined amount of time as it is available in the Data Store. Generally, this frees not only space but also allows for faster fetching of data in the CTRM as it has fewer items to retrieve.

Fetches data at any given time

One of the other benefits of a Data Store is that you can fetch data at any time your organization deems it necessary. This can free you to run valuation at different times for products that may publish data on a different schedule. You can still use a price aggregator for final or EOD prices, but “flash” prices can be fetched separately.

Enables advanced analytics

A Data Store can provide a centralized dataset so that advanced analytics can be run against it. The more data that is centralized in the Data Store, the more analysis that can be conducted. With the explosion of Machine Learning and Artificial Intelligence applications, this is a huge benefit that was previously not as easy to exploit.  The Data Store can also be incorporated as a broader data management strategy including a Data Warehouse and or Data Lake.

In conclusion, the concept of introducing a centralized Data Store into your CTRM ecosystem can feel overwhelming, however, it can be done elegantly and efficiently.  When the work is done, it will be well worth the effort. From controlling the schedule for data retrieval, simplifying support and enabling ML/AI analysis and much more, the benefits of a Data Store are very real and obtainable.

At Veritas Total Solutions, we help educate clients and design data architectures and infrastructures that best meet their needs. We offer a range of technology solutions across the business spectrum. If you are interested in learning more about our specific capabilities, contact us or subscribe to our blog to stay connected.