Definition: Reference Data contains data identifiers for two types of data – the securities also known as issues and the provider of issues called as issuers. Issue identifiers are used when making a trade. The reference data is used to settle financial transactions.
Why are industries interested in reference data?
The industries based on the creation of Golden copy or Mastered Data purchases this data from different data providers and create master copy for their clients.
In the figure above, you can clearly see that Reference Data can be obtained from different vendors or exchanges like Ex shares, Bloomberg, etc. The role of Acquisition Team is to keep polling the vendor servers and collect this data on a timely basis.
The team is responsible for the initial phase of the data processing.
The data received by Acquisition Team is in a different format. Two types of data files are received. One file contains a complete history of the security data (Let’s say whole_data.txt) and other contains information about the updates on the security data for last 24 hours (latest_data.txt).
ETL – Extract Transform and Load is the Core Team. A Team who developed procedures and core logic to extract required data from the latest_data.txt file and converts it to other formats (may be XML format data.xml).
Now, after Transforming, the security is checked in the database for its presence. If it is, then the values in their attributes are matched. If the table attribute value is old and the XML file has a new value for that attribute, a check is performed for what value to be updated.
Now different exchanges provide different data for same security information. For example, latest_data.txt can be provided by Bloomberg, Telekurs, and Reuters. So based on the priority of each attribute in the XML file, the core logic is designed. So some attributes may only update values provided by Reuters and some may update values provided by Telekurs.
After updating all the attributes, the data is loaded in the Production Tables. This makes the Mastered Data.
As traders need latest values of securities, it is the responsibility of the industry to provide the values they have to the traders.
The final stage is to use a data publishing medium like TIBCO to update the databases of the traders. This is how the traders get the latest data. This is a continuous process throughout the day and traders should be updated continuously.