Enterprise Data Management

Any talk about Data Governance is incomplete without Data Onboarding. Data onboarding is the process of uploading the customer’s data to a SaaS product often involving ad hoc manual data processes. Data Onboarding is the best use case of Intelligent Automation (IA).

If done correctly, data onboarding can result in high-quality data fabric (the golden key or the single source of truth (SSOT)) for use across back, middle, and front office for improving organizational performance, meeting regulatory compliance, and ensuring real-time, accurate and consistent data for trading.

Data Onboarding is critical for Data Governance. But what happens when Data Onboarding goes wrong?

  • Many firms struggle to automate data onboarding. Many continue with the conventional means of data onboarding such as manual data entry, spreadsheets, and explainer documents. In such a scenario, the benefits are not visible. Worse, inconsistencies during data onboarding results in erroneous reporting, leading to non- compliance.
  • Poor quality data onboarding could also be responsible for reputational damage, heavy penalties, loss of customers, etc., when systemic failures become evident.
  • Further we cannot ignore that a tectonic shift is underway in the capital markets – trading bots and crypto currency trading are becoming more common, and they require accurate and reliable data. Any inconsistency during data onboarding can have far- reaching consequences for the hedge fund or asset manager.
  • From the customer’s perspective, the longer it takes to onboard, the more frustrating it becomes as they cannot avail the benefits until the data is fully onboarded. End result – customer dissatisfaction! Prolonged onboarding processing is also a loss for the vendor as they cannot initiate the revenue cycle until all data is onboarded. This leads to needless revenue loss as they wait for months before they receive revenue from new customers.

Given the consequences of Data Onboarding going wrong, it is important to understand why data onboarding is so difficult and how it can be simplified with proper use cases.

Why is Data Onboarding so difficult?

When we talk about Data Governance, we are simply not talking about Data Quality Management, we are also talking about Reference and Master Data Management, Data Security Management, Data Development, Document and Content Management. In each of the instances mentioned, data onboarding poses a challenge because of messy data, clerical errors, duplication of data, and dynamic nature of data exchanges.

Data onboarding is all about collecting, validating, uploading, consolidating, cleansing, modeling, updating and transforming data so that it meets the collective need of the business – in our case the asset manager, fintech, bank, FI, or hedge funds engaged trading and portfolio investment.

Some of the typical challenges faced during data acquisition, data loading, and data transformation are underlined below:

Data Acquisition and Extraction

  • Constraints in extracting heavy datasets, availability of good APIs
  • Suboptimal solutions like dynamic scrapping in case API are not easily accessible
  • Delay in source data delivery from vendor/client
  • Receiving revised data sets and resolving data discrepancies across different versions
  • Formatting variations across source files like missing/ additional rows and columns
  • Missing important fields/ corrupt data
  • Filename changes

There are different formats in which data is shared – CSV files, ADI files, spreadsheets. It is cumbersome to onboard data in these varied formats.

Data Transformation

Converting data into a form that can be easily integrated with workflow or pipeline can be a time-taking exercise in the absence of standard taxonomy. There’s also the issue of creating a unique identifier for securities amongst multiple identifiers (CUISP, ISIN etc.). In many instances, developers end up cleaning messy files, which is not at all worthwhile.

Data Mapping

With data structures and formats different for Source and Target systems, data onboarding becomes difficult as data mapping – mapping the data coming in with the relevant fields in the target system poses a huge challenge for organizations.

Data Distribution/Loading

With many firms resorting to the use of spreadsheets and explainer documents, data uploading is not as seamless as it could be. File formatting discrepancies with the downstream systems and data reconciliation issues between different systems could easily be avoided with Intelligent Automation or Administrative AI.

Data Onboarding builds a bridge for better Data Governance

“Without a data infrastructure of well-understood, high-quality, well-modeled, secure, and accessible data, there is little chance for BI success.” Hugh J Watson

When we talk about the business-driven approach to Data Governance, the importance of early wins cannot be negated and hence the need for streamlining Data Onboarding with the right tools and technologies for ensuring scalability, accuracy, and transparency while keeping in mind affordability.

As the volume of data grows, data onboarding challenges will persist, unless a cohesive approach that relies on people, technology, and data is employed. We have provided here two use cases where businesses were able to mitigate their data onboarding challenges with Magic FinServ’s solutions:

After all Comprehensive Data Governance requires Crisper Data Onboarding.

Case 1: Investment monitoring platform data onboarding – enabling real-time view of positions data

Investment Monitoring Platform automates and simplifies shareholder disclosure, sensitive industries and position limit monitoring and is a notification system for filing threshold violations based on market enriched customer holding, security, portfolio, and trade files. Whenever a new client is onboarded into the application, the client’s implementation team takes care of the Initiation, Planning, Analysis, Implementation and Testing of Regulatory filings. We analyzed customer’s data during the Planning phase. Data such as the Fund and Reporting structure, Holdings, Trading Regimes, and Asset Types etc., were analyzed from the Reference Data perspective. As a part of the solution, after the analysis, the reference data is set up and source data loaded with the requisite transformation, followed by a quality vetting and completeness check. As a result of which our client was able to have a real-time view of the positions data which keeps flowing into the application in real time.

Case 2: Optimizing product capabilities with streamlined onboarding for regulatory filings

The requirement was for process improvement while configuring jurisdiction rules in the application. The client was also facing challenges in the report analysis that their client required for comparing the regulatory filings. Streamlining the product and optimizing its performance required a partner with know-how in collecting, uploading, matching, and validating customer data. Magic FinServ’s solution consisted of updating the product data point document – referred to by clients for field definitions, multiple field mapping, translations, code definitions, report requirements, etc. This paved the way for vastly improved data reconciliation issues between different systems.

The client’s application had features for loading different data files related to Security, Position, Transactions, etc., for customizing regulatory rule configuration, pre-processing data files, creating customized compliance warnings, direct or indirect jurisdiction filings, etc. We were able to maximize productivity by streamlining these complex features and documenting it. By enabling the sharing of valuable inputs across teams, the errors and omissions in data/customer were minimized while product’s capabilities were enhanced manifold times.

The importance of Data Governance and Management be ascertained from the success stories of Hedge Funds like Bridgewater Associates, Jana Partners, and Tiger Global. By implementing a robust Data Governance Approach, they have been able to direct their focus on high value stocks (as is the case with Jana Partners) or ensure high capitalization (Tiger Global).

So, it’s your turn now to strategize and revamp your data onboarding!

Paying heed to data onboarding pays enormous dividends

If you have not revamped your data onboarding strategy, it is time to do so now. As a critical element of the Data Governance approach, it is imperative that data onboarding should be done properly and without needless human intervention and the shortest span of time to meet the competitive needs of capital markets. Magic FinServ with its expertise in Client Data Processing/Onboarding with proficiency in Data Acquisition, Cleansing, Transformation, Modeling and Distribution can guide you through the journey. A professionally and systematically supervised data onboarding results in detailed documentation of data lineage, something very critical during data governance audits and subsequent changes. What better way to prevent data problems from cascading into a major event than doing data onboarding right. A stitch in time after all saves nine!

For more information about how we can be of help write to us mail@magicfinserv.com

The Buy-Side and Investment Managers thrive on data – amongst the financial services players, they are probably the ones that are the most data-intensive. However, while some have reaped the benefits of a well-designed and structured data strategy, most firms struggle to get the intended benefits primarily because of the challenges in consolidation and aggregation of data. In their defense however, Data Consolidation and Aggregation challenges are more due to gaps in their data strategy and architecture.

Financial firms’ core Operational and Transactional processes and the follow-on Middle Office, Back Office activities such as reconciliation, settlements, regulatory compliance, transaction monitoring and more depend on high-quality data. However, if data aggregation and consolidation are less than adequate, the results are skewed. As a result, investment managers, wealth managers, and service providers are unable to generate accurate and reliable insights/information on Holdings, Positions, Securities, transactions, etc., which is bad for trade and shakes the investor’s confidence. Recent reports of a leading Custodian’s errors in account set up due to faulty data resulting in less than eligible Margin Trading Limits are classic examples of this problem.

In our experience of working with many buy-side firms and financial institutions, the data consolidation and aggregation challenges are largely due to:

Exponential increase in data in the last couple of years: Data from online and offline sources must both be aggregated and consolidated before being fed into the downstream pipeline in a standard format for further processing.

Online data primarily comes from these three sources:

  • Market and Reference Data providers
  • Exchanges which are the source of streaming data
  • Transaction data from inhouse Order Management Systems or from the prime brokers and custodians, often this is available in different file formats, types, and taxonomies thereby compounding the problem.

Offline data comes also through emails for clarifications, reconciliation of the data source in email bodies, attachments as PDF’s, web downloads etc., which too must be extracted, consolidated, and aggregated before being fed into the downstream pipeline.

Consolidating multiple taxonomies and file types of data into one: The data that is generated either offline or online comes in multiple taxonomies and file types all of which must be consolidated in one single format before being fed into the downstream pipeline. Several trade organizations have invested heavily to create Common Domain Models for a standard Taxonomy; however, this is not available across the entire breadth of asset and transaction types.

Lack of real-time information and analytics: Investors today demand real-time information and analytics, but due to the increasing complexity of the business landscape and an exponential increase in the volume of data it is difficult to keep abreast with the rising expectations. From onboarding and integrating content to ensuring that investor and regulatory requirements are met, many firms may be running out of time unless they revise their data management strategy.

Existing engines or architecture are not designed for effective data consolidation: Data is seen as critical for survival in a dynamic and competitive market – and firms need to get it right. However, most of the home-grown solutions or engines are not designed for effective consolidation and aggregation of data into the downstream pipeline leading to delays and lack of critical business intelligence.

Magic FinServ’s focused solution for data consolidation and integration

Not anymore! Magic FinServ’s Buy-Side and Capital Markets focused solutions leveraging new-age technology like AI (Artificial Intelligence), ML (Machine Learning), and the Cloud enable you to Consolidate and Aggregate your data from several disparate sources, enrich your data fabric from Static Data Repositories, and thereby provide the base for real-time analytics. Our all-pervasive solution begins with the understanding of where your processes are deficient and what is required for true digital transformation.

It begins with an understanding of where you are lacking as far as data consolidation and aggregation is concerned. Magic FinServ is EDMC’s DCAM Authorized Partner (DAP). This industry standard framework for Data Management (DCAM), curated and evolved from the synthesis of research and analysis of Data Practitioners across the industry, provides an industrialized process of analyzing and assessing your Data Architecture and overall Data Management Program. Once the assessment is done, specific remediation steps, coupled with leveraging the right technology components help resolve the problem.

Some of the typical constraints or data impediments that prevent financial firms from drawing business intelligence for transaction monitoring, regulatory compliance, reconciliation in real-time are as follows:

Data Acquisition / Extraction

  • Constraints in extracting heavy datasets, availability of good API’s
  • Suboptimal solutions like dynamic scrapping in case API are not easily accessible
  • Delay in source data delivery from vendor/client
  • Receiving revised data sets and resolving data discrepancies across different versions
  • Formatting variations across source files like missing/ additional rows and columns
  • Missing important fields / Corrupt data
  • Filename changes

Data Transformation

  • Absence of a standard Taxonomy
  • Creating a unique identifier for securities amongst multiple identifiers (Cusip, ISIN etc.)
  • Data arbitrage issues due to multiple data sources
  • Agility of Data Output for upstream and downstream system variations

Data Distribution/Loading

  • File formatting discrepancies with the downstream systems
  • Data Reconciliation issues between different systems

How we do it?

Client Success Stories: Why to partner with Magic FinServ

Case Study 1: For one of our clients, we optimized data processing timelines & reduced time and effort by 50% by optimizing the number of manual overrides for identifying an asset type of new securities by analyzing the data, identifying the patterns, extracting the security issuer, and conceptualizing a rule- based logic to generate the required data. Consequently, manual intervention was required only for 5% of the records manually updated earlier in the first iteration itself.

In another instance, we enabled the transition from manual data extraction from multiple worksheets to more streamlined and efficient data extraction. We created a macro that selects multiple file source files and uploads data in one go – saving time, resources, and dollars. The macro fetched complete data in the source files even when the source files had some filters applied to the data (accidentally). The tool was scalable, so it could be easily used for similar process optimization instances. Overall, this tool enabled reduced data extraction efforts by 30-40%.

Case Study 2: We have worked extensively in optimizing reference data. For one prominent client, we helped onboard the latest Bloomberg industry classification, and updated data acquisition and model rules. We also worked with downstream teams to accommodate the changes.

The complete process of setting up a new security – from data acquisition to distribution to downstream systems, took around 90 minutes (about 1 and a half hours) and users needed to wait till then for trading the security. We conceptualized and created a new workflow for creating a skeleton security (security with mandatory fields) which can be pushed to downstream system in 15 minutes. If sec is created in skeleton mode, only the mandatory data sets/tables were updated and subsequently processed. Identification of such DB tables was the main challenge as no documentation was available.

Not just the above, we have worked with financial firms extensively and ensured that they are up to date with the latest – whether it be regulatory data processing, or extraction of data from multiple exchanges, or investment monitoring platform data on-boarding, or crypto market data processing. So,
if you want to know more, visit our website, or write to us at mail@magicfinserv.com.

“What you know cannot really hurt you” Nassim Nicholas Taleb

Enterprise data management (EDM) long ceased to be an option, in the post-pandemic business landscape, it is a necessity driven by a cross-influence of factors such as increased compliance, rising costs of operations, risk management, client relationship management, and more importantly the ability of the management to be on top of the situation, keeping in mind the fact that everyone is operating virtually, all of which depends on how smoothly and efficiently data is managed.

However, Alveo’s 2021 survey for hedge funds suggests that most buy-side firms are enmeshed in data management challenges. For most, data management continues to be a concern.

  • Nearly 23 % of the firms surveyed talked about fragmented and unreliable data and the challenges that arose due to it. 
  • Data maturity remained low: 24 % of the surveyed scored poorly in terms of data maturity (defined by CMMI Institute’s Data Management Maturity Model.) 
  • Process discipline was found lacking. Instead of focusing on ”prevention” firms focused on repair – or patchwork, when it comes to processes involved in improving the quality of data.   
  • Redundant data feeds are a problem. 77 % of the firms surveyed reported that organizations require the same data multiple times from the vendor. Adding to the costs.
  • The collection and analysis of environmental, social, and governance data sets are critical today. However, only 45% of buy-side firms surveyed said that they had a centralized mechanism that made their data secure and accessible. 

If the whereabouts of organizational data (operational, financial, strategic, and data generated via network logins and alerts) and what can be done with it are known to a firm, they not only avoid day-to-day disputes but also stay up-to-date with the latest trends and changing investor demand and competition.

That is where enterprise data management (EDM) comes in.

“Enterprise Data management is the development, execution, and supervision of plans, policies, programs and practices that control, protect, deliver and enhance the value of enterprise data and information assets.”

Provided below is how EDM functions:

A differentiator for buy-side firms

  • A centralized EDM enables data blending and golden rules creation – that goes a long way in preventing disputes between business units such as operations, trading, compliance, and risk – of a buy-side firm. 
  • As no single vendor can cater to the needs of firms that handle multiple asset classes, investment strategies, and products, therefore a holistic and enterprise-wide data management strategy is needed. Else it would be pure chaos with firms paying for data that is redundant. 
  • Enterprise data management makes it easier for buy-side firms to provide their stakeholders with real-time visibility into risk factors.   
  • Today, we are seeing a lot of new-generation portfolio management systems. These are designed to meet the ever-increasing demands of banks, asset managers, hedge funds, brokers, and insurance companies who want transparency and accountability to stay competitive. A centralized EDM ensures better management and control of data for these differentiated products

Planning for EDM

Before planning the EDM, here’s a list of the prerequisites that need to be defined to ensure that tech strategy is in sync with the business goals and objectives. 

  1. Begin with an EDM audit.
  2. Vision: Define the core values of the EDM program. 
  3. Goals: The strategic goals, objectives, and priorities behind the EDM program
  4. Governance model: Chalk out how the enterprise-wide program would be managed and implemented
  5. Choose the appropriate technology, and get backing from key executives.  
  6. Resolution of issues: Ensure what kind of mechanism would be in place for identifying, triaging, tracking, and updating data issues.
  7. Monitoring and Control: How to measure the effectiveness of the program? Identify the leaders in charge.  

Key elements or capabilities of EDM  

The objective of the EDM strategy is to create the Golden Sources or the Single version of truth and ensure that it is disseminated across the proper channels and intended signatories. So here are the key elements or pillars in reaching that end goal.  

Critical Data Inventory: All those extremely important data elements that are required for making key business decisions must be handled carefully and with the full knowledge of all stakeholders involved. 

Data Sharing and Governance: Ingrain a data-sharing culture across the board, in its right spirit. Everyone must be aware of the rules and regulations regarding data sharing securely. Whether there is limited-time access to data, or complete access, or denial of access (data is hidden), data sharing and governance streamlines the organization’s flow of data so that information is not released to anyone apart from the intended party. 

Data Architecture: In the whitepaper: A Case for Enterprise Data Management in Capital Markets, it has been suggested that the layered approach in which each ” each horizontal (technology) function/capability is managed separately as a shared service across the vertical (business) function/capability” does not lead to the chaos and confusion seen in the end-to-end functionality approach which leads to silos.

Data Integration: Data integration is a multi-pronged process, but the essence of data integration is that it ensures that data from multiple sources are integrated to provide a single amalgamated view. As underlined earlier, the benefits of a unified repository are many. But primarily it ensures that data is actionable and available to anyone who needs access to it. Data integration marginalizes costs as there is less rework and error.  Data Quality Management: The quality of data is important for ensuring optimum outcomes. Too often, when dealing with data, whether it is financial or strategic or operational, or even network logs and alerts, the quality of data is suspect. With data quality and management, organizations are assured that they have cleaner and high-quality data readily available. Data quality management ensures that organizations have high-quality data at their disposal after data processes such as data cleansing and integrity checks.

Metadata Management: In 2020, Magic Quadrant for Metadata Management Solutions  the software category was defined by Gartner, “a core aspect of an organization’s ability to manage its data and information assets.”

Metadata is information about data. Metadata primarily captures aspects of data like type, length, timestamp, source, and owner, that can be traced back. It can be created manually, or with a data processing tool. For describing, understanding, and creating an inventory of data, for data governance and risk and compliance, metadata management is a requisite. 

Master Data Management: Master data management, refers to the organization, centralization, and categorization, of master data. Simultaneously, master data management also enriches data, so that the organization uses the most accurate version – the ‘golden record’ to ensure that the data that is disseminated to applications downstream is consistent.

State of data maturity

It is also important for firms to realize the state of data maturity. Only 9 percent of the buy-side firms had high levels of data maturity as per the CMMI Institute’s scale. 

To reach higher levels of data maturity as per CMMI Institute guidelines firms must:  

  • “See data as critical for survival in a dynamic and competitive market. 
  • Optimize process performance by applying analysis on defined metrics for target identification of improvement opportunities. 
  • Ensure that best practices are shared with peers and industry.

How data is managed and controlled is important now that the work model has changed considerably

The pandemic has marked a sea change in the way businesses function today. There’s not much physical interaction between people and in such a scenario, data and how it is disseminated, managed, and controlled assumes great importance – especially in the case of capital markets. So, to be without a viable EDM strategy is like committing a “Hara-Kari.”

Magic FinServ – your data management partner     

We understand why high-quality data is of paramount importance for FIs today. With many FIs walking the tightrope between stricter regulatory compliance and rising customer expectation, with challengers in hot pursuit, quality data is all it takes retain the edge.   

Unclean, inconsistent, and poor-quality data weighs down on enterprise resources. It clogs business applications that run on data and makes it an uphill task for any organization to achieve any kind of growth. Today, true transformation begins with a clear understanding of data. Hence the need for a data model.  

  • Enabling clear understanding of data needs: We help FIs and Banks understand their data requirements by understanding what they have, what they need (in terms of the quality of data and more) and analyzing how their business processes are impacted in real-time by drawing an abstract model. And that is not all. Having partnered with multiple banks and FIs, we know that time saved is “dollars” earned and hence the importance of minimizing the amount of effort required to extract only relevant data from unstructured and offline sources, and creating a single version of truth.
  • Experts in the field: Today, data sources have proliferated. With organizations having to take account of ESG and Blockchain data as well, there is not much that traditional systems can do to ease the burden – and ensure clean, consistent, structured, and auditable data. Only an expert in the domain of capital markets like Magic FinServ with expertise not only in AI and ML (our proprietary Magic DeepSight™ is a one-stop solution for comprehensive data extraction, transformation, and delivery), but also in the cloud, APIs and Smart Contracts can rightfully create a single version of truth, after consolidating data from the multiple sources that are in existence today to create a golden source.
  • Unique perspective in reference data: Having worked extensivelywith a major global multi-class investment solution provider for more than 6 years and successfully delivering on all reference data specific implementations in the complete value chain for Security Master, Pricing, Ratings, Issuer Data and Corporate Actions projects, across the geography for various key accounts and established buy-side entities has provided Magic FinServ a unique perspective in reference data that we are happy to share with our clientele.    
  • Unmatched value: When it comes to sheer depth of knowledge and experience in working with industry-leading platforms for reference data – Simcorp Dimension, TCS BaNCS, Everest, SAXO Trader, amongst others – Magic FinServ’s expertise is unmatched. Not just Implementation, we are also your Go-to partner in Delivery, Custom Development, QA, and Support.
  • Skilled team: We are experts in delivering projects from kick-off to go-live in market data as well. Our team brings in unmatched efficiency and experience in delivering projects for different industry-renowned market data providers (for different regions) like Six, ICE, Telekurs, Bloomberg, Refinitiv, Markit, Custodian Feeds in the SWIFT ISO15022 format. The engagement further strengthens in managing requirements in relation to Data Architecture, Integration Aspects, and Management of Master Data.
  • In tune with the latest trends – AI and ML: There’s a perceptible shift in the industry towards harnessing AI and ML-based solutions. This is evident in the way industry leaders are talking about enhancing current mature product offering by introducing AI and ML based approaches to it. With the advent of modern technologies like AI and ML, multiple use cases of their adoption have also emerged in Reference Data Management. Some of the prominent use cases where Magic FinServ brings value are:
    • Automation of Business Processes
    • Lowering the cost and increasing the Operational Efficiency in the Post-trade Processes: Confirmation, Settlement of Trades, Reconciliation of Position, Cash Balances, NAV’s, etc.
    • Handling complex data issues like Missing Data, Stale Information or Erroneous Data; ML techniques can be applied to identify and flag out the issues after careful assessment and Model integration.
    • Exception Management and Increasing STP, with ML at the core
  • Due Diligence and Comprehensive Risk Assessment: However, introducing AI and ML to Enterprise Data is not a cakewalk. It requires thorough due diligence and comprehensive risk assessment. That’s where we come in. Not only do we have the experience in navigating pitfalls, but our team – competent in the latest AI and ML trends – can help clients surmount the odds. Not only do we (help) in assessing customer’s current needs but also propose and help them envisage the future landscape by owning and driving their AI and ML journey to the target state.
  • Merits of a partnership with EDM Council: The EDM council is a Global Association created to elevate the practice of Data Management as a business and operational priority – and is a leading advocate for best practices with regards data standards. As underlined earlier, accessing the state of data maturity and identification of the gaps that exist in the data environment are prime requirements for reaching enterprise maturity. Here Magic FinServ can drive a definite advantage. Thanks to our partnership with the EDM council, we are in a position to accurately access the state of client’s Enterprise Data Maturity leveraging industry leading frameworks approved by the EDM Council.
  • Going beyond pure advisory: Our services go beyond pure advisory. Our exhaustive domain knowledge and competency in managing and implementing EDM solutions are key differentiators when it comes to managing and standardizing our client’s data in line with industry approved and vetted global standards.

Upping the game with Magic FinServ

Magic FinServ brings a deep understanding of the financial services business to help FinTech and Buy-side firms build and support their EDM platform. We offer sophisticated tools, products, and services using AI and Analytics to manage data repositories, organize business glossaries, create and improve data lineage, review and optimize reporting rules, create and manage semantic frameworks and improve data quality of financial institutions. We cover the complete data lifecycle from data strategy and management, bringing in capabilities for Data Inventory, Integration, data quality, Profiling, Metadata Management and Data Privacy for our customers.

For more information about our services and solutions, connect with us today. Write to us at mail@magicfinserv.com. We’ll be glad to be of assistance.

Before we actually get into details as to how digitalization has contributed to unstructured data, we really need to understand what is meant by the terms, Digitalization and Unstructured Data.

Digitization: is the process of converting information into a digital format. In this format, information is organized into discrete units of data (called bits) that can be separately addressed (usually in multiple-bit groups called bytes).

Unstructured Data: is information that either does not have a pre-defined data model or is not organized in a pre-defined manner. Unstructured information is typically text-heavy but may contain data such as dates, numbers, and facts as well. This results in irregularities and ambiguities that make it difficult to understand using traditional programs as compared to data stored in fielded form in databases or annotated (semantically tagged) in documents. (Source: https://en.wikipedia.org/wiki/Unstructured_data)

Now to establish connections between above two, I begin with a point, that every day there is new evolution happening in Technology space, and in addition to this desire to digitalize everything around us is also gaining momentum.

However, we haven’t thought that this process will solve our problem, or will lead to a bigger problem which will be common across all the current verticals and new verticals of the future.

Actually, if we do deep thinking around this then we will realize that instead of creating a solution for the digital world or digitized economy we have actually paved the path for making data as unstructured or for that matter Semi/Quasi structured, and this heap/pile of unstructured data is growing day by day.

Certain questions crop in our minds that what are various factors which are contributing to the unstructured data pile. Some of them are mentioned below:

  1. The rapid growth of the Internet leading to data explosion resulting in massive information generation.
  2. Data which is digitalized and given some structure to it.
  3. Free availability and easy access to various tools that help in the digitization of data.

The other crucial angle for unstructured data is how do we manage it.

Some insights and facts around unstructured data problem, that stresses it is a serious affair:

  • According to projections from Gartner, white-collar workers will spend anywhere from 30 to 40 percent of their time this year managing documents, up from 20 percent of their time in 1997
  • Merrill Lynch estimates that more than 85 percent of all business information exists as unstructured data – commonly appearing in e-mails, memos, notes from call centers and support operations, news, user groups, chats, reports, letters, surveys, white papers, marketing material, research, presentations, and Web pages

(Source – http://soquelgroup.com/wp-content/uploads/2010/01/dmreview_0203_problem.pdf)

  • Nearly 80% of enterprises have very little visibility into what’s happening across their unstructured data, let alone how to manage it.

(Source – https://www.forbes.com/sites/forbestechcouncil/2017/06/05/the-big-unstructured-data-problem/2/#5d1cf31660e0Source –)

Is there a solution to this?

In order to answer the above question, I would say data (information) in today’s world is Power, and Unstructured data is tremendous power because the essence/potential is still untapped, which when realized effectively and judiciously can turn fortunes for the organizations.

On the other hand, Organizations and business houses which are trying to extract meaning/sense out of this chaotic mess will be well-positioned to reap competitive edge and will have a competitive advantage among the peer group.

Areas to focus on addressing the problem related to unstructured data are.

  1. Raising awareness around it.
  2. Identification and location in the organization.
  3. Ensure information is searchable
  4. Make the content context and search friendly
  5. Build Intelligent content.

The good news is that we, at Magic, realized the quantum of this challenge sometime back and hence have designed a set of offerings specifically designed to solve the unstructured & semi-structured data problem for the financial services industry.

Magic FinServ focuses on 4 primary data entities that financial services regularly deals with:

Market Information – Research reports, News, Business and Financial Journals & websites providing Market Information generate massive unstructured data. Magic FinServ provides products & services to tag meta data and extracts valuable and accurate information to help our clients make timely, accurate and informed decisions.

Trade – Trading generates structured data, however, there is huge potential to optimize operations and make automated decisions. Magic FinServ has created tools, using Machine Learning & NLP, to automate several process areas, like trade reconciliations, to help improve the quality of decision making and reduce effort. We estimate that almost 33% effort can be reduced in almost every business process in this space.

Reference data – Reference data is structured and standardized, however, it tends to generate several exceptions that require proactive management. Organizations spend millions every year to run reference data operations. Magic FinServ uses Machine Learning tools to help the operations team reduce the effort in exception management, improve the quality of decision making and create a clean audit trail.

Client/Employee data – Organizations often do not realize how much client sensitive data resides on desktops & laptops. Recent regulations like GDPR make it now binding to check this menace. Most of this data is semi-structured and resides in excels, word documents & PDFs. Magic FinServ offers product & services that help organizations identify the quantum of this risk and then take remedial actions.

People are often confused with the terms – Visual analytics and Visual Representations. They many times take both words for the same meaning – presenting a set of data into some kind of graphs which looks good to the naked eye. However deep down, ask an analyst and they will tell you that visual representation and visual analytics are two different arts.

Visual Representation is used to present the analyzed data. The representations directly show the output from the analysis and are of less help to drive the decision. The decision is already known with analytics already performed on data.

On the other hand, Visual analytics is an integrated approach that combines visualization, human factors, and data analysis. Visual analytics allows human direct interaction with the tool to produce insights and transform the raw data into actionable knowledge to support decision- and policy-making. It is possible to get representations using tools, but not interactive visual analytics visualizations which are custom made. Visual Analytics capitalizes on the combined strengths of human and machine analysis (computer graphics, machine learning) to provide a tool where alone human or machine has fallen short.

The Process

The enormous amount of data comes with a lot of quality issue where data would be of different types and from various sources. In fact, the focus is now shifting from structured data towards semi-structured and unstructured data. Visual Analytics combines the visual and cognitive intelligence of human analysts, such as pattern recognition or semantic interpretation, with machine intelligence, such as data transformation or rendering, to perform analytic tasks iteratively.

The first step involves the integration and cleansing of this heterogeneous data. The second step involves the extraction of valuable data from raw data. Next comes the most important part of developing a user interface based on human knowledge to do the analysis which uses the combination of artificial intelligence as a feedback loop and helps in reaching the conclusion and eventually the decision.   

If the methods used to come to conclusion are not correct, the decisions emerging from the analysis would not be fruitful. Visual analytics takes a leap step here by providing methods/user interfaces to examine the procedures using the feedback loop.  

In general, the following paradigm is used to process the data:

Analyze First – Show the Important – Zoom, Filter and Analyze Further – Details on Demand (from:  Keim D. A, Mansmann F, Schneidewind J, Thomas J, Ziegler H: Visual analytics: Scope and challenges. Visual Data Mining: 2008, S. 82.)

Areas of Application

Visual Analytics could be used in many domains. The more prominent use could be seen in

  1. Financial Analysis
  2. Physics and Astronomy
  3. Environment and Climate Change
  4. Retail Industry
  5. Network Security
  6. Document analysis
  7. Molecular Biology

Today’s era greatest challenge is to handle the massive data collections from different sources. This data could run into thousands of terabytes or even petabytes/exabytes. Most of this data is in a semi-structured or unstructured form which makes it highly difficult for only a human to analyze or only a computer algorithm to analyze.

E.g. In the financial industry a lot of data (mostly unstructured) is generated on a daily basis and many qualitative and quantitative measures can be observed through this data. Making sense of this data is complex due to numerous sources and amount of ever-changing incoming data. Automated text analysis could be coupled with human interaction and knowledge (domain specific) to analyze this enormous amount of data and reduce the noise within the datasets. Analyzing the stock behavior based on news and the relation to world events is one of the prominent behavioral science application areas. Tracking the buy-sell mechanism of the stocks including the options trading in which the temporal context plays an important role, could provide an insight into the future trend. By combining the interaction and visual mapping of automated processed world events, the user could be supported by the system in analyzing the ever-increasing text corpus.  

Another example where visual analytics could be fruitful is the monitoring of information flow between various systems used by financial firms. These products are very specific to the domain and perform specific tasks within the organization. However, there is an input of data which is required for these products to work. This data flows between different products (either from the same vendor or different vendor) through integration files. Sometimes, it could become cumbersome for an organization to replace an old system with a new one due to these integration issues. Visual analytic tools could provide the current state of the flow and could help in detecting the changes would be required while replacing the old system with a new system. It could help in analyzing which system would be impacted most based on the volume and type of data being integrated reducing the errors and minimizing the administrative and development expenses.

Visual analytics tools and techniques create an interactive view of data that reveals the patterns within it, enabling to draw conclusions. At Magic FinServ, we deliver the intelligence and insights from the data and strengthen the decision making. Data service team from Magic would create more value for your organization by improving decision making using various innovative tools and approaches.

Magic also partners with top data solution vendors to ensure that your business gets the solution that fits your requirements, this way we rightly combine the technical expertise with business domain expertise to deliver greater value to your business. Contact us today and our team will be happy to speak with you for any queries.

Reference data is an important asset in financial firm. Due to recent crisis in global market, regularity changes and explosion of derivative and structured products, the need for valuable market & reference data has become central focus for financial institutions. For any financial transaction accurate information/data is the key element and faulty data is the major component of the operation risk.

Reference data used in financial transactions can be classified as static and dynamic

  • Static Data: Data elements which have unalterable characteristics such as financial instrument data, indexes, legal entity/ counterparty, markets and exchanges.
  • Dynamic Data: Variable data such as closing and historical prices, corporate actions.

Reference data is stored and used across front office, middle office and back office systems of the financial institutions. In a transaction life cycle, reference data is used to interact with various systems and application internally and externally. Problems related to faulty reference data continue to exist and this leads to increased operations risks and cost.

To reduce data related risk & issues and contain cost, financial institutions are looking at innovative solutions to improve data management efficiency. Centralization, standardization and automations of data management process is key to achieve this goal.

Industry Challenges

  • Poor data quality; lack of global standards; presence of data silos; multiple data sources leading to inefficiency in the whole data governance process.
  • Data duplication and redundancy across various business functions.
  • Lack of data governing policies.
  • Lack of standardized data definition.
  • Time consuming data source onboarding process.
  • Inconsistent data leading to poor reporting and management.
  • High manual intervention in data capturing and validation process.

Poor data quality is leading to

Solution

  • Deploy centralized reference data management system and create data management framework.
  • Create golden copy of the reference data received from the various sources within an organization that can be accessed by all business functions.
  • Update the data daily/real time at this single point.
  • Validate data at single place before distributing to relevant business functions.
  • Resolves data exception centrally to avoid issues at downstream systems.

Benefits

  • Improve process efficiency by centralization of data management.
  • Reduced operational and data management cost.
  • More control over data quality and change management.
  • Reduced turnaround time for new business needs and meeting new regulatory requirement.
  • Early detection and resolution of potential data issues.

Reference data is the data used to classify other data in any enterprise. Reference data is used within every enterprise application, across back-end systems through front-end applications. Reference data is commonly stored in the form of code tables or lookup tables, such as country codes, state codes, and gender codes.

Reference data in the capital market is the backbone of all financial institutions, banks and investment management companies. Reference data is stored and used in the front office, middle office, and back-office systems. A financial transaction uses the reference data when interacting with other associated systems and applications. Reference data is also used in price discovery for the financials instruments.

Reference data is primarily classified into two types –

  • Static Data– Financial instruments & their attributes, specifications, identifiers (CUSIP, ISIN, SEDOL, RIC), Symbol of exchange, Exchange or market traded on(MIC), regulatory conditions, Tax Jurisdiction, trade counterparties, various entities involved in a various financial transaction.
  • Dynamic Data– Corporate actions and event-driven changes, closing prices, business calendar data, credit rating, etc.

Market Data

Market data is price and trade-related data for a financial instrument reported by the stock exchange. Market data allows traders and investors to know the latest price and see historical trends for instruments such as equities, fixed-income products, derivatives, and currencies.

Legal Entity data

The 2008 market crisis exposed severe gaps in measuring market credit and market risk. Financial institutions are facing a hard challenge to identify the complex corporate structure of the security issuer and other counterparties & entities involved in their business. Institutions must have the ability to roll up, assess, and disclose the aggregate exposure to all the entities across all asset classes and transactions. Legal Entity is the key block of this data which will help the Financial institution to know all the parties with whom they are dealing with and help to manage the risk.

The Regulation rules like The Foreign Account Tax Compliance Act (FATCA), MiFID II will require absolute clear identification of all the entities associated with the security. LEI plays a vital role to perform such due diligence.

EDM workflow

  • Data Acquisition – Data is acquired from leading data providers like Bloomberg, Reuters, IDC, Standards & Poors, etc.
  • Data Processing –Data normalization & transformation rules are applied & validation processes clean the data.
  • Golden Copy creation – Cleaned & validated data is transformed into more trusted Golden Copy data through further processing.
  • Data Maintenance –  Manual intervention if necessary to handle the exceptions that cannot be handled automatically.
  • Distribution/Publishing – Golden Copy data is published to the consumer application like Asset Management, Portfolio Management, Wealth Management, Compliance, Risk & Regulatory applications, other Business Intelligence platform for Analytics.

Importance of efficient EDM system

The fast-changing regulatory & business requirements of the financial industry, poor quality of data, competition demand a high-quality centralized data management system across the firm.

In current market situation, companies must be able to quickly process customer requests, execute trading requests quickly, identify holdings and positions, assess and adjust risk levels, maximize operational efficiency and control, and optimize cost all while implementing regulatory and compliance needs in a timely fashion.

An efficient EDM system enables the business to –

  • Establish a Centralized database management system
  • Reduced manual work
  • Decreased operational risk
  • Lower data sourcing costs
  • Having a better view of data
  • Governance & auditing needs
  • Better overview of risk management
  • Tailor-made user rights
  • Analytics & data-driven decision

Challenges need to overcome

  • Data quality & data accuracy.
  • Siloed data and disparate data across firms making it difficult to have a consolidated view of the risk exposure.
  • Data lineage.
  • Keeping the cost lower in such a fast-changing financial market.
  • Ability to quickly process customer requests, accurately price holdings, assess and adjust risk levels accordingly.
  • The complexity of the latest national and international regulations.

Get Insights Straight Into Your Inbox!

    CATEGORY