Corporate Action « FiNETIK – Asia and Latin America

October 5, 2011 • 2:02 am 1

SIBOS Toronto Round Up: LEI, TS2, Standards – A-Team

Unsurprisingly given the host’s recent market positioning, the dominant theme at last week’s Swift user conference Sibos in Toronto was standards, in many different flavours. The one at the top of the list in terms of reference data, however, had to be the legal entity identifier (LEI) and there was certainly no shortage of discussions on the subject (see the list from my preview here). A total of three sessions were dedicated to the topic and the exhibition hall was abuzz with the potential that the LEI holds for the vendor community in terms of revenue generation.

As I noted in my interview with Fabian Vandenreydt, Swift’s head of Securities and Treasury Markets, at the conference last week (see more here), the industry messaging network provider has made reference data a key part of its 2015 strategy and its selection by the Sifma led committee to act as the issuing body for the proposed ISO 17442 standard is a significant element in these endeavours. However, the US Office of Financial Research (OFR) has not made a final decision on whether the Swift, ISO and Depository Trust and Clearing Corporation (DTCC) team will get the gig (see more on which here) and there are many factors to be considered before a new data infrastructure is put in place, not least of which is governance.

Although it was not addressed at length during the three panel discussions on the LEI (in fact, it was only briefly noted), the notion of a privately run, public data utility is something of a challenge in terms of governance. Given that DTCC owned Avox is currently run as a commercial operation with a number of large customers such as Citi, how will the vendor’s technology be deployed as the backbone of a data utility without granting the DTCC unfair advantage over the market? Ditto with Swift, given that it plans to offer “value added” services on top of the basic reference data set provided by the OFR.

This was a subject I raised offline with a number of the DTCC and Swift execs at the show and the response was that it is an issue that is due to be tackled over the coming months. Given the European Commission investigations of players such as Thomson Reuters and S&P’s Cusip Global Services (CGS) on the subject of potential anti-competitive issues regarding reference data, it will be an active participant in this debate. That is, if Europe agrees to go down the utility route.

A lot seems to be predicated on this week’s Financial Stability Board (FSB) discussions in Basel; an event that nearly everybody I spoke to was planning to attend. The hope seems to be that the new global body will make a final recommendation about whether Europe and the rest of the world will adopt the LEI as it has been proposed. Given that the FSB is working on tricky issues such as tackling the shadow banking sector in a coordinated fashion (see more on which here), it seems likely that there will be some pressure to adopt such a standard.

However, whether the FSB has the teeth to be able to get the global regulatory community to listen and get on the same page as each other is another matter entirely. After all, China has already indicated it will be developing its own entity identification standard. How many more can the industry expect? As noted by UBS’ Daniel Maury, who is the global lead for the firm’s Enterprise Client Data Programme (ECDP), during the LEI session there could be 10 or more if regulators don’t agree on one; a development that could prove especially costly for the industry as a whole.

The party that these developments will prove most beneficial too, however, will be the vendor community. Maury admitted that there is no appetite within the investment banking community to build a vast library of cross references to these new standards, hence these firms will turn to vendors for the solution. Thomson Reuters’ announcement on the first day of the conference (see more here) that it has expanded its legal entity data solution is a case in point of vendors scaling their capabilities ahead of the requirements. It follows a similar move by Bloomberg earlier this year and it will certainly not be the last.

Turning away from the LEI for a second, the other main news from the conference from a post-trade perspective was the announcement by the European Central Bank (ECB) that the Target2-Securities (T2S) settlement infrastructure would be delayed by up to another year (see my guide to T2S from back in 2009 here). Rather than launching in September 2014, the pan-European settlement platform will be delayed until an unspecified date in 2015, according to T2S programme board chairman Jean-Michel Godeffroy.

Speaking during a panel debate on the Tuesday of the conference, Godeffroy said the delay was caused by a need for additional user requirements to be taken into account and for user testing with central securities depositories (CSDs) to be extended beyond the originally scheduled nine month period. However, the buzz from the conference and exhibition halls was that given the loss of T2S champions at the ECB Jean-Claude Trichet and Gertrude Tumpel-Gugerell, the central may back away entirely from the project and leave it up to the industry to sort out.

What does all of this mean for the data standardisation space? T2S has been a driver for a lot of work around corporate actions standardisation and, as such, a delay or even a complete reversal will have an impact on these developments, as well as more general data standardisation efforts (see more on which here). The main impact of the T2S developments relate to the fact it would take settlement out of the hands of CSDs and thus result in a complete re-evaluation of their business models and those of all the other players in the securities market active in Europe. Taking this pressure away could therefore have a whole host of consequences.

Of course, a roundup of the Sibos week couldn’t go without a mention of my standards forum panel, during which myself and Bob Masina, head of technology and operations for the Australian Payments Clearing Association (APCA), and Dan Retzer, CTO at corporate actions solution vendor XSP, debated whether ‘standards innovation’ was an oxymoron (see my earlier blog here). Our conclusion was that being innovative with standards is all well and good, but it takes the big players adopting these standards (and thus bringing the rest of the market with them) to make a difference. Standards development is merely the first step.

Source: A-TEAM, 26.09. 2011, Virginie O’Shea

Filed under: Corporate Action, Data Management, Data Vendor, Market Data, Reference Data, Standards

August 29, 2011 • 1:23 am 0

Data Strategy : Consolidate and integrate Reference and Markets Data

A DATA STRATEGY – Framework & Knowledge Management is a presentation to the Data Management Association in the London showing how a clear strategy and tactical implementation can:

consolidated several dozens of internal and external data feeds & legacy system,
streamlined and simplified data operations and
finally implanted a consistent data governance and change management policy, which is based on a solid metadata management.

This applies specifically to reference data, corporate action, financial reporting, client- and market data in the financial services industry.

The project spanning Asia, Europe and the Americas, went life with feeding the first multilingual product within 9 months of approved funding.

Click here for the presentation: Data Strategy for Financial Markets

Source: FiNETIK, 28.10.2008

Filed under: Corporate Action, Data Management, Data Vendor, FiNETIK Articles, Market Data, Reference Data, Standards, Asia, Business Identifyer, Capital Markets, Client Data, Corporate Action, Data Management, Data Strategy, Data Vendors, Exchanges, Identifiers, Market Data, MDM Master Data Management, Meta Data, Reference Data, Risk Management

August 19, 2011 • 1:02 am 1

Data Classifications Define Us

We use classifications in many aspects of our daily lives. In the data world, we use classifications to categorize our business data so we can effectively identify a group of similar items. A well-formed classification enables business users to locate and perform analysis on exactly the subset of data they want. A well thought-out classification enables incorporation of new categories as the environment changes.

Before diving into some guidelines for constructing and testing classifications, let’s start with terminology.

Taxonomy: The science or technique of naming, identifying and classifying. This article explains aspects of a general taxonomy.
Domain: A group of items or objects characterized by a specific feature. The two domains used as examples in this article are financial securities and songs.
Classification (noun): A collection of ordered categories (delineated along one or more features) for a specific domain.
Category: A subset of similar domain objects. A category can be of group of more detailed categories.
Level: All the categories with the same parent category (as in multilevel classifications).

As a simple example, consider a classification of music – you might identify the categories (or genres) as classical, rock, country, pop, jazz, or hip hop. Consider food: Vegetables, legumes, fruit, meat, fish and dairy are all categories of the classification. If we wish to have more specificity of vegetables, we might have root vegetables (e.g., carrots, beets), plant vegetables (e.g., collards, lettuce, peas) and so on. When we include subcategories of vegetables, we are now talking about a multilevel classification.

Classification Benefits

The benefits of classifications are numerous. We refer to classifications when we communicate with colleagues. We name classifications so that systems can label them and attach qualities to them.

Well-documented classifications become part of our language. Classifications improve our communication; they reduce probability of interpretation errors or reporting errors, and provide the ability to differentiate, compare and analyze. From a purely utilitarian standpoint, they enable us to more quickly locate a single data item we really want. A classification enables a tree-based search – for each level we traverse, the closer we get to our desired target domain object.

The larger the universe of data items being classified, the more valuable the classification. A large universe of domain objects lends itself well to a multilevel classification. For a large universe, a single level classification likely has too many domain objects associated with each category, reducing its ability to target a small, targeted group. In financial services, would an asset management firm be happy with classifying all its securities into just one level such as Equities, Fixed Income, Commodities, Currencies, and Derivatives? It is unlikely the firm would b e content with this, because a good-sized asset management firm may keep track of more than 20,000 securities.

Constructing the Classification

When building classifications, ensure all categories apply to the same domain. Getting back to music, categories such as ‘70s or ‘80s songs would not be consistent with genres. However, you could have a separate time-based classification. Both genre and decade would be fair – but separate – classifications.

Here are a few points to consider when building a new classification:

Give the classification a name so it clearly represents its domain.
When considering a category, ask yourself whether it can apply to all objects. If so, it is probably an attribute of the domain object (like the recording technique of a song or whether a security’s income is taxable).
If your new category is Boolean (usually yes or no), it may or may not be a good choice for a classification category. For example, in financial services, IS_TAXABLE is not a good category since it is really an attribute (every domain object has this attribute). However, the Boolean category IS_CONVERTIBLE is acceptable because it only applies to a subset of categories (e.g., fixed income).
Each category should not be defined too narrowly (very few domain objects fall into this category); if so, it may rarely (if ever) get assigned to the domain objects and even more importantly, used rarely in reporting.
Similar to the previous point, ensure some dispersion. If the expected time frame before a classification is first assigned is relatively long (for instance more than one year) it may be better to keep “unused/future” categories on the drawing board. With several categories having very or no few domain objects, you can expect business users to complain that a search using the category is not working. Seeing the category implies there are members.
Stay brutally focused on the domain. Consider another example from financial services: Many firms tend to consider “municipals” as a type of security, putting it alongside other fixed income categories like convertibles, mortgage backed, floating rate notes, etc. But the term “municipal” actually describes the issuer, not the security. When you test the classification (see below), you’ll see it fails the test. However, “municipal” is a very reasonable issuer classification
Socialize it, because feedback will foster consensus. Encourage multiple, frequent reviews to achieve incremental agreement and have several sample domain objects for each category in the classification. This aids understanding by the users of the classification.
As a rule of thumb, define no more than 10 to 12 categories under the level above. More than that suggests a higher-level grouping or level in between may be beneficial. For example, you may introduce a level in between, with three categories, where each spans out to four (which equals the original 12). Consider how often manual assignment of the classification will be made.
Try to select one or two words to name a category if possible, because they will likely be displayed in an application GUI tree or displayed on a report. If you make the category names too long, they’ll be harder to refer to, and users will abbreviate them with acronyms.
In a multilevel classification, create as few “Other” categories as possible and place them at the leaf (lowest) level.
In a multilevel classification, every category should be a formal subset of its parent category. If the new lower category can capture domain objects in any other level above, it is likely not a good candidate (or it may be an attribute of the domain object). An example of this is:

Level 1: Debt
Level 2: Issuer Backed
Level 2: Money Market
Level 3: Commercial Paper

Here is ambiguity because Commercial Paper could arguably fit under both Level 2 categories. One solution is to rename the category from Issuer Backed to something else (like Bonds) which does not overlap with Money Market (or remove it altogether).

A balanced classification means each branch from the root has an equal number of levels (but not necessarily categories). An example of a balanced classification is illustrated in Figure 1.

A non-balanced classification scheme might look like Figure 2.

There is no inherent problem in having a non-balanced classification, as long as your GUI/application can adapt and support variable levels. If, for some reason, you’re constrained by the GUI or application, just clone the levels downward until you have a balanced classification. To balance the unbalanced classification in Figure 2, add the following:

Under Category 1.1.1, add Category 1.1.1.1
Under Category 1.1.2, add Category 1.1.2.1
Under Category 1.1.3, add Category 1.1.3.1

The category “Other” is always problematic. When someone recommends including an “Other,” it may indicate more analysis is required to complete the classification. “Other” can be a crutch. It is too easy to assign “Other” rather than conduct the research to assign the correct category. Reclassifying later (from “Other” to the proper category) may affect reports that were previously run – business users tend to distrust a system or its data when they get differing results in repeated runs.

What if you absolutely need “Other”? Put procedures in place to flag when “Other” is assigned to any domain object so you can reassign it as soon as possible. Often, business users will not include the “Other” category in analysis/reporting – but they may in fact be missing domain objects that they should include. Depending where “Other” is in the classification, it may contain unclassified domain objects from anywhere in the classification.

Data Modeling

To model a multilevel classification it the database, consider creating X tables where X equals number of levels + 1. For example, Figure 3 represents a how the four level classification in the previous section might be modeled.

Table “Class_A_Categories” contains all the category definitions for Classification A, on all levels.
Table “Class A” contains the valid combinations of categories on each level.
Table “Domain_Object” are the objects which take a single classification from table “Class_A.”

A GUI/application can present the categories for selection by starting with ID_Level_1 (and showing the associated Class_A_categories.Desc) then when the user has selected a category, present the associated ID_Level_2 and so on until the leaf category has been selected.

Testing the Classification

Before distributing the classification for comment, find preferably three domain objects that fit into each category and display them in a table. This will give reviewers a concrete understanding of what you mean by each category. It will also give you a chance to verify your category selections.

Seek out edge cases and classify them. Can the domain object arguably fit into two categories? Explain in a short write-up. Fabricate domain objects and step through the decisions at each classification level to arrive at the correct category. This will provide the reviewer with confidence that business rules can be written to ensure accurate and consistent assignment.

Test-drive the category names with the business users. From their point of view, are the category names clear, meaningful, understandable and unambiguous?

Managing Changes

Now that you’ve successfully established and published your classification you will need rules on how to manage it. Basically, this falls under the topic of data governance.

If you have a data steward (but no data governance council), this is the person that should own, or at least sign off on any classification changes.
If you have a data governance council (DGC), the council will own the classification decisions.
Whether the classification assignment is automated or manual (via GUI), ensure all change requests are documented, including justification and examples. Often a little research will show the change request can be accommodated with the existing classification.
If your firm is large enough, record the minutes of the DGC meetings/decisions and post them on a website (like ISO does). Keep metrics about when and how many categories were adjusted. This will be informative for new users of the classification and will reveal trends as to how well the classification is serving its purpose.
Publish upcoming changes to your classification. Future date them so consumers can make appropriate changes to their systems and reports (if required).
Establish guidelines around timing, and include consideration of requests and decisions. If timelines are too long, business users are going to look for other solutions.
One of the best ways of implementing a classification change is through software. Not only will you be changing the assignment logic going forward, you may also have to reassign existing domain objects.
If you have to add categories, perform the same analysis effort and follow the same rules as when you constructed the classification. A sign of a good classification is when several requests for expansion can in fact be accommodated with the existing classification. But before adding, prove to yourself that the domain objects in question cannot be accommodated by existing categories.
Periodically (once a year or quarter, as appropriate for the rate of change in your industry), test the classification with a few of the newer or newly created domain objects to see if they were correctly classified.

One rarely builds or encounters a perfect classification. Over time, classifications need to be modernized and adjusted to stay relevant. Governing how the classification is updated must be established, published and consistent. The classification must be strictly focused on its target domain. Most of all, communicate often during the classification building process, seek feedback and stay flexible to changing conditions which require adjustment or expansion of the classification.

Source: Information Management Newsletters, August 16, 2011 by Mike Lapenna at mike@inti.ca.

Filed under: Corporate Action, Data Management, Market Data, Reference Data, Risk Management, Standards, Data Classification, Data Management, MDM Master Data Management, Meta Data, Reference Data, Risk Management, Standards

August 9, 2011 • 4:52 am 0

Bloomberg Pushes Benefits, Value of Data License New Commercial Model

Bloomberg is redoubling efforts to convince customers of the value of its new pricing model for its Bloomberg Data License service of intraday and end-of-day market and reference data—known as the New Commercial Model (NCM)—which it originally introduced in March, and which could see the cost of Data License increase by between 30 and 100 percent over three years.

The pricing model, which is part of the vendor’s new customer engagement model for enterprise Data License customers, came into effect from the start of June for existing contracts facing renewal and from April 1 for new accounts, according to a letter sent to clients in March by Bloomberg president and chief executive Daniel Doctoroff. However, in recent weeks, sources say the vendor’s sales management team has contacted Data License clients to obtain feedback on the structure of the NCM, and to visit customers in person to re-explain the model.

Although Bloomberg declines to comment on why it was revisiting customers, banks and buy-side firms have criticized the model, which will lead to unbudgeted price rises of up to—and in some cases more than—100 percent. “Originally they gave us a detailed breakdown of every single security license, back-office license, estimated dollar spend, renewal dates and all the instruments that had been consumed on the feed,” says a source at one sell-side firm. “Then in the last two weeks they came back and said they want to re-present this…. Bloomberg is keen to make sure customers understand everything and show that it is not as bad as it first looks.”

Under the old commercial model, customers paid a monthly charge per security, with prices based on six categories of instrument type and three categories of data type—a security master incorporating corporate actions and prices; derived data; and issuer data—plus a sub-category of price-only data. Under the NCM, Bloomberg has retained the monthly charges and the link between prices and data/instrument type, but has replaced existing categories with a greater number of new categories which result in higher fees overall than in the old model. For example, the security master, corporate actions data and prices for a corporate security were previously bundled together for $1.50 per security per month, but are now sold separately for $1.70, $0.50 and $0.75 per security per month, respectively—a total of $2.95 per security per month.

Bloomberg has also expanded the six instrument categories—including a category covering corporate, government, and money market assets; one for municipals; agency pools; collateralized mortgage obligations, commercial mortgage-backed securities, whole loans and asset-backed securities; equity options, futures, warrants, funds indexes and currencies; and economic statistics—to 11 categories, by splitting out different asset types into new, individual categories, such as separate categories for funds, US government and syndicated loans.

Meanwhile, the vendor has divided issuer data into three component categories—credit risk data, fundamentals and estimates—meaning that monthly fees for a corporate security have more than doubled from $2.50 to $6.50 in the NCM. The cost of derived data has risen by up to 50 percent depending on the asset class, while the vendor now charges for accompanying corporate actions data, regardless of whether a corporate action event actually occurred that month. Under the NCM, multiple requests from firms who wish to view the data more than once per month will also now be charged between one and three cents per security per day, depending on the asset class and data type, whereas previously the first multi-request was free.

More Flexible
Bloomberg officials say the new model is intended to provide more flexibility and value, and to allow clients to “only pay for the data that they want and need.” But one market data manager at a European asset manager calls the change a “pure slicing and dicing” exercise, adding that if a business needs to subscribe to all the content, “You get nothing new or extra—you just have to pay a lot more for the same data.”

To soften the impact of the changes for existing clients, Bloomberg’s Data Solutions group will provide enterprise data license consultants to help clients manage their data usage, and is phasing in the increases, so clients renewing their Data License contract this year and early next year will see stepped cost increments, limited to a total increase of no more than 7 percent in the first year and a further 7 percent in the second. Some clients praise this softly-softly approach but are concerned about the impact after that initial two-year period.

“In our peer group, we are sharing knowledge on how much it will impact us. For some, it’s 2 percent, for others it’s 30 or 100 percent, depending on what data you take and how exposed you are to certain services,” says a market data vendor manager at a second European asset manager. “Seven percent in the first year, then another 7 percent in the second is fine, but after that, when it hits you fully—that’s what we’re worrying about.”

In addition to incremental rises, Bloomberg will also offer “optimization,” whereby if a firm has multiple contracts with the vendor across different branches or business units and requests the same data on the same security in the same month via those contracts, then—excluding intraday and derived data—the vendor will only charge between one and three cents for the second request, rather than twice the full price, which it expects to deliver better value for clients.

However, Jean-Pierre Gottdiener, manager at Paris-based consultancy Lucidine Conseil, says firms who have made the biggest efforts so far to reduce costs and administration by consolidating multiple contracts across branches will not be eligible to take advantage of optimization, and will have to pay the most. “If you only have one contract because you have already rationalized your request to Bloomberg, there will be no optimization and you will support nearly the full increase of the prices,” he says. “Some firms have made no optimization on Bloomberg and their increase was only 30 percent, whereas those who have already made an investment to rationalize Bloomberg face a rise of 100 percent.”

Some acknowledge that the vendor’s prices are fair, given that data volumes have increased considerably since the last time the vendor increased prices—more than a decade ago, according to Bloomberg officials—but Gottdiener adds that Bloomberg’s leading position in the market means “the industry is facing a real issue from the policy, and will probably need to find alternative solutions.”

In fact, the NCM has prompted dissatisfied buy- and sell-side firms to reassess their data consumption. Some participants have even said they will look to alternative parties for cheaper data for some parts of the Data License, such as corporate actions, where plenty of alternative providers exist. “Often with Bloomberg, you just absorb the whole universe and pump it everywhere, so it’s good that we now have to look at what data do we use, where we use it, and why,” adds the source at the second asset manager.

Source: Waters Technology 08.08. 2011

Filed under: Corporate Action, Data Management, Data Vendor, Market Data, News, Reference Data, Standards, Bloomberg, Data License, Data Management, Data Strategy, Data Vendors, Market Data, Reference Data

July 13, 2011 • 12:09 am 0

Integration of Histroical Reference Data

Historical data is becoming more crucial to managing risk, but to make it useful, data updates must be reconciled with the moments actual changes in data occurred, says Xenomorph’s Brian Sentance.

There has been much talk recently about integrated data management, as the post-crisis focus on risk management demands a more integrated approach to how the data needed by the business can be managed and accessed within one consistent data framework. Much of the debate has been around how different asset classes are integrated within one system, or how different types of data—such as market and reference data—should be managed together.

However, there has been little discussion on how historical components can be integrated into the data management infrastructure. This will have to change if the needs of regulators, clients, auditors and the business are to be met in the future.

Why is history and historical data becoming more important to data management? There are many reasons. First, data management for risk needs historical data in a way that simply was not necessary for the reference data origins of the industry over a decade ago.

Another reason would be the increasing recognition that market data and reference data need to be more integrated, and that having one without the other limits the extent of the data validation that can be performed. For example, how can terms and conditions data for a bond be fully validated if the security is not valued by a model and prices not compared to the market?

As another example, how many data management staff were overloaded by the “false positives” of price movement exceptions during the highly volatile markets of the financial crisis? I would suggest many organizations would have saved hours of manual effort if the price validation thresholds used could have automatically adjusted to follow levels of market volatility derived from historical price data.

Regulators and other organizations in the financial markets now want to know more of the detail behind the headline risk and valuation reports. The post-crisis need for an increase in the granularity of data should be taken as a given. This is progressing to an extent where external and internal oversight bodies not only want to know what your data is now, but want the ability to see what the data was at the time of market or institutional stress. Put another way, can you easily reproduce all the data used to generate a given report at a specific point in time? Can you also describe how and why this data differs from the data you have today?

“But I already have an audit trail on all my data,” I hear you say. Yes, that is a necessary condition on being able to “rewind the tape” to where you were at a given time, but is that sufficient? An audit trail could be considered as a sparse form of historical “time series” storage for data, but as we all are aware, there are not many pieces of “static” data that do not change over time (corporate events being the main cause behind these kinds of changes). The main issue with audit trail use here is that it can only represent the times when the data value was updated in the database, which is not necessarily the same time as when the data value was valid in the real world.

So for example, for the sovereign, that forces a change in the maturity dates of its issued bonds. You can only capture when your data management team implemented the change in the database, not necessarily when the change was actually made in the market. Hopefully, the two times may turn out to be the same if your data management team is efficient and your data suppliers are accurate and timely. But don’t count on it, and don’t be too surprised if a regulator, client or auditor is displeased with your explanation of what the data represents and why it was changed when it was. We are heading into times where not knowing the data detail beneath the headline numbers is no longer acceptable, and historic storage of any kind of data—not just market data—will necessarily become much more prevalent.

Source: Xenomorph, 13.07.2011

Filed under: Corporate Action, Data Management, Data Vendor, Market Data, Reference Data, Risk Management, Standards, Compliance, Consolidation, Data Management, Data Strategy, Data Vendors, Market Data, Reference Data, Regulation, Risk Management

April 28, 2011 • 12:54 am 0

Why and How to Start a Data Governance Program

The ancient proverb “May you live in interesting times” was really underscored during 2010. In fact, the year offered some great examples of the increasing need for data governance in both the public and private sectors.

According to the Associate Press, he Federal Aviation Administration is missing key information on who owns 119,000 private and commercial aircraft in the U.S. – a gap the agency fears could be exploited by terrorists and drug traffickers.

Bloomberg BusinessWeek reported that TUI Travel, Europe’s largest tour operator, said it accepted the resignation of its CFO after having to restate its 2009 results. The tour operator said it wrote off $185 million after faults in integrating computer systems following a 2007 merger.

And finally, in an interesting paper by Louise Francis and Virginia Prevosto, Data and Disaster: The Role of Data in the Financial Crisis, the authors stated, “Since 2007 a global financial crisis has been unfolding. The crisis was initially caused by defaults on subprime loans, aided and abetted by pools of asset-backed securities and credit derivatives, but corporate defaults, such as that of Lehman Brothers, and outright fraud have also contributed to the crisis. We use the Madoff fraud and the mortgage meltdown as data quality case studies. We conclude that data quality issues have made a significant contribution to the global financial crisis.”

While your company may not have data governance horror stories on this scale, undoubtedly there are plenty of cases of bad decisions, missed revenue opportunities and increased costs due to silos that have sprung up between business units. These may include:

Inefficient processes due to bad quality data.
Systemic problems due to improperly integrated systems passing bad information to one another.
Poorly trained data entry personnel unwittingly entering incomplete data.
Problems complying with government regulations brought on by fragmented or inconsistent information.

Problems like this are remarkably common across the corporate landscape. Data governance maturity models generally have five levels and are modeled on Software Engineering Institute’s Capability Maturity Model. A recent survey sponsored by Kalido found that only 10 percent of organizations have been able to move their data governance programs beyond the lowest two levels of data governance maturity. This shows most companies haven’t gotten very far in their data governance efforts or are just getting started.

So if your company is in this position – increasingly aware of the need for data governance but not very far along the data governance maturity scale – how do you make tangible progress in a short time?

Suggestions For Getting Started

Avoid “analysis paralysis.” Since many organizations don’t know where to start, they stall and won’t do anything.
Make sure everyone realizes that although technology is important, it will at best be an enabler. It’s not a silver bullet. You can’t just implement data governance technology. You’re going to need a comprehensive program involving organizational changes, process redesign and intensive work on information quality (and yes, some technology).
Use one of the many data governance maturity models to baseline your current situation and to describe your desired future state. This will provide a framework for planning your activities and managing everyone’s expectations. Define: a) where you are now, b) where you want to be, and c) how fast you want to get there.
Start to look at project dimensions for master data management, data integration, data quality and data governance over that time period. This should lead to a realistic data governance roadmap.
Don’t start before you receive executive sponsorship and funding for your data governance organization. If you begin without high-level support, you’ll end up with a failed effort that must be explained later.

Spend Time on the Design Before Launching

Each company is different, so your data governance program should be tailored to how your company is structured and aligned with your corporate culture. There are a lot of moving parts in a data governance initiative; working from an understood design is a lot easier than making it up as you go along.

Make sure you plan for (and address in your program) all of the important aspects that come together in a data governance initiative:

Organizational design, including the organizational model, leadership and staffing. Where will the data governance function live and who pays for it?

Executive sponsorship, corporate culture and organizational change management.
Communications strategy. Create an internal campaign to support the data governance program.
Data governance lifecycle, including the design and implementation of new data governance processes.
Information quality. Build or incorporate a data quality function into the data governance program.
Technology. Implement the various types of supporting technologies required to enable a robust data governance program.
This should give you the sense for what is involved in starting a data governance program in a large corporation.

Don’t let challenges dissuade you – in the long run, not tackling data governance is far worse, since your enterprise is at risk of being a 2011 horror story. So dig in and start the hard work of governing your enterprise data. Don’t forget, if it’s done properly, data governance can produce new revenue, cost savings and more efficient processes for your company.

Source: Information Management Magazine, 03/01/2011 by Dan Power

Filed under: Corporate Action, Data Management, Market Data, News, Reference Data, Standards, BI Business Intelligence, Data Management, Data Strategy, Financial Crisis, Fraud, Governance, Meta Data, Reference Data, Risk Management

February 18, 2010 • 5:12 am 0

Managing Corporate　Actions Risk – January 2010 – IRD – Insight Reference Data

Despite industry efforts to reduce financial losses typically associated with corporate actions processing, managing risk remains one of the major challenges for the corporate actions industry. On November 18, Inside Reference Data gathered leading corporate actions professionals in a web forum to discuss what more could be done to help improve the situation.

Source: Insight Reference Data, 29.01.2010

IRD_Jan2010_ManagingCorporate Action_ Report

Filed under: Corporate Action, Data Management, Library, News, Reference Data, Risk Management, Corporate Action, Data Management, Data Vendors, Risk Management, Survey

January 29, 2010 • 11:20 pm 0

UK asset managers lack confidence in reference data quality – survey

Over a third of UK-based asset managers and banks are not confident in the quality of reference data they use to support trading activity, according to a survey from IT services firm Patni.

The survey of 100 company representatives found that 91% of asset managers do not have a single supplier of reference data, with the remainder admitting that they were not sure of their source at all. Respondents say that an average of six per cent of trades fail as a result of poor reference data.

Yet just half of those questioned say they have not considered outsourcing the management of their reference data to a third party, due to fears of a potential loss of control and security breaches. Meanwhile, the overwhelming reason cited for considering outsourcing is the potential for cost savings, followed by higher levels of accuracy.

Philip Filleul, product manager, reference data, Patni, says: “Many buy-side and sell-side firms are now uncomfortably aware of both the time and costs they devote to purchasing, cleansing and distributing reference data, as well as the risks that arise when these tasks are not performed effectively, among them failed trades and lost revenue opportunities.”

“The twin pressures of achieving regulatory compliance and straight-through processing have highlighted substantial redundancy and duplication of effort in the area of reference data management.

“One in ten trades fail on first settlement attempt – and of these, 60 per cent -70 per cent can be attributed to poor data management. “

Research from the Tower Group, which was cited by the report, showed that nearly two thirds of failed trades did so due to inaccurate data.

Source: Finextra, Bobsguide, 29.10.2010

Filed under: Corporate Action, Data Management, Market Data, Reference Data, Risk Management, Standards, Asset Management, Asset Manager, Corporate Action, Data Management, Data Strategy, Market Data, Reference Data, Risk Management, Standards

January 7, 2010 • 11:34 pm 0

Finextra and the FISD partner for Data webcasts in 2010

The Financial Information Services Division (FISD) and Finextra have forged a partnership to deliver a series of video webcasts for market data and risk management professionals worldwide with a focus on real-time data management and delivery, reference data, and standards.

The events will be hosted in-studio and broadcast via real-time or recorded streaming video to an invited audience of financial professionals. Participants will be able to interact with panelists during the live webcasts via real-time Q&As.

The FISD/Finextra partnership builds on Finextra’s established Finextra Live brand of webcast events. Past webcasts have been broadcast from London, Hong Kong and New York. Participants have included senior representatives from HSBC, Nomura, Citi, Morgan Stanely, Societe Generale, Credit Suisse, Bank of America, Barclays and Royal Bank of Scotland. Over 2,500 people working within the financial industry have registered to attend Finextra video webcasts since March 2009.

Tom Davin, managing director of FISD, says “A survey of FISD members revealed that more of them would like to interact with FISD programs and thought leadership via more sophisticated technologies. Our partnership with Finextra aims to provide timely and relevant topics and discussion to our member base and beyond via interactive video content.”

Nick Hastings, managing director of Finextra, says: “This new service will provide the industry with a forum to discuss and learn about key issues affecting global data management via emerging and innovative communication mediums.”

Source:FINEXTRA, 06.01.2010

Filed under: Corporate Action, Data Management, Data Vendor, Market Data, News, Reference Data, Standards, Data Management, Data Strategy, Data Vendors, FINEXTRA, FISD, Market Data, Reference Data, Standards

January 4, 2010 • 2:13 am 0

Data Management 2010: The Ins and Outs

Effective data management will become critical in 2010, as regulators on both sides of the Atlantic scrutinize, identify and try to limit systemic risk. This means firms will have to keep a closer tab on risk within their own shops.

What will that entail? The Enterprise Data Management Council’s latest “What’s In” and “What’s Out” list of 44 items provides a glimpse.

Creating source tagging to eliminate data scrubbing: Firms often rely on multiple vendors to find the best reference, counterparty, corporate action and market data.

To come up with a “golden copy” of the data they must then “scrub” or cleanse the data by reconciling the differences through either an internal or external engine. Wouldn’t it be easier if firms could just tag data at the source so they wouldn’t be faced with “transforming” the data?

Such a scenario is already happening with financial reporting due to the Securities and Exchange Commissions’ requirement for XBRL tagging. The same could soon apply for corporate action notifications if the efforts of XBRL U.S, the U.S. arm of XBRL International, Depository Trust & Clearing Corp. and Swift to push issuers to use XBRL tagging for corporate actions succeed.

Developing business semantics and metadata repositories instead of relying on data definitions and spreadsheets: The definitions, business rules, quality metrics, data relationships, policies and stewardship roles that enable efficient data management can all be captured and maintained through effective use of metadata – data that describes other data.

“Precise business semantics is the foundation of effective data management,” says Michael Atkin, managing director of the EDM Council in New York. The EDM Council has developed a business semantics dictionary to give firms a common data model and dictionary for the information in their securities files.

Such a repository separates the meanings behind the terms used in data fields, from the physical and logical models. Because the repository relies on metadata-based standards, it will help reduce handcoded data mapping and provide information vendors a set of tags that can be used to mark up data at the time the data is originated.

Creating a corporate data culture and understanding of the value of data enterprise-wide rather than aligning data to business lines: Too often firms store data from dozens of internal applications, vendors and counterparties in a number of disconnected databases.

The databases often do not share identical names and definitions. This makes comparing data difficult, if not impossible. Because the databases and applications are tied to individual business lines, data governance is often relegated to a particular business unit. Such a scenario often creates inconsistencies in data which doesn’t bode well for enterprise-wide risk management. Be relentless in reinforcing the importance of sharing data – which in turn means identifying data the same way, throughout the organization.

Assigning data management to Office of Data Management instead of relegating data management to the IT department: What better way to ensure a solid decision-making process behind how data is maintained, distributed and accessed than putting the job in the hands of people who understand it best. This would be a central team of data management experts who can report to C-level executives. Who wants computer geeks – even the best ones –to be in charge when it is business executives who must and will use the data to achieve bottom line objectives?

Helping to create an industry-wide repository instead of relying on disparate data silos and standards at each firm.: “Regulators are being asked to rely on data knowing that counterparties might be mislabeled, subsidiaries segregated differently or specific products defined differently,” says Alan Grody, president of Financial InterGroup, a New York consultancy specializing in risk management. “Non-standard data is adding unnecessary risk and higher costs to each financial firm and the financial industry overall. Aka – its is increasing systemic risk.

Which leads to this New Year’s question: Wouldn’t it be ideal if the financial industry rallied around using common data standards and a single centralized data repository?

Grody says that such a repository, which he calls the Central Counterparty for Data Management (CCDM) will allow firms to throw away their legacy data systems and access a single industry source for fundamental data, such as reference data on securities, counterparty data and corporate actions data. The CCDM would be funded by the “enormous savings” financial firms would realize in using a central database of common business entity identifiers; data attributes; corporate event reporting and consistent terms and conditions.

The European Central Bank has asked the EDM Council, the European Commission, U.S. regulators and lawmakers for its support in creating what it calls a global securities reference data utility. The utility would leverage the data model the European Central Bank uses for its centralized database.

How much would it cost to implement EDM Council’s objectives? Not known. But Atkin says that proving returns on investment and cost containment to justify any new data management initiative may well be passe in 2010. Doing more with less is the new mantra. As athletes might be urged: Just do it. With the teams you’ve got.

Source: Securities Industry News, 29.12.2009

Filed under: Corporate Action, Data Management, Data Vendor, Library, Market Data, News, Reference Data, Risk Management, Standards, Corporate Action, Data Management, Data Strategy, Market Data, Meta Data, Reference Data, Risk Management, Standards

December 18, 2009 • 11:01 pm 0

Ten Trading Technology Trends and Tools for 2010

Despite the continued economic downturn, many buy- and sell-side firms still opened their wallets in the search for best-of-breed technology solutions. In order to decrease latency and increase speed, countless firms both big and small, bulge-bracket and boutique, have upgraded trading platforms, invested in latency management solutions, or set themselves up at co-location facilities.

But the race to have the best technology that will slice latency down to microseconds—and eventually, nanoseconds— is far from over. In interviews with Securities Industry News, industry experts pointed at technology solutions the buy and sells sides are expected to spend their dollars in the New Year.

Networking (both intra- and inter- data center). Growing market data message rates and shrinking latency have made networks a key focus of the sell side, said Kevin McPartland, senior analyst with the Tabb Group. “Upgrades of data center network equipment and purchases of long distance bandwidth will accelerate driven by current bandwidth requirements and future capacity planning,” explained McPartland. “And looking beyond bandwidth and transmission speed, reliability is tremendously important as downtime in today’s market is unacceptable.” The core goal: Reduce the number of hops or other factors that introduce network inefficiencies.

Multi-asset-class platforms. Mutating asset classes is the future – different ways to trade traditional asset classes, going electronic, and new types of listed derivatives and structured products will be the norm, said Lloyd Altman, a senior executive in Accenture’s Capital Markets Industry division. “The multi-asset class problem is really affecting the buy side more than anyone else,” he said. “[There are traditional institutional asset management and hedge funds that are employing multi-asset strategies in order to generate alpha… everyone on the buy side is multi-asset class at this point. The question is will they need to replace what they have with something that’s new, or will they continue to modify what they have—it depends on the nature of how they use technology and whether they view themselves as technologists.”

Commoditizing high-frequency trading. Turnkey high-speed algorithmic trading systems will be a trend in 2010 as more players enter the high frequency trading business, explained Paul Zubulake, senior analyst with Aite Group. “We’re seeing a lot of people leaving large broker dealers and starting up their own small businesses related to trading,” he said. “If you’re a new group and want to start out on your own it’s not that easy to just dive into that business, so what’s happening is there are a few firms out there selling their technology and setting you up to trade… it’s an interesting story for next year.”

Latency management. The quest to squeeze more latency and provide more throughput is still creating opportunities for network, data center, and niche technology providers, said Accenture’s Altman. “It feels at times like squeezing a toothpaste tube to find one more use, and it is asymptotic on the latency front as we approach zero,” added Accenture’s Altman. “Whoever can advertise that they can get their first with the trade wins, and they can charge for that as a service. At some point it will not matter anyone, but we’re not there yet.”

Co-location. “Putting trading systems under the same roof as matching engines “is at the top of our priority list,” said Frederick Scuteri, senior vice president of prime brokerage services at institutional brokerage Cuttone & Co. “We’re seeing a lot of interest in many buy-side firms, especially the black box/high frequency trading shops looking for sponsored access to the different exchanges and alternative trading systems (ATSs). That game itself is a low-latency game, and co-location is a very big component of the success of that business. That’s something we’re full throttle on both with the NYSE and some other vendors and exchanges as well.”

Risk management for sponsored access. This ties in with the whole co-location story, said Feargal O’Sullivan, managing director of high performance messaging with NYSE Technologies, the commercial technology arm of NYSE Euronext. “Risk management for sponsored access is the idea of being able to allow buy side firms to use a broker ID and get access to markets directly without having to go through the broker systems but with the risk management that’s required before you allow them to do that,” explained O’Sullivan, noting that NYSE Technologies offers a risk management gateway. “It’s an additional step of latency that’s required to ensure that traders are not taking unjustified risks and bring the market down.” Added Aite’s Zubulake: “Pre-trade risk management in all asset classes will become a pre-requisite, or regulatory mandate, for trading.”

Central clearing. Over-the-counter, or OTC, products are going to central clearing, which will increase the demand for proper data management, said Zubulake. This is a trend that is already happening, with the Chicago Mercantile Exchange having begun clearing credit default swaps through CME Clearing on Dec. 15. “You’re taking a business that was purely a voice business… and now instead of having a one-to-one trade you’re going to have the trade done on that basis but it will be cleared through the central clearer. There will be multiple counterparties.”

Data loss prevention (DLP) technology. DLP, which is made up of systems that identify, monitor and secure data whether it’s in use, is on the upswing, according to Jim Routh, KPMG’s chief information security officer. Several major vendors including Symantec and McAfee have emerged as leaders in this relatively new market and are currently selling these offering as integrated suites rather than individual products.

Data profiling. Data profiling, which examines data in an existing database, collects statistics and information about that data and determines if it can be used for other purposes, provides a deeper, broader and speedier insight to data analysis than the more traditional approaches. Garry Katz, a senior vice president and information architect at SmithBarney/Citigroup, says this technique is getting increasing play, becoming an “essential tool’’ in trading.

Virtualized solutions. JP Morgan Chase & Co. is currently deploying technologies, which create “virtual desktops” within its network – and even virtual networks within its overall network capacity. The selling points here include reduced support costs, improved security, greater agility and more streamlined application deployment. As a result of its virtualized network, JP Morgan’s data centers will evolve “from application-based silos to unified fabrics that allow for greater agility and utilization while improving the bottom line,” said Cory Shull, VP of investment architecture at JP Morgan, in a statement.

Source: Securities Industries, 17.12.2009

Filed under: Corporate Action, Data Management, Market Data, News, Risk Management, Services, Trading Technology, Asset Management, Clearing and Settlement, Co-Location, Data Management, HFT High Frequency Trader, Low Latency, Pronósticos, Risk Management, Trading Network, Trading System

November 8, 2009 • 1:46 am 2

Asset Management: Data management is top concern

Markus Ruetimann, chief operating officer at Schroders, expects a growing debate about data and the “acceptance of liabilities” over the next year.

“Our clients want us to cover everything. Whether we do things internally or outsource, that is not their concern – they want us to stand firm if something goes wrong and to cover any losses,” says Mr Ruetimann.

“If the outsourced NAV [net asset value] is wrong and this leads to a loss, our outsource provider would compensate us.

“But what happens when our distributors have another 20,000 unit holders? Where the buck stops is something I think we will hear more about next year.”

The comments come as a growing number of investment firms have outsourced a range of non-core functions, such as fund accounting and risk management, in order to shrink their fixed cost base and turn attention towards their main competencies.

According to figures from Beacon Consulting Group, 31 per cent of managers are reviewing their middle offices, while 36 per cent have reviewed them over the past year.

The result is managers now have to cope with multiple feeds of information – such as data on fund accounting, FX rates and benchmarks – being fed back into their businesses from those providers who manage these functions on their behalf.

Mr Ruetimann says the demand for information from both internal fund managers and clients has “quadrupled” over the past year. “Our clients and managers want information on demand – they want to access it faster and across different jurisdictions.

“A lot of that data is held by our third-party providers and sometimes they struggle with the quality and access to that data,” he says.

“What we have learned with our providers is that we need a clear definition of data flows and what information is hosted by us and our third parties.

“Inevitably, there can be some overlap.”

Dan Watkins, head of European operations at JPMorgan Asset Management, agrees the use of data has become a priority for the industry, particularly with regard to client reporting.

“When you outsource some of your non-core functions, there is a multiple amount of data that is taken back in from those providers. “Our top priority is having best-in-class client reporting and performance analysis for our clients, and data plays a vital part in that,” says Mr Watkins.

With a heightened emphasis on industry transparency, managers are also under increased pressure to provide clients with detailed reports on procedures they have in place to manage risk, custody and other essential functions that have been outsourced to third parties. “You need to ensure that the quality of data to produce reports is as good as it can be,” says Mr Watkins.

“In conversations I have with people in the industry, more and more managers are turning their attention in this post-outsourced world towards client reporting – this is where managing that data layer comes in.”

The appetite for quality data was highlighted at an industry conference held by consultancy Investit in July. Asked what the highest priority was for managers, 82 per cent cited data management while client reporting was an important focus for 75 per cent.

Asset managers have identified data management as a top priority for operations in what they call a “post-outsourced” landscape.

Some fear that poor handling of information, including how managers aggregate data fed to them from third parties, could lead to disputes with outsource providers over who is responsible for potential losses.

Dave Francis, head of operations at Gartmore, also recognises the growing trend for improved data flows from third-party vendors. “The quality of data is fundamental,” says Mr Francis.

“We make sure that we have the appropriate performance indicators and service level agreements that give a guide to data integrity,” he says.

“There are monthly review schedules and we watch for signs or trends, and whether there is a need to address it or not. By looking at these indicators, it gives us a good idea of what is going on inside the vendor.”

Source: FT 08.11.2009 By David Ricketts

Filed under: Corporate Action, Data Management, Data Vendor, Market Data, News, Reference Data, Risk Management, Standards, Asset Management, Data Management, Data Strategy, Data Vendors, Fund Industry, FX Foreign Exchange, Market Data, Reference Data, Risk Management

September 20, 2009 • 11:57 pm 0

Corporate Actions Report Sept 2009 – Reference Data Review

Download: Reference Data Review Special Report Corporate Actions 2009 Edition

Rather than detracting attention away from corporate actions automation projects, the financial crisis appears to have accentuated the importance of the vital nature of this data. Financial institutions are more aware than ever before of the impact that inaccurate corporate actions data has on their bottom lines as a result of the increased focus on risk management in the market as a whole.

This renewed focus on the basics of data management has, in turn, spurred on vendors in the space to significantly up their game. The focus of this innovation has been on bringing prices down, making the implementation of these solutions easier and designing more intuitive user interfaces. This has manifested itself in a range of enhancements, not least of which are the deployment of web-based front ends and software as a service (SaaS) models.

Financial institutions and (surprisingly) issuers have also been doing their bit to improve the often complex muddle of corporate actions data via various working groups and standards initiatives. Earlier in 2009, the European issuer community agreed that a framework for shareholder communication and cross border voting is needed in the market. This was then followed by the release of the results of the Corporate Actions Joint Working Group’s standardisation initiative, which is aimed at defining each category of corporate action in the market.

Corporate actions are most certainly back in the spotlight

Filed under: Corporate Action, Data Management, Data Vendor, Library, News, Reference Data, Risk Management, Standards, A-TEAM, Corporate Action, Data Management, Data Strategy, Data Vendors, Reference Data, Standards, XBRL

August 19, 2009 • 1:17 am 0

BNP Paribas Improves Quality and Efficiency Across Silos with Data Assessment Strategy

Paris – BNP Paribas is in the middle of an enterprise-wide reference data assessment initiative as part of a larger program, which aims to increase efficiency, improve data quality and help manage data costs across silos, Inside Reference Data has learned.

The data assessment exercise, started in spring 2009 and expected to deliver savings by the end of the year, has included reviewing vendor contracts.

“We thought it could be interesting to have another view of the contracts we currently have in place,” says Paris-based Andre Kelekis, senior strategist at BNP Paribas, adding that the first area of focus, without directly impacting the systems, was to make an assessment of all vendors’ contracts and sourcing with the aim of optimizing the sourcing in terms of procurement in every area.

The merger with former Fortis Bank in May 2009, now BNP Paribas Fortis, slowed down the assessment procedure, as the revised scope of the efforts now also includes consolidating contracts and reviewing data spend at the Fortis-side of the business.

Yet, the data inventory is being done, and as part of the next phase, the bank plans to optimize the data feeds without modifying applications or the database.

The bank does not currently have a full enterprise data management (EDM) project in place, but as part of the assessment exercise it is paying close attention to the data to ensure high quality and efficiency.

“At this stage we are not claiming to want a full EDM strategy, but we do want to know if we could have a much more efficient organization, and to find out if this is possible and what needs to be done we are paying close attention to the data,” says Kelekis.

In fact, the bank does not plan to re-architecture its data management systems as part of the assessment initiative. “Modifying the infrastructure could take around two to three years,” says Kelekis, explaining that the current efforts are focused on the data itself.

One of the main drivers behind the data assessment was to be able to overcome the data challenges that come hand in hand with a typically siloed organization and be able to evaluate the levels of data quality within silos.

“By construction we are in a siloed company, business has its priorities and it’s not easy to work on data projects … some still think controlling their systems is better than having to rely on sub-parties,” says Kelekis, adding that being able to start data projects, largely transversal in nature, depends largely on the mentality and culture within the organization.

But Kelekis says that at the moment, he sees some silos developing in the right direction. “The fixed-income system, with its rationalized reference data feed enabling data optimization, for example, is advanced … it could even be used as a model for all the other silos,” he says.

The Push for Governance

The bank does not currently have a data governance program at enterprise level in place, but has facilitated data discussions by introducing a market data and reference data steering committee, which unites professionals from all the various silos within the organization to discuss data management within BNP Paribas, while raising awareness of what this means in terms of costs and systems. This group was created in the early 2000s.

“We are not ready at the moment to put in place a data governance program at the enterprise level,” says Kelekis, adding that as long as market and reference data remain difficult to understand at the top management level, it will be complex to find a global sponsor and put in place a governance strategy.

Communication is key, as Kelekis says it is complex to carry out a global-transverse data project without a global sponsor, especially if the silos do not understand the value good-quality data can bring to their operations.

“The steering committee is a means to share information across the different silos, but it is only very efficient when those representing such silos are top management and have decision-taking power,” says Kelekis, adding that if the silos are not represented at a very high level, the main purpose of the committee is just information gathering.

Source: InsideReferenceData, 18.08.2009 by Carla Mangado

Filed under: Corporate Action, Data Management, News, Reference Data, Risk Management, Standards, BNP Paribas, Corporate Action, Data Management, Data Strategy, Merger, Reference Data, Risk Management

June 27, 2009 • 5:04 am 2

Data Quality – Understanding and Managment Commitment

This column will allow you to take the message to management because without, first their understanding and then their commitment, nothing will happen of any significance. I’ve tossed in a couple of points on the cost of poor quality that should capture their attention.

What Is Data Quality?

There are a number of indicators of quality data.

The Data Is Accurate – This means a customer’ name is spelled correctly and the address is correct. If the Marketing Department doesn’t have the correct profile for the customer, Marketing will attempt to sell them the wrong products and present a disorganized image of the organization. When data on a company vehicle is entered into the system, it may be valid (a vehicle number that is in the database), but it may be inaccurate (the wrong vehicle number).
The Data Is Stored According To Its Data Types
The Data Has Integrity – The data will not be accidentally destroyed or altered. Updates will not be lost due to conflicts among concurrent users. Much of this is the responsibility of the DBMS, but proper implementation of the DBMS should not be assumed. Robust backup and recovery procedures as implemented by the installation are needed to maintain some levels of integrity. In addition, operational procedures that restrict a batch update from being run twice are also necessary.
The Data Is Consistent – The form and content of the data should be consistent. This allows for data to be integrated and to be shared by multiple departments across multiple applications and multiple platforms.
The Databases Are Well Designed – A well designed database will perform satisfactorily for its intended applications, it is extendible, and it exploits the integrity capabilities of its DBMS.
The Data Is Not Redundant – In actual practice, no organization has ever totally eliminated redundant data. In most data warehouse implementations, the data warehouse data is partially redundant with operational data. For certain performance reasons, and in some distributed environments, an organization may correctly choose to maintain data in more than one place and also maintain the data in more than one form.

The redundant data to be minimized is the data that has been duplicated for none of the reasons stated above but because:

The creator of the redundant data was unaware of the existence of available data.
The redundant data was created because the availability or performance characteristics of the primary data were unacceptable to the new system. This may be a legitimate reason or it may also be that the performance problem could have been successfully addressed with a new index or a minor tuning effort and that availability could have been improved by better operating procedures.
The owner of the primary data would not allow the new developer to view or update the data.
The lack of control mechanisms for data update indicated the need for a new version of the data.
The lack of security controls dictated the need for a redundant subset of the primary data.

In these cases, redundant data is only the symptom and not the cause of the problem. Only managerial vision, direction, and a robust data strategy would lead to an environment with less redundant data.

The Data Follows Business Rules - As an example, a loan balance may never be a negative number. This rule comes from the business side and IT is required to establish the edits to be sure the rule is not violated.
The Data Corresponds To Established Domains - These domains are specified by the owners or users of the data. The domain would be the set of allowable values or a specified range of values. In a Human Resource System, the domain of sex is limited to “Male” and “Female.” “Biyearly” may be accurate but still not an allowable value.
The Data Is Timely - Timeliness is subjective and can only be determined by the users of the data. The users will specify that monthly, weekly, daily, or real-time data is required. Real-time data is often a requirement of production systems with on-line-transaction processing (OLTP). If monthly is all that is required and monthly is delivered, the data is timely.
The Data Is Well Understood - It does no good to have accurate and timely data if the users don’t know what it means. Naming standards are a necessary (but not sufficient) condition for well-understood data.Data can be documented in the Data Dictionary/Repository, but the creation and validation of the definitions is a time consuming and tedious process. This is, however, time and effort well spent. Without clear definitions and understanding, the organization will exhaust countless hours trying to determine the meaning of their reports or draw incorrect conclusions from the data displayed on the screens.
The Data Is Integrated - An insurance company needs both agent data and policyholder data. These are typically two files, databases, or tables that may have no IT connection. If the data is integrated, meaningful business information can be readily generated from a combination of both the agent and policyholder data. Database integration generally requires the use of a common DBMS. There is an expectation (often unfulfilled) that all applications using the DBMS will be able to easily access any data residing on the DBMS. An integrated database would be accessible from a number of applications. Many different programs in multiple systems could access and, in a controlled manner, update the database.
Database integration requires the knowledge of the characteristics of the data, what the data means, and where the data resides. This information would be kept in the Data Dictionary/Repository.

An integrated database would have the following potential benefits:

Less redundant data
Fewer possibilities for data inconsistency
Fewer interface programs (a major resource consumer)
Fewer problems with timing discrepancies
More timely data

The Data Satisfies The Needs Of The Business – The data has value to the enterprise. High quality data is useless if it’s not the data needed to run the business. Marketing needs data on customers and demographic data, Accounts Payable needs data on vendors and product information.
The User Is Satisfied With The Quality Of The Data And The Information Derived From That Data – While this is a subjective measure, it is, arguably, the most important indicator of all. If the data is of high quality, but the user is still dissatisfied, you or your boss will be out of a job.
The Data Is Complete –
There Are No Duplicate Records - A mailing list would carry a subscriber, potential buyer, or charity benefactor only once. You will only receive one letter that gives you the good news that “You may already be a winner!”
Data Anomalies - From the perspective of IT, this may be the worst type of data contamination. A data anomaly occurs when a data field defined for one purpose is used for another. For example, a currently unused, but defined field is used for some purpose totally unrelated to its original intent. A clever programmer may put a negative value in this field (which is always supposed to be positive) as a switch.

Design Reviews

An important set of information to be included in design reviews is the requisite quality of the data under consideration and the actual state of the data. The basic question to be asked is “How clean, timely, etc. must the data be?” In the design review, the team members would consider the data source, the process of update and delete, and the quality controls imposed on those accessing the data.

The Design Review would review and validate that standards are being followed. The review process may make recommendations to clean up the data, establish strict controls on shared updating, and assure sufficient training for users who would query the data.

Assessment Of Existing Data Quality

As people overestimate the intelligence of their grandchildren and the sweet nature of their dogs, organizations overestimate the quality of their own data. A reality check is generally needed. Poor quality data can be detected in a number of ways:

Programs that abnormally terminate with data exceptions.
Clients who experience errors or anomalies in their reports and transactions and/or don’t trust their reports or don’t trust the data displayed on their screens.
Clients who don’t know or are confused about what the data actually means.
On-line inquiry transactions and reports that are useless because the data is old.
Data that can not be shared across departments due to lack of data integration.
Difficulty for clients to get consolidated reports because the data is not integrated.
Programs that don’t balance.
In the consolidation of two systems, the merged data causes the system to fail.

Quality may be free but data quality does require an initial investment. It takes people and resources to bring data to the desired pristine state. If data is allowed to remain in its current (dirty) state, there may be a substantial cost and disruption to the organization. Very few organizations understand the costs and exposures of poor quality data.

Impact Of Poor Quality Data

Data is an asset but it can only be an asset if the data is of high quality. Data can also be a liability if it is inaccurate, untimely, improperly defined, etc. An organization may be better off not having certain data than having inaccurate data, especially if those relying on the data do not know of its inaccuracy. A hospital would be better off not knowing a patient’s blood type than believing and trusting it to be “O+.”

Which Data Should Be Improved?

It should be obvious that it’s impossible to improve the quality of all the data in an installation. The prioritization is much like triage. The energy should be spent on data where the quality improvement will bring an important benefit to the business. Other criteria that would suggest data improvement is data that can be fixed and kept clean. Unimportant data can be ignored. Data that will become obsolete can also be bypassed. Examples are:

The business will be bought
The data will be converted because of a new application
A reengineering of the business will cause the certain data to be retired

If the Marketing Department is reviewing the demographics of their customers, the zip code (as part of the address) is important while the rest of the address is less critical.

There will be wide variations in the costs to clean different files and databases. This will enter into any decision about which data to purify. The cost of perfectly accurate data may be prohibitive and may not be cost effective. Based on the source of the data, accuracy may also be impossibility.

Users of data may be willing to settle for less than totally accurate data. Even so, it is important that the users know the level of quality they are getting. A greeting card company asked their retailers to measure the number of linear feet devoted to that company’s card products. Those who analyzed the data knew the data to be inaccurate but preferred inaccurate data to no data at all. A large computer manufacturer asked their marketing representatives and technical engineers to report on how they spent their time. It was well known that the respondents were not keeping very good records themselves and their reports reflected the lack of their concern for accuracy. Those who analyzed the data knew of the inaccuracies but were looking for trends and significant changes to indicate shifts in how jobs were being performed. The inaccurate data, in both of these cases, was acceptable.

Purification Process

To clean up the data, the following steps should be followed:

Determine the importance of data quality to the organization.
Assign responsibility for data quality.
Identify the enterprise’s most important data.
Evaluate the quality of the enterprise’s most important data.
Determine users’ and owners’ perception of data quality. – Users will convey their understanding of the data’s quality and will often indicate why the data has problems.
Prioritize which data to purify first.
Assemble and train a team to clean the data.
Select tools to aid in the purification process.
Review data standards.
Incorporate standards in the application development process to ensure that new systems deliver high quality data.
Provide feedback and promote the concept of data quality throughout the organization.

Roles And Responsibilities

The creation and maintenance of quality data is not the sole province of any one department. The responsibility touches Application Developers, Database Administrators, Data Administrators, Quality Assurance, Data Stewards, Internal Auditors, Project Managers, and most importantly, senior management. The importance of quality data must be understood by senior management and expressly communicated throughout the organization. Words are not as important as deeds. When quality measures appear in performance plans, reviews, and bonuses, people finally believe that quality is important. It is equally important that time and resources be allocated to development schedules to support management’s commitment to quality.

Impact Of Data Quality On The Data Warehouse

Bad data should never be allowed into the data warehouse unless the problems are recognized and acknowledged by those who will use the data. Whenever possible, the data should be validated and purified prior to extraction. If bad data enters the data warehouse, it may have the effect of undermining the confidence of those who access the data. Clients and IT must be able to rely on the data, regardless of whether it is detailed, summarized, or derived.The effort to clean up data once it is in the data warehouse becomes a major and never-ending task. It should not be the responsibility of those administering the data warehouse to clean up bad data. The cleanliness standard puts an additional burden on the stewards of the data to perform validations of the source data.

Assessing The Costs Of Poor Quality Data

It will be difficult to assign real dollars to most of these categories. If estimates in real dollars are possible, conservative numbers should always be used. When an organization has experience with any of the following problems and if the costs of fixing those problems have been calculated, those figures can be assigned.

Bad decisions due to incorrect data.
Lost opportunities because the required data was either unavailable or was not credible.
Time and effort to restart and rerun jobs that abnormally terminated due to bad data.
In a buyout situation, accepting too low a price for your business because you cannot properly demonstrate your business potential, or your business seems to be in disarray because your reports are inconsistent.
Fines imposed by regulating authorities for non-compliance or violating a governmental regulation as a result of bad data.
Time and resources to fix a problem identified in an audit.
Hardware, software, and programmer/analyst costs as a result of redundant data.
The costs and repercussions of bad public relations due to bad or inconsistent data. (Ex. A public agency unable to answer questions from the press or from their Board of Directors.)
Time wasted by managers arguing and discussing inconsistent reports which are the result of bad data.
Poor relations with business partners, suppliers, customers, etc. due to overcharging, underpayment, incorrect correspondence, shipping the wrong product, etc.
The time spent correcting inaccurate data. These corrections may be performed by line personnel or by IT.
The costs of lost business in operational systems because of poor quality data (data was wrong or non existent). An example is the lost marketing opportunity for an insurance company that does not have accurate information about a client and thus loses the opportunity to market an appropriate insurance product.

Data Quality Feedback To Senior Management

Unlike measurements of performance and availability, the quality of data will not be changing daily. Quality can, however, be quickly compromised by operating procedures that cause improper batch updates. Those responsible for data will want to make periodic checks to determine trends and progress in improving data quality. The results should be reported to IT management and to the departments that own the data.

The quality of data can be measured, but before any measurement takes place, the following questions should be answered:

Why is the quality of the data being measured? – The classic answer is that without measurement, management of the data is impossible.
What is being measured? – Some possibilities include:
1. trends, i.e. is the data getting cleaner or dirtier?,
2. user satisfaction with the quality of the data
What will be done with the measurements? – Some possibilities include: 1) focus on the data that needs to be purified, 2) provide a basis for cost justifying the purification effort, and 3) give information for prioritizing the cleanup process.

Summary

This column identified various categories of data quality, discussed how to identify data quality problems and how to address those problems. The column gave suggestions for incorporating data quality topics in design reviews. Roles and responsibilities were discussed. Also addressed was data quality as it impacts the data warehouse and the necessity of bringing senior management into the picture.

Data is a critical asset for every enterprise. The quality of the data must be maintained if the enterprise is to make effective use of this most important asset. Improvements in data quality do not just happen; they are the result of a diligent and on-going process of improvement.

Source: EIMInstituted.org, 20.06.2009 by Sid Adelman read full article at Enterprise Information Management Institute

About The Author

Sid Adelman is a principal consultant with Sid Adelman & Associates, an organization specializing in planning and implementing data warehouses, performing data warehouse and BI assessments, and in establishing effective data strategies. His web site is www.sidadelman.com.

Filed under: Corporate Action, Data Management, Data Vendor, Library, Market Data, News, Reference Data, Risk Management, Standards, Corporate Action, Data Management, Governance, Market Data, Reference Data, Risk Management, Standards

FiNETIK – Asia and Latin America – Market News Network