INFORMATION TECHNOLOGY -Extracting knowledge from data

For many organisations, information is power. The more they know about their customers, their products and their competitors, the more successful they can be. Yet despite a welter of complex computer systems and a PC on every desk, a surprising number of companies find themselves rich in data but poor in knowledge. In other words, they have lots of details on lots of items, but none of it adds up to very much.

One solution is to have another go at collecting and collating vital data by putting it in a so-called “data warehouse”. This is essentially a clearing house for storing all kinds of information which is currently spread around the company in dozens of different systems, and allowing anyone who needs it access. First mooted in the late 1980s, data warehouses have taken off in a big way in the 1990s for a variety of reasons.

“On the technical side, costs of storing data have become very much cheaper in recent years, and there’s new technology around which makes distributing data across different boxes much easier. That kind of client/server environment is very scaleable, plus hardware costs are now coming down very much quicker than before,” points out Charles Cowan, product director for merchandising systems at BACG, which specialises in software for the retail industry.

Developments in IT may make data warehouses more do-able, but it is the revolution in business activities which is the real driving force. In the days when most organisations expected to work in the same way for years on end, having operational computer systems which tackled discrete areas of activity made sense. But now change is the norm, and the company’s existing computer systems are struggling to keep up.

“The technology most people have invested in over the years and the systems they have are not very good at answering the questions people now want to ask. What people are interested in is ‘how are we doing as a business’ and ‘how could we be doing better’. Those two questions cause no end of grief if they are trying to make comparisons, say between one region or product group and another. Doing that over a period comparing last year’s promotion with this year’s is even harder to measure,” explains Peter Weston, business development manager for data warehousing at Informix.

Many service industries and financial institutions have an additional problem: they now want to take a customer-focused view of their activities, but their existing transaction-based systems make that impossible. A bank could have half a dozen different contacts with the same customer, but all will be recorded under different account numbers and held in completely separate systems. In this situation, relationship marketing is a non-starter, and yet it is far cheaper to sell new products to existing customers than it is to find new buyers.

Deloitte & Touche Consulting Group’s recently published third biennial Information Management Survey found that, despite advances such as electronic mail, the Internet, and CD-Rom, 40% of their respondents still have major difficulties in knowing where to look for information. Half of those polled felt too much information generated internally caused significant problems, and 38% of large organisations reported they rarely had the right amount of information to do their jobs properly.

“The premise behind the data warehouse is that the data is organised so access is much quicker and easier and users can do all kinds of slicing and dicing on what’s in there, compared to traditional, classic transactional databases where the computer has to churn through every transaction to come up with a standard report. We found that while the idea sounds great, achieving it in practice was very difficult,” warns Julia Parsons, the report’s editor.

And even the most ardent fans of data warehouses would admit the whole project needs to be approached with caution. In particular, companies have to look closely at why they want a warehouse, what is going to go in it and how they are going to ensure the data they use is accurate and reliable.

“Data warehouses are most successful when they are considered as part of a business change process. If a company finds it needs to compete on its ability to attract a micro-market, that is to identify a very specific customer base it can target with appropriate material, then it needs business information organised in a way which supports that aim. The danger comes if the IT department is the one behind the proposal, because then it’s usually motivated by a desire to get hold of some new technology, rather than part of the business strategy,” cautions James Appleby, a consultant with Druid which has advised several blue-chip clients on data warehouse implementations.

“The other key issue is that companies tend to be fairly departmental by nature, and a data warehouse, by virtue of its nature, is cross-departmental.

So people start losing control of their information – they can no longer massage the figures by putting items in or out of certain months, for example. The political issues shouldn’t be ignored,” he adds.

Business semantics is another common problem, with different departments referring to the same basic concept by a different term. In some cases, departments may have their own particular way of calculating a value which is then referred to by the same name in another department, but calculated completely differently.

“We built a data warehouse for the Royal Navy and found when we pulled the data together that the existing underlying systems had 11 different ways of defining gender. One of the hardest problems about building a data warehouse is getting everyone to decide what they are going to measure and what they are going to call those things,” Weston maintains.

Added to this, much of the data will come from old systems which may not accurately reflect changes in the way the company looks at its information.

Often those systems will have been patched over the years, with programmers colonising spare data fields for their own purposes. One Irish building society found that a quarter of its customers appeared to be 85 years old and all to share the same birthday. Closer examination revealed many users had found a shortcut to the system by inputting 11/11/11 in the date of birth field.

“Companies shouldn’t under-estimate the amount of work they will have to do in terms of data cleansing. The quality of the historical data in their existing operational systems is not usually all it’s cracked up to be,” cautions Mel Earp, director of technical services at Sema Group.

But getting the data into shape will pay dividends. “In applications such as executive information systems, people used summaries of the data, so they were working on broad indicators of what was going on. A data warehouse allows you to have all of the data, and often the devil is in the detail,” explains Gary Smith, managing director of European operations at data warehouse specialists Red Brick Systems.

“A data warehouse is a potential way of getting a much more accurate picture of how the business is performing, and one which suits every user.

It allows a manager to follow his suspicions at a certain point in time, and still get back to the raw data. That compares with the traditional approach of fixed reports and fixed views, where if a manager wants to take a different look at something he has to wait months for the IT department to come up with a new solution,” Earp confirms.

The main benefit of a data warehouse is the ability to analyse and re-analyse sales, customers or accounts in any way which makes sense to the inquirer. Thus a retail organisation might want to look at shrinkage levels in stores according to location, or by product line, or by value of items.

All are possible using a data warehouse. Managers can go back two years later and re-examine trends with the advantage of hindsight.

In some organisations, notably the telecommunications operators and financial institutions, data warehouse implementations have had a significant impact on the problem of customer “churn” – the loss of existing subscribers who defect to a rival service. By analysing the characteristics of all those customers who have left in the past 12 months, such companies can predict those most likely to follow them and offer discounts or other attractions specifically designed to encourage them to stay.

Retail companies are also big data warehouse users, because information on what stock is moving where can have a significant impact on margins.

“People can evaluate the performance of certain lines in certain stores, compared with the store size or some other demographic. It becomes much easier to look at which products are performing as expected and which aren’t, so much better decisions can be made about changes. It’s also possible to start working towards activity based costing,” reports Cowan.

But despite the undoubted potential, all commentators are unanimous on one point: the best data warehouses start small and grow bigger. At least one high street bank is rumoured to have spent u12m on a two-year “big bang” project which has yet to show any results. Small scale or departmental data warehouses, commonly referred to as “data marts” are the first step.

“Take a pilot and use it in a very low-cost start up, to see if you’ve really got a handle on the problem. It’s not necessary to make a multi-million pound investment, and certainly not upfront, while a data warehouse is also not the only way to solve an information problem,” says Pamela Pipe, product marketing manager at Information Builders.

At Informix’s Information Superstore, customers can bring in their own data and spend a five-day session trying out 60 different software products and u5m-worth of hardware from different vendors to see what kind of approach would help them. Weston says that companies such as Pizza Hut, Whitbread and Barclays have used this facility as an opportunity to iron out semantic and cleansing issues to ensure a project starts on a level playing field.

“Companies that are trying to create a more nimble or responsive organisation, provided they know what they want to achieve, can see real benefits within 60 or 90 days of starting a data warehouse,” Appleby says.

Tandem Institute research into Europe’s top 1,000 companies showed that no less than 85% of them have already installed a data warehouse or plan to develop one in the next two years. So, plenty more will be built over the next few years.

Pat Sweet is a freelance journalist.