RSS feedsInformation technology projects easily go awry because of their complexity and the different groups of people involved in their design, testing and implementation.
Fundamentally, the goals of these groups of people are in conflict. Project managers want to go-live on schedule at all costs, while top management want to stay inside budget. Business users are too busy in their day jobs to devote quality time to system design and user testing, yet will complain the loudest when the new system does not meet their needs. Meanwhile, outside stakeholders, such as customers and the audit committee do not necessarily understand what the fuss is about.
Being independent, internal audit has a central role to play in ensuring each group avoids taking risky shortcuts to achieve their own objectives at the cost of overall project success.
We have been involved in several IT implementations and this article summarises many of the lessons learned during that time. In the most recent example, the company had many legacy systems dating back around 20 years. Some were bespoke applications; some were supported externally with little in-house expertise or documentation; others were not compatible with the latest network operating system. Data was either buried in systems with poor reporting tools available for its extraction, or it resided on spreadsheets, in manual records or peoples’ heads.
When the company decided to replace the old regime with new financial and operational systems, it was an enormous undertaking and internal audit was drafted in to help guarantee successful go-live.
We have identified systematic problems with IT projects and have turned these into lessons learned for future projects.
1. Project scope
Project goals were outlined in project charters of varying quality. They were often vague in respect of project deliverables, the impact of change on existing procedures, predicted system usage and the quantifiable benefits of the new system.
Although business management signed off acceptance of the charters, their involvement in drafting them was suboptimal due to their day-to-day pressures in running the legacy systems.
The resulting imprecise and narrowly-defined charters led to dissatisfaction with the end product by the very people who failed to define what they wanted at the outset. In the worst case, when we stripped out the gobbledygook, the charter merely promised an ‘operable system’.
Lessons:
- The business must be involved at project initiation in defining project
objectives and scope.
- Deliverables should be clearly specified in the charter.
- There must be a distinct task before project closure to match the charter with
actual deliverables.
2. Business model
The new applications either overlaid existing ways of working, or ushered in completely new functionality. Either way, users were forced to change the way they worked. What the businesses discussed in concept, but never fully executed in practice, was how the manual and IT processes would interact to provide seamless data flows end-to-end.
Issues such as local procedures around a system, data primacy rules, interfaces and other data transfer mechanisms, cross-system reporting and common account structures were left out of the pre-implementation design as being ‘too hard’, although in the post-implementation world they became even harder to implement as the account structures had solidified. Project managers typically used the excuse that these issues were “not go-live critical”. Don’t believe it!
Lessons:
- Detailed definitions of ‘to be’ business processes should be drafted,
which flow directly from the scope defined in the charter.
- Before acceptance of new processes, ensure there is proof of concept through
pilot testing.
- Ensure business sign-off for acceptance and ownership of new processes.
- Draft clear documentation of new business processes and their impact on
existing processes.
3. Benefits realisation
The benefits of the new systems were not always thoroughly articulated in the project charters. Often, they were not aligned with deliverables, nor measurable (for example, some stated: “greater ease of use of the system,” which is neither articulate nor measurable).
Some projects identified greater bankable benefits than others because their definition of a benefit was broader. Because of project time constraints, no benchmarks were taken of the operating costs of the legacy systems before they were decommissioned, nor of the new systems when they first went live.
Hence, time series comparison of how costs had reduced as a result of any single implementation was not possible. Overall, it was difficult to match actual cash outlays for implementations against the bankable benefits supposedly accruing.
Lessons:
- Identify benefits at the business case stage of a proposed system –
link these to the project deliverables defined in the charter.
- Benchmark existing running costs (in legacy systems) to compare with on-going
benefits in the new production environment.
- Categorise benefits into ‘bankable’ and ‘non-bankable’ and define what is a
valid benefit.
4. Project management
Projects were supported by project managers and project plans. The plans varied in standards of detail and comprehensiveness: the better the plan, the better the project execution. To illustrate, individual tasks were specified (for example, ‘execute test scripts – five days, or by dd/mm/yy’), but the resources needed to do it were not always specific.
In one case, there was no global view of resource commitment and some staff were over-committed. When staff could not achieve everything at once, it led to a setback in milestone achievement or led to personal angst.
The need for cross-system liaison was not properly anticipated in individual project plans. Eventually, insufficient time was allocated to interface testing and end-to-end data flows, which led to a lot of manual intervention when systems went live.
Lessons:
- Build resource allocation into project plans and get management to
commit those resources.
- Build cross-system forums.
5. Stakeholder involvement
Due to business-as-usual pressures, there was variable engagement between the business users and the project teams.
Though project plans included business activities, their realisation was patchy. There was inadequate business commitment of resources, particularly in user acceptance testing (the ultimate proof of functional fit) and indifference in making decisions before project deadlines.
Directors found it particularly difficult to devote regular quality time to project board meetings. This risked a degree of cynicism among end-users before go-live as preparation for change was not taken as seriously as it should have been.
External consultants are typically used extensively on projects, but they are expensive and often their loyalty lies outside the business. In one case, a project manager left with insufficient notice two weeks before the project went live. With their single-minded focus on go-live, project managers may take decisions about system deliverables or project resources without business ratification, especially if the business takes a hands-off approach to the IT project.
Lessons:
- Enforce involvement by business managers in project implementations.
Ensure promises of loaned staff are honoured.
- Enforce attendance at project board meetings and report non-attendance.
- Ensure sign-off of key project milestones.
- Strengthen the business integration tasks in project plans – such as
communications, user training, documentation and change management.
- Use a project authority matrix to identify a process/system owner, budget
holder, design authority, end-client and project manager.
- Use terms of reference for project consultants to define their work and
authority to commit the company.
- Contractually oblige consultants to see projects through to completion and
knowledge transfer to the business.
6. Project budgets
Project budgets risk being inconsistent in their detail. Hardware costs, inflation and accommodation charges were areas where different projects took different approaches. A ‘requiring project pays’ policy could be applied. Because many project requirements are ad hoc and made at the last minute, this charging system may not always be applied.
Lessons:
- Standardise the basis of budget calculations across projects.
- Ensure budget headings match the key project processes.
- Enforce a robust and consistent ‘requiring project pays’ policy.
7. Go-live acceptance criteria
Deliverables were reviewed by the project boards before go-live. However, not every deliverable carried an associated acceptance criterion. Criteria were not always adhered to. Product deliverables were dropped from scope when it looked like they could not be achieved or were rationalised as non-critical for go live. In such scope changes, there needs to be business sign-off for the decision, impact analysis and contingency arrangements.
Having an issues log is one thing. As one of the go-live acceptance criteria, it needs to be cleared down so that all substantial issues are known to have been addressed.
Where project deliverables had acceptance criteria, these could have been more methodically analysed at project boards before a go-/no-go-live decision. In the best examples, critical success factors were identified and risk prioritised.
Lessons:
- Ensure there is formal business sign-off of deliverables against the
original project scope.
- Ensure all deliverables have acceptance criteria assigned to them.
- Ensure that non-attainment of an acceptance criterion is factored into the
go-/no-go-live decision.
- Enhance the audit trail around go-live decisions: from minuted board papers,
to acceptance criteria being met, to closure of issues and risks logs.
8. Post-go-live support
Project handover at go-live was to support functions that were not always procedurally mature nor sufficiently resourced. In the worst cases, this led to costs being shifted from project to production budgets and issues being resolved with the use of temporary, expensive external resources.
Lessons:
- Ensure that production support is adequately ramped up pre-go-live
and not just activated on go-live day.
9. Documentation standards
It was important that there was evidence of key project deliverables and milestones being signed-off as acceptable to the business. Document version control was sometimes varied and storage control was weak. Documents required by external stakeholders (such as external auditors and ISO9000 certifiers) were not always readily available upon request. In one case, project documents were lost completely in both soft copy and hard copy.
Lessons:
- Standardise project directories.
- Enhance document referencing, authorship, ownership and version control.
- Control document storage, particularly when required by third parties for
review.
- Incorporate the production of key documentation into project plans’ tasks and
go-live acceptance criteria.
Conclusions
Having a clear vision on a project – setting the scope well, sticking to it and referencing back at closure that everything required has been delivered – is the essence of a good project. Yet it is astonishing how often human factors and badly executed short-cuts can warp this straightforward approach.
The temptation in implementations is for the business to see them as IT projects, which they leave to the IT project manager, and for the IT project manager to push through deliverables without sufficient engagement from the business. If this happens, then, after go-live, each will blame the other for the system’s shortcomings.
Being truly independent of both business and the project teams, internal
audit has a role in ensuring that unacceptable risks are not taken and that both
IT and the business are sufficiently and jointly engaged to produce a successful
outcome.
David Thurston has worked in IT audit at KPMG and as internal audit manager in
public and private sector roles.
Chris Kelly FCA has worked as an internal audit consultant at Ernst
& Young and as head of audit in the private sector.
