$80-Million Failure : B of A’s Plans for Computer Don’t Add Up

SAN FRANCISCO —  In May, 1986, Bank of America was so confident about the impending success of its pioneering new computer system for trust accounts that dozens of the bank’s most important corporate clients were invited to a lavish two-day demonstration at the Santa Barbara Biltmore Hotel.

Holes were cut in the hotel roof to ventilate the rooms full of computers, and color monitors were in place to show off the bells and whistles of what B of A officials touted as the industry’s most sophisticated technology for handling trust accounts.

The party’s $75,000 tab was minor to officials anticipating the lucrative new business that the system seemed sure to generate when it went on line within the next few weeks.

End of Good Times

“There never has been a meeting that went as well as this from a point of view of inspiring the customers,” said Clyde R. Claus, a 30-year veteran of banking, who organized the session as the executive in charge of B of A’s trust department. “People were trembling with excitement.”

The bash at the Biltmore was the last thing that went right.

Last month, Bank of America acknowledged that it was abandoning the $20-million computer system after wasting another $60 million trying to make it work. The bank will no longer handle processing for its trust division, and the biggest accounts were given to a Boston bank. Top executives, including Claus, have lost their jobs already and an undisclosed number of layoffs are in the works.

If the episode involved only a handful of ruined careers and the millions lost in pursuit of a too-fancy computer system, it would merit an embarrassing but forgetable footnote in Bank of America’s legendary history.

But the story is more important than a simple footnote because it opens a rare window on what author Martin Mayer has dubbed “a decade of decline” at the San Francisco bank, an unprecedented span in which its fortunes plunged from a $643-million profit in 1980 to a $955-million loss in 1987 and the bank fell from largest in the world to No. 29.

Deeper Questions

Further, the total abandonment of a computer system after five years of development and nearly a year of false starts raises questions about the bank’s ability to overcome its technological inadequacy in an era when money is often nothing more than a blip on a computer screen.

A spokesman said last week that Bank of America officials would not respond to questions from The Times about the episode.

“Since last year, we have acknowledged that operational problems existed in our institutional trust business and our energies have been directed toward resolving them as quickly as possible,” the spokesman said in a prepared statement. “We are not interested in fixing blame. . . . We do not believe that it is productive to rehash the situation.”

The widely publicized difficulties surrounding the trust computer problems obscure the fact that the Bank of America was once a technological leader, developing the first big commercial computer in the 1950s and inventing the magnetic ink that allows machines to read the codes on checks.

By the late 1970s, however, under the leadership of A. W. Clausen, the bank was skimping on the spending required to keep up with technological advances. Instead, the money went into greater profits. By the time Clausen relinquished the helm of the parent company, BankAmerica, to Samuel H. Armacost in 1981, the bank had fallen far behind in the computer race.

Armacost launched a $4-billion spending program to push B of A back to the technological forefront. The phrase he liked was “leapfrogging into the 1990s,” and one area that he chose to emphasize was the trust department.

Financial Role

Trust departments serve as custodians and managers of investments for individuals, corporations, unions and government agencies. Investments can be real estate, cash, stocks and bonds. Accounts run from a few thousand to billions of dollars for big pension funds. Banks collect fees for their services, and the amounts can be substantial.

In return, trust departments must provide customers with extensive records and statements to explain their actions and balance the accounts. The reporting is similar to balancing a checkbook for thousands of customers with enormously varied demands.

For instance, a $300-million pension fund might cover six affiliates within the company. The affiliates would share the services of an investment manager, whose trading would be reported to the bank as trustee. The bank must allocate each purchase or sale to the proper affiliate account, keep track of dividends and provide the customer with an ongoing accounting and monthly statements.

Throw in the management of an office building or two and a picture emerges of the complexity of trust accounting.

Developing a computer system that puts all of this information on the computer screens of customers in a microsecond is enormously complex, and it is vital to be competitive.

Traditionally, the field has been dominated by a handful of big Eastern banks, such as Bankers Trust in New York and State Street Bank & Trust in Boston. Although it was the largest trust bank on the West Coast, B of A in the late 1970s was small by comparison and it was mired in a 1960s-vintage accounting and reporting system.

An effort to update the system ended in a $6-million failure in 1981 after the company’s computer engineers worked for more than a year without developing a usable system. So Armacost turned to Claus, who had spent 20 years with New York’s Marine Midland Bank before arriving at B of A in 1977.

Gets Ultimatum

Soon after Claus was named executive vice president in charge of the trust department in 1982, Armacost called him to his office on the 40th floor of the bank’s granite tower in downtown San Francisco to discuss the trust department’s problems.

“Fix it or close it,” Armacost ordered Claus.

Claus soon found that abandoning the business would damage client relationships. Many customers who relied on the bank for trust business as a convenience maintained far larger corporate accounts there. He was equally reluctant to return to the in-house technicians who had produced the stillborn system a year before.

So, Claus and two key executives in data processing, Nicholas Caputo and Thomas Anderson, embarked on a search to find an outside vendor to help develop the new system.

In true Bank of America style, the plan was grand: Create a system to surpass Bankers Trust and State Street and turn B of A into a national power in trust business.

In the fall of 1982, the trust industry held its annual convention in Atlanta. Caputo and Anderson attended, and so did Steven M. Katz, a pioneer in creating software for bank trust departments.

Katz, the computer expert, and Alfred P. West Jr., a marketing specialist, had formed a company called SEI Corp. outside Philadelphia in Wayne, Pa. They had parlayed concepts in Katz’s MBA thesis into a software system used by about 300 small banks in the 1970s.

Katz left in a dispute and, in June, 1980, he founded a rival company, Premier Systems, across the street from SEI in Wayne. Insiders referred to the pavement between the companies as “the DMZ.”

Katz was in Atlanta trying to drum up business. Caputo knew Katz and invited him to meet with the B of A officials at the Hyatt Regency Hotel. By the accounts of people who were there, it was a stormy beginning.

Basis of System

Bank of America’s existing system was based on IBM computers and the bank officials wanted to stick with the familiar hardware. Katz insisted on using Prime Computer, an IBM rival with which he had a long relationship.

There also was a clash on delivery time. According to one participant, Katz boasted that he could put together a system by 1983, and Anderson argued that the promise was ridiculously optimistic. The argument ended the meeting--but did not doom the partnership.

During the next six months, Bank of America and Katz brought together a consortium of banks that agreed to advance Premier money to develop a new, cutting-edge system for trust reporting and accounting.

The other banks, all smaller than B of A, were Seattle-First National Bank (which would later be purchased by BankAmerica), United Virginia Bank (now Crestar), and Philadelphia National Bank. The three smaller banks were using SEI’s system.

Nearly a year was spent on additional research before Claus took the proposal to the bank’s management committee and got the go-ahead to fund the project in March, 1984.

A contract was signed with Premier to provide a system called MasterNet. While the trust business was by far the biggest task, the contract also called for the bank’s technicians to develop eight smaller systems to augment it under the MasterNet umbrella.

While it was not a deadline, the goal was to have the new system, called TrustPlus, in operation by Dec. 31, 1984.

What followed was a textbook structure for designing a computer system.

A committee was formed of representatives from each B of A department that would use the system and they met monthly to discuss their requirements. Data-processing experts from the four banks gathered for a week each month in Pennsylvania to review progress and discuss their needs with the Premier designers.

Shared Risks

“The bank seemed to be doing it right,” a B of A executive involved in the project said. “The risks were shared with other banks. A proven vendor was hired. And all areas of the bank were involved.”

Some of the bank data-processing experts found Katz difficult to deal with occasionally, particularly when they offered views on technical aspects of the project. “Don’t give us the solutions. Just tell us the problems,” Katz often said.

Katz declined to answer questions for this article, saying: “It’s our policy not to talk about individual customer relationships.”

When the ambitious Dec. 31, 1984, goal passed without a system, no one was concerned. There was progress, and those involved were excited about the unfolding system and undaunted by the size of the task.

The immense size of what they confronted is contained in two minor statistics: B of A devoted 20 man-years to testing the software system and its 3.5 million lines of code; 13,000 hours of training, including rigorous testing, were provided to the staff that would run the system.

After 1985 passed without a working system, some team members detected subtle pressures from corporate brass to come up with a return on the bank’s investment, which was approaching $20 million. Customers who had been promised the best system in the world were also concerned.

“Major clients were anxious to get the system and we were real late,” one executive who was involved said. “Some of these people were threatening to leave the bank.”

Claus, the only person connected with the program who would speak for attribution, denied that he had been pressured over costs or timing. He said Thomas A. Cooper, then the president of B of A, told him in 1986: “You’re not getting any pressure from me to do it unless you’re ready.”

That spring, Claus decided the system was about ready. Some smaller parts of MasterNet were already working smoothly in other parts of the bank. Test runs for the trust system had not been perfect, but the technicians thought most bugs could be worked out soon. A demonstration run in Wayne had been successful.

Divergent Opinions

So invitations were mailed for the bash at the Biltmore. Although Claus genuinely thought that the system was about ready, others viewed the party as a means of appeasing anxious customers by giving them a taste.

The taste was good.

“It was a very well-staged function and it really did show the capabilities of a system that had great appeal to us and others attending,” said Derek Rowlett, administrator of the $350-million pension fund of the Directors Guild of America.

The plan was to first bring in the institutional trust customers. Although their accounts were larger, totaling assets of $38 billion, there were only 800 of them. The consumer division, with 8,000 smaller accounts, would be converted later.

But the promise that the bank would soon convert the institutional customers to MasterNet was unfulfilled. Technical bugs kept popping up and the system would not work efficiently enough to handle the conversion.

“There were all kinds of problems,” a former bank official said. “You could be sitting at a terminal and it would take too long to get a response, too long for the screen to pop up. Other times, the whole system crashed.”

The delays put additional pressure on bank employees, many of whom were also operating the old system and working double shifts and weekends to try to get the new system operating too.

“It was an especially heavy burden on the people involved,” one executive who worked on the conversion said.

Late in 1986, Claus received an anonymous letter from someone familiar with the system who warned against a “rush to convert” and told Claus, who was not a computer expert, that people had “pulled the wool” over his eyes.

Memo to Staff

Claus responded with a memo to the staff assuring them that there would be no conversion before it was time.

The three chief components of the system--trust department, systems engineering and the bank’s securities clearance operation--had reported to Claus at the start of the project, which gave him the authority to ensure full cooperation.

By 1986, his authority had been restricted. The systems group and the securities staff had been given their own bosses who did not report directly to Claus. It made obtaining cooperation, particularly from the securities group in Los Angeles, difficult as the pressure to perform increased in 1986.

These pressures, whether spoken or not, were felt by many involved in the project. The bank had reported severe losses in 1986 and efforts were being made throughout the giant company to cut back. Some of the bank’s most profitable businesses were sold and 9,600 jobs were cut.

One who lost his job was Armacost, and with him went his vision of 1990s technology. Clausen was brought back from retirement to run the bank again, and his perception of computers was not enhanced when he reviewed the trust troubles.

The economic cutbacks and Clausen’s reaction made it difficult to justify the continued expense of staffing two trust systems when one was mired in costly troubles.

For several months in late 1986 and early 1987, however, tests of TrustPlus had been running with only a few bugs. “There were still bugs, but the users felt they could run with it and work out the bugs as we went along,” one former executive said.

A conversion date was set: March 2, 1987.

Just as the data-processing staff was rushing to complete work for the conversion, half of the 16-member contingent was pulled off the assignment.

Trust Business Sale

In its push to raise money to offset its losses, B of A had sold its consumer trust business to Wells Fargo for $100 million. B of A was rushing to close the deal by March 31, 1987, so the proceeds could be booked in the first quarter. So, half the data-processing staff was switched to help transfer the accounts to the Wells Fargo system, which was based on SEI software.

On Saturday, Feb. 28, and Sunday, March 1, the remaining staff worked almost nonstop to complete the switch of the institutional trust accounts, which had begun a week before. They pulled it off on that Monday--and it lasted until Saturday, March 7.

That was the day the first of the 24 disk-drive units on the Prime computers blew up, causing the loss of a portion of the database and signaling the beginning of the end. Workers spent a discouraging weekend retrieving data from a backup unit. It was past midnight each night before they left the offices.

Over the next month, at least 14 more of the disk drives blew up. None had malfunctioned in the previous months of tests.

It turned out that the units were part of a faulty batch manufactured by Control Data Corp., a Minneapolis computer firm. But by the time the cause was discovered, delays had mounted and other difficulties had arisen. Taken individually, none would have caused the ensuing disaster. Together, they doomed the system.

“When the stuff hit the fan in the springtime, there was a series of really statistically impossible little disasters that became one big one,” Claus said.

At the precise time the technical team was struggling with these setbacks in April, the bank decided to move the staff from San Francisco to its data-processing headquarters across the bay in Concord, 30 miles away, in another money-saving effort.

For many who had been working under great stress for months, the move became the focus for their frustration. Several key people quit and morale sank as many who remained grumbled.

Los Angeles Troubles

The difficulties were not restricted to San Francisco. The securities clearing operation on the 18th floor of the BankAmerica building in Los Angeles was thrown into disarray by the computer woes and its own unrelated problems.

Securities clearing is critical to a trust operation. It involves reconciling thousands of stock and bond trades daily. At the end of the day, the accounts must balance--each purchase and sale recorded in the proper account and matched to the records from the brokers who actually execute the trades. Stocks, or their equivalent, must be delivered and money accepted.

One of the intended functions of TrustPlus was to both reconcile this activity and ensure that transactions were credited to the proper accounts. When it did not work, the securities group had to rely on records from the outside brokers to settle transactions. The practice, called “blind settling,” is abhorrent to any well-run operation.

The computer problems confirmed the suspicions of the securities people in Los Angeles, many of whom had never thought that TrustPlus would work. But its defenders maintain that the securities operation had unrelated problems that contributed to its difficulties.

The securities people had become reluctant to participate in the design process. When the problems erupted in the spring, Claus no longer had authority over the division, and many thought he was unable to force its cooperation. An outside consultant later told bank employees that some securities work was destroyed and some was simply stuck away in drawers during the critical weeks after the ill-fated conversion.

And some in Los Angeles were less inclined to put up with the demands of the collapsing computer system because in March the bank had announced plans to move the operation to San Francisco, which meant many Los Angeles workers would lose their jobs. Reaction was so strong against the move that the bank put it on hold three months later, but nearly 40 people had left by then.

Temporary Workers

Whatever the complex causes, dozens of highly paid “temporary” workers were brought into the securities group in Los Angeles to straighten out the reconciliation mess at an enormous cost.

In the ensuing months, there were conflicts between the bank staff and the “temps” from Ernst & Whinney and Touche Ross, and there were turf battles among the consulting firms as they jockeyed for the millions of dollars that B of A was paying in an attempt to fix the problem.

The bank’s first public acknowledgement of the problems came in a one-line notice in the earnings report it issued in July, 1987. It said $25 million was being placed in a reserve to cover anticipated losses from problems with MasterNet.

Bank officials assured reporters and clients that the problems would be resolved. But within weeks the bank was quietly seeking a buyer for the entire institutional trust department. The effort was unsuccessful because, an official at a rival bank said, there was not much to buy.

Clausen also ordered an in-house investigation of the debacle, which many staff members viewed as little more than a witch hunt. The result was a “one-copy” report that went only to Clausen. In October, Claus and Louis Mertes, the executive in charge of systems engineering, resigned.

Claus acknowledged that he was leaving over the MasterNet problems and took responsibility for them. Mertes, who had been at the bank only two years, has not spoken publicly about his departure.

Another surprise came in January when the bank announced that an additional $35 million would be reserved to “correct problems” with the system, bringing the total spent on fixing the $20-million system to $60 million.

Period of Decline

By then, the institutional customers were leaving. The number of accounts had dropped from about 800 to around 700 and assets under management had declined to $34 billion from $38 billion.

What the bank did not say was that the decision had been made to abandon the system. But over the next few days, it was disclosed that the bank was shifting 95% of its trust business to Seattle-First National Bank, now a BankAmerica affiliate, which uses an IBM-based system from SEI.

The remaining accounts, deemed too complex for the Seattle bank, were given to State Street Bank in Boston, one of the industry leaders that Bank of America had set out to overtake nearly six years and $80 million ago.

Even the decision to drop the embarrassing program is not immune to criticism, which was summarized by Claus last week.

“A lot of people lay down on the floor and spilled blood over this system, and why they abandoned it now I cannot understand,” he said. “A guy called me this morning out of the blue and said that 95% of it was working very well.”