By ROBERT T. GARRETT and EMILY RAMSHAW
The Dallas Morning News
Texas' critical computer data at risk
AUSTIN – Concern that state vendor IBM isn't backing up critical computer data extends well beyond the attorney general's office to many other agencies that build roads, pay unemployment benefits and protect the public health.
In the months before a massive computer crash destroyed thousands of the attorney general's confidential documents, more than 10 agencies complained about network breakdowns and server backup problems with the state's newly expanded outsourcing of computer services, records show.
A Dallas Morning News review of a year's worth of state agency report cards for IBM shows several agencies' documents have been in danger.
Two dozen agencies lodged complaints about the contractor, including unexpected server and e-mail outages, corrupted files and a hacking incident revealing security breaches that weren't addressed for months. They also point to severe staffing problems, poor response time for problems, and billing and invoice inaccuracies.
"Granted, all outsourcings have their rough beginnings ... but it hasn't really gotten substantially better over time as we would expect from a world class operation" like IBM, Ed Serna, a top official at the Texas Department of Transportation, said in an interview Monday.
Also Monday, Brian Rawson, the state's chief technology officer in charge of the IBM contract, released a letter in which he said the company "is not meeting expectations" and has been fined $900,000 for failure to complete timely backups as required by its $863 million, seven-year contract.
An IBM spokesman said the company is racing to correct the data-backup problems.
"We take all of this very seriously and we're taking the appropriate steps to resolve any issues," said company official Jeff Tieszen.
Mr. Tieszen said that viewed from other angles, the contract is a success and the state is on track to save $178 million by 2013.
Last week, The News reported that a server malfunction in July destroyed nearly half of eight months' worth of documents belonging to Attorney General Greg Abbott's Medicaid fraud investigators in Tyler. The loss compromised scores of prosecutions.
Texas was an early advocate of outsourcing state government computer work. Few other states have dismantled agencies' IT shops and completely outsourced computer services, though Virginia last year launched an equally ambitious outsourcing program.
Texas' outsourcing of data storage and upgraded technology, security and disaster recovery services was ordered by lawmakers in 2005. It built on a decade-old effort that involved outsourcing services at only a dozen agencies.
Currently, 27 state agencies are in various phases of transition to a new arrangement in which IBM manages their information technology services and equipment purchases. Their IT and financial officers are screaming.
"I don't feel like we're getting the emphasis that we need in customer satisfaction," said Gary Gumbert, the chief technology official at the Health and Human Services Commission. "I'm almost to the point of believing that IBM doesn't understand the term. ... They're almost an anchor in this and we're dragging them along."
IBM's Mr. Tieszen declined to respond.
In preparing budget requests, agency officials in recent months have complained that IBM is charging them more and more money – and they'll need a combined $71 million in the next two-year budget cycle to cover unexpected technology costs and overruns.
Mr. Rawson's department, the Department of Information Resources, says that most of the higher costs are driven by increased demand for computer time and services.
The recent security breaches have brought closer scrutiny of the contract.
Allison Castle, Gov. Rick Perry's spokeswoman, said Monday that the governor is "working with the agencies, DIR and legislators to address the problems and identify a solution."
According to customer satisfaction forms and other records of complaints, network outages and server backup problems have been severe:
•This spring, the transportation department reported that "crucial server backups" were not being performed, and that more than 500 backup jobs had failed. A major server outage occurred during an attempt to replace computer hardware and wasn't repaired for nearly a week. "What used to take the district support staff less than an hour can now take as long as a week [if there are no problems]," TxDOT officials reported.
•At the Department of State Health Services, which oversees the public health network that protects against outbreaks of epidemics, officials in June reported 10 backup failure incidents in the previous nine months. Four backup failures in the previous two months still had not been addressed.
•In April, the Texas Workforce Commission, which pays benefits to and retrains the jobless, reported that nine servers were not backed up for two weeks, resulting "in a loss of agency data." In August, the agency reported that IBM still wasn't regularly backing up critical services, putting the commission at "an unacceptable risk."
•In May, library patrons and academic researchers across the state lost access to the Web site of the Texas State Library and Archives Commission after a hacking incident that brought down the site for two days. State librarians believe the incident probably would not have happened if IBM heeded their requests late last year for software upgrades, said department spokesman Derick Hackett.
•The secretary of state's office has been the contractor's toughest critic, repeatedly giving it customer service scores of 0 or 1, out of a possible 5. IBM has "failed to implement a permanent solution to complete backups successfully," officials wrote in a July report, "placing the agency at risk."
"We believe honesty is the best policy, and the best way to improve service," said Randall Dillard, spokesman for the office.