IBM blames DOE contract when asked why NYC remote school system failed during snowstorm
IBM pointed the finger Wednesday at the city Department of Education for the remote learning system failure that left parents, students and teachers outraged and frustrated during last month’s snowstorm.
Officials for the tech company said its contract with the DOE didn’t cover the bandwidth required to withstand the public school system’s more than 900,000 students and their teachers logging on at the same time.
“It was not a failure of IBM technology,” IBM senior State Executive for New York Vanessa Hunt said, adding it would be a “disservice to not identify the root cause of this event.”
The revelation came during a City Council hearing about the system failure that took place when the department insisted on going remote the day of the February storm.
“The department’s use of ISV (IBM Security verify) was simply not contracted to support the nearly one million students and their teachers logging in at the same time,” Hunt told lawmakers, referring to the tech DOE uses as its remote login and identity management system.
The hearing followed weeks of finger-pointing over the fiasco, including Schools Chancellor David Banks’ claims that “IBM was not ready for primetime.”
“Our contract which was signed in 2019 established a simultaneous use threshold of 400 transactions per second — that’s simply a gauge of how many users can log in at any given time,” Hunt revealed.
“IBM has on multiple occasions increased the capacity of this system, far beyond the contracted levels at no added cost to the DOE to meet the school system’s evolving needs,” she added.
Hunt stressed that IBM has made recommendations to the DOE to amend their current contract — but in the meantime is also working with the department to find immediate solutions.
“I think that if we are going to plan for additional remote learning days where that is the plan, then absolutely not 400 TPS is not adequate. And we would need to move to that auto scaling solution,” she said.
DOE Deputy Chief Information Officer Scott Strickland echoed Hunt’s statement, saying: “The immediate term plan includes the possibility of staggering start times on remote learning day to distribute the peak load on the IBM user authentication service.”
Strickland defended the DOE’s decision not to conduct widespread testing before the remote learning day.
“In the past while we had superintendents test in all of their districts, we did not conduct a load test where all students and staff attempted to log in at the same time,” he said.
“This would have been a substantial undertaking that would have disrupted school and or families and staff time at home. That for more testing is also not an industry standard in terms of performance testing.”
About 100,000 were able to log in before 8am. Meanwhile, between 15,000 and 40,000 users were able to log in every 10 minutes from 8am till 10am totalling more than 660,000 users.
“By the end of the day, over one million students, teachers and staff were able to log into New York City public school systems,” Strickland said before adding, “we are sorry we did not prevent this issue.”
