Paul Venezia wrote:Why San Francisco's network admin went rogue
An inside source reveals details of missteps and misunderstandings in the curious case of Terry Childs, network kidnapper
By Paul Venezia
July 18, 2008
Last Sunday, Terry Childs, a network administrator employed by the City of San Francisco, was arrested and taken into custody, charged with four counts of computer tampering. He remains in jail, held on $5 million bail. News reports have depicted a rogue admin taking a network hostage for reasons unknown, but new information from a source close to the situation presents a different picture.
In posts to my blog, I postulated about what might have occurred. Based on the small amount of public information, I guessed that the situation revolved around the network itself, not the data or the servers. A quote from a city official that Cisco was getting involved seemed to back that up, so I assumed that Childs must have locked down the routers and switches that form the FiberWAN network, and nobody but Childs knew the logins. If this were true, then regaining control over those network components would cause some service disruption, but would hardly constitute the "millions of dollars in damages" that city representatives feared.
Apparently, I wasn’t far off the mark. In response to one of by blog posts, a source with direct knowledge of the City of San Francisco’s IT infrastructure and of Childs himself offered to tell me everything he knew about the situation, under condition that he remain anonymous. I agreed, and within an hour, a long e-mail arrived in my in box, painting a very detailed picture of the events. Based on this information, the case of Terry Childs appears to be much more – and much less – than previously reported.
A man and his network
It seems that Terry Childs is a very intelligent man. According to my source, Childs holds a Cisco Certified Internetwork Expert certification, the highest level of certification offered by Cisco. He has worked in the city’s IT department for five years, and during that time has become simply indispensable.
Although Childs was not the head architect for the city’s FiberWAN network, he is the one -- and only one -- that built the network, and was tasked with handling most of the implementation, including the acquisition, configuration, and installation of all the routers and switches that comprise the network. According to my source's e-mail, his purview extended only to the network and had nothing to do with servers, databases, or applications:
“Terry's area of responsibility was purely network. As far as I know (which admittedly is not very far), he did not work on servers, except maybe VoIP servers, AAA servers, and similar things directly related to the administration of the network. My suspicion is that you are right about how he was 'monitoring e-mail'; it was probably via a sniffer, IPS, or possibly a spam-filtering/antivirus appliance. But that's just conjecture on my part.”
Like many network administrators who work in the rarified air of enterprise network architecture and administration, Childs apparently trusted no one but himself with the details of the network, including routing configuration and login information. Again, from the source's e-mail:
“The routing configuration of the FiberWAN is extremely complex. Probably more so than it ought to be; I sometimes got the feeling that, in order to maintain more centralized control over the routing structure, [Childs] bent some of the rules of MPLS networks and caused problems for himself in terms of maintaining the routing.
“Because the system was so complex (and also because he didn't involve any of the other network engineers in his unit), Terry was the only person who fully understood the FiberWAN configuration. Therefore, to prevent inadvertent disruption of this admittedly critical network, he locked everyone else out. I know most of the networking equipment … does use centralized AAA, but I get the impression he may have configured the FiberWAN equipment for local authentication only.”
Childs' attitude toward other administrators is by no means unusual in the IT industry. This is generally due to the fact that admins who are tasked with constructing and maintaining networks of this size and scope care for them like children, and eventually come to believe that no one else could have the knowledge and skills to touch the delicate configurations that form the heart of the network.
Sole administrator
A key point made in the e-mail is that Childs' managers and coworkers all knew that he was the only person with administrative access to the network. In fact, it was apparently known and accepted in many levels of the San Francisco IT department. Again, quoting from the e-mail:
“This is where it gets tricky for the prosecution, IMO, because the localized authentication, with Terry as sole administrator, has been in place for months, if not years. His coworkers knew it (my coworkers and I were told many times by Terry's coworkers, 'If your request has anything to do with the FiberWAN, it'll have to wait for Terry. He's the only one with access to those routers'). His managers knew it.
"Other network engineers for the other departments of the City knew it. And everyone more or less accepted it.
"No one wanted the thing to come crashing down because some other network admin put a static route in there and caused a black hole; on the other hand, some of us did ask ourselves, 'What if Terry gets hit by a truck?' If a configuration is known and accepted, is that 'tampering'?”
My source appears to believe that Childs' motivation was the antithesis of tampering, and that Childs did everything possible to maintain the integrity of the network, perhaps to a fault:
“He's very controlling of his networks -- especially the FiberWAN. In an MPLS setup, you have 'provider edge' (PE) routers and 'customer edge' (CE) routers. He controlled both PE and CE, even though our department was the customer; we were only allowed to connect our routers to his CE routers, so we had to extend our routing tables into his equipment and vice versa, rather than tunneling our routing through the MPLS system.”
Dedicated engineer
Like so many other high-level network administrators, Childs seems to have taken his job extremely seriously, to the point of arrogance and, perhaps, burnout.
“Terry was very dedicated to his career as an engineer. He is a CCIE (probably the only one in the City government), and spent much of his free time studying and learning more -- the MPLS for the FiberWAN, VoIP some of the departments are rolling out, other new technologies for our 311 and E911 systems, etc. He worked very hard, evenings and weekends in addition to full-time 8-5 work, and rarely took vacations. His classification is 'professional,' so he doesn't earn overtime pay, only comp time -- which like many of us he never really had the opportunity to use. He was on standby more or less 24-7-365; whereas in the private sector, in a company of 20,000 or more employees, you'd expect to find multiple engineers rotating that standby status, I'm pretty sure he was always the guy on call.”
This attitude is, again, not uncommon among high-level IT administrators. Neither is the fact that they tend to eschew what they perceive to be unnecessary questioning and bureaucratic “nonsense.”
“Terry also, obviously, had a terrible relationship with his superiors. I should point out that he's not just a network engineer -- he was the lead network engineer for the entire City. His bosses were all managerial rather than technical, and while the other engineers did not actually report to Terry, they did defer to him in any technical matters. Even the network architect left it to Terry to actually figure out implementation. Terry felt that his direct superior was intrusive, incompetent, and obstructive, and that the managers above him had no real idea of what was going on, and were more interested in office politics than in getting anything done.
"[Childs] complained that they spent more time doing paperwork -- change requests, documentation, etc. -- than actually implementing or fixing anything (a common complaint among engineers, I know). He complained about being overworked (which he was, and which his colleagues are even more now) and that many of his colleagues were incompetent freeloaders (also not entirely without basis).
"You could see him getting red in the face whenever he started talking about his department. And once you were on Terry's bad side (which thankfully I never was), that's where you stayed, and you'd get only the most grudging assistance from him from then on. Whether any of his complaints were valid or not, I can't really say, but I don't think that's as relevant as how Terry felt.”
Keys to the kingdom
If Childs' sole proprietorship of the FiberWAN network was normal operating procedure, how did the tensions between Childs and his managers come to a head? Why was Childs arrested on Sunday? There have been reports that the city’s newly hired head of security may have pushed for Childs to open the FiberWAN doors to other admins. My source doesn’t know for sure, but offers some insight:
“I don't know much about his actions in the last few weeks. It's been a couple of months, at least, since I've even spoken to him, and even then it was probably only in reference to some specific request or ticket. But I can imagine that being the subject of disciplinary action by his supervisors for 'performance' issues would be absolutely infuriating to him. I can imagine that his response would be, 'How can you say my performance is poor when I've been doing what no one else here was willing or able enough to do?'"
If Childs was pressured to give up the keys to the network that he had built and tended for so long, would he go so far as to explicitly prevent anyone else from tinkering with his charge?
“I can imagine that [Childs'] response to a demand to open up authentication to the FiberWAN would be, 'Why? So you can screw it up and bring the City network crashing to a halt?' I can even imagine that, under so much pressure, he'd take steps (deleting or hiding config backups, for instance) to make sure he was the only one in control.”
These tales offer significant insight into what may have occurred between Childs and the FiberWAN network hostage situation. Rather than a case of a rogue administrator attempting to cause damage to the network by locking out other administrators, this may be a case of an overprotective admin who believed he was protecting the network – and by extension, the city – from other administrators whom he considered inferior, and perhaps even dangerous. One important fact seems to be in Childs' favor, if reports that the network has continued to run smoothly since his arrest are true. My source corroborates this.
“As for the impact of [Childs'] actions to the rest of the City, the mayor's statement basically has it right. The network is completely up and running. No servers that I'm aware of are affected. No one has had any downtime (yet). But until they get back into those routers, they can't make any changes. I don't know yet if Terry's lockout applies only to the FiberWAN or also to the other routers, firewalls, switches, etc. in the City network.”
Laying the blame
My source doesn’t appear to harbor any ill will toward Childs for this situation, and even believes that the city may be worse off with Childs out of the picture and that some of the blame should be shouldered by Childs' superiors.
“It's a real shame. The city is losing a good network engineer -- probably the best, technically, that they've ever had. Ultimately he has no one to blame but himself, but it's too bad his superiors weren't better about establishing and enforcing policies about authentication, backups, auditing, cross-training, and separation/rotation of duties.
"You'll note the papers have referred to the new information security manager. It's only been a month or so since the City even had an information security policy, and even that is a bare, unmodified template from CCISDA that's awaiting discussion and alteration by a committee that hasn't been formed yet. (When I asked Terry if we could get a copy of the City's network security policy some months ago, he told me, 'I've been trying to get them to approve one for years. I've written ones up and submitted them, but they don't want to do it, because they don't want to be held to it.')”
He also points out that by forcing the issue, the city may have significantly reduced its ability to use and control its own network.
“The one impact they haven't mentioned is that Terry was one of only two engineers assigned to special projects and to do major routing changes and perimeter firewall configuration. The service level, even after they regain control of the network, is going to be way down, until they can fill his mighty big shoes.”
My source had many good things to say about Childs, but did not shy from negative comments, noting that Childs has a bad temper and can be very defensive.
“As for Terry's character, I can imagine this happening. He takes great personal and professional pride in his work -- to a fault. He can be very defensive if someone suggests there's something wrong with the way his network is set up, and that's been a problem for us (as his customer) a couple of times. Terry has a bad temper.
"He's the sort of person who, while his bile is up, won't budge an inch – and then will call you a couple of hours later and acknowledge that maybe your suggestion was right, after all, or maybe here's an even better way to handle things.”
The inner sanctum
Later in the e-mail, my source offered some insight into what may be at the core of the issue: Childs was so paranoid about the security of the network that he even refused to write router and switch configs to flash, which would mean that if the device was powered off, all configurations would be lost.
“At one point he was concerned about the security of the FiberWAN routers in remote offices, so he had them set up without saving the config to flash. 'If they go down, I'll get alerted, and connect up to them and reload the config.' Great, except we have power outages all the time in this city, some of those devices aren't on UPSes, and what happens if you're on vacation? And what about the 15 to 60 minutes it might take you to connect up and reload? He eventually conceded and (ahem) decided that disabling password recovery was sufficient security.”
If Childs did this with some or all of the switches and routers comprising the FiberWAN network, then password recovery without significant network disruption becomes a bigger problem. Without firsthand knowledge of the state of those routers and switches, there’s no good way to know, unfortunately.
If the details given to me in this e-mail are accurate, it would appear that this case is not nearly what it seemed originally. Perhaps it comes with the pressure and responsibility of the job, or the belief that the network they’ve built is simply too complex for mere mortals to comprehend, but it’s not uncommon for highly skilled network administrators to become overprotective of their networks, or for networks of significant size to become an extension of the person who built them.
It certainly appears that Terry Childs believed San Francisco’s FiberWAN network was his baby, and that by refusing to allow others to access the inner sanctum was in the best interests of the city, the citizens, and perhaps most importantly, himself.
