Working in IT means fat paychecks, opportunities for creative problem-solving and the chance to be a hero every day. Tech pros report great experiences in their daily routines but, like every job, some days can feel like a total nightmare.
In the spirit of Halloween, we asked some hardworking IT pros about their worst days on the job and heard a handful of wild tales, from disappearing equipment to major network hacks. Can you identify with any of these Nightmare IT Stories?
Uprooted, For Nothing
Infrastructure services adviser Andrew Donga’s biggest nightmare started the morning he arrived in Maryland, accepting a promotion as a network engineer for a cybersecurity program he’d been working on.
The move was sudden, but it was a big promotion, and definitely worth the move. “I threw everything I owned in storage, filled my car with my personal essentials, and moved across country on my own dime,” Donga said.
His road trip ended in Maryland around 11 p.m. on a Sunday, the night before he was scheduled to start his new job. After a quick night sleep, he woke up to what felt like a bad dream. “When I arrived to work the next day, I found out the contract I was working on was only funded for three more weeks,” he said. “And my new boss had given his two-week notice that morning.”
On the very first day of his new job, Donga started looking for a new one.
OS Obliteration
Brian Richards, Manager of Network and Information Systems at Medina Central Schools, was doing some contracting work for an area school district when he had what he thought would be a time-saving idea.
“We had some free time coming up and a spare server, so I suggested to one of the techs that maybe we could start toying around with a new server application during our downtime,” he said. That overly eager underling took that to mean “I’m doing that this Friday” and went for it, installing the new server on his own.
What happened next would make any IT pro’s blood run cold.
“The installation completely wiped about 10 production servers,” Richards said. “They booted to ‘no operating system found.’”
The OS obliteration happened two days before the end of Richards’ contract, just shy of his first day as the school’s IT administrator. Instead of celebrating his promotion, he worked like he’d gotten a demotion.
“I spent my last day as the contractor — and my first week as the school’s IT admin — restoring the 10 servers that he blew up.”
Lost in the Pacific
That wasn’t the only time Richards experienced a nightmare managing network and IS systems for a school district.
One summer, he spent weeks waiting for hundreds of replacement computers to arrive. “We had 300 computers we needed to bring in, unbox, image, deploy and inventory with a small IT staff.”
That’s a six-week job, and as the summer wore on, the equipment failed to arrive. He called his reseller and vendor, setting off an exasperating round of the blame-game, and a week before school opened heard these foreboding words: “Your equipment is on a cargo ship somewhere in the Pacific Ocean.”
“The equipment arrived two days before school started,” Richards said. “Needless to say, they were not installed on time.”
In The Belly Of The Beast
Pete Caro, director of the cybersecurity services division at KEYW Corp. in Baltimore, once had an encounter so terrifying it permanently changed the way he finds and extracts cables.
“I was stationed in South Korea with the Army, and one day I was crawling under a raised floor pulling cables,” he said. “I’m on my back about 10 feet from either of the two floor tiles we removed when I feel something run across my chest.”
It was a big, fat rat.
“I pushed the floor tile above me out of the floor so hard it went about four feet in the air,” Caro said. “That was the last time I crawled under the floor to save time pulling cables.”
Waiting for Godot
Waking office nightmares can happen, too. Caro once spent several months evaluating a mesh network product focused on the mechanisms the company had come up with for devices to join the network after it was established.
After the “umpteenth unproductive meeting,” in which the vendor was still trying to figure out what was going on, one of the engineers made a comment about the product not being an issue in the first place.
“We went back over the entire product architecture again and it turned out he was right — there was no way they could establish the network in the first place. Let alone have devices join late,” Caro said.
The scary part wasn’t a ghost or ghoul, but the hellish nightmare of a tedious project that led to nowhere. “Two months of reading the world’s most boring documentation and sitting through meetings, down the drain.”
Hack Attack
Jacob Brady’s IT nightmare started back in 2007, when he replaced a data chief at the Officer Candidate School (OCS) in Quantico, Virginia. He was tasked with keeping more than 500 computers secure and in good working order, and to act as a liaison for repairs like failed hard drives.
“Two days after I started, they said, ‘By the way, you’re also the new webmaster. What coding languages do you know?’”
“I had no webmaster experience, like none, and knew nothing about web coding, Java, HTML, nothing,” he said. The site’s importance added to the pressure: It was the main way parents could get information on their sons and daughters going through the rigorous OCS program.
Brady got down and dirty, reading a 200-page book on FrontPage and trying to retain as much as he could during the quick tutorial he got before his predecessor shipped off on orders to Okinawa, Japan. “The site is a pretty big deal, and it was just thrown at me,” he said.
Brady skated on his middling webmaster skills for about two years, until disaster struck.
“One day I come in and my mailbox is flooded. ‘The website is down.’ I have no clue,” he said. Turns out the Marine Corps’ OCS site had been hacked. In a post-hack assessment, Brady received a 100-page report on all the vulnerabilities that had been lurking inside the OCS website.
“When we got hacked, I had to be able to explain to my boss’s boss’s boss why we got hacked, what the problem was and break it down to him to a language that he could comprehend,” Brady said. “The light was shined on me, and I had to go from zero to hero all because of this hack. It was up to me to get the site back up. All my leadership cared about was me getting the website back up.”
The site was a complete loss, so Brady was assigned to build a new one. He learned a whole new coding language and designed a replacement site in a two-week timeframe.
“It was the biggest nightmare ever.”
This article was originally published in 2016.
