Monday, December 19, 2005

Okay, so...

The aforementioned brief hiatus was not, in fact, all that brief.

All manner of crazy shenanigans at work over the last month and a half, but the site has shipped, things are vaguely stable, and I have about twelve seconds to take a breath before I get full-on into wedding logistics, then the honeymoon, and then the next batch of projects.

I've worked a lot of crazy launches, but this one is for the books. We were supposed to ship at 3 PM on a Sunday, but a few other major projects that we were dependent on were supposed to go out the door first. Those projects were delayed. So we waited. Dinnertime came and went. Bedtime came and went. The bars closed, and still, we waited.

Finally, at about 3AM on Monday, we got the go-ahead. I'd sent half the team home already, figuring that we needed people to be in the next day in case the site blew up. We pushed the code out the door, things looked good, and then we realized that it was completely impossible for users to create new accounts.

About 4:30 AM, I got to call the developer who worked on the account code, who I had sent home to get some sleep, and ask him if he had any ideas about what the problem might be. Miraculously enough he did, even though I'd woken him up and he sounded like he didn't have any particularly good ideas about what his own name was.

We tracked it down to an issue with the load-balancer configuration, and that was as far as we could take it ourselves, so we paged a bunch of sysadmins and network guys and sat on a conference call with them listening to them type and periodically say "Damn, that's not it either." This went on for about five hours, during which I and my boss started to mildly hallucinate.

At 6 AM, the half of the team that I'd sent home came in, and said "You're still here!?!?"

At 9 AM, one of the admins fixed the problem. "Hooray!" we said, ended the conference call, and started packing up to go home. At this point, we realized that the problem wasn't actually fixed. We sat back down heavily and paged the admin again, but he managed to fix the problem for real in about thirty seconds.

And then I went home. And nothing blew up.