Gulp.... whew...

Remember the Vyatta router/firewall I was configuring yesterday? Well, the site went live on the beta product yesterday afternoon and moved their mail server in-house (routed via our static NAT).

Last night, late, I realized that I needed to add one more NAT rule to the configuration, but thought I'd better not touch anything until someone was on-site this morning in case the Vyatta box rolled over and played dead. About 9am today I heard that folks were on-site so I went ahead and configured the product remotely from my house and added the NAT rule. The configuration took fine and as I was looking at the various "show" commands to see the NAT rules and statistics, my ssh connection died. Gulp.

Of course, I thought our product had died a horrible death and called someone on-site to reboot the router. They did that. I waited at home. Nothing. Gulp again.

Now you get to picture me rushing through my morning ritual at breakneck pace to get out of the house. Meanwhile, I'm watching the time thinking of when the MX records will expire and mail will start bouncing like a superball on concrete.

I get to the site about 40 minutes later and look at the console of the Vyatta router/firewall. All looks fine... the box is up, running, answering DHCP for the local LAN, and so forth. As it turns out, the T1 for the site went down - it wasn't our box at all. Still, email was bouncing, but it was not Vyatta's fault. At least not this time. Whew...

Did we mention not to run our beta 0.5 release in production yet?

Post-mortem: The T1 came back and the Vyatta box went right back into production...