The 4th of July holiday is always more fun when it falls on a weekend. This year the 4th was a Sunday so that meant 3-day weekend for yours truly. But since things rarely go as planned, this one turned into a disaster.
Despite my previous rants, we decided to go with Postini for mail filtering. I made the appropriate MX record change Friday afternoon and planned to check-in at various points over the weekend. My 1st attempt was Friday evening around 6:30. Within 2 minutes of being connected my link failed. This was VERY annoying, but not the end of the world. I had a patch from HP so I drove back to the office, applied the patch, and promptly lost both network cards in the server. A 3-hour call to support ensued at the end of which I still had no network connectivity. The tech said she would have a replacement motherboard sent but that would not happen until Tuesday because of the holiday.
So I decided to buy a pair of PCI-express network cards and install them Saturday morning. I was the 1st one in the CompUSA store at 9am and found exactly what I needed. As I pulled into the office parking lot at 9:45 I was greeted by a ComEd truck. The driver told me the power was out and that he needed me to open the back gate. When we got out to the pole we found a dead bird at the base. We assumed that to be the cause of the outage so Dexter (his real name) replaced the fuse and closed the circuit. Still no power so the problem must be further on up the line. And I’m sure Dex would have found it, too, if not for a line down in the street a couple of towns over that required his immediate attention. So I gave him my cell number and asked him to let me know when he would need me back. I locked-up and took my server home to work on it to the extent that I could.
I called the ComEd service line later on but I couldn’t get any sort of acknowledgement so I decided to return the server, with the two new NIC’s. The lights were still out but a flashlight helped me make the connections. I once again locked-up and drove home. I received a call around 4pm from another ComEd repairman but he didn’t have any update for me. The automated system said repairs should be completed by 5pm but I wasn’t about to hold my breath. When I called again at 5pm the system said repairs should be complete by 10pm. Whatever.
Sunday morning, bright and early, I called ComEd again and this time they had no reported outages for the address in question. So I drove in yet again and was pleasantly surprised to see the light on over the front door! I entered the building and happily turned-off the alarm. I was so thrilled I could finally put my server issue to bed and maybe salvage some part of my holiday weekend. When my SBS server came up I checked its network settings and made sure everything was connected. I was able to access it from my desktop and the whole thing looked OK until I tried to get on the internet. My SBS box handles all the internet traffic so I immediately thought one of the new NIC’s was messed-up. I have another server with a similar configuration and it couldn’t access the ‘net either. That meant a call to AT&T.
If there is a Hell, it is filled with people trying to reach Tech Support at AT&T. We spent a good two hours on the phone, trying to determine the exact point of failure. The initial call was not successful. It wasn’t until I returned to the office an hour later that we were able to isolate the problem: the WIC module in the router had failed. Support created a request to have a replacement sent and I made of point of asking if it would be shipped on Monday.
AT&T’s Tier 1 & Tier 2 was out-sourced to Malaysia (or someplace like it) a long time ago and they don’t know from US holidays. I was assured that it would, in fact, ship on Monday for a Tuesday delivery. I passed that info along to the boss and went about the business of enjoying what was left of my weekend.
Late in the afternoon of the 4th we looked online for fireworks displays in our neck of the woods. We decided to make the trip out to Beecher, IL for their 10pm show. As we drove into town we could see the streets lined with people and their lawn chairs, blankets, etc. Beecher hosts a 4th of July Festival that reminded me a lot of the Carnival that Dolton used to hold every year (before the gangs and the Shaws took over). We found the corner of a side street next to an open parkway and plopped-down in the middle of a dozen-or-so other families. The fireworks show was perfect! We were far enough away that the explosions weren’t deafening, while still close enough for a good, unobstructed view. I think we’ll go back to Beecher next year on the 4th. Only we’ll get there earlier in the evening so we can enjoy their Festival.
The ride home was wonderful: driving along the expressway at 65mph with the top down on a warm, summer night is almost as good as having the boat out on the lake. We stopped at a McDonald’s and ate under the stars. And a good time was had by all.
As previously mentioned, the 4th was on a Sunday this year. For most towns this works out pretty well. For the village of South Holland, this is nothing short of a crisis. There is still a strong Dutch/Hollander influence at work in South Holland so there was no celebration on the 4th. Instead, they held everything on Monday, the 5th. And that included their fireworks display. So we got to enjoy 2 shows over the weekend – nothing wrong with that! In fact, the South Holland show was quite spectacular! We found a small rise in the terrain near the firing line and parked our oversized blanket and non-matching lawn chairs. The show lasted almost 30 minutes and the finale was amazing! Great job South Holland!
Tuesday morning did not go so well. My first task was to verify if AT&T had shipped the part. They hadn’t. I then had to break the news to the boss, who was not happy. My next job was to start working the phone to find anyone with a compatible WIC or router I could get my hands on. I struck pay-dirt with a friend-of-a-friend at a telecom company in Schaumburg. I was able to borrow an older router, complete with T1 WIC, and get it in place by 1:30pm. We were still experiencing some flaky, inconsistent internet issues but we were once again able to send and receive e-mail. Finally, I could take a deep breath, look around, and assess the damage.
Wednesday came and went in a blur. The replacement WIC arrived at 10am but I did not install it until late in the afternoon. It worked as expected so I was able to focus on the internet problems. But by this time it was too late in the day to deal with and since I had a new motherboard on the way from HP I didn’t want to create any new problems.
Thursday also slipped by in a flurry of phone calls to various contacts and vendors, looking for a backup for our troubled T1. The motherboard arrived for the server but that had to wait until after hours. Sometime around 5:30pm I shut down the server and performed the motherboard swap. Windows booted after a couple of false-starts but it refused to become a part of the network. I had left the RealTek NIC’s installed so I could transition smoothly back to the built-in NIC’s. That didn’t work so well. I ended-up having to disable the built-in NIC’s. After that, the server booted with full network access. Our e-mail is moving in and out like nobody’s business and the internet has never worked better. I just wish I knew why.
~Dave