The last couple weeks were a complete nightmare. To recap – Exchange took a shit on me. One of the databases was creating insane log files. I watched it go from 4GB free to 0 in about ten minutes. Even though I was able to add 200GB in HyperV, it took hours to boot. Here’s what I learned:
- According to a bunch of articles I found re: Server 2012/Hyper V and “stuck at”, the hours long reboot was probably due to a bad update. It was stuck on “Do Not Shut Off”. Be patient. Don’t panic and try and bounce it again. Let the damn thing start. Explorer will eat shit and take forever to load everything, but it will all be up again in about two hours.
- Don’t mount the shitty database again if you can avoid it. Fortunately I only had about ten mailboxes, half of which were really, REALLY non-essential employees. I left the database dismounted, disabled the original mailboxes, recreated new ones and then restored them from .pst using Stellar Phoenix. Download it free first to make sure it will actually pull up the mailboxes. It’s around $300 to purchase. It’s not perfect, but it recovered enough for me to at least get people going and to buy me enough time to sort out the details.
Oh, and coincidentally, right after this, I had two client domains that we stopped receiving emails from. The two things they had in common were Office365 and spam protection. They were receiving different NDRs, making it really crappy to try and pin down. Noticed they never hit my internal spam filter or Exchange. After fiddling with TLS on Exchange and my WatchGuard, I took a shot at increasing the stupid WatchGuard’s incoming SMTP proxy nonsense….specifically the maximum header size, which is set to the default value of 20000 bytes. After increasing it to 80000 the emails all started coming in. Office365 must have noodled with something tiny that week, because the header sizes are all around 24000 bytes – just over the default and enough to block them at the firewall.