View Full Version : STATUS: New server up and running

02-27-2016, 12:07 PM
Hi everyone -

As you may have noticed, we had a major server malfunction this week which resulted in some fairly severe data corruption. Our only option to move forward was to set up a brand new server and do a complete system restore from our most recent set of complete backups. Unfortunately those backups were 4 weeks old. This means we've lost all data between roughly January 30th and February 27th.

If you're reading this message, it means you're on our new, upgraded server. You should be able to play normally again from here on out, though you will notice that any points/trophies accrued over the past four weeks are now gone. The good news: the new server is a major step up from our last one, and should provide much, much faster response times and fewer needed maintenance outages.

I sincerely apologize for the interruption and the headaches involved. Its been a rough week, but I'm hoping we're now through the worst of it.

Thanks as always for playing!


02-27-2016, 04:15 PM
Thank you Stephen! And thanks for working straight into the weekend on this.

02-27-2016, 04:35 PM
Wow, it must have been a rough few days. Thanks for fixing it.

02-27-2016, 04:46 PM
bless his heart--we must drive him crazy!

02-27-2016, 06:13 PM
It's great to have things back to normal. There really was not that much disruption, all thing considered, though I am sorry for those who lost records. I for one would be lost without this site!

02-27-2016, 06:25 PM
I have tremendously enjoyed my time spent on this site over the years. Thank you, Stephen, for being genuinely caring about this site and its members. You are the best admin ever, and we are all so fortunate to be in such a great place here.

02-28-2016, 08:54 AM
Thanks so much for gutting it out!

“By "guts" I mean, grace under pressure” Earnest Hemingway

02-28-2016, 05:43 PM
Stephen, thanks for the great support in keeping this and so many other puzzle sites running.

03-01-2016, 07:34 PM
Hi everyone -

Well, obviously, we're not quite past our server issues just yet - sorry about that! I'm working behind the scenes to iron out issues as they become apparent but we're running into a couple of issues I've frankly never even seen before, mostly having to do with newer versions of certain software packages and stricter requirements therein.

I'm optimizing as quickly as I can, and for what its worth I feel like we're making good progress. But it might still be a bumpy couple of days before we're completely out of the woods.

I want to assure everyone that this is my #1 priority, and I'm working non-stop to get things in order.

Thank you for your patience and understanding!


03-01-2016, 07:44 PM
thanks, stephen, for everything!

03-02-2016, 04:16 AM
You are the best, Stephen. Thanks for all your hard work.

03-02-2016, 05:46 AM
Your hard work is definitely appreciated. I'm not going back to paper-and-pencil puzzles yet!

03-03-2016, 07:56 AM
Quick update...

We found a major issue yesterday with a system process on the new server that was creating an unbelievably huge log file (700 GB by the time we caught it). That was fixed about 24 hours ago, and according to our monitoring services we haven't had any downtime issues since then. I'm keeping my fingers crossed that this was the culprit but we'll give it another few days just to be sure.

Thanks everyone!

03-03-2016, 08:00 AM
when i click on recent games, i'm still getting a blank screen that says "file not found."

03-03-2016, 08:02 AM
Hmm.... that page is working for me. Its possible you have a cached version of it stuck in your browser from when the site was buggy - can you try clearing your browser's cache and see if that helps?

03-03-2016, 08:17 AM
Hmm.... that page is working for me. Its possible you have a cached version of it stuck in your browser from when the site was buggy - can you try clearing your browser's cache and see if that helps?

hot diggety--that took care of it--thanks!

03-03-2016, 10:00 AM
Glad to hear it, thanks!

03-08-2016, 06:18 AM
We've been fiddling with various configurations over the past week and at this point I think we've got the server to a point where its running pretty smoothly. My diagnostics are showing excellent response times - just want to check with you guys to see what you're seeing in the real world. Is the site responding well for everyone now (i.e. the past 24 hours)?

03-08-2016, 07:51 AM
Yes, so far, so good, Stephen. I hope you know your hard work doesn't go unnoticed!

03-08-2016, 10:25 AM
I didn't see any problems yesterday, no slow times loading or checking, no error messages. When there are problems, it becomes apparent how dependent we are on this site. Thanks for making it run smoothly again.:)