View Full Version : Server issues and warnings

06-08-2012, 06:46 PM
Hi everyone -

Sorry about the server downtime yesterday - there was a transformer explosion which blew both primary and secondary power to our data center in Las Vegas. Our web hosts say the issue is now under control.

Unfortunately, we're not quite out of the woods just yet. As luck would have it, we're also in the middle of migrating the Cryptograms.org box to a brand new server. The new server will be significantly faster and better equipped than our current one, and will be housed in an entirely separate data center.

The new server is being set up today/tomorrow, and once its ready we're going to copy all content over to it.

What this means in practical terms... basically, at some point today or tomorrow, we'll be taking a "snapshot" of the database on the current server. Any games played on the old (current) server after that point will essentially be lost - and different players will start to see the new server at different times, depending on their webhosts and DNS settings. This also means there is potential for games to be marked as unsolved/failures, i.e. if you happen to start the game at say 6:00pm, our data dump occurs at 6:02 pm, and you finish the game at 6:04pm.

So if you're the type of player who is fanatical about playing percentages and maintaining a 100% solve rate, I highly recommend you log out of your account and play only as a guest until the entire migration is completed. It may be 2-3 days or even longer before you will start accessing the new server, though for most people it will take just a few hours.

If, on the other hand, you just play for fun and don't care much about the statistics - it's perfectly fine to continue playing as you normally do. The worst that will happen is that you may lose a day or two of solves, and perhaps accrue one or two "missed" puzzles in your solving percentage.

I will be adding a message to the old (current) server on every page that will indicate you're on the old server. When that message disappears, you'll know you're seeing the new (upgraded) server, and at that point it will be safe for you to log back in and resume normal play.

Apologies for the hassles, especially after yesterday's outages - but our server has unfortunately reached the end of its usable life span. We're at a point where it is starting to experience periodic slowdowns and failures due to increased site demands. The new server should alleviate this and everyone should experience a much faster and responsive site when this is all over.

Thank you for your patience!


06-08-2012, 07:35 PM
Thanks and a tip o the hat


Live Fanatically
Love Fanatically
Laugh Fanatically
and want 100% at Cryptograms FANATICALLY

06-08-2012, 08:38 PM
Thanks for the warning. I didn't get a chance to play yesterday so I lucked out and missed the crash. In the interest of maintaining my 100% I think I'll go mess around on facebook for a bit. Cheers.

06-09-2012, 08:46 PM
Really appreciate how well you re keeping us informed, Stephen.


So basically the last server was just a move and now we have
a brand new one?


06-10-2012, 03:02 AM
So the good news is that the site is becoming so much more active that we need a better server! Yes! Good job, everyone. Thanks Stephen.

06-10-2012, 03:07 AM
Quick update - the data migration is taking longer than expected, latest estimate is that it will be completed sometime tomorrow morning. There's some other steps I need to take care of before I can flip the switch, so if all goes as planned, I'd expect to start changing DNS sometime tomorrow afternoon/evening.

Sorry this is taking so long...

06-11-2012, 06:00 AM
New server is up! DNS has just been switched - will likely take a few hours for most people to see the new site, but a few are already getting it. If you see this message, this is on the OLD SERVER ONLY. (New server will have the warning message wiped from all non-forum pages.)

Thx for your patience!

06-11-2012, 06:06 AM
Thanks Stephen. I'll give it a go in the morning.

06-11-2012, 07:25 AM
Hi everyone -

Seems my earlier message got lost in the confusion but here it is again.... the new server just experienced a catastrophic failure. I've pushed back all the DNS changes so that everyone should be pointing back to the old server for now. The bad news is that this means we are going to need to start from scratch on yet another new server and migration. The good news, if there is any, is that this happened now, and not a month from now or later, when everyone would have been actively using the system and we would have had severe data loss as a result.

I'm going to work to get the new, new server up as quickly as possible, but it will likely require another couple of days. The same rules apply if you're obsessive on percentage completions, it's best not to play logged in until all of this is sorted.

06-11-2012, 07:20 PM
Hands Stephen some chicken wire and chewing gum, some Valium, a large bottle of
whatever beverage gets ya thru...

Hang in there little buddy. Sure hope ya had a warranty on that last one.

Thanks for keeping us up to date and thanks for all the work you re doing and
you should take the censor off and just let it out here.
We wont mind, well Lily and I wont thats for sure.

Im trying to lighten the mood. Not helping much I know.

But srsly thanks and good luck and if you need any choice swear words
you know where to ask.

06-12-2012, 04:15 PM
If you're reading this message on the website (not via email), you're on the new, NEW server. Will probably take a few hours before most people start seeing it.

06-13-2012, 12:16 AM
Okay, I saw no message when I came to the site so I logged on. I was logged off in the middle of a game and now I'm seeing the message about going to a new server shortly, again. Even though I logged backed on I can't find the puzzle I was solving when I was kicked off. The tequila offer still stands and there is also some gin, rum and triple sec.

06-13-2012, 12:17 AM
Now the message is gone and I was able to find and finish that puzzle. Thanks.

06-13-2012, 12:21 AM
This can happen if your computer is still caching old DNS data - a few things that might help:

- rebooting your computer
- clearing your browser's cache
- manually flushing your local dns
- try using a different browser

..... but in the main, this sort of thing will resolve itself after a few hours. Unfortunately there's nothing I can do on my end to alleviate it. Sorry about that.

06-13-2012, 12:22 AM
How to flush your DNS:


06-13-2012, 12:41 AM
I think it worked!. Thanks, Stephen.

06-13-2012, 01:59 AM
Okay, I was logged off for an hour or so and logged back on about a half hour ago. I was playing on the old server then lost my puzzle and I was on the new server. It seems I've lost somewhere between 30 and 35000 total points. I happened to write down my points earlier today. Will this correct itself or am I stuck.:(

06-13-2012, 02:33 AM
Hi damsel22 -

Unfortunately most everyone's lost some points in the transfer, there's no way to avoid it, at least without shutting down the entire site during the 3-4 days it took to migrate everything over.

But now that the new box is up there should be no further problems.

06-13-2012, 02:40 AM
okay, thanks for all your hard work, Stephen.

06-13-2012, 02:45 AM
I lost most of the puzzles I played yesterday, too. I had kind of figured there was a 50-50 chance of that happening. Whatever. I do love the points and contests, but I also love to learn about new quotes and authors. That's why I've been playing logged-out all week. Thanks for the site, Stephen.

06-13-2012, 03:15 AM
I think I was fine up until just recently. I may have lost a puzzle here or there, but that was about it. It was like in a few seconds most of today's puzzles were gone. Oh, well, I rolled the dice and lost. I still really enjoy the site and playing the games anyway.

06-13-2012, 04:18 AM

I'm fairly new to the site, but I want to thank you, also. The puzzles seem to load much faster. I also lost at least one day's points and had the puzzle I had just completed charged as a non-completion. However I am in it for the sport and play catch and release. Thanks again for your dedication over this last harrowing weekend.

Lily H
06-13-2012, 04:59 AM
Thank you, Stephen, for all your hard work the past several days. The new server seems to working just fine. I was able to access the site from my favorites just like I usually do. The only thing is that my points from last night didn't carry over, but I'm okay with that because at least I still have my 100% solve rate.

06-13-2012, 05:10 AM
Thanks everyone! It's been a bit more brutal than previous migrations, but hopefully we'll get some much better performance as a result.

So far the server has performed even better than I'd hoped... whereas we'd normally be maxing out at around 75-85% resource usage on an average Tuesday night, this new server is barely even registering a load of 5-10%. This should translate to faster load times and fewer failed page loads for everyone. It also gives us some breathing room for adding new features down the line.

Happy solving!


06-13-2012, 05:29 AM
That is impressive, Stephen!

06-13-2012, 05:34 AM
I'm still waffling between servers if I put my computer to sleep and come back. I hope it's just my computer having a hissy fit today, because it is taking a reboot, cache dump, and Safari reset to find the new server every time I come back.

Anyone have any idea why a Mac is being this difficult?

06-13-2012, 03:25 PM
Hi Lurker -

Some systems can be fussy for up to 24-48 hours after a DNS switch and switch back and forth a few times before the new info becomes permanent... hopefully today it will be better for you.

06-13-2012, 06:12 PM

06-14-2012, 04:43 AM
I've never made a complaint about the general operations of this site before, and everyone else's complaints seemed like trivialities beneath my concern. A new problem has arisen that I find greatly annoying, and I feel it will have an averse effect on people's solution times. Before the change, if you highlighted an incorrect letter, you could type the new letter over it and the incorrect letter would disappear. Now, it is necessary to highlight the incorrect letter and delete it before typing over it. Keyboards may be mostly in a neatly ordered QWERTY, but the delete key can appear in various places. The aughties are over and we're in the '10s. This is so unwieldy, I feel like I'm using a computer in the '70s again. I want innovation in computers, not nostalgia.

(is this secretly bansaisequoia on prom night???)

06-14-2012, 04:49 AM
Hi bansai -

None of that code has been changed in over 2 years... and I'm not seeing that on my system. What browser/OS are you using? I'll try my best to replicate it on my end.

06-14-2012, 04:51 AM
Bansai, that problem isn't happening to me, and I think we're on similar computers and browsers. Mac Leopard & Safari for you too?

06-14-2012, 05:12 AM
Just discovered the problem, Stephen. It seems to be unique to the Comodo Dragon browser, but I never had this problem before the change. I just used Maxthon and Sea Monkey, and they're working just fine. Sorry to jump the gun.


06-17-2012, 11:29 AM
Bansai, Your speed in solving cryptograms without any known letter astonishes me. What is your i.q., and what else is your secret? Maggiemouse

06-18-2012, 08:57 AM
If I were to tell my IQ here in the forum, I would feel like a certain type of hygiene product.


My darkest secret is that I frequently use lame humor to divert people from the fact that I'm just a meager 108.

06-18-2012, 11:50 PM
The only thing IQ tests measure is how well a person does on tests like that. A person can take tests like that again and again, and improve their "intelligence." The tests were never meant to be a difinitive measure of intelligence: they were designed to predict success in school. But, people are infatuated with the idea that they can nail down a person's intelligence with a single number. In reality, it's not that stupidly easy.

06-19-2012, 07:48 PM
It also gives us some breathing room for adding new features down the line.

Maybe the mailbox returning up at the top?
or or or even...

Possibly showing our current rank in the monthly stats on the
my score page, even?? :D Because it would be great to see where
you are before you hit the board at a 100th.

Oh and and and, fixing the discrepancy between the time in "my scores"
to the time on the board. Its off by a tenth. Thats just a round up thingie I know that
but yanno if yer in there dickerin around you could give it a tweak.

Thanks Stephen. Site is running much faster.
Appreciate the hard work.:)

06-20-2012, 05:30 AM
I second the mailbox and monthly ranking. I'm actually starting to like the mismatched times. It means that, when I go into the official rankings, I'm faster than I thought I was. Like keeping your car's clock set a minute or two ahead so you won't be late for work.

06-20-2012, 06:56 AM
Whatever the secret ingredient is, I would like some.

06-20-2012, 06:08 PM
The secret ingredient is...




06-20-2012, 11:38 PM
Actually, the secret is acupuncture, followed by a mud bath with a cucumber facial, chanting "Nom myoho renge kyo" while listening to the Turangalila Symphony on your iPod while the moon is in Scorpio.

(bansaisequoia's IQ has just slipped to 106)

06-20-2012, 11:48 PM
So, there is a "slip" in straight linear logic when one thinks in a "creative" logical way?

06-21-2012, 02:11 AM

Whut if ya dont gots one of them thar fancee Ipod deevices
ken i listen to my transistor radio instead??

It only gets one station. Country.
Can y'all tell?

now whoa nelly ...not that thar's nuffin wrong abowt country.
no siree billybob i dint mean to say there wuz.
cuz i got the facial muck on me im chanting them thar
funny wurds but danged if i ken find the Trumangila simfony
anywhars on this here lil bitty thing.

(heatstroke, sorry folks)

06-21-2012, 06:14 PM
I love this forum.. people have actually heard of the Turangalila Symphonie. Now, who owns an ondes Martenot?

06-23-2012, 01:54 AM