## Forum

 Cryptograms Forums Difficulty Ratings
 FAQ Members List Calendar Search Today's Posts Mark Forums Read

#11
04-06-2016, 02:14 PM
 abra Senior Member Join Date: Jun 2009 Posts: 148

Wow, that one seems to have suffered, under the new ratings. I would think that a quote with a solve rate that low, should get an automatic Difficult rating.
#12
04-06-2016, 05:44 PM

Right now we have a three-tiered difficulty system, and one of the pitfalls of that system is that there are a lot of "edge cases". This is a prime example. We're marking as "Easy" all puzzles that fall into the 1-33 percentiles overall (based on a combination of completion rate, median solve time and user feedback). This puzzle here falls exactly on the 33rd percentile, so it literally straddles the line between Easy and Medium. It made it to "Easy" just by the hair on its chinny chin chin.

Most likely we'll beef up the scoring system so that we can break it down by 5-10 percentiles at a time (instead of 33 at a time) so that things will be a bit more reflective in the points department.

Thanks!
__________________
http://www.puzzlebaron.com/blog
#13
04-06-2016, 06:37 PM
 abra Senior Member Join Date: Jun 2009 Posts: 148

It's great to know that you're working on it. Thank you!
#14
04-06-2016, 06:55 PM
 nanrich Member Join Date: Feb 2008 Posts: 38

Yes, thanks Admin for the explanation. And also for having such a great site!
#15
04-06-2016, 08:18 PM
 LLapp Occasional Ranter Join Date: Aug 2014 Posts: 321
Confusion about the 1-to-10 star rating system

Stephen,

Thank you for all the attention you've been giving this site lately. We players appreciate it.

Since you plan to further refine the difficulty ratings, I want to point out that there may be a critical glitch in how you calculate user feedback via average star ratings for Difficulty.

Based on reading many quote comments versus what you said recently about how you treat a quote's average star rating, I'm pretty sure that many (or even most) users have misinterpreted the meaning of the 1-to-10 star scale.

I get the impression that you read the meanings of our "Difficulty" ratings in terms of "Easy vs. Difficult," like this:

1 (extremely easy) -- 3 (easy) -- 5 (neither difficult nor easy) -- 7 (difficult) -- 10 (extremely difficult)

(Did I get that right?)

And meanwhile, I believe that many of us use that same "Difficulty" rating to rank the quote only for difficulty, literally -- like so:

1 (not difficult) -- 3 (somewhat difficult) -- 5 (pretty challenging) -- 7 (very difficult) -- 10 (nearly impossible; I only got here by googling the quote)

(Other players - Have you had this confusion?)

If I'm right about this disconnect, then you can see how it might lead you to misread many users' intended rating of "somewhat difficult" to mean "quite easy" -- which could result in you assigning a difficulty rating out of sync with reality. And it gets more complicated when you consider that the misunderstanding may be widespread but not universal.

This confusion about the star-rating scale is most apparent in how people respond to star ratings for "Offensive." When an innocent quote gets an average "Offensive" rating of 2 stars, for example, commenters typically remark, "How could anyone find this quote offensive?" The perception is that even one star equals "slightly offensive" -- whereas I think you are reading one star to mean "completely innocent." I suspect the same disconnect happens in the "Errors" rating: I might indicate a real-but-not-fatal error by clicking four stars, but now I wonder if you read that feedback to mean "Quote seems okay as is."

So, if you plan to use our star ratings for feedback data, I strongly encourage you to first update the feedback section of the solution page, to clarify the star rating scales. I bet you would get significantly different feedback after the scales are clarified.

Thank you for reading this long message!

Lisa
#16
04-07-2016, 05:22 AM
 LurkerSmurf Junior Member Join Date: Jan 2010 Posts: 28

As long as we're commenting on the star system, I've always had problems with putting "edgy" in the same category as "offensive". Edgy can mean clever, not just "on the border of good taste". I wonder how many so-called offensive quotes were removed because people rated them high for being somewhat rude but funny.
__________________
La, la, la - la - la - la....
#17
04-07-2016, 05:39 AM
 wvwoman Senior Member Join Date: May 2012 Posts: 324

Quote:
 Originally Posted by LurkerSmurf As long as we're commenting on the star system, I've always had problems with putting "edgy" in the same category as "offensive". Edgy can mean clever, not just "on the border of good taste". I wonder how many so-called offensive quotes were removed because people rated them high for being somewhat rude but funny.
i've always thought the same thing. i don't think of edgy as a synonym of offensive--i think of it as offbeat, out of the ordinary, or wickedly funny.
#18
04-07-2016, 06:31 AM
 Annamariah Member Join Date: Nov 2013 Posts: 34

One more thing about the difficulty stars - if the solving rate is less than 10%, it means that most of the people who found the quote extremely hard never got the chance to give it any stars at all.
#19
04-08-2016, 05:31 AM
 montyb Senior Member Join Date: Nov 2011 Posts: 195
For Lurker

"He whose inclination prompts him to cultivate your friendship of his own accord will love you more than one whom you have been at pains to attach to you."
— Samuel Johnson

Lurker on 2012-09-26 14:41:53
So not fair that an "easy" one has a 14.8% solve rate.

The new system has this rated as "difficult" now.
__________________
"Great spirits have always encountered violent opposition from mediocre minds." ~ Albert Einstein

 Thread Tools Display Modes Linear Mode

 Posting Rules You may not post new threads You may not post replies You may not post attachments You may not edit your posts vB code is On Smilies are On [IMG] code is On HTML code is Off
 Forum Jump User Control Panel Private Messages Subscriptions Who's Online Search Forums Forums Home Cryptograms.org Discussion     Main Forum Cryptograms iPhone Application     iPhone Application Questions/Comments

All times are GMT. The time now is 06:02 AM.

The Puzzle Baron family of web sites has served millions and millions of puzzle enthusiasts since its inception in 2006. From cryptograms to acrostics, logic puzzles to drop quotes, patchwords to wordtwist and even sudoku, we run the gamut in word puzzles, printable puzzles and logic games.