Reassessing exercise difficulty levels

[tasxatzial]

It's about time to reassess the difficulty of the exercises. Some of them are way off; for example, the change exercise is labeled as easy when, in fact, it's closer to hard compared to the rest of the exercises. This might not seem like a big problem until we realize that incorrect difficulty ratings can easily dissuade people who choose exercises based on difficulty and cause them to abandon the track.

The purpose of this adjustment isn't to determine the absolute difficulty of the exercises, as that's impossible, but to judge their relative difficulty. More precisely, the classification into easy, medium, and hard is intended to suggest which exercises should be attempted before others. A good classification should assist those who select exercises based on difficulty.

Here's how I would approach it:

• First, do a quick pass over all exercises and rate them as easy, medium, or hard. We don't need to be entirely accurate at this stage.
• Second, do another pass and check if any exercises in a set should move up or down a level. To do this, each exercise should be compared with the others in its set. For example, if an "easy" exercise is harder than most of the other easy exercises, it might be safer to label it as "medium".
• Lastly, take each set and compare all exercises relative to one another. For example, the easiest exercises in the easy category will be assigned a 1, and the hardest a 3.

Any other ideas on how we should be approaching this?

[tasxatzial]

@ErikSchierboom Many exercises have difficulty levels that are significantly off. I'll review the config and update them to something more appropriate.

[ErikSchierboom]

Great! Thanks.

[ErikSchierboom]

The https://exercism.org/tracks/clojure/build page can help here too, as you can find percentages for completions.

[tasxatzial]

Thanks, I'll check that page as well. However, I'm concerned that an incorrect difficulty level might affect how many people attempt the exercise, ultimately skewing the completion rates.

[ErikSchierboom]

It will be skewed, definitely, but it might give us some help.

[tasxatzial]

Went over the config and did a first-pass adjustment using my judgement, my solutions, community solutions, and example solution.

I'll do a second pass using the stats page.

Then i'll re-arrange the config and do some micro-adjustments.

[ErikSchierboom]

Thanks for doing that

[tasxatzial]

After reviewing solutions multiple times, consulting the completion rates, and fine-tuning the difficulties, I've finalized results that I believe will offer a smoother user experience. These reflect the proposed difficulties only. The ordering of exercises with the same difficulty is still random and will be adjusted as a final step. If you have questions about a specific difficulty, ask, and I'll explain how I determined it. Note that exercises without completion rates are those recently added.

Exercise	Old Difficulty	Completion	New Difficulty	Proposed Order
two-fer	1	87	1
darts	2	-	2
etl	2	85.3	2
hamming	2	92.1	2
high-scores	2	-	2
leap	3	90.8	2
resistor-color	2	-	2
reverse-string	1	84.3	2
rna-transcription	2	93.8	2
space-age	2	90.3	2
accumulate (depr)	2	85.5	3
acronym	2	82.5	3
anagram	2	92.3	3
armstrong-numbers	1	63.7	3
bank-account	7	76.4	3
collatz-conjecture	3	84.7	3
complex-numbers	2	80.4	3
difference-of-squares	4	87.4	3
eliuds-eggs	2	86.7	3
flatten-array	3	80.9	3
grade-school	3	90.4	3
grains	3	88.8	3
isogram	3	83	3
nucleotide-count	3	87.6	3
pangram	2	89.1	3
pascals-triangle	3	75.3	3
protein-translation	2	82.5	3
proverb	3	73.3	3
raindrops	2	90.6	3
resistor-color-duo	3	-	3
scrabble-score	2	90.6	3
series	2	81.1	3
square-root	3	-	3
strain	2	86.2	3
sum-of-multiples	2	83.8	3
triangle	2	90.8	3
word-count	2	93.5	3
beer-song	3	81.1	4
bob	2	87.2	4
roman-numerals	2	85.9	4
allergies	4	76.6	4
atbash-cipher	3	86	4
binary-search-tree	3	67.7	4
binary-search	3	78.6	4
binary (depr)	3	87.8	4
clock	5	72.9	4
dnd-character	4	-	4
gigasecond	3	81.2	4
hexadecimal (depr)	3	82	4
isbn-verifier	3	78.5	4
kindergarten-garden	3	77.7	4
largest-series-product	4	81.9	4
matrix	4	-	4
octal (depr)	4	84	4
perfect-numbers	3	86.4	4
phone-number	3	86.5	4
prime-factors	3	82.4	4
saddle-points	4	-	4
secret-handshake	2	81.8	4
simple-cipher	4	-	4
sublist	2	59.5	4
trinary (depr)	3	86.7	4
twelve-days	4	-	4
run-length-encoding	2	72.7	5
all-your-base	2	60.9	5
diamond	5	65.9	5
food-chain	5	-	5
game-of-life	5	-	5
knapsack	5	-	5
list-ops	5	-	5
luhn	5	75.7	5
matching-brackets	7	79.1	5
nth-prime	2	70.1	5
pig-latin	2	66.8	5
resistor-color-trio	5	-	5
robot-name	2	83	5
sieve	5	76.4	5
state-of-tic-tac-toe	5	-	5
transpose	5	-	5
yacht	5	78.6	5
crypto-square	4	70.5	6
killer-sudoku-helper	6	-	6
meetup	4	77.8	6
minesweeper	6	63.2	6
pythagorean-triplet	6	-	6
queen-attack	6	75.4	6
robot-simulator	5	59.7	6
spiral-matrix	4	54.5	6
wordy	6	65.3	6
say	3	57.5	7
change	3	55.2	7
dominoes	4	50.5	7
poker	7	42.3	8
zebra-puzzle	8	-	8
zipper	9	51	9
go-counting	9	36	9

Old Difficulty	Exercise count
1	3
2	29 (+1 depr)
3	18 (+3 depr)
4	12 (+1 depr)
5	13
6	5
7	3
8	1
9	2
10	-

Summary

• Easy: 54
• Medium: 34
• Hard: 3

---

New Difficulty	Exercise count
1	1
2	9
3	26 (+1 depr)
4	22 (+4 depr)
5	17
6	8
7	4
8	2
9	2
10	-

Summary

• Easy: 37
• Medium: 55
• Hard: 4

Edit: Removed the plots, will try to recreate them

[ErikSchierboom]

Love this!