Constraint and Or Graders

[T-Brick]

Adds a constraint grader functor which takes in a Constraint and a Standard grader. When ran, it first checks to make sure the Constraint grader gives a point value above the threshold. If it does then it has the same results as the Standard grader. If not, then it awards zero points and gives some message.

The or grader functor simply takes in two graders and takes the maximal score of the two. Potentially useful for if we want to accept off-by-one errors (so we don't have to make the original grader handle this in some way).

[HarrisonGrodin]

These seem like great abstractions to have in the general autograding library! Have a few thoughts/comments.

I'm now also thinking... these both seem a bit similar to ProdGrader; perhaps there's a better abstraction that would encompass both functionalities? ProdGrader just does weighted sums, but I wonder if it would make sense to parameterize it on some arbitrary combination operation. Then, OrGrader would just instantiate the more general abstraction at operation max; ConstraintGrader would instantiate with * maybe; what we have now would instantiate at a weighted average operation? (Could have others, too, like min, unweighted average, etc.) Thoughts?

[HarrisonGrodin]

Out of curiosity - what's the benefit of having structure Constraint : GRADER rather than, say, constraint : bool (or unit -> bool)?

[T-Brick]

I'm not sure how you would do this with just bool or a unit->bool?

My idea was that it simply runs the Constraint grader and if it passes that then it would run the normal grader (rather than running the normal grader and having a TA manually grade the code). I'm unsure of any other clean way to use the existing infrastructure to run student code.

[HarrisonGrodin]

Ah, gotcha. I was thinking you could use Result.evaluate, but I guess you'd have to rewrite some boilerplate (run a list of tests, etc.).

[HarrisonGrodin]

This feels a bit odd, since description is a student-facing. Instinctively, I would imagine val description = Standard.Rubric.description would be reasonable (a la functor Preamble) if constraint : bool, but not sure if that makes sense.

[T-Brick]

Where exactly is this student facing?

Oh when its in the prod grader is one example maybe? It doesn't seem like it shows if you are using the grader directly.

[HarrisonGrodin]

Yeah, it's expected that the student can see description. As an edge case, the 150 infrastructure doesn't show the description at the top level (since graders are per-problem anyway so descriptions wouldn't be very useful), but other grader combinators can show it.

In retrospect, the more graders we write, I think this might be a design flaw: I suspect the right thing to do would actually be make val weights in ProdGraderN be (string * int) * (string * int) * ..., where the strings are descriptions. Then, graders wouldn't need descriptions (because they just "exist"), but combining graders would require that you specify a description for each component.

[T-Brick]

I'm now also thinking... these both seem a bit similar to ProdGrader; perhaps there's a better abstraction that would encompass both functionalities? ProdGrader just does weighted sums, but I wonder if it would make sense to parameterize it on some arbitrary combination operation. Then, OrGrader would just instantiate the more general abstraction at operation max; ConstraintGrader would instantiate with * maybe; what we have now would instantiate at a weighted average operation? (Could have others, too, like min, unweighted average, etc.) Thoughts?

Fair point. I think constraint grader would be more complex than * because it should give 0 points if the constraint is violated... but that would just a different function.

I'm not sure how useful some of those would be. Constraint grader has it's uses for offloading work from TAs/better organising some of the graders. The Or grader can be used for cases where we want to accept off-by-one errors, etc (bbs on lazy/imperative comes to mind). Definitely worth considering though.

[HarrisonGrodin]

I think constraint grader would be more complex than * because it should give 0 points if the constraint is violated... but that would just a different function.

I was thinking that if the constraint grader always gave 0 if the constraint were violated, it would be able to "knock out" the other score via multiplication by 0. It also works nicely with the style grader (which it'd be nice to express via the GRADER signature), since you could knock out only part of the correctness points.
Or, for a subtly-different behavior, constraint graders could use min instead of * - the zeroing-out behavior would stay the same, but the style grader wouldn't deduct points if the correctness was low anyway.

The design I was imagining is approximately as follows:

functor NewIdeaForProdGrader2 (
  val description : string  (* well, might want to rethink this per other discussion, but anyway *)
  structure Grader1 : GRADER
  structure Grader2 : GRADER
  val combine : (string * Rational.t) * (string * Rational.t) -> (string * Rational.t)
) = (* ... *)

Then:

functor CurrentProdGrader2 (
  val description : string
  structure Grader1 : GRADER
  structure Grader2 : GRADER
  val weights : int * int
) =
  NewIdeaForProdGrader2 (
    val description = description
    structure Grader1 = Grader1
    structure Grader2 = Grader2

    val (w1, w2) = weights
    val combine = fn ((s1, r1), (s2, r2)) => ((* format nicely *), weightedAverage [(r1, w1), (r2, w2)])
  )

functor OrGrader2 (
  val description : string
  structure Grader1 : GRADER
  structure Grader2 : GRADER
) =
  NewIdeaForProdGrader2 (
    val description = description
    structure Grader1 = Grader1
    structure Grader2 = Grader2
    val combine = fn ((s1, r1), (s2, r2)) =>
      case Rational.compare (r1, r2) of
        LESS => (s2, r2)
      | _    => (s1, r1)
  )

[T-Brick]

Ooooo yeah that would be quite nice.

I like the idea of moving the descriptions into the weights since that makes more sense to where they are used.

Would the style grader also just be a *? So if a student has 2 style errors, then the style grader would return 0.9 and we'd multiply that by whatever their actual score is.

[HarrisonGrodin]

Yeah, that's what I was thinking! :)

[qtgeo1248]

Sooo what's the tldr

I think to add to the conversation, I would prefer the constraint grader to not be just simple multiplication, since that is not how 150 grading works (at least not anymore). We deduct x amount of points, and if you reach below 0, we add it back so you get 0.

[qtgeo1248]

Oh in the case of memofib, we explicitly say we deduct 10 points (out of 15) for violating constraints, so would also like it if we could make it so that we can change the number of points we deduct

[T-Brick]

i think the tl;dr is that ideal the prod graders are rewritten to take in a score "combining" function which can be used to handle all these different use cases

[qtgeo1248]

Will this be worked on sometime in the near future?

[qtgeo1248]

It seems like it needs a review?

[HarrisonGrodin]

(I think you both should have the ability to review/merge, if it looks good to you!)

[HarrisonGrodin]

Also, should this be closed as superseded by #7, or both make sense?