Understanding Grit

[shntnu]

@gwaygenomics

In https://github.com/broadinstitute/grit-benchmark/blob/main/1.calculate-metrics/cell-health/0.calculate-grit.ipynb, it says

Some perturbations have only one guide per gene, these cannot have grit scores

But aren't you computing grit at the replicate level? And you have 6 replicates for most guides so this should be possible.

If you're computing grit at the perturbation level, are you first aggregating the profiles (to a consensus) somewhere?

(Meta: Consider adding Discussions to this repo, and move this there?)

[gwaybio]

For this application, I am not calculating grit at the replicate level (i.e. the profile, or well, level). I am calculating grit at the perturbation level (aka the guide level). See cell 5 in that notebook you linked.

Therefore, for a single guide (which has multiple replicates) all the r's from figure 1A are profiles from different guides targeting the same gene. No aggregation to consensus.

But this is a very good point, I can easily also calculate grit at the individual level. Would that help your analyses?

[shntnu]

@gwaygenomics so is this correct? (you don't need to count all the lines :D)

Each dot is a well

[gwaybio]

yep. That is the grit formulation in a CRISPR experiment, or when comparing compounds against MOAs. The original Figure 1A linked above is for a general case.

[shntnu]

Got it

We look at profiles that belong to the same "group" (in the case above that means guides to the same gene) while ignoring pairwise correlations to the same "replicate" (in this case replicates of the same guide perturbation).

If I take this literally, that would mean you would also include pairwise correlations between guide-2 and guide-3 reps (in my diagram) when computing the grit for guide-1, given that guide-2 and guide-3 belong to the same group (i.e. same gene) as guide-1, but you don't do that, correct?

[gwaybio]

Correct. We are zooming in on the specific guide's grit. We do not compare guide 2 and 3 in guide 1's grit calculation.

[shntnu]

Got it. This makes a lot of sense.

I was wondering if we could simplify the definition by computing grit on the consensus instead, but it gets messy because we do need to have several control data points for this to work well, where doing consensus won't allow for that (easily).

I'm nearly there with you. More tomorrow.

[shntnu]

@gwaygenomics Another general, non-urgent question related to Grit: Did you ever consider doing the "completely different way" described below
cytomining/cytominer-eval#38 (comment). Wondering if there's a strong reason NOT to do it that way. I find the z-score approach (the current grit) more intuitive but I was wondering if there is anything particularly problematic about using AP / PRAUC.

I'd note that there are other, completely different ways of reporting the comparison of the two distributions (1. correlation to replicates and 2. corrections to negative controls) e.g. Average Precision (replicates = class 1, negative controls = class 2), but @gwaygenomics's choice of average z-score of class 1, using class 2 as a reference, is a defensible choice.

Context: This discussion broadinstitute/DeepProfilerExperiments#5 (comment) led me to ask this q.