Scout Substrate Dataset - Add patch or diff field

[arturoBeccar]

Issue

The reviewer requested us to include a field called "patch" or "diff" in the Hugging Face Dataset, including a diff between audited and remediated commits, when available.

Steps

• Create a script in scripts (can be called add_diff_to_linearized_findings.py) that generates a json findings-linearized-with-diff.json in the dataset folder that contains the patch or diff field requested by PAL reviewers.
  • It can have as input the findings-linear.json generated with the make dataset command through the script linearize_findings.
  • The patch or diff field should be included for each finding. It should be a string with the git diff between the audited_commit and the remediated_commit for the specified repository for the finding. If audited_commit or remediated_commit is not available, then complete the patch/diff field with null.
• Add the add_diff_to_linearized_findings.py execution to the Makefile when calling make dataset.
• The generated json should be linearized and uploaded to a Dataset in Hugging Face.

Caveats

Identifying the fix for each issue without simply putting the whole diff of the repository is a complicated task that requires a manual review.