Subclass `PyTorchYolo` and `PyTorchDetectionTransformer` off `PyTorchObjectDetector` #2321

f4str · 2023-11-09T00:58:12Z

Description

Modify both the PyTorchYolo and PyTorchDetectionTransformer estimators to be subclassed off the PyTorchObjectDetector estimator. This reduces a lot of redundant code and allows most functionality to be shared among the estimators. Additionally, this adds the option to train the PyTorchDetectionTransformer since the code is inherited accordingly.

The PyTorchObjectSeeker class was also cleaned up accordingly since there is now one unified superclass for all PyTorch object detectors.

Additionally updated the unit tests for all PyTorch object detection estimators to use a fixed input. This reduces randomness and ensures that outputs can remain consistent throughout tests.

Fixes #2267

Type of change

Please check all relevant options.

Improvement (non-breaking)
Bug fix (non-breaking)
New feature (non-breaking)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Testing

Please describe the tests that you ran to verify your changes. Consider listing any relevant details of your test configuration.

Updated unit tests for PyTorchObjectDetector
Updated unit tests for PyTorchFasterRCNN
Updated unit tests for PyTorchYolo
Updated unit tests for PyTorchDetectionTransformer

Test Configuration:

OS
Python version
ART version or commit number
TensorFlow / Keras / PyTorch / MXNet version

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes
My changes have been tested using both CPU and GPU devices

codecov-commenter · 2023-11-09T01:22:41Z

Codecov Report

Attention: 21 lines in your changes are missing coverage. Please review.

Comparison is base (80fd393) 85.39% compared to head (9e347e6) 85.54%.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@              Coverage Diff               @@
##           dev_1.18.0    #2321      +/-   ##
==============================================
+ Coverage       85.39%   85.54%   +0.14%     
==============================================
  Files             327      327              
  Lines           30205    29888     -317     
  Branches         5589     5528      -61     
==============================================
- Hits            25793    25567     -226     
+ Misses           2964     2900      -64     
+ Partials         1448     1421      -27

Files	Coverage Δ
...ators/certification/object_seeker/object_seeker.py	`99.15% <100.00%> (+1.00%)`	⬆️
...estimators/object_detection/pytorch_faster_rcnn.py	`100.00% <ø> (ø)`
...mators/object_detection/pytorch_object_detector.py	`89.41% <100.00%> (+3.62%)`	⬆️
art/estimators/object_detection/pytorch_yolo.py	`86.95% <100.00%> (+4.00%)`	⬆️
.../object_detection/pytorch_detection_transformer.py	`85.00% <77.50%> (+21.73%)`	⬆️
.../estimators/certification/object_seeker/pytorch.py	`81.92% <77.77%> (-1.95%)`	⬇️

... and 2 files with indirect coverage changes

art/estimators/certification/object_seeker/pytorch.py

beat-buesser · 2023-11-30T16:00:21Z

art/estimators/object_detection/pytorch_object_detector.py

@@ -247,8 +249,41 @@ def _preprocess_and_convert_inputs(

        return x_preprocessed, y_preprocessed

+    def _translate_labels(self, labels: List[Dict[str, "torch.Tensor"]]) -> Any:


Can we define the return type more accurately?

Since _translate_labels() and _translate_predictions() need to be overridden for each object detector model, this can take many different types (e.g., dictionary for FRCNN, tensor for Yolo, list of tensor for Detr). Replacing the Any would require a very long Union with every possible type for all subtypes which would need to updated if any new object detector is added. Therefore, it makes the most sense to keep this an Any type.

beat-buesser · 2023-11-30T16:03:10Z

art/estimators/object_detection/pytorch_object_detector.py

+        labels_translated = [{k: v.to(self.device) for k, v in y_i.items()} for y_i in labels]
+        return labels_translated
+
+    def _translate_predictions(self, predictions: Any) -> List[Dict[str, np.ndarray]]:  # pylint: disable=R0201


Can we remove the pylint comment and decorate this function with @staticmethod?

Since _translate_predictions needs to be overridden for all subclasses, it cannot be a static method. For the specific cases of Faster RCNN, it does not use any of the properties, but this is not the case for YOLO and Detr.

beat-buesser

Hi @f4str Thank you very much for your pull request! I have added a few questions to get a better understanding of all the changes. Please let me know what you think?

beat-buesser · 2023-11-30T16:12:23Z

art/estimators/object_detection/pytorch_object_detector.py


        if isinstance(x, torch.Tensor):
-            return loss
+            return loss  # type: ignore


Why is there a type ignore?

This should be resolved now

beat-buesser · 2023-11-30T16:12:36Z

art/estimators/object_detection/pytorch_object_detector.py


-        return loss.detach().cpu().numpy()
+        return loss.detach().cpu().numpy()  # type: ignore


Why is there a type ignore?

This should be resolved now

beat-buesser · 2023-11-30T16:20:36Z

tests/estimators/object_detection/conftest.py

+
+
+@pytest.fixture()
+def get_pytorch_detr(get_default_cifar10_subset):


Please add a docstring.

art/estimators/certification/object_seeker/pytorch.py

beat-buesser · 2023-11-30T16:35:17Z

art/estimators/object_detection/pytorch_object_detector.py

@@ -247,8 +249,41 @@ def _preprocess_and_convert_inputs(

        return x_preprocessed, y_preprocessed

+    def _translate_labels(self, labels: List[Dict[str, "torch.Tensor"]]) -> Any:


Is there a way to ensure that inheriting classes remember to update the label and precision translations?

beat-buesser · 2023-11-30T16:36:14Z

art/estimators/object_detection/pytorch_object_detector.py

+        self.set_dropout(False)
+        self.set_multihead_attention(False)


Is this new?

Yes, these are required for PyTorchDetectionTransformer since it is a subclass of PyTorchObjectDetector. Since PyTorchObjectDetector and PyTorchYolo do not have attention layers, this is just a no-op and does not affect functionality.

beat-buesser · 2023-11-30T16:38:26Z

tests/estimators/object_detection/test_pytorch_detection_transformer.py

-    from art.utils import load_dataset
-    from art.estimators.object_detection.pytorch_detection_transformer import PyTorchDetectionTransformer
+@pytest.mark.only_with_platform("pytorch")
+def test_predict(art_warning, get_pytorch_detr):


Why did the expected values for predictions change?

PyTorchFasterRCNN previously did not support the preprocessing parameter. This is probably legacy before the original refactor from before as it is fully supported now. Therefore, this test case is no longer applicable.

art/estimators/object_detection/pytorch_detection_transformer.py

beat-buesser · 2024-01-09T12:02:55Z

Hi @f4str It looks like that one of the new unit test is change the prediction when the test runs on the GitHubActions machine. I have tried to fix it with #2371 but was not yet successful. Do you have an idea why the mode changes its predictions in that failing test?

Signed-off-by: Farhan Ahmed <[email protected]>

beat-buesser

Hi @f4str Thank you very much! The changes look good to me.

f4str force-pushed the pytorch-yolo-rebase branch from baeec58 to 008679e Compare November 9, 2023 01:01

f4str marked this pull request as draft November 9, 2023 06:03

github-advanced-security bot found potential problems Nov 9, 2023

View reviewed changes

art/estimators/certification/object_seeker/pytorch.py Fixed Show resolved Hide resolved

f4str marked this pull request as ready for review November 10, 2023 00:58

beat-buesser self-requested a review November 30, 2023 15:56

beat-buesser self-assigned this Nov 30, 2023

beat-buesser added the improvement Improve implementation label Nov 30, 2023

beat-buesser added this to the ART 1.17.0 milestone Nov 30, 2023

beat-buesser linked an issue Nov 30, 2023 that may be closed by this pull request

Subclass the PyTorchYolo and PyTorchDetectionTransformer estimators off the PyTorchObjectDetector estimator class #2267

Closed

beat-buesser reviewed Nov 30, 2023

View reviewed changes

beat-buesser requested changes Nov 30, 2023

View reviewed changes

github-advanced-security bot found potential problems Nov 30, 2023

View reviewed changes

art/estimators/object_detection/pytorch_detection_transformer.py Fixed Show fixed Hide fixed

beat-buesser mentioned this pull request Dec 23, 2023

Fix #2321 Subclass PyTorchYolo and PyTorchDetectionTransformer off PyTorchObjectDetector #2371

Closed

beat-buesser modified the milestones: ART 1.17.0, ART 1.18.0 Dec 27, 2023

f4str added 12 commits January 18, 2024 16:00

rebase pytorch yolo off pytorch object detector

50cb7c0

Signed-off-by: Farhan Ahmed <[email protected]>

fix style checks

a0344de

Signed-off-by: Farhan Ahmed <[email protected]>

cleanup faster rcnn and yolo refactor

cbe65f8

Signed-off-by: Farhan Ahmed <[email protected]>

rebase pytorch detr off pytorch object detector

0663a1e

Signed-off-by: Farhan Ahmed <[email protected]>

finalize pytorch object detector rebase

77d1a8a

Signed-off-by: Farhan Ahmed <[email protected]>

update pytorch detr unit tests

9fb75d2

Signed-off-by: Farhan Ahmed <[email protected]>

fix style checks

b4415ce

Signed-off-by: Farhan Ahmed <[email protected]>

modify object seeker accordingly

b5ce4b1

Signed-off-by: Farhan Ahmed <[email protected]>

fix object seeker issues

41bb273

Signed-off-by: Farhan Ahmed <[email protected]>

fix mypy errors

a1410ce

Signed-off-by: Farhan Ahmed <[email protected]>

fix bug

b50471e

Signed-off-by: Farhan Ahmed <[email protected]>

address review comments

17df575

Signed-off-by: Farhan Ahmed <[email protected]>

f4str added 3 commits January 18, 2024 16:03

fix object seeker codeql warning

a4eb974

Signed-off-by: Farhan Ahmed <[email protected]>

add additional typing

c38fb2d

Signed-off-by: Farhan Ahmed <[email protected]>

fix style checks

400f839

Signed-off-by: Farhan Ahmed <[email protected]>

f4str force-pushed the pytorch-yolo-rebase branch from 750bd30 to 400f839 Compare January 19, 2024 00:03

f4str changed the base branch from dev_1.17.0 to dev_1.18.0 January 19, 2024 00:03

attempt to fix test cases

a3a44a2

Signed-off-by: Farhan Ahmed <[email protected]>

f4str requested a review from beat-buesser January 19, 2024 18:22

f4str and others added 8 commits January 26, 2024 15:31

create new inputs for pytorch object detector test cases

41a0143

Signed-off-by: Farhan Ahmed <[email protected]>

update pytorch faster rcnn test cases

b70cf34

Signed-off-by: Farhan Ahmed <[email protected]>

update pytorch object detector test cases

4465875

Signed-off-by: Farhan Ahmed <[email protected]>

fix test cases for pytorch object detectors

e50e5e5

Signed-off-by: Farhan Ahmed <[email protected]>

fix test cases for pytorch detection transformer

1a62aa0

Signed-off-by: Farhan Ahmed <[email protected]>

fix test cases for pytorch yolo

66e0e5a

Signed-off-by: Farhan Ahmed <[email protected]>

increase testing tolerence

2b79aa9

Signed-off-by: Farhan Ahmed <[email protected]>

Merge branch 'dev_1.18.0' into pytorch-yolo-rebase

9e347e6

beat-buesser approved these changes Feb 27, 2024

View reviewed changes

beat-buesser merged commit 3597228 into Trusted-AI:dev_1.18.0 Feb 27, 2024
31 checks passed

f4str deleted the pytorch-yolo-rebase branch February 27, 2024 16:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Subclass `PyTorchYolo` and `PyTorchDetectionTransformer` off `PyTorchObjectDetector` #2321

Subclass `PyTorchYolo` and `PyTorchDetectionTransformer` off `PyTorchObjectDetector` #2321

f4str commented Nov 9, 2023 •

edited

Loading

codecov-commenter commented Nov 9, 2023 •

edited

Loading

beat-buesser Nov 30, 2023

f4str Nov 30, 2023

beat-buesser Nov 30, 2023

f4str Nov 30, 2023

beat-buesser left a comment

beat-buesser Nov 30, 2023

f4str Nov 30, 2023

beat-buesser Nov 30, 2023

f4str Nov 30, 2023

beat-buesser Nov 30, 2023

f4str Nov 30, 2023

beat-buesser Nov 30, 2023

beat-buesser Nov 30, 2023

f4str Nov 30, 2023

beat-buesser Nov 30, 2023

f4str Nov 30, 2023

beat-buesser commented Jan 9, 2024

beat-buesser left a comment

		@@ -247,8 +249,41 @@ def _preprocess_and_convert_inputs(

		return x_preprocessed, y_preprocessed

		def _translate_labels(self, labels: List[Dict[str, "torch.Tensor"]]) -> Any:


		return loss.detach().cpu().numpy()
		return loss.detach().cpu().numpy() # type: ignore



		@pytest.fixture()
		def get_pytorch_detr(get_default_cifar10_subset):

Subclass PyTorchYolo and PyTorchDetectionTransformer off PyTorchObjectDetector #2321

Subclass PyTorchYolo and PyTorchDetectionTransformer off PyTorchObjectDetector #2321

Conversation

f4str commented Nov 9, 2023 • edited Loading

Description

Type of change

Testing

Checklist

codecov-commenter commented Nov 9, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

beat-buesser left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

beat-buesser commented Jan 9, 2024

beat-buesser left a comment

Choose a reason for hiding this comment

Subclass `PyTorchYolo` and `PyTorchDetectionTransformer` off `PyTorchObjectDetector` #2321

Subclass `PyTorchYolo` and `PyTorchDetectionTransformer` off `PyTorchObjectDetector` #2321

f4str commented Nov 9, 2023 •

edited

Loading

codecov-commenter commented Nov 9, 2023 •

edited

Loading