Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for IOS Matching Metric. Introduced the mask_non_max_merge function for handling non-maximum merging of masks #1774

Open
wants to merge 4 commits into
base: develop
Choose a base branch
from

Conversation

SunHao-AI
Copy link

Description

When I used the yolo11-seg model for slice inference, I found that the merging did not work well. After analysis, it is found that for slender objects, its mask area only accounts for a small part, and the box area may be very large, so it is necessary to use mask for iou calculation. aaa
When I implemented the mask_non_max_merge function, I found that the merge effect was still not satisfactory, as shown in the figure:
NMM-IOU10
I found that the sahi library implemented the non-maximum merge algorithm of the IOS version, so I made changes to the code.
NMM-IOS10

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

How has this change been tested, please provide a testcase or example of how you tested the change?

import cv2
import numpy as np
from PIL import Image
from ultralytics import YOLO

import supervision as sv

model = YOLO("yolon11-seg.pt")
image = cv2.imread(<your_image_path>)


def callback(image_slice: np.ndarray) -> sv.Detections:
    results = model(image_slice)[0]
    return sv.Detections.from_ultralytics(results)

# slicer = sv.InferenceSlicer(callback=callback, slice_wh=(960, 960), overlap_ratio_wh=None, overlap_wh=(50, 50), overlap_filter="non_max_merge", iou_threshold=0.1, match_metric="IOU")
slicer = sv.InferenceSlicer(callback=callback, slice_wh=(960, 960), overlap_ratio_wh=None, overlap_wh=(50, 50), overlap_filter="non_max_merge", iou_threshold=0.1, match_metric="IOS")
# slicer = sv.InferenceSlicer(callback=callback, slice_wh=(960, 960), overlap_ratio_wh=None, overlap_wh=(50, 50), overlap_filter="non_max_suppression", iou_threshold=0.1, match_metric="IOU")
# slicer = sv.InferenceSlicer(callback=callback, slice_wh=(960, 960), overlap_ratio_wh=None, overlap_wh=(50, 50), overlap_filter="non_max_suppression", iou_threshold=0.1, match_metric="IOS")
detections = slicer(image)

box_annotator = sv.BoxAnnotator()
mask_annotator = sv.MaskAnnotator()
label_annotator = sv.LabelAnnotator()
labels = [f"{class_name} {confidence:.2f}" for class_name, confidence in zip(detections["class_name"], detections.confidence)]

annotated_image = box_annotator.annotate(scene=image, detections=detections)
annotated_image = mask_annotator.annotate(scene=annotated_image, detections=detections)
annotated_image = label_annotator.annotate(scene=annotated_image, detections=detections, labels=labels)
Image.fromarray(annotated_image[..., ::-1]).save("test.jpg")

Any specific deployment considerations

For example, documentation changes, usability, usage/costs, secrets, etc.

Docs

  • Docs updated? What were the changes:

- Added the `match_metric` parameter to the `with_nms` and `with_nmm` methods in the `Detections` class.
- Modified the `box_non_max_suppression` and `box_non_max_merge` functions to support IOS calculation.
- Introduced the `mask_non_max_merge` function for handling non-maximum merging of masks.
- Updated the `box_iou_batch` and `mask_iou_batch` functions to support both IOU and IOS calculations.
- Enhanced the `InferenceSlicer` class to accommodate the IOS matching metric.

### Notes
- IOS (Intersection over Smaller): A variation of IOU that compares intersection over the smaller area, useful in scenarios with varying object sizes.
- Refinements: Improved terminology for clarity and consistency, ensuring technical accuracy.
- Added the `match_metric` parameter to the `with_nms` and `with_nmm` methods in the `Detections` class.
- Modified the `box_non_max_suppression` and `box_non_max_merge` functions to support IOS calculation.
- Introduced the `mask_non_max_merge` function for handling non-maximum merging of masks.
- Updated the `box_iou_batch` and `mask_iou_batch` functions to support both IOU and IOS calculations.
- Enhanced the `InferenceSlicer` class to accommodate the IOS matching metric.

Notes
- IOS (Intersection over Smaller): A variation of IOU that compares intersection over the smaller area, useful in scenarios with varying object sizes.
- Refinements: Improved terminology for clarity and consistency, ensuring technical accuracy.
@CLAassistant
Copy link

CLAassistant commented Jan 9, 2025

CLA assistant check
All committers have signed the CLA.

@SkalskiP
Copy link
Collaborator

SkalskiP commented Jan 9, 2025

Hi @SunHao-AI 👋🏻 thank you so much for your interest in supervision. This looks like a useful feature. I will try to get back to you with a PR review as soon as possible.

@SkalskiP SkalskiP added the priority:high Tasks rated as high priority by Roboflow team label Jan 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority:high Tasks rated as high priority by Roboflow team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants