Mask "Bleeding out"

[DonggeonKim1012]

Hello,
I'm currently trying to make a real time sam2 with interactive input.
I stop tracking, use the add_new_points_or_box to give points for that frame, and resume tracking.
I run track_step() every frame, before giving new inputs (if any), so that I can keep track of objects with no input in that frame.

For example, I gave points to a block(green mask) earlier, then gave points to a box(orange).
But after a few frames with no input, the mask of the block, first highlighted in green, 'bleeds out'.
Here are some pictures to explain.

Does anyone have an idea what might be causing this phenomena?
I think there is a way to make the mask tighter, but I'm still not quite familiar with the hyperparameters.

I used the expression 'bleed out' because I remember reading it somewhere along the way, but I can't find the reference.
Any suggestions would be highly appreciated. I haven't made any changes to the sam2_base file.