[Discussion] Section 3 (of the paper) should have pseudocode

[dinhanhx]

Discussion

I know the paper is being reviewed and will likely be modified. However, I think some sort of pseudocode would be nice. Few chunks of paragraphs make things a bit hard to follow. The pseudocode also would help other people implement this technique onto their current models.

[mu-cai]

Thanks for your advice!
The core of M3 is here https://github.com/mu-cai/matryoshka-mm/blob/main/llava/model/llava_arch.py#L147

Let me know if you have further questions!

[dinhanhx]

matryoshka-mm/llava/model/llava_arch.py

Lines 147 to 157 in 8ca825d

	def matryoshka_vis_token_process(self, image_features, matryoshka_vis_token_scale):
	N, H_W, C = image_features.shape
	H = W = int(H_W ** 0.5)
	reshaped_tensor = image_features.view(N, H, W, C)
	reshaped_tensor = reshaped_tensor.permute(0, 3, 1, 2)
	pool_size = stride = int( np.sqrt(H_W / matryoshka_vis_token_scale) )
	pooled_tensor = F.avg_pool2d(reshaped_tensor, kernel_size=pool_size, stride=stride)
	image_features = pooled_tensor.permute(0, 2, 3, 1)
	image_features = image_features.reshape(N, -1, C)
	# print('image_features.shape :', image_features.shape)
	return image_features

What is the value range of matryoshka_vis_token_scale? From 1 to infinity? Or 0.0 to 1.0?

[mu-cai]

Hi, the range is shown here: https://github.com/mu-cai/matryoshka-mm/blob/main/scripts/v1_5/finetune.sh#L36