[Bug Report] Llama 3.2 11B (vision) performance lower than expected

[milank94]

Describe the bug
The decode performance of the Llama 3.2 11B (vision) model is lower than expected:

Expected: 14.8 t/s/u, 2880 ms (ttft)
Actual: 11.4 t/s/u, 3813 ms (ttft)

More details: https://docs.google.com/spreadsheets/d/1Mdn3mBIOHYRC0ETsMJdO9dXtSVNJn_tFaR6QipbpEXU/edit?usp=sharing

To Reproduce
Steps to reproduce the behavior:

1. Go to https://github.com/tenstorrent/tt-inference-server/tree/tstesco/dev/benchmarking

Please complete the following environment information:

• OS: Ubuntu 20.04
• Version of software
  • tt-metal: v0.55.0-rc12
  • vllm: 2f33504

[tstescoTT]

I'll add that this is for N300 only, T3000 performance was as expected.

[uaydonat]

@skhorasganiTT to re-measure the perf

[skhorasganiTT]

Hey @milank94 @tstescoTT, are you running 3.2-11b on N300 with max_num_seqs=16? (that is the batch size which we reported the perf for on N300)

[tstescoTT]

Yes, measured 16 concurrent requests. When testing with more than 16 requests sent the vLLM backend queues the additional requests as desired.