How to achieve speed-ups in Colab (T4 GPU)?

[infinite-darkness108]

In Colab T4 GPU setting, I noted that on an average, a moonshine tiny encoder layer takes 1.64ms, while whisper tiny 5.10ms. (beckett.wav ~10 sec). And a decoder layer is faster owing to smaller input matrix its self-attention receives, and key-value caching at its cross-attention.

I expect that for both the models, regardless of the size of the input speech waveform, their whole decoders should take around the same time. 288 vs 384-dimension size but 6 vs 4 # of decoder layers. If the reasoning is correct, all of the speed-up comes from the encoder.

However, considering first few iterations as warmup,

for i in range(11):
    t0 = time.time()
    ids = model.generate(**inputs)
    print(time.time()-t0)

Whisper tiny has a latency of 0.12 seconds, while Moonshine tiny has 0.18 seconds. This puzzles me. Furthermore, on trying with 10 sec silence (all zeros), suddenly I notice speed ups. In this case, both the decoders stopped with 1 decoded token, like eos and all of the speed up came from the encoder. I believe that in my case, the decoder layer of Moonshine tiny is not matching in speed with that of Whisper tiny.

Any suggestion on how to achieve similar latency will be greatly valued. If fused kernel implementation is only employed in whisper, then this is the core.