Classification Output Issue

I am getting the majority of the images classified as two lables only. The evaluation accuracy is 36% as mentioned and I am using my latest checkpoint only for classification. Is someone else also facing the same/similar issue?
And can anyone help with it?

Thanks.