Instead of using the only CLS token embedding as input features, a common approach in literature is to concatenate the CLS embedding with the mean aggregation of all patch token embeddings.