Hey @ishaan-jaff, I'd like to get assigned and work on this issue. I will update the Code, add test cases and Documentation for this. Let me know if this sounds good!!

yes @NiharP31 , please send the proposed litellm interface / request on this issue before building out the feature

Based on my understanding, here is the solution:

1. File Type Classification and Handling

In litellm/types/files.py, I'll add comprehensive support for all Gemini-supported MIME types by:

• Creating a classification system to distinguish between:
  • Binary file types (images, PDFs, videos, etc.) - requiring base64 encoding
  • Text file types (text/markdown, application/json, text/csv, etc.) - no base64 encoding needed
• Expanding the is_gemini_1_5_accepted_file_type() function to include all MIME types supported by Gemini's API
• Adding a helper function requires_base64_encoding() to determine proper handling

2. Transformation Logic Updates

In litellm/llms/vertex_ai/gemini/transformation.py, I'll:

• Rename _process_gemini_image() to _process_gemini_file() to reflect its broader purpose
• Modify the file processing logic to conditionally apply base64 encoding based on MIME type
• Fix token counting for text-based files by properly handling them without unnecessary encoding

3. Implementation Benefits

• No UI changes required - users will use the existing interface
• Fixes the token miscalculation issue by preventing unnecessary base64 encoding of text files
• Enables support for all Gemini-supported MIME types as documented in Google's API

If this approach aligns with your expectations, I'd be happy to proceed with implementation.

Hi @NiharP31 can you define a clear success criteria for this issue ? Ideally 3-4 test cases you're hoping to pass

Test cases to validate the implementation & Success Criteria:

Test Case 1: Markdown File Processing

• Input: A Markdown file (lets assume 2-3KB in size)
• Expected Output:
  • File processed without ";base64" appended to MIME type
  • Token count accurately reflects actual content size (appx. ~500-1000 tokens, not millions)
  • Gemini model successfully processes and responds to the Markdown content

Test Case 2: JSON Data File

• Input: A JSON file (structured data is assumed here)
• Expected Output:
  • File properly processed with "application/json" MIME type without base64 encoding
  • Model can correctly reference and analyze the JSON structure in its response

Test Case 3: Mixed Content Input

• Input: A request containing both text (Markdown) and binary (image) files
• Expected Output:
  • Text files processed without base64 encoding
  • Binary files properly encoded with base64
  • Model successfully references both content types in its response

Test Case 4: Large Text File Handling

• Input: A larger text file (lets assume ~50KB or even more in size)
• Expected Output:
  • File processed without token count errors
  • Response acknowledges content without hitting artificial token limits caused by encoding issues

All the above test cases would verify that the system got correctly integrated with MIME type classification and conditional base64 encoding, ensuring Gemini can properly access the full range of file types it supports. More cases can be CSV files (Structured format), calling data from Google-Cloud Storage, etc.

ok, you can go ahead on implementing this @NiharP31

@johann-petrak what do you think of the test cases ?

Looks like a really good plan to me, thank you for proposing this!

One thought I have is that given the LiteLLM package provides a very consistent API for using a wide range of models,
if there is a way to make using file somewhat consistent across the models?
This is complicated by the fact that there exist basically two approaches for doing this: 1) upload a file in a separate request, get an ID, then send the prompt and reference the file ID in a special kind of prompt message and 2) directly send the file as part of the prompt message
As far as I can see the details of these approaches differ between providers (and also the kind of files supported) e.g. here is how this works with OpenAI and PDF https://platform.openai.com/docs/guides/pdf-files?api-mode=chat

Sorry if all this has been clear anyways ...

For me the most important aspect right now is support for Gemini, but I think especially with models very likely supporting much larger contexts and also with models that support multimodal prompting, sending files as part of a prompt will become much more widely used soon.

@johann-petrak got your point. I'm currently exploring the code-base. The current PR should provide a foundation for a more consistent file handling approach across different platforms. Once I'm done with the existing PR, I'd love to work on further integration if that would be helpful for the project.
@ishaan-jaff let me know your take on this.

if there is a way to make using file somewhat consistent across the models?

Hey @johann-petrak openai just added support for a new files message content type, it maps quite similarly to vertex's FileData part type - which would make sending gs://, etc. url's much easier. I'm using it in our Gemini audio file input implementation as well