DOC: "First Token" details and explanation

[khteh]

Issue with current documentation:

https://docs.smith.langchain.com/reference/data_formats/run_data_format

first_token_time	datetime	Time when the first token was generated.

When is the start and stop of the measurement and the events that trigger them?

Idea or request for content:

No response

[tanushree-sharma]

Thanks! Docs PR: langchain-ai/langsmith-docs#888

Do you have a specific question about this that we can help with? Otherwise will close this issue out.

[khteh]

When does the clock start ticking? Or what triggers the start of the time measurement?

[tanushree-sharma]

The start time gets set here. It is determined by examining the runs associated with the trace and finding the first new_token event

[khteh]

Still unclear to me. It's not a trick question. I just don't understand what this "Frist Token" measures.

[tanushree-sharma]

Time to First Token is the time between sending a prompt to an LLM and receiving its first generated token. It’s an important metric for user experience because it leads to a more responsive app. Hope that clarifies things.

[khteh]

receiving its first generated token.

Which party receives its first generated token? The end user who sends it the prompt or the tool which the LLM sends its query (token) to?