huggingface text-generation-inference · Discussions · GitHub

This repository was archived by the owner on Mar 21, 2026. It is now read-only.

Is there a way to get GenerateResponse full json when calling sagemaker endpoints
Q&A nth-attempt

Discussions

You must be logged in to vote

Load multiple models in a single TGI container

Rictus started Apr 18, 2025 in Ideas

1
You must be logged in to vote

TGI service, how to pass sampling parameters using OpenAI interface？

zyt1024 asked Aug 1, 2025 in Q&A · Unanswered

1
You must be logged in to vote

Nvidia Spark / ARM64

daz-williams started Nov 2, 2025 in General

2
You must be logged in to vote

Default model precision in TGI

jerin-metrum-ai started Oct 30, 2023 in General · Closed

2
You must be logged in to vote

How to use TGI for a finetuned LORA adapter of llama-3.3-70B using unsloth?

InderjeetVishnoi started Jun 26, 2025 in General

0
You must be logged in to vote

Does HF TGI support Multi Node -Multi GPU server set up ?

ansSanthoshM asked Feb 14, 2024 in Q&A · Unanswered

9
You must be logged in to vote

I want to express my gratitude for this project

zacksiri started Apr 21, 2025 in General · Closed

0
You must be logged in to vote

Is it possible to get all_input_ids in forward or prepare_inputs_for_generation for a custom model?

Kurokabe asked Apr 14, 2025 in Q&A · Unanswered

0
You must be logged in to vote

Do we have any Grafana Dashboard example on top of TGI metrics ? Could metrics be explained in more detail ?

luk-cer started Oct 10, 2023 in Ideas · Closed

15
You must be logged in to vote

Add Sentry / Glitchtip error tracking integration

andresC98 started Mar 25, 2025 in Ideas

0
You must be logged in to vote

Setting up an environment for Langchain development with TGI Containers

penard-monkey asked Mar 17, 2025 in Q&A · Unanswered

0
You must be logged in to vote

Keras Support Needed

chintan-ushur asked Feb 24, 2025 in Q&A · Unanswered

0
You must be logged in to vote

Installation stucked on "build wheel for flash-attn"

celsowm asked Feb 14, 2025 in Q&A · Unanswered

0
You must be logged in to vote

Huggingface-TGI: Local model not recognized

n-morgan asked Aug 14, 2024 in Q&A · Unanswered

2
You must be logged in to vote

Is TGI not available for ARM64 platform?

SMAntony asked Jul 18, 2024 in Q&A · Unanswered

1
You must be logged in to vote

Other generation strategies

dachenlian asked Dec 11, 2023 in Q&A · Unanswered

1
You must be logged in to vote

How to control request size within a batch?

cringelord000222 asked Jun 6, 2024 in Q&A · Unanswered

2
You must be logged in to vote

Question about the temperature parameter in the Hugging Face Inference API

xufengduan asked Jan 6, 2025 in Q&A · Unanswered

1
You must be logged in to vote

Error for exllamav2_kernel, running TGI on Google Colab

andychoi98 asked Apr 18, 2024 in Q&A · Unanswered

6
You must be logged in to vote

an beginner for TGI, for to trigger the op which is written by triton, use which model-id, thanks

alanguo1234 asked Nov 19, 2024 in Q&A · Unanswered

0
You must be logged in to vote

Are there benchmarks comparing Tgi performance on gpus like the rts 2080ti and amd 6800xt ?

phoenixtomcat asked Nov 16, 2024 in Q&A · Unanswered

0
You must be logged in to vote

Where can I see the chat template that a model is using?

Simon-Stone asked Oct 22, 2024 in Q&A · Unanswered

0
You must be logged in to vote

Function calling structure

ankush13r asked Oct 21, 2024 in Q&A · Unanswered

0
You must be logged in to vote

How to truncate using TGI? input tokens exceeded max_input_tokens.

LanSnowZ asked Oct 16, 2024 in Q&A · Unanswered

0
You must be logged in to vote

Repetition Aware Sampling in VALL-E 2

zyt1024 asked Oct 9, 2024 in Q&A · Unanswered

0