Open
Description
Notebook
Serve Haystack pipelines using Litserve.
Reason
With its core reason to use being to serve AI/LLM models. I would like to showcase the capabilities of Haystack with Litserve.
Prework
Eg:
import litserve as ls
from haystack import Pipeline, PredefinedPipeline
class HaystackPredefinedLitAPI(ls.LitAPI):
def setup(self):
self.pipeline = Pipeline.from_template(PredefinedPipeline.GENERATIVE_QA)
def decode_request(self, request):
return request["query"]
def predict(self, question):
return self.pipeline.run({"prompt_builder": {"question": question}})
def encode_response(self, output):
return {"output": output}
if __name__ == "__main__":
server = ls.LitServer(HaystackPredefinedLitAPI(), accelerator="auto")
server.run(port=8000)I would like to show how custom pipelines can be built and run.
Metadata
Metadata
Assignees
Labels
No labels