[BLOCKER] API Gateway timeout breaking all service!

[supreme-gg-gg]

This was very unexpected and I did not see this anywhere when implementing API Gateway #36...

It seems like there's a default timeout for 60 seconds and a maximum of 60 seconds for API Gateway. This is a HUGE problem because some services take longer like inference because it runs GPU computation. while the Cloud Run can be run for longer up to 3600s timeout, the gateway does not wait and cuts the request.

THIS IS VERIFIED TO BE A BLOCKING ISSUE on various endpoints, mostly inference and preprocessing. Both services produced valid response (seen in logs), but it breaks on frontend / api testing because the gateway already cut the request...

I do not know how to fix this to be honest... 😢 😢 😢 might need to remove the entire gateway if this is the case, i don't have a choice here if this is the case, I will just build another gateway with cloud endpoints on our own managed Cloud Run router to do auth and user IAM to do trigger similarly

[inclinedadarsh]

Though I'm still triaging this issue, another possible solution could be simple JWT auth.

So rather than having a gateway, we do JWT validation right in our services. A pro is that we remove the gateway entirely, so no such dependencies. I believe we'll be able to scale to thousands of users like this without any issue.

A con is that we'll have to implement the same JWT auth in all the services. If we make design changes in the future in auth, we'll have to implement the same change in every service.

However, even if we increase the gateway time limit in it, it might be possible that we implement another API endpoint in the future that exceeds that time limit, which again will be an issue. As we're working with LLMs, I feel this is a very possible case.

All in all, I think implementing JWT in the services will be a quicker solution for now, unless we are able to have different testing and production environment, and more flexibility on how we can test.

[supreme-gg-gg]

Yes simple JWT auth was original plan and most solid but repeated code is not very good plus frontend needs to manage 3 urls. That's why I prefer having a gateway still now, maybe our custom one with Cloud Run + Cloud Endpoints if you can investigate