I had an interview question regarding LLM. How exaclty do you deploy LLM, what are your consideration in terms of speed, resource, imbalance load, and all that stuff?
You must log in or register to comment.
If you’re asking that question here, you ma not be qualified for the job.
I think you’re looking at the problem wrong by focusing just on the LLM aspect of it. If you’re deploying any type of application then it will depend on the demand you’re expecting from users and the use cases the application will be used. A failure rate for medical applications is probably a lot more important than a low budget game service.
Call the API using requests.post(…)
Troglodyte