I had an interview question regarding LLM. How exaclty do you deploy LLM, what are your consideration in terms of speed, resource, imbalance load, and all that stuff?
I had an interview question regarding LLM. How exaclty do you deploy LLM, what are your consideration in terms of speed, resource, imbalance load, and all that stuff?
I think you’re looking at the problem wrong by focusing just on the LLM aspect of it. If you’re deploying any type of application then it will depend on the demand you’re expecting from users and the use cases the application will be used. A failure rate for medical applications is probably a lot more important than a low budget game service.