You suggest that hosting your LLM will help but is it truly the case? Why wouldn’t you experience similar issues with a hosted model?
You did not say much about building a comprehensive set of automated LLM test evals and observability/monitoring services in PROD. Isn’t it critical to detect such issue?
The other big question is how much does it add to your TCO? It seems so critical that I’d mot be surprised to see dedicated teams to monitor just that. Thoughts?
Super interesting article and research papers!
You suggest that hosting your LLM will help but is it truly the case? Why wouldn’t you experience similar issues with a hosted model?
You did not say much about building a comprehensive set of automated LLM test evals and observability/monitoring services in PROD. Isn’t it critical to detect such issue?
The other big question is how much does it add to your TCO? It seems so critical that I’d mot be surprised to see dedicated teams to monitor just that. Thoughts?