What is a primary challenge in evaluating ChatGPT models using automated metrics?

A primary challenge in evaluating ChatGPT models using automated metrics is their inability to fully capture the nuances of human language and understanding, specifically in areas like context, coherence, and factual accuracy. Automated metrics, such as BLEU and ROUGE, typically assess the similarity between the generated text and a reference text based on n-gram overlap (sequences of n words). While these metrics can provide a quick quantitative assessment of fluency and grammatical correctness, they often fail to capture higher-level aspects of text quality. For instance, a generated response might have a high BLEU score because it uses similar words and phrases as the reference text, but it may still be incoherent, irrelevant, or factually incorrect. Automated metrics struggle to assess whether the generated text makes sense in the given context, maintains a consistent flow of ideas, or provides accurate information. They also struggle with paraphrasing, where the generated text expresses the same meaning as the reference text using different words. Therefore, while automated metrics can be useful for initial screening, they should be complemented by human evaluation to obtain a more comprehensive and reliable assessment of the model's performance. A model could achieve a high automated score, but completely miss the underlying context that a human would immediately grasp.