RACE

In recent years, researchers have proposed numerous benchmarks to evaluate the impressive coding capabilities of large language models (LLMs). However, current benchmarks primarily assess the accuracy of generated code while neglecting the LLMs' multi-dimensional instruction-following abilities, resulting in a gap between benchmark performance and real-world development requirements. Therefore, this paper proposes the RACE benchmark, which focuses on evaluating LLMs' instruction-following ability across four dimensions: Readability, mAintainability, Correctness, and Efficiency. We select specific, actionable attributes for each dimension and establish verifiable instructions grounded in common software development practices. We then develop deterministic evaluation methods to ensure objective and reliable assessment.

We analyze 24 representative LLMs based on RACE and find that: 1) Using correctness as the only guiding indicator is insufficient to steer code LLMs towards comprehensive advancement. 2) RACE not only provides valuable insights that reveal the defects of current LLMs across multiple dimensions, but also serves as an effective tool for mitigating the risks of data contamination. These findings highlight the need for a multi-dimensional instruction-following evaluation for LLMs' code ability, emphasizing metrics beyond correctness for real-world applications.