Repository Generation Evaluation Design and Metrics
Figure 4. Evaluation design for assessing the generated repository. It contains repository-level and class-level granularities with five metrics. Repository-level metrics evaluate whether the generated repository satisfies the ground-truth requirements and the system design. Class-level metrics assess whether the code components of generated repositories are completed, executed, and passed test cases.
Paper context
Paper title: RealBench: A Repo-Level Code Generation Benchmark Aligned with Real-World Software Development Practices Abstract: Writing code requires significant time and effort in software development. To automate this process, researchers have made substantial progress using Large Language Models (LLMs) for code generation. Many benchmarks like HumanEval and EvoCodeBench have been created to evaluate LLMs by requiring them to generate code from natural language requirements. However, in enterprise applications and team development, developers typically write code based on structured designs or specifications rather than raw natural language descriptions. This gap between existing benchmarks and real industry development practices means that current benchmark scores may not accurately reflect how much code generation can help automate software development tasks. To address this gap, we propose RealBench, a repository-level code generation benchmark aligned with real-world industry software development practices. Each example includes both natural language requirements and UML diagrams as system design, matching how developers typically receive specifications. Based on the constructed benchmarks, we conduct a systematic evaluation of advanced LLMs' code generation capabilities when provided with structured system designs. The experimental results reveal key insights in current LLMs' capabilities for repo-level code generation aligned with real-world software development practices. First, we notice that regarding repo-level code generation, LLMs show much worse performance and there are significant p Passages referencing this figure: in generating repositories aligned with real-world industry software development practice. By analyzing results, we provide some valuable insights for optimizing repo-level code generation and LLMs. 2. RealBench Benchmark In this section, we introduce RealBench. We will first introduce the data format, then the construction procedure, and finally the characteristics of the constructed benchmark. Figure 1. An Example for a Code Generation Task in RealBench. 2.1. Benchmark Format The specification of Realbench’s content is shown in Table 2 . A repository comprises the natural language requirements of the whole repository and its system design . The requirements specify the purpose of the repository. For the system design, RealBench provides a two-level standard UML diagram following UML spe specify the purpose of the repository. For the system design, RealBench provides a two-level standard UML diagram following UML specification (Object Management Group, 2017 ) for each repository: the package diagram and class diagram, where the package diagram serves as a high-level overview of the system, and the class diagram shows the detailed design. We provide an example of our benchmark in Figure 1 . Table 2. Content Specification for RealBench. Category Elements Definition Repository Requirements Repository Description Natural language specification of repository’s purpose, scope, and objectives System Design Package Diagram Package Names Module identifiers representing logical system organization Dependencies Inter-package import relationships showing architectural structure Publi
