Description
LLMEval is a research series from Fudan NLP Lab dedicated to creating comprehensive, fair, and robust evaluation frameworks for large language models (LLMs). It spans over 13 academic disciplines, medical AI, and includes more than 220,000 generative questions. The initiative aims to provide rigorous and fair evaluation frameworks for LLMs, ensuring their performance is accurately assessed across different domains.
Read More
One of the key components of LLMEval is LLMEval-Logic, an open-source benchmark for logical reasoning in LLMs. This benchmark includes 196 public base items and 154 adversarially hardened items, with Z3-verified answers and expert rubrics. The evaluation pipeline is reproducible, allowing researchers to test and improve LLMs' logical reasoning capabilities.
LLMEval-Fair is another significant part of the initiative, focusing on robustness and fairness in LLM evaluation. It involves a 30-month longitudinal study using a proprietary bank of 220,000 graduate-level questions. The automated pipeline ensures data integrity through contamination-resistant curation and a novel anti-cheating architecture. This study has revealed performance ceilings and data contamination vulnerabilities in nearly 60 leading models.
Additionally, LLMEval-Med provides a real-world clinical benchmark for evaluating medical LLMs. It covers five core medical areas and includes 2,996 questions from electronic health records and expert-designed scenarios. The evaluation pipeline is validated through human-machine agreement analysis, ensuring high accuracy in medical AI evaluation.
LLMEval is open to the public, with code and data available on GitHub. Researchers and developers are encouraged to explore and collaborate on improving LLM evaluation frameworks.
- CategoryAI Models & LLMs
- Pricing ModelOpen source
- DeploymentCloud
- PlatformsWeb
LLMEval's Core Features
Comprehensive evaluation frameworks
Open-source logical reasoning benchmark
Fairness and robustness evaluation
Medical AI benchmark
220,000+ generative questions
Z3-verified answers
Contamination-resistant data curation
Human-machine agreement analysis
How to use LLMEval?
Explore: Visit the LLMEval website
Access: Download datasets from GitHub
Evaluate: Use benchmarks for LLM testing
Collaborate: Contact for research partnerships
LLMEval's Use Cases
- Academic Research
- Medical AI Evaluation
- Logical Reasoning Testing
- Fairness Assessment
- Collaborative Research
