swt-bench
[NeurIPS 2024] Evaluation harness for SWT-Bench, a benchmark for evaluating LLM repository-level test-generation
Open the source (opens github.com) https://github.com/logic-star-ai/swt-bench
What it is
Datasets & benchmarks from logic-star-ai.
The problem it solves and who it suits haven't been reviewed by InnerVault. The description above is the source's own.
How to get it
Get the code
git clone https://github.com/logic-star-ai/swt-bench.git
Standard git access to a public repository
Setup steps: not provided by the source. Follow the instructions on the source.
Related
-
WritingBench
Datasets & benchmarksWritingBench: A Comprehensive Benchmark for Generative Writing
git clone https://github.com/X-PLUG/WritingBench.git -
AgentLab
Datasets & benchmarksAgentLab: An open-source framework for developing, testing, and benchmarking web agents on diverse tasks, designed for scalability and reproducibility.
git clone https://github.com/ServiceNow/AgentLab.git -
UHGEval
Datasets & benchmarks[ACL 2024] User-friendly evaluation framework: Eval Suite & Benchmarks: UHGEval, HaluEval, HalluQA, etc.
git clone https://github.com/IAAR-Shanghai/UHGEval.git -
ollama-benchmark
Datasets & benchmarksLLM Benchmark for Throughput via Ollama (Local LLMs)
git clone https://github.com/aidatatools/ollama-benchmark.git -
swe-sweep
Datasets & benchmarksHow many bugs can LMs find & fix in large codebases?
git clone https://github.com/facebookresearch/swe-sweep.git -
llm-colosseum
Datasets & benchmarksBenchmark LLMs by fighting in Street Fighter 3! The new way to evaluate the quality of an LLM
git clone https://github.com/OpenGenerativeAI/llm-colosseum.git