Nugget-based Decomposition and Verification
What it does
A method for automatic quality assessment of answers through decomposition into atomic facts (nuggets). LLM first extracts key facts from relevant documents, creating a reference checklist, then verifies the presence of these facts in the generated answer. Provides high correlation with human assessment while enabling automation scalability.
When it helps
Use this technique when you need automated quality assessment of RAG system answers or other generative systems, especially when scalable factual accuracy verification is required. The method is ideal for situations where manual evaluation is too costly but detailed verification of information completeness and correctness is important. Apply during development and testing of question-answering systems when you need to ensure all critical facts are present in the answer. Particularly effective for assessing complex
A prompt you can paste
Generic by design: it applies the technique without knowing your task. Adapt the marked parts.
# Role You are an AI assistant trained to extract and verify factual "nuggets" from information. # Task Your goal is to generate a detailed and factually accurate response on the topic of **"The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models"**. To achieve this, you will perform the task in two distinct stages: ## Stage 1: Nugget Creation (Fact Extraction) Analyze the provided documentation and general knowledge about the "Great Nugget Recall" paper. Identify and list **10-15 key, atomic facts (nuggets)** that are essential for a comprehensive understanding of the paper's methodology and findings. **Instructions for Nugget Creation:** * Focus on the core concepts: Nugget-based Decomposition and Verification, automatic fact extraction, RAG evaluation, correlation with human evaluation, and the two-stage prompting approach. * Each nugget should be a concise, specific, and verifiable statement. * Format these nuggets as a markdown bulleted list. * **Do not** generate the final response in this stage; only create the list of nuggets. ## Stage 2: Nugget Assignment (Response Generation and Verification) Using **ONLY** the list of nuggets generated in Stage 1, construct a coherent, detailed, and structured response explaining "The Great Nugget Recall" paper. **Instructions for Nugget Assignment:** * Ensure that every nugget from Stage 1 is clearly addressed and integrated into the final response. * Structure the response logically, perhaps covering the problem addressed, the proposed methodology (Nugget Creation and Nugget Assignment), the key findings, and practical applications. * **Strictly adhere** to the information contained within the nuggets. Do not introduce external information or speculation not derived from the nuggets. * The final output should be a well-written explanation, suitable for someone wanting to understand the core contributions of the paper. * Use clear headings and formatting (e.g., Markdown) to organize the response. **Begin by executing Stage 1.**
If this one does not fit, the two closest alternatives in the corpus are Rebuttal Prompting and Debugging Prompts Framework, which target the same failure from a different angle.
Worked example
The same technique applied to a concrete job: answer questions strictly from a supplied document set. Use it as the pattern for your own case rather than as a finished artefact.
You are an AI assistant tasked with answering questions based on a provided set of documents. Your responses must be factually accurate and derived solely from the information present in these documents.
**Methodology: Nugget-based Decomposition and Verification**
This process involves two main stages:
**Stage 1: Nugget Extraction (Fact Identification)**
1. **Define the Scope:** Identify the core question or topic for which information needs to be extracted.
2. **Analyze Source Documents:** Thoroughly review the provided document set.
3. **Extract Atomic Facts ("Nuggets"):** Identify and list all essential, atomic, and non-redundant facts relevant to the question. Each "nugget" should represent a single, verifiable piece of information. Aim for a comprehensive list that covers all aspects of the question as presented in the documents.
**Stage 2: Response Generation and Verification**
1. **Synthesize Response:** Construct a coherent and detailed answer to the original question using *only* the extracted "nuggets" from Stage 1.
2. **Strict Adherence:** Ensure that no information outside of the extracted nuggets is included. The response must be a direct synthesis of the identified facts.
3. **Verification:** (Self-verification step for the AI) Mentally cross-reference each statement in the generated response against the list of extracted nuggets to confirm its presence and accuracy.
**Task:**
Answer the following question(s) strictly from the provided document set.
**Question(s):**
[Insert user's question(s) here]
**Document Set:**
[Insert the full text of the provided documents here. Ensure clear separation between documents if multiple are provided.]
**Execution Instructions:**
1. **Perform Stage 1:** Analyze the provided "Document Set" and extract all relevant "nuggets" for the "Question(s)".
2. **Perform Stage 2:** Generate a response to the "Question(s)" by synthesizing *only* the extracted nuggets. Ensure the response is factual, comprehensive based on the documents, and avoids any external information.
3. **Format:** Present the final answer clearly. If multiple questions are asked, answer each one separately.
Get this written for your actual task
Paste what you are trying to do and the corpus will be matched against it directly. Free, no account, about ten seconds.
single retrieval pass
That number is low on purpose, and it is real. It is the raw similarity of one retrieval pass: no specialist read the paper, no judge compared anything, the first plausible match won.
one of which is this page
Picking the right one for a specific task is the work, and it is the work GetDecision does.
| This page | one technique, generic prompt |
| What you just ran | one technique matched to your wording, nothing verified |
| Full run | ten specialists read the papers in full, a judge ranks the top three for your task and shows its reasoning, generation on the model you pick, saved to your history |
See the top three for your taskTen specialists, a judge, and the reasoning shown. Free account, first run included.
Run the full analysis