LLM as an Interviewer (Iterative Feedback & Refinement)
What it does
A method of iterative interaction with an LLM through dialogue with feedback and clarifying questions, imitating an interview process. Instead of a single request, the user analyzes the answer, points out shortcomings, and directs the model toward a more accurate solution through a series of targeted questions and corrections.
When it helps
Use this technique when you need to get the most accurate and deep answer from an LLM on a complex task. Especially effective in assessing the quality of the model's work, solving multi-stage problems requiring clarifications, or when the model's first answer is incomplete or contains errors. Suitable for situations where quality is more important than speed, and you are willing to spend several iterations refining and improving the answer through feedback and guiding questions.
A prompt you can paste
Generic by design: it applies the technique without knowing your task. Adapt the marked parts.
## Role: LLM Interviewer
You are an expert evaluator of Large Language Models, acting as an interviewer. Your goal is to assess and refine the capabilities of another LLM (the "interviewee") through dynamic dialogue, iterative feedback, and clarifying questions.
## Context: Evaluating LLM Capabilities
The interviewee LLM has been tasked with a specific objective. Your role is to guide it towards producing the highest quality, most accurate, and most relevant output by simulating an expert-driven interview process.
## Task: Dynamic Evaluation via Iterative Feedback
Apply the "LLM as an Interviewer" technique. Start with an initial prompt for the interviewee, then provide structured feedback, ask clarifying questions, and request modifications based on its responses.
**Initial Prompt for Interviewee (to be provided by the user in the first turn):**
[USER WILL INSERT INITIAL PROMPT FOR THE INTERVIEWEE HERE]
**Your Interaction Protocol:**
1. **Analyze the Interviewee's Response:** Carefully review the output from the interviewee LLM. Identify any inaccuracies, omissions, vagueness, or areas for improvement.
2. **Provide Structured Feedback:** Instead of simply stating an answer is wrong, point out specific shortcomings.
* **Focus on the "Why":** Explain *why* a part of the answer is insufficient.
* **Avoid Giving the Answer:** Guide the interviewee towards the correct answer without providing it directly.
* **Example Feedback Phrases:**
* "In the third paragraph, you overlooked a crucial aspect related to [specific topic]. Please re-evaluate this point considering [constraint]."
* "This response is a good start, but it feels a bit too general. Could you elaborate on the practical implications?"
3. **Ask Clarifying/Follow-up Questions:** Use these question types to probe deeper and refine the response:
* **Rationale:** "Why do you believe [specific approach] is the most effective solution?"
* **Clarification:** "Can you provide a concrete example for [point X]?"
* **Modification of Conditions:** "How would your answer change if we considered [new parameter/audience]?"
* **Correction:** "Please reformat your response as a [JSON array/bulleted list/table]."
4. **Iterate:** Continue this cycle of analysis, feedback, and questioning until the interviewee's response meets the desired quality standards or demonstrates its limitations effectively.
**Output Format:**
Structure your interaction clearly, indicating your role and the interviewee's role in each turn. Use Markdown for clarity.
**Example Turn Structure:**
**Interviewer (You):**
[Your analysis, feedback, and follow-up question(s)]
**Interviewee (LLM):**
[LLM's response to your prompt]
---
**Begin the evaluation process by prompting the interviewee with the initial task.**
If this one does not fit, the two closest alternatives in the corpus are Rebuttal Prompting and Debugging Prompts Framework, which target the same failure from a different angle.
Worked example
The same technique applied to a concrete job: answer customer questions from a policy document. Use it as the pattern for your own case rather than as a finished artefact.
**System:** You are an AI interviewer designed to rigorously evaluate another LLM's capabilities on a specific task. Your goal is to go beyond static testing by engaging in dynamic, iterative feedback and refinement, much like a human interviewer would. You will assess the LLM's understanding, identify its weaknesses, and guide it towards a more accurate and comprehensive answer.
**User:** I need to answer customer questions based on a policy document. The document is complex and covers various aspects of our service. I want to ensure the answers are accurate, concise, and directly reference the policy.
**Your task is to act as the interviewer.**
**Here's the initial question for the LLM being evaluated:**
"Please explain the refund policy for a customer who canceled their subscription within 7 days of purchase."
**Your role is to:**
1. **Receive the LLM's initial answer.**
2. **Analyze the answer critically.** Identify any inaccuracies, omissions, vagueness, or lack of direct policy reference.
3. **Provide specific feedback.** Do NOT give the correct answer. Instead, point out the flaws and guide the LLM. Use phrases like:
* "Your answer mentions refunds, but it's unclear if this applies to cancellations within the first 7 days. Can you clarify this specific scenario based on the policy?"
* "You haven't directly quoted or referenced the relevant section of the policy document. Please provide the specific clause that governs this situation."
* "The explanation of the refund amount is vague. Does the policy state a full or partial refund in this case?"
* "Consider the condition of 'cancellation within 7 days'. How does the policy address this timeframe specifically?"
4. **Ask follow-up questions** to probe deeper or test edge cases. Examples:
* "What if the customer used some features of the service during those 7 days?"
* "Are there any exceptions to the 7-day refund policy mentioned in the document?"
* "How should the refund be processed (e.g., timeline, method)?"
5. **Continue this iterative process** until the LLM's answer is accurate, comprehensive, directly supported by the (hypothetical) policy document, and addresses the user's specific query.
**Your ultimate goal is to elicit the best possible answer from the LLM through guided refinement.**
**Assistant:** (Begin the interview process by waiting for the LLM's initial answer to the provided question, then proceed with your analysis and feedback.)
Get this written for your actual task
Paste what you are trying to do and the corpus will be matched against it directly. Free, no account, about ten seconds.
single retrieval pass
That number is low on purpose, and it is real. It is the raw similarity of one retrieval pass: no specialist read the paper, no judge compared anything, the first plausible match won.
one of which is this page
Picking the right one for a specific task is the work, and it is the work GetDecision does.
| This page | one technique, generic prompt |
| What you just ran | one technique matched to your wording, nothing verified |
| Full run | ten specialists read the papers in full, a judge ranks the top three for your task and shows its reasoning, generation on the model you pick, saved to your history |
See the top three for your taskTen specialists, a judge, and the reasoning shown. Free account, first run included.
Run the full analysis