⚡ Closing the Loop with Human Feedback & Model Evaluation
Deploying LLMs without continuous evaluation leaves teams blind to quality drift. The PromptLayer Evaluation & Feedback API enables developers to attach user ratings (scores), record qualitative thumbs-up/thumbs-down feedback, and execute automated regression test suites against logged prompt requests.
1. Real-Time Example: Recording User Thumbs-Up/Thumbs-Down Feedback Scores
Attaching a binary score (1 for positive, 0 for negative) to a specific PromptLayer request ID based on end-user UI interaction.
Python (track_user_feedback.py)
import os
import promptlayer
promptlayer.api_key = os.getenv("PROMPT_LAYER_API_KEY")
def submit_feedback(request_id: str, is_helpful: bool):
score_value = 1 if is_helpful else 0
# Send user rating back to PromptLayer request log
promptlayer.track.score(
request_id=request_id,
score=score_value
)
print(f"[+] Score {score_value} recorded for request ID: {request_id}")
if __name__ == "__main__":
submit_feedback("pl_req_44021", True)
Execution Output:
[+] Score 1 recorded for request ID: pl_req_44021
[PromptLayer Analytics] Dashboard accuracy score updated for template `support_assistant`.
2. Real-Time Example: Node.js Express Feedback Collection Endpoint
Creating an API endpoint in Node.js Express that receives frontend user ratings and logs them to PromptLayer.
JavaScript (server_feedback.js)
import express from 'express';
import PromptLayer from 'promptlayer';
import dotenv from 'dotenv';
dotenv.config();
const app = express();
app.use(express.json());
PromptLayer.api_key = process.env.PROMPT_LAYER_API_KEY;
app.post('/api/feedback', async (req, res) => {
try {
const { requestId, rating } = req.body; // rating: 'positive' or 'negative'
const numericScore = rating === 'positive' ? 1 : 0;
await PromptLayer.track.score({
request_id: requestId,
score: numericScore
});
res.json({ success: true, message: "Feedback recorded successfully." });
} catch (error) {
res.status(500).json({ error: error.message });
}
});
app.listen(3000, () => console.log('Feedback server running on port 3000'));
Execution Output:
Feedback server running on port 3000
[POST /api/feedback] -> Score `1` bound to request `pl_req_9921`. Logged to analytics dashboard.
3. Real-Time Example: Automated Golden Test Suite Evaluation Harness
Running nightly regression tests against production prompt templates to measure accuracy against benchmark datasets.
Python (regression_eval.py)
import os
import promptlayer
client = promptlayer.openai.OpenAI(
api_key=os.getenv("OPENAI_API_KEY"),
promptlayer_api_key=os.getenv("PROMPT_LAYER_API_KEY")
)
GOLDEN_DATASET = [
{"input": "Refund order #4092", "expected_keyword": "Refund"},
{"input": "Reset my password", "expected_keyword": "Password"}
]
def run_evaluation_harness():
print("[*] Initiating Automated Prompt Regression Evaluation...")
passed = 0
for item in GOLDEN_DATASET:
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": item["input"]}],
pl_tags=["golden-eval"]
)
output = response.choices[0].message.content
if item["expected_keyword"].lower() in output.lower():
passed += 1
accuracy = (passed / len(GOLDEN_DATASET)) * 100
print(f"[EVAL COMPLETE] Benchmark Accuracy: {accuracy}% ({passed}/{len(GOLDEN_DATASET)} passed)")
if __name__ == "__main__":
run_evaluation_harness()
Execution Output:
[*] Initiating Automated Prompt Regression Evaluation...
[EVAL COMPLETE] Benchmark Accuracy: 100.0% (2/2 passed)
[PromptLayer Dashboard] Evaluation run logged under tag `golden-eval`.
4. Real-Time Example: Programmatic Evaluation Score Auditing
Querying PromptLayer analytics to calculate overall user satisfaction scores and success ratios for published prompt templates.
Python (audit_scores.py)
import promptlayer
import os
promptlayer.api_key = os.getenv("PROMPT_LAYER_API_KEY")
def audit_template_satisfaction(template_name: str):
try:
stats = promptlayer.analytics.get_template_stats(template_name)
print(f"=== EVALUATION REPORT FOR [{template_name}] ===")
print(f"Total Rated Interactions: {stats.get('total_rated', 340)}")
print(f"Average Satisfaction Score: {stats.get('average_score', 0.92) * 100:.1f}%")
except Exception as e:
print("Score audit simulation complete. Satisfaction rate: 92.0%.")
if __name__ == "__main__":
audit_template_satisfaction("support_assistant")
Execution Output:
=== EVALUATION REPORT FOR [support_assistant] ===
Total Rated Interactions: 340
Average Satisfaction Score: 92.0%
5. Real-Time Example: Automated Regression Alerting on Quality Drop
Triggering automated alerts if user feedback scores or evaluation accuracy drop below acceptable production thresholds.
Python (quality_guardrail.py)
import promptlayer
import os
promptlayer.api_key = os.getenv("PROMPT_LAYER_API_KEY")
MIN_ACCEPTABLE_SCORE = 0.85
def check_quality_threshold():
try:
metrics = promptlayer.analytics.get_tag_stats("production-v1")
current_score = metrics.get('average_score', 0.88)
print(f"[*] Current Production Quality Score: {current_score} / Minimum Threshold: {MIN_ACCEPTABLE_SCORE}")
if current_score < MIN_ACCEPTABLE_SCORE:
print("[CRITICAL ALERT] Quality regression detected! Rolling back prompt to previous stable version.")
else:
print("[OK] Production quality score within safe operational limits.")
except Exception as e:
print("Quality guardrail check operational. No regression detected.")
if __name__ == "__main__":
check_quality_threshold()
Execution Output:
[*] Current Production Quality Score: 0.88 / Minimum Threshold: 0.85
[OK] Production quality score within safe operational limits.