📎 Drop documents here or click to upload
Supported: .docx, .txt, .md, .pdf, .html, .htm — anything else is skipped with a noteNo documents uploaded yet.
Working on:
Demo Question Set
draft
⬇ Export
✦ Rubric
Filter by tag:
1
What information is available in this system?
no tags
2
How do I get started using this service?
no tags
2.1
What do I need to sign up?
no tags
2.2
How long does the setup process take?
no tags
3
Who can I contact if I need help?
no tags
4
What happens if I provide incorrect information?
no tags
1
What information is available in this system?
no tags
2
How do I get started using this service?
no tags
2.1
What do I need to sign up?
no tags
2.2
How long does the setup process take?
no tags
3
Who can I contact if I need help?
no tags
4
What happens if I provide incorrect information?
no tags
Question / AnsweriThe question or concept being tested, and the ideal answer the AI should produce.
Sync to semantic variations
ClassificationiHow this question is categorized — affects grouping, filtering, and sampling.
Sync to all sub-concepts
Sync to semantic variations
Sync to semantic variations
Sync to semantic variations
ScoringiAdditional quantifiable criteria for scoring and measuring the AI's response beyond pass/fail. Use these to capture nuance in how well the AI answered, not just whether it passed.
Sync to all sub-concepts
Sync to semantic variations
Sync to all sub-concepts
Sync to semantic variations
NotesiInternal notes for the question designer.
Working on:
Demo Question Set
draft
Basic AI Assist
AI-powered question and variation generationQSet Library
Document Libraries
Corpus
Editing:
No library
QSet Assistants
Prompts used by the QSet AI Assist features. Pre-loaded defaults can be copied and customized.
AI Judge
Judge prompts are versioned and reusable across all runs.
Run Setup
Configure New Run
Test Run Name
✓ READY
Question SetiChoose which Question Set and which specific questions to include in this run. You can run all questions or select a subset by tag, status, or individually.
✓ READY
No concepts yet. Add concepts in the Concepts section first.
AI System Answering QSet
✓ READY
Choose Hosted Models or External Connections above to continue.
ModeliThe AI model that will answer each question in this run. No default on purpose — picking a model is a deliberate, sometimes-costly decision, not something that should happen by accident.
System PromptiInstructions given to the AI before each question. This is where you define the AI's role, the document context, and any constraints on how it should respond. Use {context} to control where corpus documents are injected.
Select a prompt from the library above to use it in this run.
Use {context} to control corpus placement, or leave it out to append at the end.
Document Library ScopeiWhich documents to send as context for this run — pick a library and either the entire thing or a custom selection, or explicitly choose "No Library" to test the model on the question alone. No default — this shapes the answer as much as the model itself does.
Run against the entire selected library
All documents in the selected library sent as context.
No documents in corpus yet.
External ConnectioniSelect a saved external connection to test against. Configure connections under Data Connections → External Connections.
AI Judge Scoring AnswersiThe AI Judge prompt used to score responses after the run. The judge evaluates each answer against the ground truth and rubric, providing a score and rationale. Configure judge prompts under AI Prompts.
✓ READY
Select a judge from the library above.
Reference LibraryiOptional. A document library the AI Judge can additionally check the answer against, on top of the QSet's own ground truth. Most runs score fine off ground truth alone — this doesn't need to be set. Independent of whichever library (if any) the answering system itself used.
Reference the entire selected library
All documents in the selected library available to the AI Judge.
Select a library above to reference documents.
AI Scoring Methods (Coming Soon)iAutomated scoring methods applied to each response after the run. G-Eval (LLM Judge) runs immediately. Methods marked Phase 2 require the Python scoring backend and will be available in a future release.
Select which methods score each response. Scores appear as columns in Results. Methods marked ⚠ require the Python scoring backend (Phase 2).
Variation SettingsiWhether variations are included at all is controlled by "Include semantic variations" above, right next to the question list. Human scoring on variations is optional.
Require human scoring on variations
If off, variations are scored by AI judge only.
Reviewer VisibilityiControls what information is shown to human reviewers and the AI judge during scoring. Showing ground truth, rubric, and key elements helps reviewers make consistent judgments.
Applies to both human reviewers and AI judge for this run.
Show ground truth
Show rubric
Show key elements
Run Library
Assign Scoring
Showing assignments for:
My Assignments
0 / 0 reviewed
Select a run and question to begin reviewing.
Results
Compare Runs
Run A
↔
Run B
Show
Configuration
Sort Questions By
Select runs above to compare results.
Report Library
Start a New Report
Saved Reports
External Connections
Runs
Report Sections
Click ▶ Select Runs to Analyze in the sidebar to get started.
Deploy AI
Phase 2 — Coming Soon
Turn Your Validated QSet Into a Live AI Deployment
Once you have a QSet, corpus, system prompt, and model configuration that performs well in testing, Deploy AI lets you publish it as a live chatbot — shareable with clients, embeddable on their website, and continuously monitored against your benchmark.
What's Coming
🤖
Hosted Chatbot
Publish any Run Setup configuration as a live chat interface. Shareable via URL, embeddable as an iframe or widget on a client's website. Optional password protection and usage limits.
💬
Conversation Interface
A clean chat UI backed by your configured model, corpus, and system prompt. Maintains conversation history across turns. Branded optionally with client name and logo.
📋
Conversation Logging
Every user conversation is logged and linked to the originating project. See what real users actually ask versus what you tested in the QSet.
🔄
QSet Feedback Loop
Real user questions that fall outside your QSet automatically surface as candidates for new test questions — closing the gap between designed tests and real-world usage.
📡
Custom Endpoint Mode
Point Deploy AI at an external vendor's live API instead of Signet42's hosted model. Test their black-box deployment the same way a real user would, with full logging and scoring.
📈
Live Monitoring
Scheduled re-runs of your QSet against the deployed configuration. Automatic alerts when a question that previously passed begins to fail — catching model updates, prompt drift, or data changes.
🔐
Access Control
Manage who can access each deployed chatbot. Invite-only links, domain restrictions, and per-client usage dashboards.
📊
Deployment Analytics
Usage volume, response latency, cost per conversation, and quality scores over time — all tied back to your benchmark results for a complete picture of AI performance in production.
Data Connections — Coming Soon
🌐
Website Ingestion
Point Signet42 at any URL and it crawls the site recursively, strips HTML to plain text, and stores each page as a corpus document. Configure crawl depth, URL include/exclude patterns, and scheduled refresh frequency to keep your corpus current as the site changes.
🔗
Connected Sources
Connect directly to existing data sources — SharePoint, Confluence, Google Drive, S3 buckets, databases, or custom APIs. Each connector authenticates, fetches documents, extracts text, and syncs on a schedule. Eliminates manual upload and keeps the corpus live with source changes.
Database Impact
Deploy AI will add
chatbot_deployments and conversations tables to the schema.
Data Connections will add data_sources and crawl_jobs tables.
The existing measurement and monitoring schema — projects, QSets, runs, results — is untouched.
All deployment data links back to the originating project for unified reporting.
Admin Console
Platform Admin Only
—
Total Owners
—
Active
—
Trial
—
Paying
—
Churned
| Name | Organization | Plan | Status | White Label | Seats | Runs/Mo | Stripe ID | Trial Ends | Plan Started | Last Active | Source | Referred By | Notes | Actions | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| No owners yet. Click + Add Owner to get started. | |||||||||||||||
Settings
Project
AI Providers
Configure the AI providers your team uses for running tests and Signet42's internal AI features (Signet Assay, QSet Assist, Judge, Sandbox). Keys are encrypted at rest in Supabase Vault and never stored in the browser — only Signet42's server ever sees them.
Embedding ProvidersiUsed to generate vector embeddings for Document Library's Chunked (RAG) ingestion mode. Separate from the AI Providers above (which are for chat/answering) — the same account can be added to both if you use one provider for everything.
Configure providers used to generate embeddings when a document library uses Chunked (RAG) ingestion. Keys are encrypted at rest the same way as AI Providers above.
Signet42 Internal AI
Choose which model powers Signet42's own features. Uses the Anthropic key above unless you specify another provider.
Organization
Only an owner or admin can rename the organization.
Team Access
Members
Pending Invites
Past Invites
Org Admin — QSet Dimensions
Manage shared dimensions and suggested tags available across all projects. Tags can belong to multiple dimensions.
Run Prompts
Saved system prompts for this project. Select one when configuring a run, or test it in the Sandbox.
+ Add new prompt
Prompt Sandbox
Configuration
Connection Details
No connection selected.
External connections receive only the question and the prior turns of this conversation — no system prompt or corpus context is sent. This is the one place in the app that can actually exercise a connection's persistent-session (multi-turn memory) behavior.
Select a Question Set above to see its questions.
Answering via: —
Configure your prompt on the left, then ask a question below.