Playground Feedback — Next Steps

Compiled from the Ray Tools Feedback PDF (Mar 2–5, 2025). Feedback from Morgan McGowan, Benjamin Scott, and Eric Zwierzynski.

Opportunities Generator

  1. Allow starting without a file — Morgan wants to describe a product scenario and get ideas without being locked to existing data or uploading a file
  2. Align outputs with business goals — outputs need to be more closely tied to business goals

Transcript Summarizer

  1. Fix missed themes — missed several key themes that Morgan and cGPT both caught; produced signals based on only one interview's responses
  2. Shift from signal generation to data extraction — Ben suggests the tool should focus on pulling out data and enabling questions about the data, rather than producing signals directly
  3. Produce signals, not themes — currently outputs "themes" rather than signals; prompt needs clearer signal definition
  4. Reference Morgan's cGPT example — Morgan shared a good input/output example to benchmark against: https://chatgpt.com/g/g-p6839fab2a6f481918b613585b9bcf422/c/69a64091-0400-8327-8948842d0457788f
  5. Closest to useful — both Morgan and Ben agree other team members could use this easily if signal quality improves; prioritize accordingly

Signal Extractor

  1. Simplify the UI — Bryan had many confusion points: what are "entries"? Why does it say "pending"? Green/red bar unclear, no state change on "Review Entries" action, concept numbers in sidebar aren't clear, purpose between hunches/questions/recommendations isn't clear
  2. Reduce output volume — does what it's supposed to, but shows more than necessary
  3. Scope down to signals only — focus on extracting signals for now; hunches, recommendations, etc. can be held as metadata within a signal

Report Agent

  1. Fix quote extraction — won't pull quotes; likely the same root issue as the report AI panel
  2. Restructure system prompt for signal writing — currently producing observations that call out data rather than actual signals. Morgan shared the target structure: https://share.zight.com/YEu2K8lk
  3. Include qual data in outputs — didn't provide qual data, which is critical for someone who can't quickly comb through data report responses for quotes

AI Workbench

  1. Remove or radically simplify the pre-generate questionnaire — causes hesitation and takes too much time. Morgan says this questionnaire wasn't producing better signals than just giving context to cGPT. The answers are very difficult to provide (e.g., no single most important segment) and can negatively index the outputs
  2. If keeping a questionnaire, only ask for 3 things — audience, design tested, goal of the survey. Multiple choice format preferred over open-ended
  3. Fix signal generation volume — only 1 signal was generated for a full survey. Morgan's direction: start with 1 signal per question in the survey (revised from earlier suggestion of 8–10)
  4. Fix heavy weighting on success number — the success number from the pre-generate questionnaire is over-indexing the output
  5. Report on all UX metric scores — currently doesn't report on all UX metric scores in the survey
  6. Fix qualitative quote extraction — same issue as Report Agent

Suggested Priority

Immediate (high impact, multiple people flagged)

  • AI Workbench fixes (#14–19) — questionnaire, signal volume, UX metrics, quote extraction
  • Report Agent signal quality (#11–13) — quotes, prompt restructure, qual data
  • Signal Extractor simplification (#8–10) — UI clarity, scope to signals only

Short-term

  • Transcript Summarizer (#3–7) — closest to useful per team consensus; fix themes and signal quality
  • Opportunities Generator (#1–2) — freeform mode, business goal alignment

Open questions

  • Transcript Summarizer direction: should it produce signals, or focus on data extraction + Q&A? (Ben and Morgan may have different visions — needs alignment)