Two real-time clinical agents. The diagnostic agent went through 4 versions.
Role
Software Developer, Full-Stack & AI
Domain
Healthcare
Stack
OpenAI Agents SDK
OpenAI Realtime API
WebSockets
RAG
Vector stores
AWS Lambda
DynamoDB
Cognito
# Real-time healthcare agents
Jay Gadhiya. Vibgyor Technologies, Oct 2024 to now. Software Developer, Full-Stack & AI.
Two real-time clinical agents. The diagnostic agent went through 4 versions.
## Problem
Clinical intake needs low-latency, stateful sessions that stay grounded in structured clinical data.
## Approach
- Built a diagnostic agent on the OpenAI Agents SDK, served over WebSockets, that extracts symptoms from natural language.
- Mapped those symptoms to clinical Finding IDs with RAG over a custom vector store.
- Built a voice intake agent on the OpenAI Realtime API for paediatric nutrition assessments. It saves structured outputs to DynamoDB through AWS Lambda.
Architecture diagram: Two agents side by side. Text intake: a session over WebSockets, authenticated with a Cognito JWT, reaches a diagnostic agent on the OpenAI Agents SDK, which maps symptoms to clinical Finding IDs with RAG over a custom vector store. Voice intake: a paediatric nutrition assessment runs on the OpenAI Realtime API, and its structured outputs are saved to DynamoDB through AWS Lambda.
## Key decisions
- **WebSockets first.** Intake is a conversation, and a stateful, low-latency session suits it better than one-shot requests.
- **Structured outputs.** Answers stay in a fixed clinical format that can be stored and used downstream.
- **Cognito JWT auth, per-session context and token cost tracking.** Every session is authenticated, keeps its own state and records what it cost.
## Result
Symptoms described in plain language become clinical Finding IDs, and voice assessments are saved as structured data. The diagnostic agent went through 4 versions.
## Stack
OpenAI Agents SDK, OpenAI Realtime API, WebSockets, RAG, Vector stores, AWS Lambda, DynamoDB, Cognito
Page: https://jaygadhiya.com/work/real-time-healthcare-ai/
Contact: hello@jaygadhiya.com
1Problem
Clinical intake needs low-latency, stateful sessions that stay grounded in structured clinical data.
2Approach
Built a diagnostic agent on the OpenAI Agents SDK, served over WebSockets, that extracts symptoms from natural language.
Mapped those symptoms to clinical Finding IDs with RAG over a custom vector store.
Built a voice intake agent on the OpenAI Realtime API for paediatric nutrition assessments. It saves structured outputs to DynamoDB through AWS Lambda.
3Key decisions
WebSockets first.
Intake is a conversation, and a stateful, low-latency session suits it better than one-shot requests.
Structured outputs.
Answers stay in a fixed clinical format that can be stored and used downstream.
Cognito JWT auth, per-session context and token cost tracking.
Every session is authenticated, keeps its own state and records what it cost.
4Result
Symptoms described in plain language become clinical Finding IDs, and voice assessments are saved as structured data. The diagnostic agent went through 4 versions.
Want to go deeper on this project?
I can walk you through the decisions and trade-offs behind it.