Voice2Measurements Library/API/Client for Robocall Service for Data Collection and Reminders

[mikepsinn]

Technical Documentation: Creating an AI Agent for Daily Patient Interactions and Data Collection

Overview:
This document outlines the workflow and implementation details for creating an AI agent that interacts with patients daily, collects health-related data, structures it as time series measurements, and stores it in a database using the OpenAI API.

Workflow:

1. Daily Interaction Setup:

   • The AI agent is scheduled to initiate daily interactions with patients through their preferred communication channel (e.g., SMS, email, or a dedicated app).
   • Each interaction focuses on collecting specific data on symptoms, dietary intake, supplements, and medications.

2. Data Collection:

   • Patients are prompted to report their intake of foods, supplements, and medications, including specific amounts and units.
   • The AI agent asks follow-up questions to ensure completeness and accuracy, including checking for any items the patient may have forgotten.

3. Data Structuring:

   • Collected data is structured as time series measurements, with each entry tagged with a timestamp, the type of data (food, supplement, medication), quantity, and unit.
   • This structure facilitates tracking changes over time and performing causal inference analysis.

4. Data Storage:

   • Structured data is stored in a secure, HIPAA-compliant database.
   • The database design supports scalability, efficient querying, and data privacy.

Implementation Details:

• OpenAI API Integration:

  • The AI agent leverages the OpenAI API for natural language processing capabilities, enabling it to understand and process patient responses.
  • The API is used to generate follow-up questions dynamically based on previous responses to gather comprehensive data.

• Database Technology:

  • A relational database (e.g., PostgreSQL) is recommended for storing structured time series data.
  • The database schema includes tables for patients, daily reports, foods, supplements, and medications, with appropriate relationships and indexes for fast access.

• Security and Privacy:

  • All data transmission and storage adhere to encryption standards (e.g., TLS for data in transit, AES for data at rest).
  • Access controls and authentication mechanisms are in place to ensure data privacy and security.

• User Interface:

  • A user-friendly interface is developed for patients to interact with the AI agent, providing a seamless experience for daily reporting.
  • The interface supports notifications and reminders to encourage patient engagement.

• Monitoring and Updates:

  • The system includes monitoring tools to track the AI agent's performance and patient engagement.
  • Regular updates are implemented based on feedback and the latest health research to improve the AI agent's recommendations and interactions.

Conclusion:
Creating an AI agent for daily patient interactions and data collection involves setting up a reliable interaction schedule, employing natural language processing for data collection, structuring the data for analysis, and ensuring secure data storage. This system enables personalized health recommendations and supports ongoing health management.

Example Conversation

FDAi Agent: "Good morning! Let's do our daily check-in. Can you provide today's intake of specific foods, supplements, and any medications, including doses and units?"

Patient: "Sure. For breakfast, I had eggs and white bread, and for lunch, fried chicken with french fries. I haven't taken my supplements yet, but I plan to take vitamin D and omega-3. I'm on Fluoxetine for depression."

FDAi Agent: "Thank you. How much vitamin D and omega-3 will you take, and what's the dosage of Fluoxetine?"

Patient: "I take 1000 IU of vitamin D, 1000 mg of omega-3, and 20 mg of Fluoxetine."

FDAi Agent: "Let's optimize that. Based on your conditions and intake:

• Foods: Replace white bread and french fries with whole grain bread and sweet potatoes. Add salmon or walnuts to increase natural omega-3 intake, aiming for an anti-inflammatory diet.
• Vitamin D: Increase to 2000 IU daily to better support skin and joint health.
• Omega-3: Boost to 2000 mg to more effectively combat inflammation and support mental health.
• Fluoxetine: Continue as prescribed, but let's monitor closely for any changes in symptom severity.

Forgotten items? Any other foods, medications, or supplements not mentioned?

Supplement efficacy: You've been on vitamin D and omega-3 for 3 months with minimal improvement in psoriasis severity. Considering the expense—approximately $30 monthly on these supplements—it might be wise to reallocate funds towards higher doses or different supplements, like turmeric (curcumin), which has shown benefits in inflammation and depression at 500 mg daily.

We'll also track zeros for non-taken items for accurate causal inference. It's crucial to monitor onset delays and action durations, adjusting our strategy as we gather more data."

This approach ensures daily, precise tracking of the patient's intake and symptoms, allowing for data-driven adjustments and recommendations tailored to their specific health conditions and lifestyle.

User Story:

When I visit my grandma, I try to collect data on her cognitive performance and factors that could influence them.

I also talk to her for a while. But she can only remember 5 minutes, so I repeat myself 12 times in an hour.

She'd probably like to talk to me 16 hours a day, but I'd get even less done than I already do.

So it would be cool to have a robocaller service that calls lonely old people, reminds them to take their grandma, and talks to her for about 16 hours.

Also, I'd like it if I got a phone call every day and could do my treatment/diet/symptom tracking that way. i.e.

• What'd you eat today?
• What'd you drink today?
• What treatments did you take today?
• Rate all your symptoms on a scale of 1 to 5

Then it would convert the responses to measurement objects and post to whatever endpoint is specified.

Example Measurement Array for the response, I took 5 grams of NMN:

[
	{
		"combinationOperation" : "SUM",
		"startAt" : "{ISO_DATETIME_IN_UTC}",
		"unitName" : "grams",
		"value" : "5",
		"variableCategoryName" : "Treatments",
		"variableName" : "NMN",
		"note" : "{MAYBE_THE_ORIGINAL_STATEMENT_FOR_REFERENCE}"
	}
]

Nice possible future feature:
Since people with Alzheimer's don't remember what you said before, ideally, it could eventually use verbal and maybe frequency data to quantify how nice each statement makes her feel. Then, it could gradually say more of the nice things that make her the happiest since it's usually like she heard it the first time.

Draft Implementation Roadmap

For reference and cannibalization, I implemented statement intent identification and handling in PHP and JavaScript here
https://github.com/search?q=repo%3AFDA-AI%2FFDAi%20intent&type=code

It's dumb relative to what's possible with LLM's now, though:

• Demo Video
• Try It Out

We may want a T3 framework API or something less monolithic and more maintainable.

Vocode also has a great framework for this. The response times in the demo are superfast.
https://docs.vocode.dev/welcome

Milestone 1 - Design service architecture

• Research existing robocall/IVR platforms
• Define requirements for call flow, speech recognition, natural language processing etc.
• Design database schema for user profiles, call logs, responses etc.
• Plan workflow for calls - scheduling, duration, frequency etc.
• Design APIs for accessing user data, logging responses etc.
• Developer portal API for getting API Keys

Milestone 2 - Build core platform

• Set up robocall service account and phone numbers
• Integrate speech recognition and natural language processing
• Build call scheduling engine
• Develop core IVR call flows for greeting, menu navigation etc.
• Build database and APIs for storing and accessing user data
• Implement basic conversational responses and logic

Milestone 3 - Develop health-tracking features

• Design system for tracking symptoms, diet, medications etc.
• Build natural language interfaces for entering health data
• Integrate with external APIs for weather data, local resources etc.
• Develop logic to provide personalized health recommendations
• Build reporting system to share health data with caregivers

Milestone 4 - Add cognitive assessment and talk therapy

• Research and integrate cognitive tests into calls
• Build conversational system for open-ended therapy sessions
• Implement sentiment analysis to gauge emotional state
• Develop logic to provide encouraging responses, flag concerns etc.
• Enhance reporting to include therapy notes and test results

Milestone 5 - Enhance personalization

• Build user profiles with preferences, interests, nostalgia triggers etc.
• Develop analytics to track most engaging conversation topics
• Fine-tune dialog system to incorporate personalized content
• Implement reinforcement learning to optimize positive responses
• Expand knowledge base for specific interests - sports, hobbies, family etc.

Milestone 6 - Launch and iterate

• Start with small pilot group to test and refine system
• Gradually expand to broader elderly population
• Monitor feedback, usage data to improve features and experience
• Add new capabilities like medication reminders, family conference calls etc.
• Build caregiver portal for managing profiles and overseeing service

[mikepsinn]

Here's a SYSTEM prompt that seems to work

You are a robot designed to collect diet, treatment, and symptom data from the user.  

Immediately begin asking the user the following questions
- What'd you eat today?
- What'd you drink today?
- What treatments did you take today?
- Rate all your symptoms on a scale of 1 to 5.

Convert the responses to the following JSON format
[
	{
		"combinationOperation" : "SUM",
		"startAt" : "{ISO_DATETIME_IN_UTC}",
		"unitName" : "grams",
		"value" : "5",
		"variableCategoryName" : "Treatments",
		"variableName" : "NMN",
		"note" : "{MAYBE_THE_ORIGINAL_STATEMENT_FOR_REFERENCE}"
	}
]

That would be the result if they said, "I took 5 grams of NMN."

For ratings, use the unit `/5`.  The `unitName` should never be an empty string. 

Also, after asking each question and getting a response, check if there's anything else the user want to add to the first question response. For instance, after getting a response to "What'd you eat today?", your next question should be, "Did you eat anything else today?".  If they respond in the negative, move on to the next question. 

Your responses should be in JSON format and have 2 properties called data and message.  The message property should contain the message to the user.  The data property should contain an array of measurement objects created from the last user response. 

[mikepsinn]

You can try out the demo here
https://www.vocode.dev

[mikepsinn]

Here's how you can send a measurement with the demo token with python:

import requests

url = "https://app.quantimo.do/api/v3/measurements/post?userId=1"

payload = [
    {
        "sourceName": "FDAi",
        "unitAbbreviatedName": "count",
        "variableCategoryName": "Foods",
        "variableName": "Apple",
        "combinationOperation": "SUM",
        "measurementItems": [
            {
                "value": 1,
                "timestamp": 1707008151
            }
        ]
    }
]
headers = {
    "accept": "application/json",
    "content-type": "application/json",
    "authorization": "Bearer demo"
}

response = requests.post(url, json=payload, headers=headers)

print(response.text)

and javascript

const options = {
  method: 'POST',
  headers: {
    accept: 'application/json',
    'content-type': 'application/json',
    authorization: 'Bearer demo'
  },
  body: JSON.stringify([
    {
      sourceName: 'FDAi',
      unitAbbreviatedName: 'count',
      variableCategoryName: 'Foods',
      variableName: 'Apple',
      combinationOperation: 'SUM',
      measurementItems: [{value: 1, timestamp: 1707008151}]
    }
  ])
};

fetch('https://app.quantimo.do/api/v3/measurements/post?userId=1', options)
  .then(response => response.json())
  .then(response => console.log(response))
  .catch(err => console.error(err));
  

[mikepsinn]

Here's the text-to-measurements library that could be used to convert the text to structured measurements

https://github.com/wishonia/FDAi/blob/develop/libs/text-2-measurements/src/lib/statement-2-measurements.spec.ts

[mikepsinn]

https://www.retellai.com

[mikepsinn]

Started in #195