The State of Oregon Enterprise Information Services requires an AI System Disclosure Card for public-facing chatbots. The details provided below aim to provide plain-language context and description of the AI system, including what it does, identified risks and mitigations, how it was tested, how it is monitored, and any privacy implications for users of the chatbot.
What the AI system is and does
|
What does the service do? | Explore Health's Automated Virtual Agent and Chatbot provides users with fast, convenient, and accessible support for navigating health insurance-related information and services. The virtual agent uses conversational technology to answer common questions, guide users to relevant resources, and help streamline interactions with Explore Health. Designed to enhance the user experience, the chatbot is available 24/7 and can provide consistent assistance while helping users find the information they need more efficiently. The chatbot does not make any decision about an applicant's benefits. |
|
Who can use the service? | The Explore Health Automated Virtual Agent and Chatbot are designed for Explore Health users, applicants, enrollment partners, and other individuals seeking assistance with health insurance-related information and services. It can be used by anyone who needs quick, convenient guidance or help navigating available Explore Health resources. Access and available features may vary depending on the user's role, eligibility, and the services offered through Explore Health. The services are available in English and Spanish. |
|
How it is accessed? | The Explore Health Automated Virtual Agent is available through the Customer Assistance Center (855-268-3767). When accessing the service by phone, users can interact with the automated virtual agent to receive assistance, find information, and navigate available Explore Health services. The Chatbot can be accessed through the Explore Health website (ExploreHealthOR.gov) and its digital platform. Users can open the virtual chatbot and interact with it using a conversational interface to ask questions, find information, and receive guidance. Access may vary depending on the specific Explore Health service or platform being used. |
|
Who is the System Owner? | The Oregon Health Authority (OHA) is the System Owner of this chatbot. The chatbot was created and is managed by Vimo under contract with the Oregon Health Insurance Marketplace. |
|
What data is used? | The Explore Health Automated Virtual Agent and website Chatbot use information necessary to assist users with customer service inquiries and help them navigate available Explore Health services. -
Automated Virtual Agent: Available through the Customer Assistance Center, the virtual agent uses information provided by the caller and authorized information available to the system to assist with inquiries and service navigation.
-
Website Chatbot: Available through the Explore Health website, the chatbot uses information entered by the user and authorized information available to the system to provide responses and direct users to appropriate resources.
Both tools are designed to minimize the use and disclosure of sensitive information. Personal information, protected health information (PHI), and other regulated or restricted data should not be intentionally provided to the AI model unless specifically authorized and required for the service. The AI model does not receive Health Insurance Portability and Accountability Act (HIPAA)-protected medical details, Family Educational Rights and Privacy Act (FERPA), Criminal Justice Information Services (CJIS), or Federal Tax Information (FTI). |
|
Where does the data reside? | No state production data, including prompts, responses, embeddings, logs, telemetry, or backups, is stored, processed, or accessed outside the United States. |
|
AI model and model card | AVA uses OpenAI GPT-5 series models, accessed directly through OpenAI's US data residency endpoint under an enterprise agreement that includes zero data retention for training. Any change to the AI model or model version triggers the re-testing thresholds described below |
|
No model training | No state data, member input, or AVA output is used to train, fine-tune, or otherwise update AI models. OpenAI is contractually prohibited from using this data for model training, and AVA does not learn from user interactions. |
|
Major risks identified for mitigation | The primary risks identified for this AI system are as follows: - Incorrect, incomplete, or outdated responses by AVA
- Inadvertent disclosure of personal information or personal health information, either to the member or to an assisting agent
- Failure to correctly escalate or hand off to a human agent when the interaction exceeds AVA's intended scope
- Unmanaged token use by AVA
- Over-reliance by assisting agents on AVA-generated content without independent verification
A robust testing and monitoring framework has been developed to mitigate the above primary risks, as well as other identified risks. A high-level snapshot of how the System Owner is working to mitigate these risks is listed below. |
|
Testing snapshot | AVA is tested using a structured plan aimed at mitigating risks identified in the
OWASP Top 10 for Large Language Models and to meet the requirements in the state's Responsible AI Usage Policy attachment on AI Risk Management. Specific test batteries for the following have been developed: - Accuracy and quality: testing against a series of questions and answers validated by human subject matter experts
- Fairness and equity: testing to confirm consistent answers across demographic and life-system variations
- Privacy: personal information and personal health information redaction testing
- Security and adversarial testing, including prompt injection and jailbreak attempts
- Token usage testing
- Escalation and handoff testing: confirming AVA correctly routes out-of-scope interactions to a human agent
- Data residency verification: confirming all production traffic resolves to the US-only OpenAI endpoint
Thresholds for re-testing have been established. Re-testing will occur when any of the following occur: - significant policy changes occur
- changes to the GenAI model used
- changes to prompts
- changes to the model endpoint or data residency configuration
- other significant system changes
- a defined increase in reported issues
|
|
User feedback | Collecting and reviewing feedback on AVA's performance is critical to both managing and improving the system. Feedback is captured through an established quality control (QC) process: - A sample of AVA interactions is reviewed by human quality control reviewers against a defined rubric covering accuracy, completeness, appropriateness of tone, correct escalation, and absence of personal information or personal health information in the response
- Reviewer findings are logged, categorized, and triaged by a designated team within the System Owner's organization
- Issues identified through QC feed directly into prompt updates, knowledge base corrections, and the re-testing thresholds described above
- Escalated or member-reported issues arising through normal customer service channels are routed into the same QC review and remediation queue
At the end of a voice or chat interaction, users are offered a short satisfaction survey; survey responses are collected, reported alongside the quality control findings, and reviewed by the same designated team.
System metrics: - Latest quarter: will be populated after the system has been live for three months
- Latest 12 months: will be populated after the system has been live for a year
|
|
Monitoring | In addition to user and agent feedback, AVA is monitored for a series of established alerts, including cybersecurity attacks, redactions of personal information or personal health information, token usage, escalation and handoff failures, and model endpoint or API errors. |
|
Rollback and deactivation | AVA can be turned off at any time. Interactions can also be routed entirely to human agents without disabling the wider system. Specific criteria requiring a rollback or deactivation are in development but have not yet been finalized. |
What are the privacy implications for users of the AI system?
|
Public records, privacy and data retention | The system retains the following information for three (3) years, which is subject to disclosure under Oregon public records law:- Text entered into the chat by the user,
- Responses from AVA, across both the chat and voice channels,
- Transcripts of voice interactions,
- Audio recordings of voice interactions,
- IP address from the user (chat channel),
- Originating telephone number from the user (voice channel),
- Satisfaction survey responses,
- Quality control review records, including reviewer findings and remediation notes, and
- Personal information and personal health information detected in user input or AVA responses is redacted in accordance with the redaction controls described above.
|
Policy Review
This policy shall be reviewed at least annually by the Explore Health Compliance Officer, or designee, to ensure continued compliance with applicable federal and state laws, regulations, contractual requirements, and Marketplace operational needs.
The policy may be revised at any time to reflect changes in legal or regulatory requirements, organizational responsibilities, operational processes, identified risks, audit findings, or other program integrity considerations.
Approved revisions shall be communicated to affected staff, contractors, enrollment partners, and other individuals or organizations subject to this policy, as appropriate.