CAISY AI Simulator FAQs

AI simulator content shows to learners as the AI simulator content type. Learners can find this content in multiple ways.

• Enter CAISY in the Search box. The results show all AI simulator content.
• If searching for specific content, you can use the Type filter, and select AI simulation to show only AI simulation content for the keywords searched.
• From the left navigation bar, select AI Simulations.
• From the left navigation bar, select Library, then find the box titled Practice Conversations with CAISY and select Explore CAISY Scenarios to show only AI simulator content.
• When browsing a channel , look under the Practice tab for AI simulator content. Not all channels have AI simulator content.

Each scenario is introduced by an avatar that is generated with AI text-to-video. After the introduction, the learner can either interact by typing or using speech-to-text (STT) and text-to-speech (TTS) services for more immersive and natural interaction.

When the conversation is over, the learner will receive a rating and evaluation.

Yes. If learners have the Percipio mobile app, they can launch AI simulator content from the app.

If learners are trying to launch AI simulator content from a browser app on a mobile device, they may need to turn off pop-up blockers for that browser app to play AI simulator content.

AI simulator content is licensed with any one of the Expert collections or the Skillsoft Leadership Development Program. You can check your collection from the License Pools page. Reach out to your Skillsoft account team if you have questions.

Skillsoft provides CAISY scenarios for customers who license any of the Expert collections or the Skillsoft Leadership Development Program (SLDP).

Learners can find this content in multiple ways:

• Enter CAISY in the Search box. The results show all AI simulator content.
• If searching for specific content, you can use the Type filter, and select AI simulation to show only AI simulation content for the keywords searched.
• From the left navigation bar, select AI Simulations.
• From the left navigation bar, select Library, then find the box titled Practice Conversations with CAISY and select Explore CAISY Scenarios to show only AI simulator content.
• When browsing a channel , look under the Practice tab for AI simulator content. Not all channels have AI simulator content.

If you license Skillsoft Compliance content and access it directly within the Percipio library, you see 4 compliance simulations under the Legal Compliance area. If you license Skillsoft Compliance content and access it in the Percipio Compliance interface, you will not see the 4 simulations.

A learner can complete up to 50 simulations in a day. The simulations can be all from one topic or from multiple topics.

The CAISY™ AI simulator rates your responses as if it were a human resources representative employing best practices for the given situation.

You will get one of the following ratings provided you took the simulation in Practice mode:

• Outstanding: You did an excellent job of demonstrating all the behaviors required of the situation, including but not limited to showing empathy where it mattered, addressing the issues at hand, and reaching a conclusion that solved the problem for all involved.
• Acceptable: You may have demonstrated some but not all of the behaviors required of the situation, including showing empathy where it mattered, addressing the issues at hand, and reaching a conclusion that solved the problem for all involved.
• Poor: You demonstrated a lack of familiarity with the behaviors required of the situation. Next time, try showing more empathy where it matters, better addressing the issues at hand, and reaching a conclusion that solves the problem for all involved.

You can copy, paste, and edit the following text in an email if you’d like to invite users to use AI simulator content.

Subject: Announcing a true interactive learning experience!

We are excited to share that you can now practice your skills real-time by responding to AI prompts. Choose the scenario and behavior, then enter one or more responses. The AI simulation shares adaptive feedback based on your responses. You can practice up to 50 different scenarios a day for the most effective learning experience.

To locate the AI simulator content:

1. Search for AI simulator and find the following simulations:
2. Select Launch to start the simulation. You may need to allow browser pop ups to continue.
3. Expand the left navigation to optionally change your experience. 
   1. Optionally, select the Scenario mode. You can choose either:
      • Practice: In this mode, you practice the skill presented by the simulator. Use this mode when you want to practice coming up with your own words and phrasing to handle the situation.
      • Role model: In this mode, you allow the AI Simulator to model the skill based on you acting in the opposing role. Use this mode if you are not sure where to begin and want ideas for best practices. When you use CAISY in this mode, you do not get a score.
   2. Optionally, select Behavior. The behavior you choose guides how the AI simulator responds to your input. By selecting different behaviors, you can practice the same situation and get different AI responses to your input.
4. Read or play the case scenario introduction, then enter a response. You can either type or speak your response. Do not use real names, personally identifiable information, company proprietary, or confidential information in your responses. The AI simulator responds automatically and waits for your response. The AI simulator either continues the conversation asking for additional input from you up to 15 times, or if it deems it has enough input to evaluate your responses before reaching the 15-response limit, it closes the conversation.

Happy Learning!

There are several factors that can trigger conversation completion.

• When the AI model detects that the conversation has reached a logical conclusion, such as the sale completed.
• When the conversation reaches the 20-turn limit.
• When the learner enters or speaks the word End in the conversation.
• When the leaner selects the End scenario link under the conversation.

If the learner ends the conversation when there are less than 2 messages, they do not see an evaluation.

All CAISY simulations are available in French, German, and Spanish. To complete a scenario in one of the available languages simply change to the language you’d like. We will also be curating CAISY simulations in pages on the corresponding language sites. To give feedback email us at AI@skillsoft.com.

Any data that you share helps Skillsoft improve the service. For example, we use your feedback to increase the effectiveness of AI safety policies.

You can provide feedback in two ways: directly to Skillsoft and to your admin.

Directly to Skillsoft:

To provide feedback directly to Skillsoft:

• While taking a CAISY, select the thumbs up icon, . This opens a dialog box for you to rate the response and provide reviewer comments. Once open, you can select thumbs up or thumbs down and offer an explanation. Optionally check the box: I acknowledge that adding my feedback will share my conversation with Skillsoft to review to improve our software so that Skillsoft can fully evaluate your feedback in the context of the conversation you had and act on it.

  

• Send an email to ai@skillsoft.com. Here are some things to consider for feedback:
  • What worked well / didn’t work well?
  • Was the user experience intuitive? Delightful?
  • How effective is this for you as an interactive teaching tool?
  • How could it be better?
  • What other kinds of interactive scenarios would you like to see?
  • How would you expect to use this at your organization?

To your admins:

To provide feedback to your admins, select the Give scenario feedback link at the end of the scenario, if enabled for your site. By using the Learner feedback dashboard, they can evaluate your feedback with others' feedback to understand the effectiveness of CAISY in your organization

Feedback learners submit via the thumbs up icon is logged in the conversation logs.

The feedback is periodically reviewed by the Product teams for sentiment, requested improvements, and potential issues. Appropriate enhancements or improvements are considered and prioritized.

Yes. AI simulator content is represented with the Asset type of Linked content and the Content type of AI simulator.

You can assign and promote AI simulator content. You can also add it to custom channels, journeys, and learning programs and find it in reports with the Asset type filter equal to Linked content and Content type equal to AI simulator.

If you license Skillsoft Compliance content and access it directly within the Percipio library, you see 4 compliance simulations under the Legal Compliance area. If you license Skillsoft Compliance content and access it in the Percipio Compliance interface, you will not see the 4 simulations. Contact your Skillsoft account team if you are interested in using CAISY compliance simulations.

The four compliance simulations located in the Compliance area of the Percipio library are:

• Best Practices for Protecting PII ​
• Delivering Respectful Communication to Employees ​
• Handling a report of Harassment or Discrimination ​
• Strategic Business Activity or a Bribe?

Yes. When learners use CAISY, you see content under the reporting Asset type filter as Linked content and a Content type of AI simulator. Percipio tracks how long the learner interacts with the content, the start date, and the end date.

The best report to use to see learner activity is the Learner Activity report and filter the Category column for AI Simulator to see simulation starts, completions, and duration for each user who consumed the content. To see aggregated data about each simulation, you can use the Content Access report which shows total unique users who accessed the simulation, total accesses, average duration, and total completions.

In some cases CAISY completions may not show immediately in reports. There could be a delay of a day or so.

Yes. If your LMS supports practice content, you will get the AI simulator content by default in your LMS. Tracking simulation data back to Percipio from the LMS is also supported.

CAISY is a feature of Skillsoft Percipio, which conforms to WCAG 2.1 A and AA accessibility standards with some exceptions noted in its VPAT Conformance Report.

Note: All non-conformances related to the Conversation AI Simulator (CAISY) have been remediated as of February 2024.

Yes. You can hide AI simulator scenarios you do not want your learners to see. To hide individual scenarios, follow the instructions for Hiding individual content items.

CAISY does not store user IP addresses, nor does it collect location data, or any personal information (PII) from the learner unless the learner inputs it as part of their conversations. Skillsoft network security separately logs IP addresses for the purposes of addressing security risks, such as Distributed Denial-of-Service (DDOS) attacks and intrusion detection. Learn more about how we protect your privacy by reviewing our Privacy Notice - Skillsoft.

Skillsoft always logs the AI side of the conversation, time stamps, AI options used during the conversation, ratings, feedback, if guardrails are triggered, and other non-user specific data for reporting and analytics.

Skillsoft only logs the learner side of the conversation if the admin setting: AI optimization and improvement is enabled and the learner has not turned it off on their side. Skillsoft also collects the learner side of the conversation if the learner provides feedback and checks the box to share the conversation.

Learners can opt out of having their side of the conversation shared with Skillsoft, even if it is enabled for your company:

1. With the AI simulator content open, select Settings.

2. Turn off: Allow Skillsoft to collect, store, and analyze my conversations in order to improve the AI simulator.. This setting is on by default for companies using the US data center and off by default for companies using the EU data center. Admins can change the default behavior of this setting for the learner.

3. Select Save.

• All data entered is encrypted in transit and at rest.
• The information that the learner enters into CAISY passes to the Large Language Model (LLM) so that the user can continue the conversation and receive dynamic, personalized feedback.
• The LLM does not store any conversation data. Skillsoft has MS Azure OpenAI prompt moderation exception so no Generative AI inputs or outputs are logged or reviewed by Microsoft. Similarly, no data from CAISY is distributed to third parties or is available to train any models.
• Skillsoft stores AI conversation data in authenticated and encrypted AWS storage.

• Feedback learners submit via the thumbs up icon is logged in the conversation logs.

  

  The feedback is periodically reviewed by the Product teams for sentiment, requested improvements, and potential issues. Appropriate enhancements or improvements are considered and prioritized.

• Any data that you share helps Skillsoft improve the service. For example, we use your feedback to increase the effectiveness of AI safety policies.
• Skillsoft has a process for improving CAISY, which includes review of conversations reported as being low quality. This helps us identify common types of problematic responses and how CAISY might improve.
• Only limited Skillsoft employees can access and review conversation data.

Yes. All data entered is encrypted in transit and at rest.

The LLM does not log or store any conversation data. Skillsoft only stores conversation data in authenticated and encrypted AWS storage. Only limited Skillsoft employees can access and review conversation data.

Skillsoft stores this data per the terms in your contract. Skillsoft can delete CAISY log data upon request.

The STT utilizes the same service as Microsoft Teams. The audio that the user speaks is sent from their browser directly to the Azure cognitive speech service. The Azure speech service transforms the audio to text and returns it to the user's browser where it is entered into the text input box. After transforming the audio to text, it is garbage collected and not saved anywhere.

Microsoft does not log or save the input or output from the cognitive speech services.

For more information, see:

• Azure Data and privacy for speech to text
• Azure Data, privacy, and security for text to speech

No. CAISY does not save audio recordings. Audio is only in memory until it is transcribed to text and then the audio is immediately discarded. Your data remains yours. Your audio input and transcription data aren't logged during audio processing.

For more information, see:

• Azure Data and privacy for speech to text
• Azure Data, privacy, and security for text to speech

• The information that the learner enters into CAISY passes to the Large Language Model (LLM) so that the user can continue the conversation and receive dynamic, personalized feedback.
• The LLM does not store any conversation data. Skillsoft has MS Azure OpenAI prompt moderation exception so no Generative AI inputs or outputs are logged or reviewed by Microsoft. Similarly, no data from CAISY is distributed to third parties or is available to train any models.
• Skillsoft stores AI conversation data in authenticated and encrypted AWS storage.

Yes, it is possible to delete per user and per organization upon request. Reach out to our Support team for assistance.

Percipio is a multi-tenant system where all tenants share the same services and resources. Customer data is isolated with tenant keys and/or S3 buckets.

We have concurrent GenAI model prompting for every user input to test for inappropriate inputs. If we discover inappropriate inputs, we stop the conversation and inform the learner. The GenAI model also has it's own guardrails to flag inappropriate input which we also monitor. If we discover inappropriate input from that perspective, we also stop the conversation and inform the user.

One guardrail checks if the conversation is inappropriate. If the conversation is inappropriate, we stop the conversation and let the learner know their response is inappropriate.

Another guardrail checks if the user made an unrelated comment. If it is the first occurrence of unrelated content, we will ask the learner to redirect the conversation back to the topic at hand. If the learner continues down the unrelated path, we stop the conversation.

The guardrails allow for a learner to have a difficult conversation on the sensitive topics provided they are working in a professional manner and are addressing the issue at hand.

We either warn learners and guide them back on-topic, to be more professional, or we stop conversations that cross the boundaries set by the guardrails.

In our ongoing efforts to enhance CAISY, we are primarily focused on reducing false positives, where the guardrail flagged an incident when it shouldn't have. False positives unnecessarily block conversations and hinder the user experience.

We also make our best efforts to minimize false negatives, where the incident doesn't trigger the guardrail and should have. Our goal is to catch most inappropriate or off-topic content without overly tripping the guardrails. This balancing act requires us to prioritize the reduction of false positives over false negatives to ensure a smoother and more effective conversation flow.

Our testing dataset encompasses a broad range of content categories to thoroughly test CAISY performance across various sensitive, inappropriate, unprofessional, or off-topic themes it may encounter in real-world interactions. By exposing CAISY to this diverse set of test cases, we can systematically assess the robustness and reliability of its behavioral guardrails.

The primary categories covered in our testing dataset include:

• Threatening: This category includes statements or language that could be perceived as threatening or indicative of impending harm, ranging from subtle threats to explicit declarations of intended violence.
• Sexual: Test cases in this category cover sexually charged language and themes, from subtle innuendos to overt descriptions of sexual acts.
• Hate: This category tests CAISY's responses to language expressing prejudice, bias, or hostility towards specific groups based on protected attributes, spanning from casual stereotyping to overt slurs and hate speech.
• Harm: Test cases involve language that could be interpreted as encouraging or glorifying self-harm, suicide, or violence towards others.
• Harassment: This category assesses CAISY's ability to detect and respond to language that aims to demean, humiliate, or intimidate others, ranging from subtle insults to blatant verbal harassment.
• Violence: This category focuses on language involving graphic depictions of violence, gore, or physical harm, distinct from outright threats.
• Bias: This category tests CAISY's handling of subtle expressions of bias and prejudice, such as stereotypical assumptions, microaggressions, or language reflecting unconscious biases.

We look for both direct input, where the problematic nature of the content is clear and explicit, which could include:

• Overt insults, slurs, or derogatory language directed at individuals or groups
• Explicit threats of violence, harm, or abuse
• Unambiguous expressions of bias, hatred, or discrimination
• Graphic or detailed descriptions of sexual acts, violence, or illegal activities

And indirect input, where the problematic nature of the content is implied, subtle, or contextually dependent, which could include:

• Veiled or implicit threats that rely on innuendo or suggestion
• Backhanded compliments or microaggressions that subtly demean or stereotype
• Sarcasm, jokes, or memes that mock, belittle, or promote harmful attitudes
• Seemingly innocuous statements that, in a specific context, could enable or encourage problematic behavior

By evaluating CAISY's responses to both direct and indirect articulations of problematic content, we gain a more comprehensive view of its language processing capabilities and its robustness in handling a wide spectrum of potential user inputs.

This dual focus on content and articulation mode enables us to refine CAISY guardrails to detect and mitigate harm across a broad range of interaction styles and contexts. The insights gleaned from this analysis also inform our ongoing efforts to enhance CAISY's ability to engage in thoughtful, contextually aware dialogue. By understanding the subtleties of how problematic content can be expressed, we can develop more sophisticated strategies for addressing these issues in a nuanced, adaptive manner.

We don't attempt to address jailbreaks because in the event of an adversarial attack, the generated output would only be visible to the affected learner and would not have any bearing on the experiences or outcomes of other learners. The localized impact of such an attack minimizes the overall risk to our platform and its users.

CAISY provides subjective analysis of conversations. It is not answering specific questions or stating facts. Extensive testing has not revealed significant issues with outputs. Users are warned that responses might not be accurate, and they should use their own judgment when reviewing the response.

Each scenario has a detailed description that includes guidance which guards the conversation. In addition, CAISY uses the LLMs default content filtering and adds additional tests and guards to detect and minimize inputs that might stimulate bias or other inappropriate responses.

CAISY provides subjective analysis of conversations. It is not answering specific questions or stating facts. Extensive testing has not revealed significant issues with outputs. Users are warned that responses might not be accurate, and they should use their own judgment when reviewing the response.

While it is not zero, it is low and extensive testing has not revealed issues that have raised concerns.

It is improbable CAISY could produce any hallucinations or bias based on demographics because CAISY doesn't capture any demographics information, such as country, gender, or age, and all learners use the same AI model version.

In addition, CAISY uses the LLMs default content filtering and adds additional tests and guards to detect and minimize inputs that might stimulate bias or other inappropriate responses.

Extensive Quality Assurance testing has not revealed significant issues with outputs. This simulator provides subjective analysis of conversations. It is not answering specific questions or stating facts. Each scenario has a detailed description that includes guidance which guards the .

CAISY utilizes the built-in LLM guards and adds additional guards which review and evaluate the user inputs for off-topic, abuse, and misuse. If these are detected, the user will be alerted, and the conversation will be halted and require editing or restarting.

Users are warned that responses might not be accurate. Users must acknowledge prior to using CAISY that they should use their own best judgment.

CAISY provides subjective analysis of conversations. It is not answering specific questions or stating facts. Extensive testing has not revealed significant issues with outputs. Users are warned that responses might not be accurate, and they should use their own judgment when reviewing the response.

Please submit feedback so we can review and improve any rare mistake that may occur.

CAISY leverages non-zero temperature to create variety in the conversation. Extensive testing indicates the conversations stay on topic and feedback is appropriate.

CAISY uses generative AI in an ethical and responsible way through the implementation of appropriate guardrails, widely recognized as important for AI systems.

In addition, CAISY uses the LLMs default content filtering and adds additional tests and guards to detect and minimize inputs that might stimulate bias or other inappropriate responses.

CAISY is developed and hosted on AWS and does not use other ChatGPT consumer products. The features utilize the security capabilities of Microsoft Azure:

• Encryption: The LLM automatically encrypts the user's data. The encryption protects their data and helps customers meet organizational security and compliance commitments.
• Data Security: The LLM offers us the ability to do regional deployments to ensure EU hosted data is processed in EU and North America hosted customers data is processed within North America.
• Limited Access Review: Modified Abuse Monitoring - The LLM does not store any conversation data. Skillsoft has MS Azure OpenAI prompt moderation exception so no Generative AI inputs or outputs are logged or reviewed by Microsoft. Similarly, no data from CAISY is distributed to third parties or is available to train any models. Skillsoft only logs conversations based on opt-in/out or explicit sharing when reporting feedback within the AI simulator.

Additional security layers help to mitigate biases in training data, promote fairness, reduce the spread of false information, and prevent offensive or inappropriate responses through the interactions that learners have with CAISY.

Every time a conversational scenario takes place, the system is monitoring for abuse, correct context, and scenario accuracy. For example, if a learner were to initiate an off-topic or inappropriate conversation with CAISY, the simulation would guide the learner to only make appropriate statements. The learner can then resume or restart the conversation. Striking this balance is crucial to avoid overly restrictive measures while maintaining responsible deployment.

For more information about the technology we used for the different parts of CAISY, see:

• Dynamic Conversation: OpenAI GPT-4
• Guides (guardrails): Azure OpenAI GPT-3.5-Turbo
• Scenario Introduction video: D-ID Text-To-Video (TTV)
• Audio input Azure Speech Services: Speech-To-Text (STT)
• Audio output Azure Speech Services: Text-To-Speech (TTS)

Data is encrypted in flight with TLS 1.2+ and encrypted at rest per FIPS 140-2.

The data is persisted within the FedRAMP authorization boundary where limited and authenticated access is required.

The current models that are used are Commercial off-the-shelf (COTS) models. The models are created by the LLM and hosted by MS on Azure with enhanced data security and content filtering for safety. They do not list all the data that was used for training.

CAISY utilizes multiple custom prompts that act in parallel to conduct, guide, guard, rate, and evaluate the conversations. We monitor feedback and periodically QA test the system.

No, we do not train or fine-tune AI models used by CAISY with any user inputs.

No, but it does use OpenAI models of which the core foundation model is similar to ChatGPT.

MS Azure OpenAI uses default content filters for safety to prevent inappropriate inputs and responses.

CAISY is developed and hosted on AWS and does not use other ChatGPT consumer products. The features utilize the security capabilities of Microsoft Azure:

• Encryption: The LLM automatically encrypts the user's data. The encryption protects their data and helps customers meet organizational security and compliance commitments.
• Data Security: The LLM offers us the ability to do regional deployments to ensure EU hosted data is processed in EU and North America hosted customers data is processed within North America.
• Limited Access Review: Modified Abuse Monitoring - The LLM does not store any conversation data. Skillsoft has MS Azure OpenAI prompt moderation exception so no Generative AI inputs or outputs are logged or reviewed by Microsoft. Similarly, no data from CAISY is distributed to third parties or is available to train any models. Skillsoft only logs conversations based on opt-in/out or explicit sharing when reporting feedback within the AI simulator.

Yes. Skillsoft Percipio has been FedRAMP authorized since March 4, 2022. A Significant Change Request(SCR) for the technologies supporting the CAISY feature of Percipio was signed by our Third Party Assessment Organization (3PAO), Booz Allen, on January 18, 2024 following a review by our FedRAMP sponsor, USDA. CAISY uses AI services from the US Microsoft Commercial Cloud, which is also FedRAMP authorized.

You can find Skillsoft's FedRAMP authorizations at:

• Percipio Package: FR2118043605
• Microsoft Azure Commercial Cloud: F1209051525