CAISY™ AI Simulation Dashboard

The CAISY™ AI Simulation dashboard shows a visual summary of how your learners are using AI simulations to gain skills. You can answer questions such as:

  • Which and how many AI simulations are my learners taking?
  • How many learners are engaged in learning skills through AI simulations?
  • Which behaviors and modes are my learners engaging with?
  • What are the top 10 most-used AI scenarios?
  • How are my learners' results distributed among the different proficiency levels?
  • How many learners are practicing these scenarios more than once and how much time are they spending?

How to Use

With this dashboard, you can gauge where your learners are on the proficiency spectrum for each AI simulation at any given point in time within the specified date range. The dashboard only reports on the best proficiency rating of each learner within that date range if they have taken the simulation more than once.

For example, this dashboard helps you see how you might need to tailor programs based on results. If many users rank in the Poor category for a specific scenario, you might consider assigning additional contextual content to support the skill the scenario is teaching. Conversely, if many learners score in the Outstanding category, and only a few score in the Poor category, you can tailor programs just for learners scoring poorly, while assigning learners who score outstanding to the next highest skill set or challenging them with a different scenario behavior.

If you haven't yet assigned CAISY™ AI scenarios or added them to custom journeys or learning programs, you can see what scenarios your learners are focusing on and if they align to your corporate business objectives. If you configured audiences for department and job role, you can get a sense of the proficiency level of different departments or job roles by using the audience filter to see the scenarios accessed and the proficiency levels achieved.

Access the Dashboard

To access the CAISY™ AI Simulation Dashboard: from the Analytics menu, select CAISY AI Simulation. The dashboard page displays.

  1. You can filter data on this dashboard by date range, audience, and content title and scenario category. After you set filters, make sure to select Update to update the data shown on the dashboard. Filters you can set include:

    • Date Range: Filters the data by a specified range of dates that you select from the drop-down list. You can also select a custom range from the beginning of one month to the end of the same month or the end of another month.
      • Start: Instead of using a Date Range preset, use this to specify a start date. When you enter a Start date, the Date Range Presets filter changes to Custom. Percipio started recording CAISY activity on March 1, 2024. Activity prior to that date for CAISY activity is unavailable.
      • End: Instead of using a Date Range preset, use this to specify an end date. When you enter an End date, the Date Range Presets filter changes to Custom.
    • Audience: Predefined group of users. Enter the name of an audience and then select it from the list. You can display the data for multiple audiences at the same time by entering more than one audience name.
    • Content title: The title of the AI scenario. Use this filter if you want to just review specific scenarios. The scenarios you select here impact what you see on the rest of the page.
    • Scenario category: AI scenarios are grouped by category. If you want to see AI scenarios for one or more categories, select those categories from the list. What you select here impacts what you see on the rest of the page.
    • Include Inactive users: Select if you want to include inactive users in your counts. If you include inactive users, you may not get an accurate picture of your organization current proficiency levels specially if those inactive users left your company.

  2. In this section you see:

    • Total simulations completed: The total number of simulations completed in the specified date range. If one learner completed a single simulation 8 times, each time the learner took the simulation is added to the overall count, in this case 8. If one learner took 6 different simulations one time each, the count goes up by 6.
    • Unique simulations completed:The number of unique simulations completed in the specified date range. If one learner completed a single simulation multiple times, that is only counted as one unique simulation. If multiple users completed the same simulation, that is only counted as one unique simulation.
    • Unique learners: The number of unique learners who completed a simulation in the specified date range. If one learner completed more than one simulation or one simulation multiple times, they are only counted once.

  3. In this section you see a list of the top 10, most-accessed AI scenarios based on the filters set. The list is initially sorted by the highest number of unique learners who accessed each scenario. You can sort by either of the columns.

    For each benchmark you see:

    • SCENARIO TITLE: The name of each AI simulation. If you select the name, you can view additional details of that one scenario.
    • Unique learners: The number of unique learners within the specified date range who completed each scenario title.
    • Score range: Of all the unique learners who completed the scenario title within the date range, you see the breakdown by level of each users' highest proficiency level. Levels mean the following for the learners:
      • Outstanding: You did an excellent job of demonstrating all the behaviors required of the situation, including but not limited to showing empathy where it mattered, addressing the issues at hand, and reaching a conclusion that solved the problem for all involved.
      • Acceptable: You may have demonstrated some but not all of the behaviors required of the situation, including showing empathy where it mattered, addressing the issues at hand, and reaching a conclusion that solved the problem for all involved.
      • Poor: You demonstrated a lack of familiarity with the behaviors required of the situation. Next time, try showing more empathy where it matters, better addressing the issues at hand, and reaching a conclusion that solves the problem for all involved.

    You can select the name of the scenario to get more details.

    The detailed CAISY AI Simulation page includes the following information:

    • Date and audience filters: Initially carried over from the main dashboard, you can change them. What you specify here impacts the results shown on the page.
    • Unique learners: The number of unique learners within the specified date range who completed each scenario title.
    • Learner results:For the date and audience selected, you see how many times those unique learners completed the simulations and how each completion was rated: Outstanding, Acceptable or Poor. Lets assume 201 scenarios were completed and of which 70 got a rating of Outstanding, 120 got Acceptable, and 40 got Poor. Then percentages calculation for each category. Outstanding: (70/201)*100= 35% Acceptable: (120/201)*100= 60% Poor: (40/201)*100= 20%.
    • Results distribution by CAISY behavior: When learners complete a scenario, they choose whether they want to interact in Role model or Practice mode. If they choose Practice mode, they can then choose which behavior they want to interact with: Defensive, Dismissive or Accepting. In this box, you see all completions grouped by mode and behavior and the proficiency level for each. The data shown here is based on the filter set.
    • Simulation results table: This table gives you the detail of every user who completed the scenario, the mode and behavior they used, and their proficiency level each time they completed the scenario.

      The columns in this table are:

      NOTE: If you are not able to see the user detail columns, ask your admin for permissions.
      • USER ID: The value entered in the User ID attribute. It is used to identify a user inside Percipio.
      • LAST NAME: The value entered in the Last name user attribute. Usually the user's last name.
      • FIRST NAME: The value entered in the First name user attribute. Usually the user's first name.
      • LOGIN NAME: The value entered in the Login name attribute for the user.
      • EMAIL ADDRESS: The value entered in the Email address user attribute; the user's email address.
      • USER STATUS: The status of the user in Percipio. Valid values are active or inactive.
      • AUDIENCES: Shows the audiences where the learner is a member.
      • SCENARIO MODE:The mode of the AI simulation: role model or practice.
      • SCENARIO BEHAVIOR: When using an AI simulation in practice mode, this is the behavior you are practicing against: dismissive, defensive, or accepting.
      • RECURRENCE: The number of times a learner returned and re-completed the content.
      • FIRST RATING:The rating given the first time the learner completed the item.
      • FIRST RATING DATE:The date the first rating was given.
      • LAST RATING:The rating given the last time the learner completed the item.
      • LAST RATING DATE:The date the last rating was given.
      • ESTIMATED DURATION (SECONDS): Total estimated amount of time it takes a learner to complete the specified content. Time is listed in total seconds.
      • LEARNING HOURS (SECONDS): The total number of hours a learner spent learning in this content item. If the item is a Skillsoft course, the total includes time spent watching videos, working on knowledge checks, and taking tests. Time is listed in total seconds.
      • User Attributes (varies): These fields are defined by (and are unique to) your organization. For more information, see User Attributes. If you are using an attribute type of Percipio User, the value in the field can be either first and last name, email address, login ID, or external user ID. The value that appears depends on what has been specified for the individual user.

      From this table, you can:

      • Download as CSV: download the entire list based on filters set into a CSV format that you can use for further data analysis.
      • Customize your view by sorting, filtering, ordering and choosing columns of data to show. For instructions, see Organize columns in tables.

  4. In this section you see a list of all AI simulations and their corresponding data based on the filters set. With this list you can:

    • Download as CSV: download the entire list based on filters set into a CSV format that you can use for further data analysis.
    • Customize your view by sorting, filtering, ordering and choosing columns of data to show. For instructions, see Organize columns in tables.

    The columns available for this section include:

    • SCENARIO TITLE: The name of each AI simulation. If you select the name, you can view additional details of that one scenario.
    • SCENARIOS COMPLETED: The total number of times the scenario was completed. If one learner completed it 3 times, the count would be 3.
    • TOOK MORE THAN ONCE: Of the total number of scenarios completed, how many unique learners completed it more than once.
    • UNIQUE LEARNERS: The number of unique learners who completed the scenario at least once.
    • Poor %: the percent of unique learners whose best rating was poor.
    • Acceptable %: the percent of unique learners whose best rating was acceptable.
    • Outstanding %: the percent of unique learners whose best rating was outstanding.
    • Poor #: the actual number of unique learners whose best rating was poor.
    • Acceptable #: the actual number of unique learners whose best rating was acceptable.
    • Outstanding #: the actual number of unique learners whose best rating was outstanding.
    • ESTIMATED DURATION (SECONDS): Total estimated amount of time it takes a learner to complete the specified content. Time is listed in total seconds.
    • LEARNING HOURS (SECONDS): The total number of hours a learner spent learning in this content item. If the item is a Skillsoft course, the total includes time spent watching videos, working on knowledge checks, and taking tests. Time is listed in total seconds.
    • CATEGORY: The category grouping for this scenario title.
    • SCENARIO UUID: This is an internal Percipio code for a scenario.