Virtual Assistant Windows 11

[Nella Mcnairy]

VirtualAssistant is a Microsoft open-source template that enables you to create a robust conversational solution while maintaining full control of user experience, organizational branding, and necessary data. The Virtual Assistant core template is the basic building block that brings together the Microsoft technologies required to build a Virtual Assistant, including the Bot Framework SDK, Language Understanding (LUIS), and QnA Maker. It also brings together the essential capabilities including skills registration, linked accounts, basic conversational intent to offer a range of seamless interactions and experiences to users. In addition, the template capabilities include rich examples of reusable conversational skills. Individual skills are integrated in a Virtual Assistant solution to enable multiple scenarios. Using the Bot Framework SDK, skills are presented in source code form, enabling you to customize and extend as required. For more information on skills of Bot Framework, see What is a Bot Framework skill. This document guides you on Virtual Assistant implementation considerations for organizations, how to create a Teams focused Virtual Assistant, related example, code sample, and limitations of Virtual Assistant.The following image displays the overview of virtual assistant:

The decision to add a Virtual Assistant includes many determinants and differs for each organization. The supporting factors of a Virtual Assistant implementation for your organization are as follows:

Microsoft has published a Microsoft template for building Virtual Assistants and skills. With the template, you can create a Virtual Assistant, powered by a text based experience with support for limited rich cards with actions. We've enhanced template to include Microsoft Teams platform capabilities and power great Teams app experiences. A few of the capabilities include support for rich Adaptive Cards, dialogs (referred as task modules in TeamsJS v1.x), teams or group chats, and message extensions. For more information on extending Virtual Assistant to Microsoft Teams, see Tutorial: Extend Your Virtual Assistant to Microsoft Teams.The following image displays the high level diagram of a Virtual Assistant solution:

To dispatch requests properly, your Virtual Assistant must identify the correct LUIS model and corresponding skill associated with it. However, the dispatching mechanism can't be used for card action activities, as the LUIS model associated with a skill, is trained for card action texts. The card action texts are fixed, pre-defined keywords, and not commented from a user.

This drawback is resolved by embedding skill information in the card action payload. Every skill should embed skillId in the value field of card actions. You must ensure that each card action activity carries the relevant skill information, and Virtual Assistant can utilize this information for dispatching.

Next, SkillCardActionData class in the Virtual Assistant template is introduced to extract skillId from the card action payload.A code snippet to extract skillId from card action payload is shown in the following section:

Virtual Assistant can handle interruptions in cases where a user tries to invoke a skill while another skill is active. TeamsSkillDialog, and TeamsSwitchSkillDialogare introduced based on Bot Framework's SkillDialog and SwitchSkillDialog. They enable users to switch a skill experience from card actions. To handle this request, the Virtual Assistant prompts the user with a confirmation message to switch skills:

To add dialog capabilities to a Virtual Assistant, two additional methods are included in the Virtual Assistant activity handler: OnTeamsTaskModuleFetchAsync and OnTeamsTaskModuleSubmitAsync. These methods listen to dialog-related activities from Virtual Assistant, identify the skill associated with the request, and forward the request to the identified skill.

A similar approach is followed for card action dispatching and dialog responses. Dialog fetch and submit action data is updated to include skillId.Activity Extension method GetSkillId extracts skillId from the payload, which provides details about the skill that needs to be invoked.

Teams apps can exist in multiple scopes including 1:1 chat, group chat, and channels. The core Virtual Assistant template is designed for 1:1 chats. As part of the onboarding experience Virtual Assistant prompts users for name and maintains user state. Since the onboarding experience isn't suited for group chat or channel scopes, it has been removed.

The commands for a message extension are declared in your app manifest file. The message extension user interface is powered by those commands. For a Virtual Assistant to power a message extension command as an attached skill, a Virtual Assistant's own manifest must contain those commands. You must add the commands from an individual skill's manifest to the Virtual Assistant's manifest. The command ID provides information about an associated skill by appending the skill's app ID through a separator :.

Once the commands are invoked by a user, the Virtual Assistant can identify an associated skill by parsing the command ID, update the activity by removing the extra suffix : from the command ID, and forward it to the corresponding skill. The code for a skill doesn't need to handle the extra suffix. Thus, conflicts between command IDs across skills are avoided. With this approach, all the search and action commands of a skill within all contexts, such as compose, commandBox, and message are powered by a Virtual Assistant.

Some message extension activities don't include the command ID. For example, composeExtensions/selectItem contains only the value of the invoke tap action. To identify the associated skill, skillId is attached to each item card while forming a response for OnTeamsMessagingExtensionQueryAsync. This is similar to the approach for adding adaptive cards to your Virtual Assistant.

The following example shows how to convert the Book-a-room app template to a Virtual Assistant skill:Book-a-room is a Teams that allows users quickly to find and reserve a meeting room for 30, 60, or 90 minutes starting from the current time. The default time is 30 minutes. The Book-a-room bot scopes to personal or 1:1 conversations.The following image displays a Virtual Assistant with a book a room skill:

A skill manifest is a JSON file that exposes a skill's messaging endpoint, ID, name, and other relevant metadata. This manifest is different than the manifest used for custom app upload in Teams. A Virtual Assistant requires a path to this file as an input to attach a skill. We've added the following manifest to the bot's wwwroot folder.

We have built a LUIS model by understanding these two commands. Corresponding secrets must be populated in cognitivemodels.json. The corresponding LUIS JSON file is found here.The corresponding .lu file is shown in the following section:

With this approach, any command issued by a user to Virtual Assistant related to book room or manage favorites are identified as a command associated with Book-a-room bot and is forwarded to this skill.On the other hand, Book-a-room room bot needs to use LUIS model to understand these commands if they aren't typed full. For example: I want to manage my favorite rooms.

Virtual Assistant uses SetLocaleMiddleware to identify current locale and invoke corresponding dispatch model. Bot framework activity has locale field, which is used by this middleware. You can use the same for your skill as well. Book-a-room bot doesn't use this middleware and instead gets locale from Bot framework activity's clientInfo entity.

We've added claimsValidator to restrict callers to the skill. To allow a Virtual Assistant to call this skill, populate AllowedCallers array from appsettings with that particular Virtual Assistant's app ID.

Updating activity, such as card refresh isn't supported yet through Virtual Assistant (github issue). Hence, we've replaced all card refresh calls UpdateActivityAsync with posting new card calls SendActivityAsync.

To forward card action or dialog activities to an associated skill, the skill must embed skillId to it.Book-a-room bot card action, dialog fetch and submit action payloads are modified to contain skillId as a parameter.

Book-a-room bot is designed for private chats, such as personal or 1:1 scope only. Since we have customized Virtual Assistant to support group chat and channel scopes, the Virtual Assistant must be invoked from the channel scopes and thus, Book-a-room bot must get activities for the same scope. Hence Book-a-roombot is customized to handle those activities. You can find the check in OnMessageActivityAsync methods of Book-a-room bot's activity handler.

You can also use existing skills from Bot Framework Solutions repository or create a new skill altogether from scratch. For creating a new skill, see tutorials to create a new skill. For Virtual Assistant and skills architecture documentation, see Virtual Assistant and skills architecture.

You can also use existing skills from Bot Framework Solutions repository or create a new skill altogether from scratch. Tutorials for the later can be found here. Refer to documentation for Virtual Assistant and skills architecture.

In 2019, Microsoft began reducing the prevalence of Cortana and converting it from an assistant into different software integrations.[9] It was split from the Windows 10 search bar in April 2019.[10] In January 2020, the Cortana mobile app was removed from certain markets,[11][12] and on March 31, 2021, the Cortana mobile app was shut down globally.[13] On June 2, 2023, Microsoft announced that support for the Cortana standalone app on Microsoft Windows would end in late 2023 and would be replaced by Microsoft Copilot.[14] Support for Cortana in the Microsoft Outlook and Microsoft 365 mobile apps was discontinued in fall of 2023.[15]

The development of Cortana started in 2009 in the Microsoft Speech products team with general manager Zig Serafin and Chief Scientist Larry Heck. Heck and Serafin established the vision, mission, and long-range plan for Microsoft's digital personal assistant and they built a team with the expertise to create the initial prototypes for Cortana.[16] Some of the key researchers in these early efforts included Microsoft Research researchers Dilek Hakkani-Tr, Gokhan Tur, Andreas Stolcke, and Malcolm Slaney, research software developer Madhu Chinthakunta, and user experience designer Lisa Stifelman. To develop the Cortana digital assistant, the team interviewed human personal assistants. The interviews inspired a number of unique features in Cortana, including the assistant's "notebook" feature. Originally, Cortana was meant to be only a codename, but a petition on Windows Phone's UserVoice site proved to be popular and made the codename official.[17][18]

3a8082e126