Merge pull request #28870 from MicrosoftDocs/main

[AutoPublish] main to live - 07/03 15:31 PDT | 07/04 04:01 IST

Author: m365-skilling-repo-management[bot]

copilot/employee-self-service/TOC.yml

@@ -5,6 +5,8 @@ items:
   items:
     - name: Overview of the deployment process
       href: deploy-overview-alm.md
+    - name: Prerequisites
+      href: prerequisites.md
     - name: Prepare for deployment
       href: prepare.md
     - name: Install the Employee Self-Service agent

copilot/employee-self-service/customize.md

@@ -20,7 +20,7 @@ appliesto:
 # Customize the Employee Self-Service agent
 
 >[!NOTE]
->The Employee Self-Service agent is currently in on-demand preview. Deployment processes are subject to change before this product becomes generally available.
+>The Employee Self-Service agent is currently in limited public preview. Deployment processes are subject to change before this product becomes generally available.
 
 Customization is how you make the Employee Self-Service (ESS) agent work in the best way to suit your organization's needs.
 

copilot/employee-self-service/deploy-overview-alm.md

@@ -20,7 +20,7 @@ appliesto:
 # Employee Self-Service agent deployment overview
 
 >[!NOTE]
->The Employee Self-Service agent is currently in on-demand preview. Deployment processes are subject to change before this product becomes generally available.
+>The Employee Self-Service agent is currently in limited public preview. Deployment processes are subject to change before this product becomes generally available.
 
 There are several steps necessary to deploy the Employee Self-Service (ESS) Agent. Refer to the table for an overview of the deployment process.
 
@@ -29,9 +29,9 @@ There are several steps necessary to deploy the Employee Self-Service (ESS) Agen
 |Power Platform administrator |- Create environment for development and testing.</br> - Create a preferred custom solution. |Create an environment for production rollout. |
 |Copilot Studio maker |- Install the ESS agent.</br> - Install ISV extension packs and set up connectors.</br> - Customize and test.</br> - Export as a managed solution. |-Install the ESS agent.</br> - Import customization solution.</br> - Test.</br> - Publish. |
 
-## Development stages
+## Deployment stages
 
-There are four stages in developing the ESS agent.
+There are four stages in deploying the ESS agent.
 
 1. Prepare:
     1. Set up roles.
@@ -88,3 +88,69 @@ The ESS Agent has three distinct persona experiences:
 
 These experiences are within the boundary of a single tenant, whereas the environment maker’s experience installing and publishing the agent can vary among different environments within Power Platform. It’s therefore recommended to establish ALM environments such as Developer, Test, and Production for publishing and testing the ESS Agent. Each of the third-party ISV applications can also be connected to the respective environments if each application has their own ALM instances (such as Developer, Test, and Production).
 It's recommended to have end users using a live production tenant to make use of recent and meaningful user-object interactions in Microsoft Graph.
+
+## Quality Assurance Strategy (QA)
+
+Deploying Copilot agents is a transformative initiative, but without rigorous testing, even the most promising agent can fall short of expectations.  Here’s why testing is essential:
+
+- Ensures accuracy, relevance, and trust
+  - *simulate real-world scenarios, including edge cases, to uncover blind spots early.*
+- Supports security, compliance, and governance
+  - *Involve compliance, security, governance, and corporate communications teams early in the testing phase.*
+- Refines user experience and tone
+  - *Using Copilot Studio’s instruction field to define tone, language, and behavior explicitly*
+- Enables performance benchmarking and continuous improvement
+  - *Define success metrics (e.g., resolution rate, user satisfaction) and track them from pilot to production.*
+- Validates technical integration and readiness
+  - *A phased rollout – starting in dev/test environments before scaling to production*
+- Aligns with Microsoft's deployment framework
+  - Scenario-based testing
+  - CI/CD pipelines for version control
+  - Final DLP and RBAC validations
+  - Simulated test data for controlled evaluations
+  - *Adopt a proven, enterprise-grade approach.*
+
+### Golden prompt testing framework
+
+This framework is one of the most common testing frameworks for quality testing the agent that serves as the final gate before production deployment.  This is shared here ONLY for guidance and please align with the standard QA process that’s established for your enterprise.
+
+**What are golden prompts?**
+
+Golden prompts are a curated set of test scenarios that: 
+
+- Represent critical user workflows and edge cases 
+- Have known, expected responses (golden responses) 
+- Cover core functionality that must never break 
+- Serve as regression detection mechanisms 
+
+The following **Process flow** can help you understand testing and golden prompts. Think of this flow as circular rather than linear, as testing is iterative and continuous.
+
+1. Scenarios
+1. Golden prompt testing
+1. Go or No go
+1. Production release
+1. Feedback
+1. Repeat
+
+### Implementation guidelines
+
+#### Golden prompt maintenance
+
+|Responsibility |QA Team |
+|---------------|--------|
+|Activities |- Maintain comprehensive library of golden prompts</br> - Regularly review and update prompts based on:</br> --New feature additions</br> --Historical failure patterns</br> -- Customer feedback</br> --Usage analytics |
+|Prompt categories |- **Core functionality:** Basic system operations</br> - **Integration points:** API calls, database operations</br> - **Edge cases:** Boundary conditions, error scenarios</br> - **Performance:** Response time and resource usage</br> **Security:** Authentication, authorization, data validation |
+
+#### Pre-release testing protocol
+
+|Trigger |Quality gate for production deployment |
+|--------|---------------------------------------|
+|Process |1. **Testing strategy** Begin by testing the updated package in the Copilot Studio test window as an initial validation step. Additionally, consider using a ring deployment strategy to progressively test the package across lower environments before promoting it to production.</br> 2. **Automated execution** Run all golden prompts against the modified package.</br> 3. **Response verification** Compare actual responses with expected golden responses.</br> 4. **Tolerance checking** Apply acceptable variance thresholds where applicable.</br> **Failure analysis** Investigate any mismatches or failures. |
+|Success criteria |- All golden prompts must pass</br> - Response times within acceptable ranges</br> - No security vulnerabilites introduced</br> - Memory/resource usage within limits |
+
+#### Go/ No go decision framework
+
+|Decision authority |Release management qualifying QA team approval |
+|-------------------|-----------------------------------------------|
+|Go criteria |- 100% golden prompt pass rate</br> - All critical functionality verified</br> - Performance benchmarks met</br> - Security checks passed |
+|No go triggers |- Any golden prompt failure</br> - Performance degradation beyond threshold</br> - Security vulnerability detected</br> - Resource usage exceeds limits |

copilot/employee-self-service/design-best-practices.md

@@ -20,7 +20,7 @@ appliesto:
 # How to craft agent instructions for high-quality responses in Employee Self-Service
 
 >[!NOTE]
->The Employee Self-Service agent is currently in on-demand preview. Deployment processes are subject to change before this product becomes generally available.
+>The Employee Self-Service agent is currently in limited public preview. Deployment processes are subject to change before this product becomes generally available.
 
 The Employee Self-Service (ESS) agent is a template in Microsoft Copilot studio that can be infinitely customized to assist users in your organization with HR and IT tasks. Use guidance here to learn more about how to craft high-quality responses using a mix of techniques to fit your business needs. Learn more about agents in Copilot Studio.
 

copilot/employee-self-service/install.md

@@ -20,7 +20,7 @@ appliesto:
 # Install the Employee Self-Service agent
 
 >[!NOTE]
->The Employee Self-Service agent is currently in on-demand preview. Deployment processes are subject to change before this product becomes generally available.
+>The Employee Self-Service agent is currently in limited public preview. Deployment processes are subject to change before this product becomes generally available.
 
 After completing the required steps in the [preparation](prepare.md) stage, you can install the Employee Self-Service (ESS) agent in a specific Power Platform environment.
 

copilot/employee-self-service/known-issues-limitations.md

@@ -19,6 +19,9 @@ appliesto:
 
 # Known issues and limitations for the Employee Self-Service agent
 
+>[!NOTE]
+>The Employee Self-Service agent is currently in limited public preview. Deployment processes are subject to change before this product becomes generally available.
+
 As you consider previewing the Employee Self-Service (ESS) Agent, we want to provide a clear view of known limitations—along with workarounds and roadmap visibility—to help you plan effectively and set the right expectations with your stakeholders.
 
 This guide includes:
@@ -69,6 +72,7 @@ Applies to:
 |User experience |Agent names have length constraints i.e. 42 characters max |No plans – dependent on Copilot Studio |Follow corporate naming conventions for agents that are not long |
 |Licensing and payment |Any custom agent flows outside the standard ESS package incurs charges based on each execution. |Support for custom agent flows beyond the standard ESS package is planned for release by July 2025. |Until support for custom agent flows becomes available, we recommend using the prebuilt Agent Flows included in the ESS package. These flows cover common scenarios and are designed to work out-of-the-box, without requiring more configuration or extra charges. They’re a great way to get started and deliver value quickly while ensuring compatibility with the current ESS experience. |
 |Licensing and payment |Pay-as-you-go billing users (non-M365 Copilot license) won't be able to access personal graph data (chat, emails, calendar, etc.), but will be able to access tenant graph like SharePoint and Copilot connector (graph connectors) |By design – there are no plans to change this in near roadmap |Plan suitable licensing for each scenario or persona to ensure employees can access the information they need. |
+|Licensing and payment |Billing reports in Microsoft Admin Center (MAC) and Power Platform Admin Center (PPAC) will always show the name of the agent matching with the name that’s published in the app store i.e. for now it’s “Employee Self-Service (Preview)” – even if the maker have renamed it in Copilot Studio |By design – evaluating options with Platform teams to show the customized name for customers |Coordinate with billing stakeholders communicating the agent’s name to identify easily for any billing implications that might come up |
 |Content related |Semantic indexing is limited to roughly 200 pages of content; anything beyond this limit won't be processed. |By design - there are no plans to include it in the future roadmap. |To stay within the ~200-page semantic indexing limit, consider breaking up your knowledge content into smaller, topic-focused libraries or SharePoint sites. Prioritize including only the most relevant and frequently used content in the initial knowledge source. Less critical or long-tail content can be stored separately and accessed through links or added later as needed. This helps ensure the most valuable content is indexed and searchable by the ESS Agent. |
 |Content related |If a page only has links and no actual content, it won’t be part of the summary the agent creates. |By design - there are no plans to include it in the future roadmap. |To ensure these pages are included in the summary, add a brief description or context alongside the links. Even a short paragraph explaining what the links are for or summarizing the content they point to can make the page eligible for summarization. This helps the ESS Agent better understand and include the information during indexing. |
 |Content related |You might notice lower-quality responses when the agent pulls information from Graph Connector content. |Enhanced response quality for Graph Connector content is planned for release in Summer 2025. |To improve response quality, prioritize using SharePoint or directly integrated knowledge bases as your primary content sources, especially for high-impact topics. If you must use Graph Connector sources, ensure the content is well-structured, includes clear headings, and avoids overly long pages or lists of links. Adding short summaries or key takeaways at the top of each page can also help the agent generate better answers. |

copilot/employee-self-service/overview.md

@@ -20,7 +20,7 @@ appliesto:
 # Overview of the Employee Self-Service agent
 
 >[!NOTE]
->The Employee Self-Service agent is currently in on-demand preview. Deployment processes are subject to change before this product becomes generally available.
+>The Employee Self-Service agent is currently in limited public preview. Deployment processes are subject to change before this product becomes generally available.
 
 The Employee Self-Service (ESS) agent helps your employees by answering policy related questions, simplifying HR tasks, and assisting with IT help. The agent makes it easier to:
 

copilot/employee-self-service/prepare.md

@@ -20,7 +20,7 @@ appliesto:
 # Prepare to deploy the Employee Self-Service agent
 
 >[!NOTE]
->The Employee Self-Service agent is currently in on-demand preview. Deployment processes are subject to change before this product becomes generally available.
+>The Employee Self-Service agent is currently in limited public preview. Deployment processes are subject to change before this product becomes generally available.
 
 Preparation is the first step to deploying the Employee Self-Service (ESS) agent. First, you need to meet the [prerequisites](prerequisites.md) The following roles are required to prepare the agent for deployment.
 

copilot/employee-self-service/prerequisites.md

@@ -20,7 +20,7 @@ appliesto:
 # Prerequisites to deploy the Employee Self-Service agent
 
 >[!NOTE]
->The Employee Self-Service agent is currently in on-demand preview. Deployment processes are subject to change before this product becomes generally available.
+>The Employee Self-Service agent is currently in limited public preview. Deployment processes are subject to change before this product becomes generally available.
 
 You'll need to confirm the following prerequisites have been met before deploying the Employee Self-Service agent (ESS) to your organization.
 
@@ -49,7 +49,7 @@ If ESS Agent is going to be consumed by users without Copilot licenses, it requi
 
 The ESS agent performs LLM compute for the following scenarios in addition to the included Topics:
 
-- Enabling Alchemy incurs LLM compute for determining the type of user query.
+- Enabling Microsoft 365 Self-Help incurs LLM compute for determining the type of user query.
 - Live agent hand-off package invokes LLM compute for summarizing the chat history and handing off to the live agent.
 
 Use the following sample benchmarks to understand capacity planning for ESS agent deployment:

copilot/employee-self-service/publish.md

@@ -20,7 +20,7 @@ appliesto:
 # Publish the Employee Self-Service agent to your organization
 
 >[!NOTE]
->The Employee Self-Service agent is currently in on-demand preview. Deployment processes are subject to change before this product becomes generally available.
+>The Employee Self-Service agent is currently in limited public preview. Deployment processes are subject to change before this product becomes generally available.
 
 Publishing the Employee Self-Service (ESS) agent makes it available to your users. Currently, the ESS agent supports publishing to Copilot Business Chat.