Copilot Studio has its own analytics page. Which is fair enough if you have one agent, but if you have many agents, you probably need something more. From that perspective, Agent Insights Hub came up. It’s part of the Copilot Agent Kit, and it’s still marked as a preview feature.

In this post, I’ll walk through setup, what each tab shows, and what tripped me up.

Prerequisites

To ensure a smooth configuration process, please ensure the following prerequisites have been completed:

  • Copilot Studio Kit installed
  • Azure Application Insights resource configured
  • MS Entra ID App registration that allows reading Azure Application Insights
  • CSK – Admin security role assigned
  • Copilot Agent Kit code app is available to users who need access.

To view your Usage History (how you used credits), you need special admin rights and an extra connection.

Let’s start the configuration.

Connect your agents with Application Insights

The steps below need to be followed for each agent you want to monitor.

Navigate to https://copilotstudio.preview.microsoft.com (preview portal) or https://copilotstudio.microsoft.com (classic experience)

Select the agent, then click Settings in the top-right corner of the overview page.

Then select Advanced and choose Application Insights.

Copy the connection string from the Azure Application Insights resource

Paste the connection string into the Application Insights – Connection string text box and choose Save.

Register an MS Entra ID app to access Application Insights.

In the Azure Portal, go to Microsoft Entra ID – App registrations and create a new App registration. Then, in the left-hand pane, go to Manage – API permissions and select + Add a permission.

Choose the APIs my organization uses, search for the Application Insights API, and select it.

Select Delegated permissions, then tick Data. Read, and select Add permissions.

Grant access registered app to the Application Insights resource

Open the Application Insights resource in the Azure portal. Then, select Access control (IAM)

Select Role – Job function roles, search for the Reader role, select it, and choose Next.

Next, select Members, locate the app registration you created earlier, and click Select.

Finally, click on the Review and Assign button.

Agent configuration

Access the Agent Insights Hub and click on + Add Agent.

Then, fill in the following information

Field Description
Agent Choose an agent from the list.
Name Type a unique name for the agent configuration
App Insights Application ID Find it in Azure under Application Insights in the API Access section
App Insights Tenant ID Type your Entra tenant ID
App Insights Client ID Type the Application ID of the App registration
App Insights Secret Location Choose Dataverse for now. Key Vault is not yet available.
App Insights Secret Type the client secret of the App registration

 

After saving the configuration, you can view it under Agent Configurations.

To generate metrics, choose the configuration you have created in the previous steps, select a date range, and choose Run.

Once you run the flows, you will see the informative message below if everything is configured correctly.

 

Related links

Monitor agent performance with Agent Insights Hub in Copilot Agent Kit

Share This