Dylan Morley · Follow
Published in · 8 min read · May 9, 2019
--
Azure Monitor is the unified experience in Microsoft Azure for the collection, analysis and monitoring of logs and metrics from a number of sources. Azure Monitor is a managed service for working with large scale telemetry data, giving you the capability to create alerting and automated responses based on metric thresholds and other calculations.
The teams at Microsoft work closely with Azure Monitor and make metrics from its particular products available as platform metrics. Need to know how many request units a Cosmos instance is using? Or how many dead letters you have on a Service Bus topic? No problem, there are metrics available that you can visualise and alert on, with minimal effort from your engineering team.
More metrics are being made available within Azure on a regular basis, a list of those currently available can be seen here. With the release of the Azure Monitor plugin for Grafana, you can now achieve a best-in-class graphing experience over almost real time, managed metrics.
But what if a metric you require isn’t available as part of platform metrics? Perhaps it’s something specific to your business domain, or something that the product team hasn’t (yet) made available for your use-case. The Metrics API supports a REST endpoint that allows you to store custom metrics alongside platform metrics, which means your data is stored at the resource level and can be managed as per the standard metrics for the resource.
At ASOS, monitoring and alerting is a vital part of our operation. Any downtime that impacts our ability to take customer orders must be avoided, we therefore require metrics and logs from event producing systems with a low level of end-to-end latency. This is so we can understand when systems are about to go wrong and take proactive action. Gathering telemetry from a large number of distributed micro-services, running at high scale, and turning that into something that can be acted upon quickly is a big-data problem. We aim to take advantage of the products Microsoft make available to manage the collection and storage of this data, allowing us to perform the analysis and focus on our business requirements. The Monitor Metrics API helps us achieve this, as it gives us the ability to store time-series data at high precision and low latency.
This article will show you how to create an Azure function that will make use of the Azure Monitor API to create a custom metric against a service bus namespace. The source code for this article is available in GitHub.
Consider an application that makes heavy use of Azure Service Bus (ASB), with a number of queues, topics and dependent systems subscribing to information. We want to set up alerts to make sure that our capacity limits aren’t being reached on queues and topics. The platform metrics made available from ASB allow us a view of Size, which is the total size of items currently in the entity. Without knowing the maximum size allowed, this doesn’t enable us to raise a meaningful alert without manually configuring alert thresholds.
A better metric for this requirement is a calculation that gives ‘Percentage of Capacity Used’ at the entity level. With a percentage-based metric, an alert can be triggered when any entity is reaching a threshold, e.g 85% capacity, without requiring any knowledge about the underlying maximum storage size.
We want to alert when something is running out of space so we can be proactive. Using Action Groups and Logic apps, there are opportunities to run self-healing operations on certain signals, before a capacity problem becomes critical.
We’ll deploy a function application into a subscription that will make use of the Azure Management API (Microsoft.Azure.Management.Fluent) to retrieve details about service bus namespaces in that subscription. The function will enumerate the topics and queues to retrieve the maximum sizes allowed for each entity and what the current size is, then calculate the percentage of capacity used, before finally storing the metric data alongside the service bus namespace by making a REST call to the Monitor API.
To enumerate a subscription and create custom metrics, you’ll need a service principal (SPN) within your tenant that has both Reader and Monitoring Metrics Publisher permissions to the subscription. At run-time, the function will authenticate as the SPN and use both the Azure Management API and the Monitoring API in the context of the principal.
We’ll therefore need to obtain two access tokens for working with the different Azure resources (one for each API) with the resource/audience correctly scoped.
To run the project locally, make sure you have set the values in local.settings.json for the SPN, your Azure tenant and subscription details.
The solution is a V2 functions project in Net Standard 2.0 that makes use of the Willezone nuget package for dependency injection. It does the job nicely while the Functions team finish making DI a native experience.
The startup classes in the Application directory wire up the required dependencies, which are injected into the function using the [Inject] attribute.
There’s a timer triggered function (ServiceBusPercentageUsed) that will execute every five minutes. An instance of ServiceBusPercentageUsedMetricsGenerator is injected and the single method that class exposes is executed – that’s the only thing done in the function entry point itself.
As the signature for Run is static, I find it preferable to delegate all the work the function will perform to a container created instance, allowing us to take advantage of DI and instance level variables within the class that does the work. This should lead to a cleaner overall design.
The metrics generator class takes all required dependencies from the container and begins enumerating the service bus namespaces in the subscription, kicking off a number of tasks that process each namespace.
The work done in ProcessServiceBusNamespace enumerates the queues and topics, looking at the MaxSizeInMB and CurrentSizeInBytes properties on the entities and calculating the current percentage of capacity consumed by using the values. Each calculation is then added to the series for the payload we will transmit to the Azure Monitor metrics API.
To monitor API schema supports a dimension names and series values section . The payload being generated will look like the following.
As the queues and topics are enumerated, the payload is built using the Model objects that represent the schema.
There will be one dimension name (EntityName), and many series for the topics and queues – processing all the items in a service bus namespace should result in one call to the Monitor API.
To send a custom metric to the API, a URL in the format https://{azureRegion}.monitoring.azure.com/{azureResourceId}/metrics must be built. The metrics transmitted for a resource are sent to the monitoring ingress endpoint in the same region as the resource. A list of the supported regions can be found here.
azureRegion – this will be the region of the service bus namespace, which can be retrieved from the Region.Name property of IServiceBusNamespace.
azureResourceId – the unique resource identifier, again available on the IServiceBusNamespace object as Id.
A class named AzureMonitorMetricsClient exposes one method, which takes the region, resource Id and the payload. It’s responsible for constructing the request to send to the API, which requires the correct endpoint to transmit to and appropriate authentication context.
To authenticate with the Monitor API, we need an access token that has been scoped for https://monitoring.azure.com/. MetricsApiAccessTokenProvider provides this functionality for us as a simple wrapper around the ADAL library, which takes care of token management for us.
We can then send the data and the Azure Monitor API will ingress the information and take care of creating the metric name for the resource.
Once you have transmitted some data, you can view the metrics in the portal alongside the platform metrics for a resource.
- In the resource blade, choose the Metrics option under the Monitoring section
- In the main screen, choose the namespace for your custom metrics and the metric you want to view
- As the metric created is multi-dimensional, there’s an option to split the data by the dimension name, which was set to EntityName
Finally, you can view the results, which will be the percentage of capacity used, split by queue and topic name
When viewing the graph over smaller time windows, e.g. one hour, solid lines indicate times where data points have been received, dotted lines represent periods where no data exists. As the function is on a five-minute interval, you’ll only expect to see data points at that frequency, but could obviously lower this value if required. Storing metrics at one-minute precision is supported in Azure Monitor.
Now we’re capturing the metric, alerting on it is as simple as setting up a standard Azure Monitor alert – a full walk through is available here.
When choosing Signal Source, the metric displays against the namespace we gave it when creating the payload, which was ‘ASOS Custom Metrics’
If you end up automating alert creation using an ARM template, you must specify the namespace for custom metric alerts, like so.
The last thing to do is deploy the function into the subscription it will be calculating metrics for. It will then execute as per the timer interval, calculate metrics for all service bus namespaces in the subscription and store them in Azure as custom metrics for the resource.
About me
I’m Dylan Morley, one of the Principal Software Engineers at ASOS. I primarily work on the back-end commerce APIs that enable our shopping experience.
FAQs
Using the Azure Monitor Metrics API to create custom metrics? ›
Azure Monitor Metrics can only store numeric data in a particular structure, whereas Azure Monitor Logs can store a variety of data types that have their own structures. You can also perform complex analysis on Azure Monitor Logs data by using log queries, which can't be used for analysis of Azure Monitor Metrics data.
What is the difference between Azure metrics and Azure Monitor? ›Azure Monitor Metrics can only store numeric data in a particular structure, whereas Azure Monitor Logs can store a variety of data types that have their own structures. You can also perform complex analysis on Azure Monitor Logs data by using log queries, which can't be used for analysis of Azure Monitor Metrics data.
What are Azure Monitor metrics? ›Azure Monitor Metrics is a feature of Azure Monitor that collects numeric data from monitored resources into a time-series database. Metrics are numerical values that are collected at regular intervals and describe some aspect of a system at a particular time.
How do I Create a custom log in Azure Log Analytics? ›Open the Custom Log wizard
In the Azure portal, select Log Analytics workspaces > your workspace > Tables. Select Create and then New custom log (MMA-based). By default, all configuration changes are automatically pushed to all agents. For Linux agents, a configuration file is sent to the Fluentd data collector.
- Go to the Azure Monitor page and select Alerts from the pane on the left.
- Select Action groups > Create.
- Select values for Subscription, Resource group, and Region.
- Enter a name for Action group name and Display name.
- Select the Actions tab.
- On the Actions tab under Action type, select Logic App.
Collect, analyze, and act on telemetry data from your cloud and hybrid environments. Azure Monitor supports your operations at scale by helping you maximize the performance and availability of your resources and proactively identify problems.
How do I create metrics in Azure? ›- Select the Select a scope button to open the resource scope picker. ...
- For some resources, you must pick a namespace. ...
- Select a metric from a list of available metrics.
- Open metrics explorer for an Azure resource.
- Select a metric to plot on a chart.
- Perform different aggregations of metric values.
- Modify the time range and granularity for the chart.
Azure Monitor stores structured and unstructured log data of all types in Azure Monitor Logs. You can route data to Log Analytics workspaces for querying and analysis.
What are the different types of monitoring in Azure? ›- Application monitoring data. ...
- Azure resource monitoring data. ...
- Azure tenant monitoring data. ...
- Azure subscription monitoring data. ...
- Virtual machine and cloud services data. ...
- Application insight data. ...
- Azure Active Directory reporting data. ...
- Activity logs.
What is custom logging? ›
Custom logs are logs that contain diagnostic information from custom applications, other cloud providers, or an on-premise environment. Custom logs can be ingested in the following ways: By using PutLogs to ingest custom logs directly. See the Logging Ingestion API and REST APIs for more information.
Where are Azure monitor logs stored? ›Logs in Azure Monitor are stored in a Log Analytics workspace that's based on Azure Data Explorer, which provides a powerful analysis engine and rich query language.
How do I add custom logs to Azure Sentinel? ›Configure the Log Analytics agent
From the connector page, select the Open your workspace custom logs configuration link. Or, from the Log Analytics workspace navigation menu, select Custom logs. In the Custom tables tab, select Add custom log.
Azure Monitor builds on top of Azure Log Analytics, the platform service that gathers log and metrics data from all your resources. The easiest way to think about Azure Monitor vs Log Analytics is that Azure Monitor is the marketing name, whereas Azure Log Analytics is the technology that powers it.
How do I Create a dashboard in Azure Monitor data? ›- In the menu dropdown on the left in the Azure portal, select Dashboard.
- On the Dashboard pane, select New dashboard > Blank dashboard.
- Enter a name for the dashboard.
- Look at the Tile Gallery for various tiles that you can add to your dashboard.
We create an action group for sending alert notification from azure monitor. Then set the email to a group-mail address and a single mail address.
What are the three alert states in Azure Monitor? ›Azure Monitor collects data from various sources. These sources include logs and metrics from the Azure platform and resources, custom applications, and agents running on virtual machines.
What is the difference between Azure Monitor workbook and dashboard? ›Azure dashboards are useful in providing a "single pane of glass" of your Azure infrastructure and services. While a workbook provides richer functionality, a dashboard can combine Azure Monitor data with data from other Azure services.
How do you Create a set of metrics? ›- Consider your objectives. The first step in developing metrics for your business is to consider your goals. ...
- Use SMART goals to establish objectives. ...
- Define benchmarks for each metric. ...
- Develop a measurement plan.
What are the 4 steps involved in metrics program? ›
- Step 1: Articulate Your Goals. This is obvious, but you should always start by defining your goals for your product. ...
- Step 2: List the Actions That Matter. ...
- Step 3: Define Your Metrics. ...
- Step 4: Evaluate your Metrics.
- Lead time for changes. One of the critical DevOps metrics to track is lead time for changes. ...
- Change failure rate. The change failure rate is the percentage of code changes that require hot fixes or other remediation after production. ...
- Deployment frequency. ...
- Mean time to recovery.
- From the Azure portal menu, select Create a resource. You can also select Create a resource on the Azure Home page.
- On the Create a resource page, select Integration > API Management.
- In the Create API Management page, enter settings. Setting. Description. ...
- Select Review + create. Tip.
In the Azure portal, navigate to your API Management instance. On the Overview page, on the Monitor tab, review key metrics for your APIs. To investigate metrics in detail, select Metrics from the left menu. From the drop-down, select metrics you're interested in.
When should I use Azure API Management? ›Why do we require Azure API Management? There is a necessity for Azure API Management because API Management provides a highly scalable and secure platform for developers to quickly expose APIs to internal and external consumers. It helps manage, secure, and scale APIs to use in mobile, web, and internal applications.
What should you create for Azure Monitor? ›- Set up a Log Analytics workspace.
- Design a Log Analytics workspace architecture.
- Manage tables in a Log Analytics workspace.
- Monitor workspaces with Log Analytics Workspace Insights.
- Analyze data in Log Analytics.
- Get started with KQL log queries.
Collect, analyze, and act on telemetry data from your cloud and hybrid environments. Azure Monitor supports your operations at scale by helping you maximize the performance and availability of your resources and proactively identify problems.
Which is the best monitoring service in Azure? ›- Comparison of Best Azure Monitors.
- #1) Serverless360 (Best Overall)
- #2) Site24x7.
- #3) ManageEngine M365 Manager Plus.
- #4) Application Insights.
- #5) Azure Monitor.
- #6) Service Bus Explorer.
- #7) Cerebrata.
Four basic types of monitoring can be readily distinguished by the nature of questions that the particular monitoring effort is designed to address—(1) surveillance monitoring, (2) implementation monitoring, (3) effectiveness monitoring, and (4) ecological effects monitoring (Table 1).
What are the 3 types of data that can be stored in Azure? ›Azure storage types include objects, managed files and managed disks.
What are the 4 types of active monitoring? ›
Safety inspections, sampling, surveys and tours are four active monitoring methods that can be used to check conformance to standards. Workplace inspections play an important role in active monitoring. Various factors must be considered when setting up an inspection system, such as: – Type of inspection.
What is the difference between logging and tracking? ›You require both logging and tracing to understand the root cause of the issue. Logs help you identify the issue, while a trace helps you attribute it to specific applications.
What are the two main types of logging? ›Logging is generally categorized into two categories: selective and clear-cutting. Selective logging is selective because loggers choose only wood that is highly valued, such as mahogany. Clear-cutting is not selective.
How long does Azure Monitor keep logs? ›You can keep data in interactive retention between 4 and 730 days. You can set the archive period for a total retention time of up to 2,556 days (seven years). To set the retention and archive duration for a table in the Azure portal: From the Log Analytics workspaces menu, select Tables.
How long are Azure Monitor activity logs kept? ›Activity log events are retained in the Azure platform for 90 days.
How do I export logs from Azure Monitor? ›- On the Log Analytics workspace menu in the Azure portal, select Data Export under the Settings section. Select New export rule at the top of the pane.
- Follow the steps, and then select Create.
Under Resources, click Logs, and then click the Enable Logging slider to create and enable a new API deployment log in the Oracle Cloud Infrastructure Logging service in the Create Log entry panel: Compartment: By default, the current compartment. Log Group: By default, the first log group in the compartment.
How do I create a custom event in Azure? ›- Select your Azure subscription.
- Select an existing resource group or select Create new, and enter a name for the resource group.
- Provide a unique name for the custom topic. ...
- Select a location for the Event Grid topic.
- Select Review + create at the bottom of the page.
Microsoft Sentinel security analytics data is stored in an Azure Monitor Log Analytics workspace. Billing is based on the volume of that data in Microsoft Sentinel and the Azure Monitor Log Analytics workspace storage.
What is the difference between metrics monitoring and log monitoring? ›Metrics can be used to monitor performance, recognize events of importance, and facilitate prediction of future lapses. Logs are usually used for troubleshooting issues, but also for analyzing user behavior, application metrics and a growing variety of additional use cases.
What is the difference between data monitoring telemetry and logging? ›
Logging versus telemetry
In short, logging is how you collect data about your app in the lab; instrumenting your app for telemetry, on the other hand, is how you collect data once the app is released into the wild.
Whereas log monitoring is the process of tracking logs, log analytics evaluates logs in context to understand their significance. This includes troubleshooting issues with software, services, applications, and any infrastructure with which they interact.
How to create dashboard with API? ›- Method 1: Loading Into a Database. ...
- Method 2: Built-in API Connectors. ...
- Method 3: Cloud Database Platform. ...
- Add your resources using the API connector. ...
- Create a project for your resources. ...
- Transform your data. ...
- Build your dashboard.
- Decide on the Goal of Creating Dashboards.
- Identify Your Data Sources.
- Choose the Right Chart Type.
- Define What Metrics You Want to Track.
- Pick Your Colors When Building Dashboards.
- Ensure that Your Dashboards are Mobile-Optimized.
- Onboard Your Team Members.
- Go to the Azure portal.
- Select the Monitor pane.
- Select Metrics.
- Select a resource that you've emitted custom metrics against.
- Select the metrics namespace for your custom metric.
- Select the custom metric.
- Go to the Azure Monitor page and select Alerts from the pane on the left.
- Select Action groups > Create.
- Select values for Subscription, Resource group, and Region.
- Enter a name for Action group name and Display name.
- Select the Actions tab.
- On the Actions tab under Action type, select Logic App.
You can alert on any metric or log data source in the Azure Monitor data platform. This diagram shows you how alerts work. An alert rule monitors your data and captures a signal that indicates something is happening on the specified resource.
How do I create a metric alert in Azure? ›- Select a scope for the alert rule.
- Set the conditions for the alert rule.
- Set the actions for the alert rule.
- Set the details for the alert rule.
- Finish creating the alert rule.
Azure Monitor collects logs for most Microsoft Azure services and streams the data to an Azure Event Hub. Azure Event Hubs is a data streaming platform and event ingestion service. In this pipeline, an Event Hub streams the logs collected by Azure Monitor to an Azure function.
Is Azure Monitor and Log Analytics same? ›Log Analytics is a service offered by Microsoft for analyzing and querying log data in Azure. It is a component of Azure Monitor, a solution for collecting and analyzing telemetry data from both cloud and on-premises settings.
What is the difference between Azure Monitor Dashboard and Workbook? ›
Azure dashboards are useful in providing a "single pane of glass" of your Azure infrastructure and services. While a workbook provides richer functionality, a dashboard can combine Azure Monitor data with data from other Azure services.
Is Azure Monitor an API? ›Azure Monitor APIs are a part of the Azure Management APIs. I will, therefore, use these names interchangeably.
Which query language does Azure Monitor use? ›Azure Monitor Logs is based on Azure Data Explorer, and log queries are written by using the same Kusto Query Language (KQL). This rich language is designed to be easy to read and author, so you should be able to start writing queries with some basic guidance.
Is Azure Monitor PaaS or SaaS? ›Azure Monitor is a software as a service (SaaS) offering, so its supporting infrastructure runs in Azure and is managed by Microsoft.
How do I create a dashboard in Azure monitor data? ›- In the menu dropdown on the left in the Azure portal, select Dashboard.
- On the Dashboard pane, select New dashboard > Blank dashboard.
- Enter a name for the dashboard.
- Look at the Tile Gallery for various tiles that you can add to your dashboard.
- In the Azure portal, select the menu at the top left of the screen.
- Select Dashboard from the menu. Your default Azure dashboard will appear.
- Select + New Dashboard and then Blank dashboard.
- Give your dashboard a name.
- Select Save.
Build custom dashboards based on projects, tasks, or user roles, for example. The Azure portal provides a default dashboard as a starting point.