Additional Topics > Reference Information > General Reference Information > Data Aggregation
Data Aggregation
Mercury Business Availability Center uses data aggregation to make data handling and management more efficient and to improve the speed and performance of report generation. Mercury Business Availability Center data aggregation tasks are performed on the Data Processing Server.
Mercury Business Availability Center aggregates various types of data that it collects for reports (for example, response time data collected by Business Process Monitor, infrastructure machine performance data collected by SiteScope, and user traffic data collected by Real User Monitor). Data aggregation involves combining individual measurements into manageable chunks. The result is improved speed and performance of report generation.
Categories of Data
Mercury Business Availability Center groups data into three categories:
How Data Is Aggregated
Mercury Business Availability Center aggregates data collected by Business Process Monitor, Client Monitor, and SiteScope data collectors (not including SiteScope Integration Monitors using the event data template).
- Hourly aggregation is performed every hour. The default schedule is as follows:
Hourly aggregation for Business Process Monitor, SiteScope, and Client Monitor data is done every hour for the time period that began 2 hours earlier (for example, at 12:00 PM Mercury Business Availability Center aggregates the data collected between 10:00 AM and 11:00 AM).
- Daily aggregation is performed once a day after the hourly aggregation. The default schedule is that aggregation begins at 1:00 AM for the previous day (for example, at 1:00 AM Mercury Business Availability Center aggregates the data collected between 12 am, 25 hours ago and 12 am, 1 hour ago). When configuring profile properties, you can specify the time zone that Mercury Business Availability Center uses to determine when to perform daily aggregation in GMT Offset. For more details, see Times and Time Zones Used in Mercury Business Availability Center.
You can define a different delay for when the aggregation begins in the Offline Aggregator context of the Infrastructure Settings page (Admin > Platform > Setup and Maintenance > Infrastructure Settings > Foundation > Offline Aggregator). You can specify a separate delay for Business Process Monitor, custom, and SiteScope data. The delay you specify is for both the hourly and daily aggregations. It is recommended that only advanced users change the defaults set in Infrastructure Settings and after first consulting Mercury Customer Support or your Mercury Services representative. For details on using the Infrastructure Settings page, see Infrastructure Settings in Platform Administration.
How Reports Use Aggregated Data
Whether Mercury Business Availability Center displays a report using raw data, daily aggregated data, or hourly aggregated data depends on several factors:
- If the requested data in the report is for the past 30 hours, Mercury Business Availability Center uses raw data for Business Process Monitor and Client Monitor data.
- For reports that use custom data—trend reports, Real User Monitor reports, Service Level Management reports containing custom data, and Diagnostics reports (if installed)—for the requested time range and granularity, Mercury Business Availability Center chooses the optimal combination of data categories (so that the least amount of rows must be retrieved from the database).
For example, for a time range Jan. 1 10:40 AM - Jan. 3 10:40 AM and granularity of 1 day, Mercury Business Availability Center:
- for Jan. 1 10:40 AM - Jan. 1 11:00 AM - raw data is used
- for Jan. 1 11:00 AM - Jan. 1 11:59 PM - hourly aggregated data is used
- for Jan. 2 12:00 AM - Jan. 2 11:59 PM - daily aggregated data is used
- for Jan. 3 12:00 AM - Jan. 3 10:00 AM - hourly aggregated data is used
- for Jan. 3 10:00 AM - Jan. 3 10:40 AM - raw data is used
- If the requested data in the report does not use custom data and the report range is not 30 hours, Mercury Business Availability Center uses an aggregation threshold—by default, 93% of the maximum—to determine whether to use daily aggregated data, hourly aggregated data, or raw data. When a report is generated for a specific time range, Mercury Business Availability Center calculates the amount of each type of data that exists over that time range and considers only the data types that are above the aggregation threshold.
For example, Mercury Business Availability Center might determine that, for the requested time range "Past Week," raw data exists for 100% of the time range, hourly aggregated data exists for 96% of the time range, and daily aggregated data exists for 86% of the time range. In this case, Mercury Business Availability Center only considers raw data and hourly aggregated data as possible options since both exceed the aggregation threshold of 93%.
When more than one option exists, Mercury Business Availability Center gives priority to the data with the highest granularity (that is, daily is chosen over hourly or raw, hourly is chosen over raw). Thus, in the above example, Mercury Business Availability Center would display the report using hourly aggregated data.
- If the time granularity for the report (or report time range in reports that are not over time) is set to less than a day, regardless of the chosen time range, Mercury Business Availability Center never uses daily aggregated data.
- If the time granularity for the report (or report time range in reports that are not over time) is set to less than an hour, regardless of the chosen time range, Mercury Business Availability Center always uses raw data.
Note: (not relevant for Mercury Managed Services customers) If you select a report time range that includes the past day (for example Past Month), and for which Mercury Business Availability Center chooses daily aggregated data, the data for the past day may be missing, as it may not yet have been aggregated into its one day chunk. In such cases, you can "force" Mercury Business Availability Center to use one hour chunks, instead of one day chunks, by increasing the aggregation threshold from its default setting of 93% to 98%. To do so, open the file <Centers Server root directory>\AppServer\DataEngine\conf\ TAS_consts.properties in a text editor, and search for the line defConf.aggrReasonableDiff=7. Modify the value from 7 to 2, save the file, and restart Mercury Business Availability Center on the Centers Server machine. (If you have multiple Centers Servers, repeat this procedure on all the servers.) Keep in mind that, once you make this change, it will apply for all generated reports.
Note: (not relevant for Mercury Managed Services customers) The following setting can be configured in the Infrastructure Settings Manager, Foundations > Generic Data Engine context, Generic Data Engine - Aggregation table to modify the way aggregated data is used in reports (in general these settings should not be modified without first consulting Mercury Customer Support):
Aggregation Policy setting - Defines the aggregated data usage policy for reports that use custom data. By default, reports use all available data (raw and aggregated); however in certain circumstances it may be necessary to modify this setting—for example, if the aggregation engine is not working (aggregator process on the Data Processing Server is down), you could modify the setting so that only raw data is used.
Effect of Outlier Setting on Data Aggregation
When aggregating data, if the Ignore outlier data in reports setting is cleared in a profile's properties, in the Transaction Threshold Settings section, in Monitor Administration, Mercury Business Availability Center excludes transaction instances whose response time exceeds the outlier value, also set in the Transaction Threshold Settings section. If the Ignore outlier data in reports setting is selected, Mercury Business Availability Center ignores outlier transactions, and therefore the data is not considered. Note that the default outlier value for all transactions is 45 seconds. For details on configuring outlier values, see Transaction Threshold Settings in End User Management Data Collector Configuration.
Data Aggregation Limitations
The following limitations apply, as a result of data aggregation:
- When viewing aggregated data, you may not always be able to see results when drilling down to individual transaction instances.
- When viewing aggregated data for non-rounded time periods, there might be inaccuracies for the time period close to the starting and ending times of the report. For example, if you generate a report on a Thursday based on data collected between 9:45 AM the previous Monday and 9:45 AM the previous Tuesday, the time period between 9:45 AM and 10:00 AM Monday will not contain any data, even if data was originally collected.
- Mercury Business Availability Center is unable to display both aggregated and raw data simultaneously in the Multi-Profile report.
- When viewing SiteScope data that is aggregated hourly, keep in mind that the if the aggregator did not finish aggregating the necessary data, the latest hour may not include all the data. This may happen only on rare occasions when there is a large amount of data needing hourly aggregation.
Data Aggregation and Service Level Management
Service Level Management aggregates data differently. For details, see Aggregated Data in Using Service Level Management.
| Copyright 2006 Mercury Interactive Corporation |