Re: aggregation and reporting
Scott Johnson
I have a question that is not typical for this discussion group, but I’m running into this issue more and more. First, a little background. I work for a communications company, and deal with the data warehousing and reporting for PM / CM metrics for the network elements. I can be collecting and storing anything from processor occupancy on a UNIX or Linux servers to count of messages on SS7 link sets.
Over the years the typical approach for aggregating and reporting this data has been to take the statistical source data (anything in the 5, 10, 15, 20, or 30 minute range) and aggregate it to the hour. Then for reporting purposes a “busy hour” is determined: typically at the leaf level of the data (Element – Card – CPU #) using some metric representing a volume of usage. That BH is then used for PM reporting (the number of errors encountered during peak usage) as well as CM reporting (capacity used versus max defined capacity). There is more activity to this style of reporting, and I can go into it if needed, but this covers the basics.
Recently I’ve been encountering a push from another area in the company that wants to do the aggregation in a completely different way. Their approach is to take that data (at a 15 minute level) and perform a TOP 5 aggregation for the week, using the measurement itself for the ranking function. For example, if I have a table of data that represents CPU usage, Memory usage, disk usage, etc, the metrics (pseudo-code) would be;
CpuUsage = TOP 5 AVG(CpuUsage) RANK by CpuUsage DESC WHERE DATETIME = WeekNbr,
MemUsage = TOP 5 AVG(MemUsage) RANK by MemUsage DESC WHERE DATETIME = WeekNbrAnd so on. I’ve worked in the industry doing this type of reporting since the last 1990’s and it’s always been defined using the BH approach. I decided to research where this came from, which “standards” defined it, and I’ve had no real luck. A lot of documentation talks about using hourly data, and BH, but they don’t define why they do it. The only documentation I can find that comes close was from a CISCO book on how they are doing PM and CM, and in it they reference RFC 1857. That is the extent of it, at least from my efforts so far; I have been searching through stuff from the OpenGroup, TMForum, OMG, and 3GPP.
So… does anyone have any insight into this side of PM / CM; the data warehouse, aggregations, storage and reporting of the data? What do you to address it, and why that approach?
Thanks…
--
You received this message because you are subscribed to the Google Groups "Guerrilla Capacity Planning" group.
To unsubscribe from this group and stop receiving emails from it, send an email to guerrilla-capacity-...@googlegroups.com.
To post to this group, send email to guerrilla-cap...@googlegroups.com.
Visit this group at http://groups.google.com/group/guerrilla-capacity-planning?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
--
Regards,
Scott Johnson
Greg Hunt
DrQ
Scott Johnson
--
You received this message because you are subscribed to the Google Groups "Guerrilla Capacity Planning" group.
To unsubscribe from this group and stop receiving emails from it, send an email to guerrilla-capacity-...@googlegroups.com.
To post to this group, send email to guerrilla-cap...@googlegroups.com.
Visit this group at http://groups.google.com/group/guerrilla-capacity-planning?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
DrQ
To unsubscribe from this group and stop receiving emails from it, send an email to guerrilla-capacity-planning+unsub...@googlegroups.com.
To post to this group, send email to guerrilla-cap...@googlegroups.com.
Visit this group at http://groups.google.com/group/guerrilla-capacity-planning?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
Scott Johnson
To unsubscribe from this group and stop receiving emails from it, send an email to guerrilla-capacity-...@googlegroups.com.
To post to this group, send email to guerrilla-cap...@googlegroups.com.
Visit this group at http://groups.google.com/group/guerrilla-capacity-planning?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
steve jenkin
Why have Telecomms Engineers been so hung up on "Busy Hour"?
Why is the importance of it not easy to find?
I remember Telecomms Engineers using their "Engineers Handbook". I never looked at the exact title, after using (Perry's) Chemical Engineering Handbook, I'd *assumed* there was one of these with a similar title for every Engineering Discipline.
There is a book, The Communications Handbook, that might be the bible my engineer friends used...
[vs Fink, Donald G; Beaty, H. Wayne, "Standard handbook for electrical engineers"]
<http://books.google.com.au/books?id=Tokk5bZxB0MC&printsec=frontcover&hl=en#v=onepage&q&f=false>
<http://www.amazon.com/Communications-Handbook-Electrical-Engineering/dp/0849309670>
It does mention "busy hour" and "erlangs", the Amazon search is far better than the Google book.
The BSTJ piece you pointed to does give some clues:
�- The Engineering Goal is to always meet a "Quality of Service", defined as the probability of a call request resulting in a call establishment. The reverse of "busy". Queuing Theory obviously applies, we know the Erlang was developed for this. The BSTJ article is about extending this theory to complex networks with dynamic routing. Just solving the equations for a minima on circuits per link is hard enough.
�- The *cost* of the various paths has to also be minimised.
[Their solution arises partly from Time Zones shifting periods of peak demand across a network.]
[The fact they're excited about a 7% saving tells you something about the margins they are dealing with.]
The incremental cost of add a circuit during installation is small, but after installation is very high.
The financials of Telecommunications is dominated by Fixed Costs. The variable cost of an additional subscriber or call is very close to zero: the cost of the energy consumed.
Engineers must maximise the profitability of the network, they can't affect Revenues, but *can* reduce costs, which means minimising the cost of time of the fixed costs, including accounting for interest (time cost of capital) and depreciation.
�- If a population is forecast to grow at 20%/annum, is it economically better to install One Big Cable that will handle the traffic forecast for 25years time, meaning you've got a massive debt to service but never have to lay more cable, or do you run a series of upgrade projects for "just in time" provisioning?
�- the Optimal solution of this 3-dimensional problem, {Time, Capacity, Cost} is far from easy because costs and delays are not uniform, nor is demand predictable.
The discipline of Engineering is far wider than "just make it work", there are at least two other *equally* important dimensions:
�- Safety/Security, and
�- Cost
My *guess* is that Erlang "just knew" like the BSTJ authors that the Rate-Determining-Factor of the network, the primary constraint, was when the *demand*, not load, was highest: busy-hour, for it was typically a specific hour or two (and they had very crude measuring equipment).
The simplifying assumption there is: Demand follows the same pattern a) for every day and b) for every area.
Load is what the system handles, Demand is the total requests on the system, a potentially much higher figure.
The uniform demand assumption is false, but like Physics "assume an infinitely small/large/long ...", useful in arriving at 1st order solutions:
�- public holidays & events (Mothers' Day, Christmas, Easter, Thanksgiving) have *much* higher peak demand and completely different demand profiles
�- different areas, say the CBD with only businesses, vs suburbs with no businesses, have different demand profiles
�- different suburbs with different age & income demographics have different demand profiles & peak demand times
�- All this goes to pot when there's some random large-scale event: a radio or TV show makes people pick-up the phone or a natural disaster or ...
Good Network Engineering is about catering for predictable events and mitigating the effects of foreseeable problems, *within* a cost-envelope.
I've only seen one IT Performance paper that links the internal dimension, "compute capacity" with the other Engineering constraint, an external dimension: Cost.
"Busy Hour" is "self-evidently important", which is a red-flag to any good scientist or engineer.
In process-flows, we first had to identify the Primary Constraint (or Rate Limiting Factor), then had to hang everything else off that. If that constraint was changed, then the whole design had to be modified.
Was "Busy Hour" an intuitive identification of Constraint-driven modelling?
Could it benefit from a more rigorous and complete analysis: you bet!
Can I do that? NO... Don't have the rigour or Maths competency.
All the best
steve
---------------
it's worth reading the Table of Contents of Perry's to see what Engineers worry about and why they need to be licensed to practise (people die if they get it wrong):
<http://www.knovel.com/web/portal/basic_search/display?_EXT_KNOVEL_DISPLAY_bookid=48>
-- Steve Jenkin, Info Tech, Systems and Design Specialist. 0412 786 915 (+61 412 786 915) PO Box 48, Kippax ACT 2615, AUSTRALIA stev...@gmail.com http://members.tip.net.au/~sjenkin
DrQ
Greg Hunt
--
James Newsom
For example, during the middle of the year the customers would hit anywhere from 1,000 to 1,500 page clicks per order but on Black Friday that ratio dropped down to about 150 page clicks per order. The pages that those customers hit also become a much smaller set and we prepared for Black Friday/Cyber Monday by making sure that the most common hit pages were already cached in Akamai ready to go.
This change in customer habits really killed us in 2010. Back in 2010 one of the checkout related pages had a change that made a connection to the back end to a persistence layer and a bug was introduced where the connection per click increased ever so slightly and when Black Friday came the concentration of hits to this page killed the back end performance and the site ground to a halt.
During the post-mortem I was able to identify the ever so slightly increase in back end connections that had we noticed earlier would have saved the day. DrQz might remember this data being presented and analyzed on the fly by Jim Holtman at one of his GDAT classes in 2011. One of DrQz's former students made a presentation on Friday on the subject of machine learning and in particular Support Vector Machines. If we had been running SVM analysis of this particular metric it might have stood out and allowed us to fix the problem before it killed the site in 2010.
James
Doesn't it come back to purpose? �I can understand the use of the idea of a busy hour in the context of a telephone network, optimising link capacities and assuming that the spatial pattern of demand does not move around too much (the paper about multiple busy hours seems to be addressing the logical next step, multiple spatial patterns of demand with different time peaks, but in the context of a general purpose computer system I just can't see it as clearly. �The mix of demand types (IO, CPU, memory bandwidth, network) changes over time in a heterogenous system. �If it was a large online retail system it would be a lot simpler, but across a mixed workload environment the mix will change in interesting ways. �If they are reporting free capacity and error rates then picking one time period or one measure seems less useful. �In fact for error rates I'd be starting with peak error periods. �
On Sun, Apr 28, 2013 at 5:45 AM, DrQ <red...@yahoo.com> wrote:
I'm not familiar with either the Busy Hour or the Top 5 methods of sampling and aggregation but,�after googling around for a bit, here's what I've concluded (so far):
1. There is no such thing as the Busy Hour.2. The Top 5 per week is the same as the busy hour.
OK. Perhaps I should explain.�
1. There is no such thing as the BH
Wikipedia�states: "In telecommunications... busiest hour of the day (peak hour) ..."
This just says is that there's little virtue in doing perf and cap analysis for traffic�volumes that do not include maximal or peak data. Moreover, since the traffic�intensity�fluctuates randomly, a single peak measurement is meaningless. Therefore, you need to sample over a range or window. How big should the�window�be? How long is a piece of string?�It also depends on the data collection rate. Fastest rate for Unix-type perf metrics is 1 min,�but most data collectors default to something coarser, e.g, 10 min, 15 min, 30 min.�So, in order to capture more than a single sample, the daily time sub-unit of an hour is a�reasonable convention and the sampling rate can be adjusted to fit that convention.
But which hour is the peak hour each day?�
The "Wireless Network Performance Handbook"�declares (p. 284) "The system busy hour, barring fraud, is usually between 4 to 6 P.M. in the�United States."�But it also says later�(p. 286): "We recommend using a quarterly average using the system busy�hour, most likely 4 to 5 P.M. for weekday traffic."
But it could (and probably does) vary with season.
According to CCITT standard E.500 there are multiple possible definitions for the BH or peak hour:ADPH = average daily peak hourTCBH = time-consistent busy hourFDMH = fixed daily measurement hour
and also
ADPQH = average of daily peak quarterly defined hourADPFH = average of daily peak full hour
FIGURE 2/E.500 shows a decision diagram for selecting the appropriate measurement method.
Looking at the schematic plots on this slide,�it looks to me like the "classic" busy hour corresponds to FDMH, which may or may not�be sampling a maximum value of the�metric.�On the other hand, the ADPH busy hour is determined by when the�maximum value��actually�occurs (post facto, presumably).
2. The Top 5 is the same as the BH
Why 5 and not 10 or ...? One explanaton could be that there are 5 business days in the week?
Another could be: The "Wireless Network Performance Handbook" (p. 273)�"Some reports that facilitate focusing on the performance of the system are...�2. Current top-5 worst-performing cells and top-10 worst-performing sectors�in the network and each region using the statistics metrics."
It strikes me that The Top 5 sampling is close to the ADPH busy hour�in that it is based on where�the metric maxima (5) occur rather than fixing�the time window (BH) when�the samples are taken.
SummaryBH is a documented standard in the telecoms industry, so you can play that�formal standard against the undocumented Top 5 scheme. At least, I didn't see any explanation or documentation for Top 5.
Given the arbitrariness of all these definitions (standards or not), the differences may be relatively�insignificant over the long haul. Perhaps the most important thing is that�scheduled measurements are being taken at all.
I don't know if any of these conclusions give the answer you're looking for, but maybe they can provide a new starting point for further questions.
On Thursday, April 25, 2013 3:33:37 PM UTC-7, DEK46656 wrote:
I have a question that is not typical for this discussion group, but I�m running into this issue more and more.� First, a little background.� I work for a communications company, and deal with the data warehousing and reporting for PM / CM metrics for the network elements.� I can be collecting and storing anything from processor occupancy on a UNIX or Linux servers to count of messages�on SS7 link sets.
Over the years the typical approach for aggregating and reporting this data has been to take the statistical source data (anything in the 5, 10, 15, 20, or 30 minute range) and aggregate it to the hour.� Then for reporting purposes a �busy hour� is determined: typically at the leaf level of the data (Element � Card � CPU #) using some metric representing a volume of usage.� That BH is then used for PM reporting (the number of errors encountered during peak usage) as well as CM reporting (capacity used versus max defined capacity).� There is more activity to this style of reporting, and I can go into it if needed, but this covers the basics.
Recently I�ve been encountering a push from another area in the company that wants to do the aggregation in a completely different way.� Their approach is to take that data (at a 15 minute level) and perform a TOP 5 aggregation for the week, using the measurement itself for the ranking function.� For example, if I have a table of data that represents CPU usage, Memory usage, disk usage, etc, the metrics (pseudo-code) would be;
CpuUsage = TOP 5 AVG(CpuUsage) RANK by CpuUsage DESC WHERE DATETIME = WeekNbr,
MemUsage = TOP 5 AVG(MemUsage) RANK by MemUsage DESC WHERE DATETIME = WeekNbr
And so on.� I�ve worked in the industry doing this type of reporting since the last 1990�s and it�s always been defined using the BH approach.� I decided to research where this came from, which �standards� defined it, and I�ve had no real luck.� A lot of documentation talks about using hourly data, and BH, but they don�t define why they do it.� The only documentation I can find that comes close was from a CISCO book on how they are doing PM and CM, and in it they reference RFC 1857.� That is the extent of it, at least from my efforts so far; I have been searching through stuff from the OpenGroup, TMForum, OMG, and 3GPP.
So� does anyone have any insight into this side of PM / CM; the data warehouse, aggregations, storage and reporting of the data?� What do you to address it, and why that approach?
Thanks�
--
You received this message because you are subscribed to the Google Groups "Guerrilla Capacity Planning" group.
To unsubscribe from this group and stop receiving emails from it, send an email to guerrilla-capacity-...@googlegroups.com.
To post to this group, send email to guerrilla-cap...@googlegroups.com.
Visit this group at http://groups.google.com/group/guerrilla-capacity-planning?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
�
�
--
You received this message because you are subscribed to the Google Groups "Guerrilla Capacity Planning" group.
To unsubscribe from this group and stop receiving emails from it, send an email to guerrilla-capacity-...@googlegroups.com.
To post to this group, send email to guerrilla-cap...@googlegroups.com.
Visit this group at http://groups.google.com/group/guerrilla-capacity-planning?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
�
�