F# Scribbles

My experiments with F#

Analyze Azure Cloud Service Performance Counters using F# and Deedle library

Analysing Windows performance counters is an interesting job for that we use many tools importantly Microsoft Excel. To share the trends and improvements with graphical charts, Ops people rely on many tools. As a developers, we shouldn’t :).

Here, I use Deedle – exploratory data library, this is equivalent to Python’s Pandas library. Assume that one set of results are logged in a CSV file.

Usually Microsoft Azure cloud service applications logs to table storage that can be exported to CSV files.

Import Deedle library

Deedle can be downloaded as Nuget package.

1
2
3
4
#I "DEEDLE-LIB-PATH\Deedle.1.0.0";;
#load "Deedle.fsx";;
open System;;
open Deedle;;

Import data and Analyze

Frame is a tabular based data structure. Let us import the CSV.

1
2

let perfCounters = Frame.ReadCsv(@"..\WADPerformanceCountersTable.csv")

will return

1
2
3
4
5
6
7
8
9
10
Binding session to '..\Deedle.1.0.0\../FSharp.Data.2.0.8/lib/net40/FSharp.Data.dll'...

val perfCounters : Frame<int,string> =
  
       PartitionKey RowKey                                                                                                                                                   Timestamp        EventTickCount:int64 DeploymentId                     Role                       RoleInstance                    CounterName                                       CounterValue 
0   -> PKVALUE    ROWKEYVALUE 31-01-2014 06:39 6.35267E+17          84011bf63b79412a86b0087cab91bdfe ROLE_NAME ROLE_INSTANCE_NAME \Web Service(_Total)\Bytes Total/Sec              55.204232    
1   -> PKVALUE    ROWKEYVALUE 31-01-2014 06:39 6.35267E+17          84011bf63b79412a86b0087cab91bdfe ROLE_NAME ROLE_INSTANCE_NAME \ASP.NET Applications(__Total__)\Requests/Sec     0.011111     
2   -> PKVALUE    ROWKEYVALUE 31-01-2014 06:39 6.35267E+17          84011bf63b79412a86b0087cab91bdfe ROLE_NAME ROLE_INSTANCE_NAME \ASP.NET Applications(__Total__)\Errors Total/Sec 0            
3   -> PKVALUE    ROWKEYVALUE 31-01-2014 06:39 6.35267E+17          84011bf63b79412a86b0087cab91bdfe ROLE_NAME ROLE_INSTANCE_NAME \Memory\Available MBytes                          384
...

Let us group the result by counter name

1
2

let groupedPerf = perfCounters.GroupRowsBy<string>("CounterName");;

will results

1
2
3
4
5
6
val groupedPerf : Frame<(string * int),string> =
  
                                            PartitionKey RowKey                                                                                                                                                   Timestamp        EventTickCount:int64 DeploymentId                     Role                       RoleInstance                    CounterName                          CounterValue 
\Web Service(_Total)\Bytes Total/Sec 0   -> PKVALUE ROWKEYVALUE 31-01-2014 06:39 6.35267E+17          84011bf63b79412a86b0087cab91bdfe ROLE_NAME ROLE_INSTANCE_NAME \Web Service(_Total)\Bytes Total/Sec 55.204232    
\Web Service(_Total)\Bytes Total/Sec 7   -> PKVALUE ROWKEYVALUE 31-01-2014 06:39 6.35267E+17          84011bf63b79412a86b0087cab91bdfe ROLE_NAME ROLE_INSTANCE_NAME  \Web Service(_Total)\Bytes Total/Sec 799.596468   
\Web Service(_Total)\Bytes Total/Sec 14  -> PKVALUE ROWKEYVALUE 31-01-2014 06:39 6.35267E+17          84011bf63b79412a86b0087cab91bdfe ROLE_NAME ROLE_INSTANCE_NAME  \Web Service(_Total)\Bytes Total/Sec 33.555829

You can do whatever the analysis you want. For example, I want to calculate average of “Available MBytes”.

1
groupedPerf.Rows.[Lookup1Of2 @"\Memory\Available MBytes"] //Lookup1Of2 is F# 3.0 compatiility

this results

1
2
3
4
5
6
val it : Frame<(string * int),string> =

                                PartitionKey RowKey                                                                                                                                                   Timestamp        EventTickCount:int64 DeploymentId                     Role                       RoleInstance                    CounterName              CounterValue
\Memory\Available MBytes 3   -> PKVALUE ROWKEYVALUE 31-01-2014 06:39 6.35267E+17          84011bf63b79412a86b0087cab91bdfe ROLE_NAME ROLE_INSTANCE_NAME \Memory\Available MBytes 384
\Memory\Available MBytes 10  -> PKVALUE ROWKEYVALUE 31-01-2014 06:39 6.35267E+17          84011bf63b79412a86b0087cab91bdfe ROLE_NAME ROLE_INSTANCE_NAME \Memory\Available MBytes 390
\Memory\Available MBytes 17  -> PKVALUE ROWKEYVALUE 31-01-2014 06:39 6.35267E+17          84011bf63b79412a86b0087cab91bdfe ROLE_NAME ROLE_INSTANCE_NAME \Memory\Available MBytes 390

The Stats module in Deedle provides statistical functions operate on series. For example,

1
groupedPerf.Rows.[Lookup1Of2 @"\Memory\Available MBytes"] |> Stats.mean

results take average of Available MBytes.

1
2
3
4
5
6
7
8
9
10
11
val it : Series<string,float> =
  
PartitionKey         -> 6.35267776223776E+17 
RowKey               -> <missing>            
Timestamp            -> <missing>            
EventTickCount:int64 -> 6.35267776223776E+17 
DeploymentId         -> <missing>            
Role                 -> <missing>            
RoleInstance         -> <missing>            
CounterName          -> <missing>            
CounterValue         -> 674.888111888112

Note that instead of applying Stats.mean on all columns, you can apply only on CounterValue.

For example,

1
2
groupedPerf.Rows.[Lookup1Of2 @"\Memory\Available MBytes"].Columns.[["CounterName"; "CounterValue"]] 
|> Stats.mean;;

results

1
2
3
4
val it : Series<string,float> =
  
CounterName  -> <missing>     
CounterValue -> 318.286696141