Anton Lebedevich's Blog

My Top 10% Solution for Kaggle Rossman Store Sales Forecasting Competition

Sat, 16 Jan 2016 00:00:00 +0000

This is the first time I have participated in a machine learning competition and my result turned out to be quite good: 66th out of 3303. I used R and an average of two models: glmnet and xgboost with a lot of feature engineering.

The goal of the competition was to predict 6 weeks of daily Sales in 1115 stores located in different parts of Germany based on 2.5 years of historical daily sales.

The first thing I tried after importing data was to convert it into multivariate regular time series and run SVD. The result highlighted several interesting details:

the majority of stores didn’t have upward or downward trends
seasonal variation was present but mostly as a Christmas effect
Sunday was a non-working day in a majority of stores
there was a strange 2 week cycle which was an effect of running Promo actions every other week
there were group of stores that didn’t close on Sunday in summer
some stores had strong yearly pattern
some stores showed continuous sales increases and other decreases over time
several stores were missing data from the second half of 2014

I sampled several stores from different groups to check various ideas on them first.

In the beginning my idea was to check how good a single interpretable model could be. There were two simple benchmark models (median and geometric mean) on the competition forum which I used as a starting point.

To validate model quality I implemented time-based cross-validation as described in Forecasting: principles and practice

Interactive visualization helped a lot in identifying features and sources of errors.

Initially I tried forecast::tbats (a separate model for each store) but the results were quite bad. The influence of non-seasonal factors was big but tbats can’t use external regressors. Next I considered using ARIMA, as it can use regressors, but for long-term forecasts it decays to constant or linear trends. So I continued to evaluate different kinds of linear models. As more and more features were added, the simple linear model started to get worse so I switched to glmnet which is able to select subsets of features.

There was some similarity between Sales and count data so I tried Poisson regression as suggested in Generalized Linear Models in R. This, however, resulted in a larger error in cross-validation than predicting log(Sales) using Gaussian family of generalized linear model.

RMSPE evaluation criteria is asymmetric (see discussion of MAPE) and sensitive to outliers. The typical range for different models and different stores was between 0.08 and 0.25. If a model predicted a sales value of 1000 on a specific day (for example) and the actual sales were 10 because there was an unaccounted holiday, then RMSPE would be equal to 99 for that day which would make an otherwise good model look really bad on average.

The best per store glmnet model scored worse than xgboost, also published on the forum. Tree based regression models don’t extrapolate well because they predict with constant value anything outside their training ranges. The number of stores with long-range trends was small and the majority had quite stable sales over time, so I decided to give xgboost a try and feed it with the same features as I did for linear model (without one-hot encoding for categorical features).

Feature engineering:

5 fourier terms generated by forecast::fourier with frequency=365;
Days and log(Days) since the beginning of the training set to capture trends (exponential and linear because log(Sales) is predicted);
exponential and linear growth before events or decay after events such as starting Promo or state holidays (similar to foregen which I discovered later);
binary features which took value 1 for several days before or after events including the start and end of Promo, Promo2, StateHoliday, and refurbishments;
binary features like ClosesTomorrow, WasClosedYesterday, WasClosedOnSunday;
day of week, day of month, month number, year as categorical features for xbgoost and n-1 binary features for glmnet (described at https://www.otexts.org/fpp/5/2 ).

For some stores with large error in cross-validation I dropped data before manually selected (by examining Sales time series graphs) changepoints.

The training set contained more stores than were present in the test set. I dropped those extra stores from the training set for xgboost.

I dropped outliers from the training set for glmnet. Outliers were selected by > 2.5 * median absolute residual from lm trained on a small set of features per store.

Initially I used 10 cross-validation folds with 6 weeks length starting from the end of the training set with 2 weeks step (~4.5 months total) but then found that closest to 2014 folds produce large errors for stores with missing in 2014 data. Then I switched to 15 folds with 3 days step to avoid being too close to 2014 which improved predictions for those stores.

RMSPE was quite different for different prediction ranges. For the same store it could go from 0.103 to 0.125 with the same model. It made me think that public leaderboard position is going to change a lot in private leaderboard because they have time based split. It turned out to be true.

Grid search was used to find glmnet alpha parameter. The best alpha was 1 which corresponds to Lasso regularization. Choice of lambda is implemented in cv.glmnet but it uses a standard k-fold cross-validation. I reimplemented it with a time-based cross-validation.

0.985 correction was insignificant on cross-validation (effect was less than standard deviation of RMSPE from different folds) but helped on both private and public leaderboards.

Pairwise feature combinations had positive effect for glmnet on cross-validation but didn’t work on leaderboard.

As a result single per store glmnet model gave prediction error (RMSPE) on private leaderboard 0.11974 (516th place), single all stores xgboost model - 0.11839 (379th), their average - 0.11262 (66th). Complicated ensemble models are good for competitions but in practice it might be better to have 0.007 increase in error and simple interpretable model.

Source code is available at github.com/mabrek/kaggle-rossman-store-sales

Machine Learning is not BS in Monitoring

Sat, 09 Jan 2016 00:00:00 +0000

Recently I came across provocatively titled “Machine Learning in Monitoring is BS” and decided to reply but the response came out longer than typical comment so I posted it separately.

Ambiguity of the data - true point. It’s impossible to build a single universal model that eats any data and alert when it’s wrong. But it’s possible to build different models for different cases with the help of humans. And yes, it’s possible for machine learning to find if a bump on EEG is good or bad, see Epilepsy Seizure Prediction Challenge
Multiplicity of variables. It’s true that mass of metrics doesn’t make a lot of sense without context but adding context back and applying dimensionality reduction methods (SVD) allows to reduce the mass into handful. If ECG of one cow is quite different from ECGs of all other cows then it’s quite likely to be sick.
User experience. Not all machine learning methods are opaque. Deep learning and gradient boosting models are black boxes but there are many interpretable models like regularized linear and logistic regression. They can give answers like “sales tomorrow will be higher than the previous Monday because it’s a first working day after long holidays and sales are usually higher in that case”.
Visualization. Traditional monitoring that is built around dashboards and staring at graphs doesn’t scale. At the same time there are different ways to present loads of data in compact visual form that makes sense to human (like MDS, T-SNE). Machine learning could be used to produce better visualization to help humans understand what’s going on.

There is a lot of hype around machine learning and as it usually happens some of overhyped technologies fail to deliver. The same problem was faced by information security community as explained in “Secure because Math” (pdf, video) talk by @alexcpsec

Machine learning is not a magic that would digest uncleaned and unlabeled metrics and point out when and how the system was broken. It’s rather a set of tools which when used appropriately will reduce manual labor for devops looking after production systems.

Prototyping Long Term Time Series Storage with Kafka and Parquet

Sun, 25 Oct 2015 00:00:00 +0000

Another attempt to find better storage for time series data, this time it looks quite promising

Last time I tried to switch from Graphite time series storage to Cyanite/Cassandra but the attempt failed and I stayed with Whisper files. After struggling with keeping disk IOPS sane while ingesting hi-resolution performance data I ended up putting Whisper files into tmpfs and shortening data retention interval to just one day because my load tests usually don’t last more than several hours. Then I export data into R and do analysis. Large scale projects like Gorilla(pdf) and Atlas do similar things. They store recent data in RAM for dashboards and real-time analytics and then dump it to slow long term storage for offline analysis.

Whisper file format is relatively good in terms of storage size (12 bytes per datapoint). It’s columnar because it saves each metric in its own file. It contains redundant data because it saves a timestamp with each value and many values from different metrics share the same timestamp. There is no compression. I need to find something better than Whisper.

Gorilla(pdf) paper inspired me to look into column storage formats with efficient encoding of repeated data. I decided to try Parquet. Unfortunately floating point compression is not there yet but my values have type double (as a side note ORCFile lacks it too). Many metrics (counters) actually have integer type but there is no information about their types upfront.

To achieve good compression data needs to be written in large chunks so I needed something to buffer data. The way Kafka works with streaming writes and read offsets made me think that it’s a good fit for storing data until it’s picked up by periodical job. That job would start, read all the data available from the last read offset, compress and store it, and sleep until the next cycle.

My data comes in Graphite plaintext protocol which is quite verbose:

<metric_name> <value> <timestamp>

Metric names and timestamps are repeating. Metrics are sent periodically at the same time so many lines share the same timestamp. The same set of metrics is being sent each time. Values for many metrics are not changing a lot over time (like disk usage) which makes it a good target for dictionary or delta encoding.

Feeding graphite data into Kafka

I set up single node Kafka as described in manual

Feeding graphite data into Kafka turned out to be one-liner with nc and kafkacat:

nc -4l localhost 2003 | kafkacat -P -b localhost -t metrics -K ' '

Metric name is used as message key in kafka and value timestamp is a payload.

Then I started collectd with write graphite plugin pointing to localhost and reporting interval of 1 second. After several hours I got 1.8 Gb queue in kafka.

Dumping the data back into text format is a one-liner too:

kafka-console-consumer.sh --zookeeper localhost:2181 --topic metrics \
  --from-beginning --property print.key=true

Text file had size of 1.4Gb which means kafka has some overhead for storing uncompressed data. There were ~19000000 lines in the file.

Fetching data from Kafka

I needed to handle read offsets manually so I chose SimpleConsumer. Its API turned out to be quite confusing and not that simple. It doesn’t talk to Zookeeper and allows to specify offsets. Handling all corner cases requires lots of code but simple prototype turned out to be quite short in Scala:

val consumer = new SimpleConsumer("localhost", 9092, 5000,
    BlockingChannel.UseDefaultBufferSize, name)
val fetchRequest = new FetchRequestBuilder().clientId(name)
    .addFetch(topic, partition, offset, fetchSize).build()
val fetchResponse = consumer.fetch(fetchRequest)
val messages = fetchResponse.messageSet(topic, partition)

It just reads all messages from specified offset up to fetchSize.

Saving data into Parquet

Parquet API documentation doesn’t seem to be published anywhere. Javadoc for org.apache.parquet.schema.Types contains several schema examples. Writing local files from standalone application is not described anywhere but parquet-benchmarks module contains class org.apache.parquet.benchmarks.DataGenerator which writes several variants of local files. It depends on Hadoop classes so you’ll need it as a project dependency (issue)

I decided to use ‘wide’ schema when each metric has its own column to make delta encoding possible and a single column for timestamps (like xts does for multivariate series):

val types = mutable.Set[Type]()
...
// for each message collect unique keys as types
types += Types.optional(DOUBLE).named(key)
...
val schema = new MessageType("GraphiteLine",
    (types + Types.required(INT64).named("timestamp")).toList) 

Boilerplate to create ParquetWriter object:

val configuration = new Configuration
GroupWriteSupport.setSchema(schema, configuration)
val gf = new SimpleGroupFactory(schema)
val outFile = new Path("data-file.parquet")
val writer = new ParquetWriter[Group](outFile, 
    new GroupWriteSupport, UNCOMPRESSED, DEFAULT_BLOCK_SIZE, 
    DEFAULT_PAGE_SIZE, 512, true, false, PARQUET_2_0, 
    configuration)

For each unique timestamp a row (called group in Parquet) is added which contains values for all metrics (columns) at that time:

for (timestamp <- timestamps) {
    val group = gf.newGroup().append("timestamp", timestamp)
        for ((metric, column) <- columns) {
            column.get(timestamp).foreach(group.append(metric, _))
        }
    writer.write(group)
}
writer.close()

And that’s it.

Effect of compression

Enabling gzip compression in Parquet reduced file size 3 times compared to uncompressed. The result took 12Mb for ~19000000 input lines which is quite impressive. Storing the same data in whisper format would take at least 230Mb (actually more because it reserves space for whole retention interval).

I tried enabling Snappy compression for Kafka publisher:

kafkacat -P -b localhost -t metricz -K ' ' -z snappy < metrics.txt

and got ~ 500Mb queue size for 1.4Gb original data. The only problem is that log compaction is not yet compatible with compressed topics.

The result looks quite good: temporal buffer in Kafka needs 1/3 size of original data and long term storage takes ~ 0.6 bytes per datapoint (while whisper takes 12 bytes).

Open questions

Can Parquet handle very wide schema with 100k columns and more?

How to guess effective metric type (double or integer) from incoming graphite data?

How to get data from Parquet into R without Spark? (issue)

Source code

Available on github kafka-timeseries with build instructions. Actually it’s just a single Scala file with 100 lines of code 20 of which are import statements.

Visual Exploration of Performance Monitoring Time Series with MDS and t-SNE

Tue, 25 Aug 2015 00:00:00 +0000

This time I’m going to introduce tools for visual data exploration. They represent each time series as a point in 2-dimensional space. When original time series are similar the corresponding points will be close to each other.

The same data was used here as in previous post about SVD and PCA

Multidimensional Scaling (MDS) produces this kind of image:

It has put hill shaped series (1 - 5) together in the left corner which seems to be the most dense one. Different kinds of spikes landed on the upper corner (7 - 10). Series with strong increasing or decreasing trend landed on the right corner (13, 14) and step-like changes on the bottom (15, 16).

This image was generated using classical (metric) multidimensional scaling using cmdscale function from R and distance matrix calculated as 1 - abs(cor(a, b)) for each pair of time series. Metric here corresponds to the set of properties of distance function required by the algorithm.

Not all time series (dis)similarity measures have these properties. There are non-metric variants of multidimensional scaling like isoMDS. It is much slower than metric MDS and in my case it produced results visually indistinguishable from cmdscale. Actually isoMDS starts by running cmdscale first and then iteratively refines its result.

t-distributed stochastic neighbor embedding (t-SNE) arranges the same data (using the same distance matrix) in a different way:

There are several clusters of step-like shapes (1 - 3, 13) and spikes (4, 11, 12, 16) around edges. Hill-like shapes are in the middle (5 - 10). Valley-like shape (6) is in the middle because the absolute value of correlation was used as the similarity function. It’s a flipped hill and abs(cor(...)) is the same for flipped series. Bottom-right side contains close to linear trend series (14, 15).

t-SNE seems to find more groups of smaller size than MDS. Large table-hill shaped group (which is in the middle) is not that clear as in MDS results. It doesn’t collapse points with zero distance into one but tries to spread them around which produces small and dense groups (1, 2). t-SNE is much slower than MDS.

When used interactively these tools are better for exploratory analysis of time series data than clustering that I used before.

I’ve made a quick demo for both methods.

Exploring Performance Monitoring Data with Multivariate Tools: SVD and PCA

Thu, 20 Aug 2015 00:00:00 +0000

Most methods that were presented here so far are dealing with a single time series (performance metric) at a time. Now I’d like to make a quick overview of methods which allow to glance over a whole collection of time series at once.

Data used here was gathered during a load test of an application which consists of several components: http server, messaging server (RabbitMQ), database (Cassandra). That application uses 5 hosts and number of system metrics + application metrics is about 3300 after filtering (~32000 before). Think of it as a number of graphs to get through while exploring results of the test.

The load applied to the http server looks like this (number of identical clients sending requests):

The idea behind the table hill shape of the load is that the upwards slope shows when the system breaks (how it scales), flat top shows how stable it is (if it didn’t break on upwards slope), and the downwards slope shows how it recovers.

The service didn’t do very well this time. Here is a plot of successful and error response rates:

And response latencies:

Error rate is not zero and 99th percentile of response latency has spikes close to allowed by SLA maximum. At least it recovered and continued to serve requests at a lower rate.

Here’s what a result of SVD (Singular Value Decomposition) looks like (left-singular vectors sorted by decreasing singular values):

In time series context SVD decomposes original set of series into set of uncorrelated base series (left-singular vectors), set of singular values, and a matrix of weights (right-singular vectors). These matrices could be used to reconstruct the original set of series. The nice feature is that you can take only several base series corresponding to the top singular values to get quite good (in terms of squared error) reconstruction result.

First 6 singular values (sorted by decreasing value) contribute most and the rest is a background noise.

When the data is centered (mean subtracted) and scaled (divided by standard deviation) before applying SVD then the top (by singular values) base series represent the most common shapes in the data with some caveats. Sometimes it can change sign (flip shape vertically) or mix several common shapes into one. Outliers distort extracted base series due to the scaling used and the least-squares nature of the decomposition (which amplifies outliers).

In this case the first extracted series is a slightly skewed table hill shape of the load applied because most metrics follow that pattern. A lot of metrics comes from Cassandra which uses exponentially decaying sampling for latencies. This algorithm smoothes and moves the shape to the right (delays signal). The second extracted series corresponds to growing counters and caches. The third series looks like a flipped shape of RabbitMQ disk activity during the test. There are some spikes and drops visible on several base series which corresponds to errors and latency spikes.

Closely related PCA (Principal Component Analysis) produces set of principal components (which are base series from SVD scaled by singular values) and the same weights (loadings) from SVD. Here the first 2 original series selected by maximum absolute loading per each principal component.

It selects original series which have largest contribution from top components (base series).

These methods are quite fast and produce meaningful results: they extract the most common shapes and group original series by these shapes.

They are sensitive to outliers. The usual way of scaling data (by standard deviation) doesn’t make a lot of sence for long tailed distributions which are quite common in performance monitoring data. It might be a good thing for exploratory data analysis because if you see some spikes or step-like changes in several first base series then it definitely means some abrupt changes at that time in system being monitored.

I’ve tried to center data by subtracting median and scale by MAD (Median Absolute Deviation) but discovered that zero MAD is quite common in my data (when it’s mostly constant with a few spikes).

What SVD/PCA are good for: if you have a lots of data, slow anomaly detection algorithm and interested mostly in the time when anomaly happens then running the algorithm on several first principal components might be much faster than feeding it all the original data.

Spark vs. Cassandra (Cyanite) for Metric Timeseries

Mon, 11 May 2015 00:00:00 +0000

An unsuccessful attempt to scale performance metrics processing by switching from Graphite/R to Spark/Cassandra/Cyanite

My typical workflow is to run load tests until the system fails (or discover production failure), extract data from Graphite (Whisper files produced by Carbon), import it into R, and then run a battery of statistical tests and visual explorations to find a root cause of the problem. There is a strong need for hi-resolution data in load tests (because systems fail really fast under load) but the tools I use now are not really good at ingesting and processing data with granularity finer than 10 seconds. It’s been a long time since I first tried to escape ‘Graphite world’ but all attempts failed so far and here I’ll tell about another failed attempt.

‘Graphite world’ (Carbon), Whisper, Graphite-Web) has a lot of issues and is moving slowly. Feeding metrics with 1s resolution to carbon works for one host but doesn’t work at practical scale. This could be done but needs more hardware than monitored hosts.

R has a great number of timeseries related libraries but it runs on a single core and is limited by available memory on a single box. There are packages that allow R to fork several computation processes and move data between them but it’s still limited to a single box.

Cassandra is said to be good at timeseries data. Spark is said to be fast at distributed data processing and there is spark-cassandra connector. Cyanite is a simple app written in Clojure that ingests metrics in the same format as Carbon does and saves them into simple table in Cassandra. There is a Graphite-Cyanite project that allows to run Graphite-Api on top of Cyanite and use all Graphite-compatible dashboards like Grafana.

So I wired them all together and tested with one collectd instance gathering data with 10s interval. It worked OK so I proceeded by adding more hosts and switching to 1s interval. Cyanite ate all cpu and failed. Profiling showed that the default in-memory index of metric names is the bottleneck so I installed Elasticsearch and switched Cyanite path store to es-native. This time both Cyanite and Elasticsearch ate all cpu and failed again. It turns out that for every metric update Cyanite tries to send metric name to Elasticsearch (cyanite/#96) which amounts to thousands requests per second. It’s mostly a time waste because metric names don’t change that often. I ended up disabling metric name indexing and data flowed to Cassandra with a little complaints (cyanite/#73).

Spark version was 1.3.1. Released version of spark-cassandra-connector supported only Spark 1.2 but the master branch was able to work with Spark 1.3.1 (see SPARKC-98 for more details). There are a lot of changes in internal Spark APIs between 1.2 and 1.3 so DataFrames and SQL was not working with connector (SPARKC-112) but RDD operations were quite usable.

First problem encountered was poor locality of spark jobs caused by missing reverse DNS in my virtual machines where spark and cassandra were running. There is a task SPARK-5113 to help with that but in the mean time if your reverse DNS is broken (as it usually is in intranets) then it’s better to force spark to use only ip addresses for everything (set SPARK_LOCAL_IP, SPARK_PUBLIC_DNS, SPARK_LOCAL_HOSTNAME to an ip address in spark-env.sh)

Then there was not enough job parallelism. Number of partitions was equal to number of Cassandra hosts (replication factor was 1). Data in cyanite table is partitioned by metric name and there were about 1500 unique names. Default connector setting spark.cassandra.input.split.size was too large and I lowered it to 5000.

Then I noticed quite high number of context switches and cpu usage of Cassandra process. There were too many queries executed by connector so I increased spark.cassandra.input.page.row.size to 10000. Each datapoint is a single number and at 1s resolution 10000 points equals to about 3 hours of data for a single metric.

These parameters (split size and page row size) are better to set individually per table because number of partitions, rows per partition, cells per row, cell size are different. Setting these parameters manually in Scala seems ugly because you can’t provide one implicit for ReadConf and not provide for other parameters:

val rdd = sc.cassandraTable("metric", "metric")(CassandraConnector(sc.getConf), 
                                                ReadConf(5000, 10000), 
                                                implicitly[ClassTag[CassandraRow]], 
                                                implicitly[RowReaderFactory[CassandraRow]], 
                                                implicitly[ValidRDDType[CassandraRow]])

Some kind of auto-tuning based on optimal data size per job might be better than global one-size-fit-all defaults.

Job speed increased but Cassandra cpu usage remained quite high. perf top profiler showed LZ4 decompression routines so I tried disabling sstable compression. Compression level was about 0.3 and increasing data size 3 times was not a big deal. It didn’t help at all. Cassandra was still hogging cpu. JMC profiler showed that Cassandra was reading sstables (quite expected behavior). That prompted me to check if the storage format was good enough for the data.

It takes about 100 bytes to store 1 timestamp and 1 value of type double without compression and about 30 bytes with compression enabled in the table. It’s a lot compared to whisper format which takes 12 bytes for that. For each data cell cassandra stores column name and write timestamp. Column names are identical for the majority of cells stored and compression takes care of that on disk but memory overhead is still high (see discussion at CASSANDRA-4175). Saving another timestamp for data cell which has a timestamp inside seems redundant. While it’s possible to set cell write time by insert ... using timestamp ... this internal timestamp can’t be used for sorting and filtering data.

After all of that I arrived to the point when reading 51000000 datapoints took about 40 seconds on 6 cores. That data was produced by 4 hosts with collectd reporting with 1 second interval for about 10 hours. Number of unique metric names was 1500. Data size on disk with compression enabled was 1.3Gb total.

With the data ready to be processed in Spark RDD I got stuck on what’s next. Usually I’d run positive-only-diff on known counters, filter out flat metrics, calculate different percentiles on moving windows and run Tukey Method to find spikes. But Java/Scala toolbox for time series is almost empty. There is nothing like xts, ggplot2, TSclust, forecast. The only thing that I found is a recently started spark-timeseries

There are tools available beyond Java/Scala ecosystem at the price of cpu cycles spent on another round of data serialization/deserialization (first one is moving data from Cassandra into Spark). PySpark could give access to Pandas and IPython (with python 3 support in upcoming Spark 1.4). SparkR is going to be released in Spark 1.4 too.

I took a several days break between my experiments and then I found that none of my queries work anymore. They either failed or returned empty results. Cassandra was dying with out of memory errors. One log file had a warning about too many tombstones being scanned. The problem was caused by data expiration because I set 1 week TTL and forgot about that. Cassandra doesn’t really delete data but places special records (tombstones) to mark it as deleted. Large delete operations cause known problems. Consequences could be reduced by decreasing gc_grace_seconds and switching to DateTieredCompactionStrategy.

It turns out that using single Cassandra cell for a single timeseries value is not that efficient both from storage overhead and from cpu cycles lost on doing extra housekeeping. Buffering incoming data and packing several timestamp-value pairs into a single Cassandra cell would be better.

Service Flapping and Load Oscillations

Wed, 22 Apr 2015 00:00:00 +0000

Case 1: Users were experiencing slow page loads with periodical 5xx errors but the backend service looked OK with average cpu utilization about 50%. Usually it happens when some other service required to process the request is too slow but that was not the case. Sampling profiler attached to the backend showed nothing interesting in call stacks and time distribution between calls to other services was typical. The interesting thing was found on thread activity graph produced by the profiler. All the threads were busy doing something for 2 seconds and then idle for another 2 seconds like no requests were coming for 2 seconds out of 4. Load balancer logs showed that it was sending all the load to half of the cluster then declaring it down and switching to another half back and forth. Half of the cluster was unable to process all the load applied which lead to timeouts and being marked as down by load balancer.

Case 2: JMeter reported that http latency is unusually high compared to requests executed manually via curl while the load test was running. If was possible though to catch a long response once in about 10 curl requests. Profiling didn’t show any obvious bottleneck. It looked like the service was hitting cpu but average cpu usage was low. Reading JMeter csv report revealed interesting pattern. Test plan was configured with several hundred threads with ConstantThroughputTimer between requests. All threads were sending requests almost at the same millisecond then slept for some time and then again burst of requests at the same time. The service was OK to handle 100 requests per second but micro-bursts of more than 1000 per second slowed it down. Somehow delays used by ConstantThroughputTimer got synchronized between threads and they started firing at the same time.

Case 3: Maximum and 99th percentile of service response time was too high under flat request rate generated by JMeter (now it was using UniformRandomTimer). Initially latency data was aggregated with 10s resolution and looked like there is some oscillation in median:

Re-aggregating data with 1s resolution confirmed it:

That oscillation was traced down to heavy query running every 30 seconds and slowing down the database.

What’s common in all these cases is that periodic fluctuation in load caused performance degradation but it was only visible in hi-resolution data (second or milliseconds). Visual inspection of hi-res data doesn’t scale so some method is needed to find if there is a periodical structure in a time series.

FFT periodogram (spec.pgram) produces quite unconvincing result on 10s aggregated data:

There is a spike at the expected place (frequency 0.033 which corresponds to the period of 30s) but it’s not alone and hard to distinguish from background noise. 30 seconds is larger that 20 seconds (which is a Nyquist frequency for sampling rate 10s) so it should be OK to use 10s sampling rate to catch it but it’s not. Another problem with FFT periodogram is that it doesn’t like missing data. It’s quite easy to do linear interpolation in case of several missing points though.

0.033 frequency spike is more pronounced on periodogram generated from 1s sampled data but the noise around it is still strong:

There is a method called Lomb–Scargle periodogram which produces more readable results on 10s data:

It handles missing data and irregularl samles but it’s quite slow compared to FFT.

So it’s possible to find periodic structure by observing peaks on periodograms but you still need to eyeball it. Something is needed to tell if time series has a periodical component and how significant is the component compared to background noise.

Multitaper method comes to the rescue. R implementation allows to estimate significance of spectral line in comparison to surrounding noise via F-test.

There are 22s, 30s, 45s periodical components with significance more than 99.9%

Multitaper method is still based on FFT which means that the frequency found might be a little bit off the true frequency if it’s not equal to any of component frequencies of transformation.

Due to a fixed sampling rate and a lack of low-pass filter before sampling the effect called aliasing could add more periodical components which are aliases of higher frequencies.

More on Outlier Detection

Mon, 08 Dec 2014 00:00:00 +0000

Several features of outlier detection in real world performance monitoring data.

Outlier score is useless. It could be defined as number of standard deviations from the sample mean to the value in question (or number of IQRs from median depending on method used). Due to extremely long tails of data distribution there would be outliers with scores 10 and 1000 but it doesn’t mean that one of them is 100 times worse than another. The only thing that seems to matter here is if the value is an outlier or not.

Mean and standard deviation values are misleading for non-gaussian distributions which are typical in performance monitoring data. Median and IQR (or interdecile range) are more robust estimates for center and dispersion of data.

IQR or interdecile range could be 0 for a flat metric with rare spikes. It makes methods based on these values useless.

Many metrics have small number of unique values which skews calculation of quantilles (breaks IQR-based methods) and produces steep steps in empirical CDF (breaks KS-test).

Asymmetric distribution (typical for latency data) suggests using asymmetric ranges around median but it’s often impossible to calculate range on one side because there is only one value there (e.g. mostly constant metric with rare positive spikes).

We can’t ignore outliers when reasoning about overall system performance. Imagine a single threaded service that has typical request latency about 1ms and one in 1000 requests takes 1s (outlier) to complete. If we drop the outlier (maybe because of Coordinated Omission) then the service seems to be capable of serving 1000 rps. In reality that service works for one second and then gets stuck for another second resulting in total performance less than 500 rps.

There are several events which are outliers from practical point of view but outlier detection methods don’t handle them well (though they are quite simple to detect):

Metric appeared. Some libraries might start reporting metric only after application provides value for the first time. For rare events (like errors) it means that event’s counter will be undefined until the event happens for the first time.
Metric disappeared. It might mean that metric was removed with latest update of application or that there was no single appearance for some rare event within observation window.
Constant changed. Observation window contains only equal values and the last obtained one is not equal to previously seen values.

Handling Seasonal Data with Outliers

Mon, 27 Oct 2014 00:00:00 +0000

Most anomaly detection methods for time series expect it to be flat (not changing over time) and they find when it stops being flat. But real data that comes from system monitoring is not flat. People are active during business hours and sleep at night so system usage metrics are higher during daytime. Weekends usually have lower usage metrics for business apps and higher for entertainment. That difference between daytime and nighttime (and weekend vs. middle of the week) needs to be removed somehow before applying anomaly detection algorithm.

Holt-Winters is popular for such data but it puts more weight into last observations so if you had public holiday or outage last week it’ll expect the same dip next week leading to false positive.

There are several methods described at chapter 6 Time series decomposition of Forecasting: principles and practice that allow to split time series into seasonal and non-seasonal components.

I needed to process several thousands metrics so the most simple and fast Classical decomposition was chosen first. Here’s an example weekly decomposition of 35 days of Wikipedia pageviews data using decompose function from R:

The decomposition removes most of daily and weekly variation from the data and allows to see outliers more clearly in the remainder component (labeled as ‘random’).

In the next example I injected 2 hours long outage (zero pageviews) into the data and did the same decomposition:

The single outage managed to corrupt extracted trend and seasonal component and introduced false outlier in the remainder for each week. It happens because the method uses moving average to get trend component out and then averages values for the same time during several seasons (e.g. average 10AM value on Mondays if period length is 1 week). (Moving) average is not robust in presence of outliers so I decided to try median instead.

This decomposition looks much better than the previous one but there are several caveats.

High resolution data doesn’t mean better seasonal extraction because current 10s don’t have to be the same as 10s exactly one week ago. But the current hour has to be similar to the same hour of day week ago if there is no holidays, outages, extremely successful marketing campaigns, etc.

Median filter leaves abrupt steps in extracted trend. That means that if you are looking for step-like anomalies then you can’t just process remainder. Step might be hiding in the trend component so it’s better to remove only seasonal component from signal (use trend + remainder for step detection).

Periods of odd length are better for median calculation because there is no interpolation in that case. Same reason applies to seasonal component extraction. Since one season is lost due to left and right padding during filtering then even number of periods is better.

R code and data used to make these graphs.

Statistics for Monitoring: Correlation and Clustering

Mon, 12 May 2014 00:00:00 +0000

Finding metrics with similar behavior and analyzing internal system dependencies.

There are a lot of situations when you see an unexpected change in one metric (e.g. increased latency or error rate) and need to find the cause. It could be solved by applying prior knowledge about dependencies in the system but there are still a lot of unknowns especially in case of poorly documented legacy applications. It would be great to have a tool that given a metric produces a list of other metrics it depends on based on time series data.

There are some papers (e.g. “Causality and graphical models in time series analysis”) suggesting that it’s possible to infer dependencies from time series data but I haven’t done any experiments with it yet (R package vars might be useful).

For practical purposes it’s often enough to find metrics that have similarly shaped graphs. Changes in computer systems are quite fast (seconds or milliseconds) to propagate between components making it impossible to find out what changed first with typical polling intervals (10s, 60s) so it looks like dependent metrics are moving at the same time.

There are several methods for finding similar time series:

Actually there are much more targeted at different use cases and the list above contains only most popular ones. The good news is that performance monitoring data is quite small in comparison with ECG and EEG data. It gives a hope that more effective methods from medical field will be used for performance monitoring too.

First two are quite simple to understand and fast in runtime. Pearson correlation coefficient has a simple relation to Euclidean distance for normalized data. Absolute value of correlation coefficient allows to find mirrored graphs (like disk used vs. disk free space).

Dynamic time warping has a nice property to find slightly misaligned in time graphs but its R implementation was too slow to be useful when I tried it.

My experiments with DFT failed because computer-generated metrics rarely have sparse representation in frequency domain. Their spectra are wide and contain a lot of frequencies. There is not so many periodical things (maybe cron jobs) going on in online request processing. There are seasonal daily/weekly/yearly changes but the typical need to compare graphs is limited by short ranges (hours or even minutes).

DWT looks promising because Haar wavelet’s shape is very similar to step-like changes usually found in performance metrics but I haven’t tried it yet.

Sometimes production system misbehaves but you have no idea where to start from because known dependencies and similar graphs for usual suspects (metrics like error rate) lead nowhere. Going through all metrics and watching all graphs works for small systems (up to several hundreds metrics) but doesn’t work when there are thousands metrics. While trying to do that I noticed that many graphs are almost the same making it unnecessary to get through all of them.

Clustering allows to group similar metrics and reduce amount of data to analyze. It makes possible to take only single representative from each group of similar metrics. I used Partitioning Around Medoids clustering algorithm and absolute value of correlation coefficient as a distance function.

These 4 graphs were taken from a set of cluster representatives.

Contents of a cluster represented by the top right one:

There are 2 quite similar graphs on top (actually there were more omitted from illustration) but it contains some noisy graphs (bottom) which don’t look really similar to others. Clustering algorithm used requires exact number of clusters to be set upfront which leads to the result above because it has to assign graphs which are not similar to anything to some clusters.

This is another cluster with different contents. Top right graph looks like periodical exponential charge while others are more like sawtooth wave or triangle wave. Relation between those graphs is clealy non-linear.

Spearman’s rank correlation coefficient was used to create these clusters because it allows to catch non-linear relationships between metrics. It’s more computationally difficult than Pearson but produces better results because there are a lot of non-linear dependencies in computer-generated data. It allowed me to find misconfigured cache expiration which wiped too much data from the cache and periodically overloaded service behind the cache (note to self: don’t forget to monitor cache-miss ratio next time).

Problems with clustering monitoring data:

Non-euclidean (ultrametric) space. Many clustering algorithms require distance function to be Euclidean metric while useful time series comparison functions (like correlation coefficient) are not. It limits set of available algorithms.
Many small clusters. There a lot of almost independent metrics which produce large number of small clusters. It makes harder to set number of clusters for algorithms that require it. If you set number of clusters too low there will be a lot of unrelated noise in each cluster but if you set the number too high the original purpose (to reduce amount of graphs to watch) will be missed.
Local clusters around events. Each event like service restart tend to gather a lot of metrics into single big cluster. It might be good for investigating outages but it’s bad for finding dependencies in a steady state. It’s better to choose quiet time range without any significant events for the latter use case.
Correlations which are not dependencies. If two metrics rise and fall at the same time their correlation coefficient will be close to 1 (or -1 for mirrored graphs) but in general correlation doesn’t imply causation. There are a lot of cases where independent events happen at the same time like:
- cron jobs (e.g. log rotation)
- human actions (cluster restarts, reconfigurations)
- timed cache expirations

Cluster structure often tells obvious things like grouping all metrics related to a single service into one cluster. But sometimes it uncovers something new like dependency between latency of one application and disk IO of another unrelated one which turned out to be using the same storage array.