CLML, Wine, PCA, K-Means and Hierarchical Clustering¶

(C) 2015 Mike Maul -- CC-BY-SA 3.0

This document is part series of tutorials illustrating the use of CLML.

Caveat¶

Anyone wishing to run this notebook or the code contained there in must take note of the following:

This tutorial relies on the github version of CLML https://github.com/mmaul/clml.git or a quicklist-dist CLML> than 20150805
The plotting portion of this code requires the system clml.extras https://github.com/mmaul/clml.extras.git cloned after 2015-09-10, which is not currently in quicklisp.
While the above git repositories are not in quicklisp they be loaded by quickload by placing the repositories in $HOME/quicklisp/local-projects

Introduction¶

This article will discuss two clustering techniques, k-means and Hierarchical Clustering. Clustering is an unsupervised learning technique, with the goal bing to group to samples into a given number of partitions. Clustering uses uses the similarity between examples and groups examples based of their mutual similarities.

Wine¶

Wine such a multideminensional beverage. A feast for the senses, taste, smell even for the eyes. It also pairs well with statistical analysis techniques and even Lisp. What does wine and statistical analysis have to do with one another you may ask. Well the Wine dataset at it the 'goto' dataset used in just about every introduction cluster analysis. The Wine dataset is a chemical analysis of three types of wines wines grown in a region of Italy. The dataset contains an analysis of 178 samples, with 13 results of chemical assays for each sample. It is small enough yet contains enough complexity to be interesting.

While were on the subject of wine if you find yourself in eastern Washington, I highly recommend stopping by the Parisisos del Sol winery, good wine, down to earth atmosphere and a really interesting and knowledgeable proprietor.

But back to the cluster analysis...

Lets get started by loading the system necessary for this tutorial and creating a namespace to work in.

(progn
    (ql:quickload '(:clml.utility ; Need clml.utility.data to get data from the net
                :clml.hjs ; Need clml.hjs.read-data to poke around the raw dataset
                :clml.pca
                :clml.clustering
                :iolib
                :clml.extras.eazy-gnuplot
                :eazy-gnuplot
                ))
    (defpackage #:wine
      (:use #:cl
        #:cl-jupyter-user
        #:clml.hjs.read-data
        #:clml.utility.data
        #:clml.hjs.vector
        #:clml.hjs.matrix
        #:clml.hjs.k-means
        #:clml.pca
        #:clml.clustering.hc
        #:eazy-gnuplot
        ))
)

To load "clml.utility":
  Load 1 ASDF system:
    clml.utility

; Loading "clml.utility"
....
To load "clml.hjs":
  Load 1 ASDF system:
    clml.hjs

; Loading "clml.hjs"

To load "clml.pca":
  Load 1 ASDF system:
    clml.pca

; Loading "clml.pca"

To load "clml.clustering":
  Load 1 ASDF system:
    clml.clustering

; Loading "clml.clustering"

To load "iolib":
  Load 1 ASDF system:
    iolib

; Loading "iolib"
.....
To load "clml.extras.eazy-gnuplot":
  Load 1 ASDF system:
    clml.extras.eazy-gnuplot

; Loading "clml.extras.eazy-gnuplot"

To load "eazy-gnuplot":
  Load 1 ASDF system:
    eazy-gnuplot

; Loading "eazy-gnuplot"

#<PACKAGE "WINE">

(in-package :wine)

#<PACKAGE "WINE">

Take time to get to know your wine¶

This tutorial illustrates how to use clustering, as well as how to use Principal Component Analysis in support of k-means and also hierarchical clustering in CLML. Just like swilling a bottle of wine is not a good thing neither is turning loose analysis techniques willy nilly on data. It is necessary to understand your data first, as the saying goes, garbage in garbage out...

Another take away from this tutorial is working with diverse data. Our dataset comes UCI Machine learning archive, the convention there is a headerless CSV file with seperate file ending in .names containing the a description of the dataset and the column names.

So armed with some information about the origin and meaning of the data, the orignization and the location we can begin. The dataset is located at http://archive.ics.uci.edu/ml/machine-learning-databases/wine/wine

Well almost ready another thing that is necessary is to asses the data so we can ingest it properly. This is something that generally needs to be done manually or can be skipped with prior knowldege of the dataset.

The wine.names file is rather verbose so We will just list the column names:

Alcohol
Malic acid
Ash
Alcalinity of ash
Magnesium
Total phenols
Flavanoids
Nonflavanoid phenols
Proanthocyanins
Color intensity
Hue
OD280/OD315 of diluted wines
Proline

So lets take a peek at the data.

(defparameter uci-wine 
    (read-data-from-file
        (fetch "http://archive.ics.uci.edu/ml/machine-learning-databases/wine/wine.data")
        :type :csv))

UCI-WINE

(format nil "~A" (head-points uci-wine))

"#(#(1 13.2 1.78 2.14 11.2 100 2.65 2.76 .26 1.28 4.38 1.05 3.4 1050)
  #(1 13.16 2.36 2.67 18.6 101 2.8 3.24 .3 2.81 5.68 1.03 3.17 1185)
  #(1 14.37 1.95 2.5 16.8 113 3.85 3.49 .24 2.18 7.8 .86 3.45 1480)
  #(1 13.24 2.59 2.87 21 118 2.8 2.69 .39 1.82 4.32 1.04 2.93 735)
  #(1 14.2 1.76 2.45 15.2 112 3.27 3.39 .34 1.97 6.75 1.05 2.85 1450))"

So is is all numeric data, The first column from the dataset definition is the class so we don't want to give that to our clustering algorithm as that would be cheating. The numeric columns should be of type double float

The k-means, and hierarchal clustering implementations require a numeric-dataset which we create by using pick-and-specialize data

(let ((wine-unspecialized (read-data-from-file
    (fetch "http://archive.ics.uci.edu/ml/machine-learning-databases/wine/wine.data")
    :type :csv
    :csv-type-spec '(integer
                     double-float double-float  double-float
                     double-float double-float  double-float
                     double-float double-float  double-float
                     double-float double-float  double-float
                     double-float)
    :csv-header-p ( list "Class"
                         "Alcohol"            "Malic acid"           "Ash"
                         "Alcalinity of ash"  "Magnesium"            "Total phenols"
                         "Flavanoids"         "Nonflavanoid phenols" "Proanthocyanins"
                         "Color intensity"    "Hue"                   "OD280/OD315 of diluted wines"
                         "Proline")
    )))
  (defparameter wine
    (pick-and-specialize-data 
       wine-unspecialized   
       :range '(1 2 3 4 5 6 7 8 9 10 11 12 13) :data-types (make-list 13 :initial-element :numeric )))
  (defparameter wine-with-classifications
    (pick-and-specialize-data 
       wine-unspecialized   
       :data-types (make-list 14 :initial-element :numeric )))
       
  (defparameter wine-classifications (loop for r across (dataset-points wine-with-classifications) 
                                           when (= (elt r 0) 1d0) count r into one 
                                           when (= (elt r 0) 2d0) count r into two 
                                           when (= (elt r 0) 3d0) count r into three 
                                           finally (return (list (list 1 one) (list 2 two) (list 3 three)))))

)

WINE-CLASSIFICATIONS

Letting the wine breathe¶

The process of letting wine breathe is to exposure to the surrounding air. Letting allowing wine to mix and mingle with air and surrounding temperature. Similarly our data needs to "mix and mingle" so that the scales become more uniform which is helpful to clustering algorithms.

So we need to consider if the data needs to be standardized, generally the answer will be yes, unless the columns are all of the same units and have lov variance. Specifically with this datasets since the columns are using different units scaling is necessary. Our our implementation of k-means can do our standardization for us however we will need to do it for the principal component analysis. The standardize function used here will be the z-score which is $(c - \mu) / \sigma$.

Savoring the wine¶

Savoring, the the process of detecting the essential properties generally using taste, we wiil however use principal component analysis instead.

So what columns should be evaluated? We could evaluate all of them that might be okay, and would work in this case. However in the case of large datasets it may be necessary to reduce the dimensionality to decrease execution time. It could also be the case that some relatively unimportant columns could be throwing off the results.

Principal Component Analysis¶

Well one technique we can use to select the relevant columns for analysis is principal component analysis. Principal Component Analysis is a set of orthogonal transformations that reduce dimensionality in a dataset. One of the things that PCA can tell us is the magnitude of the contribution of each column to the variation in the data.

PCA is one of those algorithms that needs regular data so we are going to standardize the dataset first.

Lets compute a few things we will be needing.

standardized-wine: The standardized copy of wine dataset
pca-result: The result of the PCA analysis of the standardized wine dataset

(progn 
  (defparameter standardized-wine (copy-dataset wine))
  (setf (dataset-numeric-points standardized-wine) (standardize (dataset-numeric-points standardized-wine)))
  
  (defparameter pca-result (princomp standardized-wine))
)

PCA-RESULT

One of the methods of evaluating the choice of principal components is the 'elbow' method. The elbow method involves choosing the number of components that lie in the elbow of the graph. In the graph below three or four would be a reasonable choice.

Specifically contributions returns the variances of the components.

(let ((png "wine-pca-contributions.png"))
    (clml.extras.eazy-gnuplot::plot-series (contributions pca-result) 
    :term '(:png) :output png :plot-title "PCA Contributions" :ylabel "Variance" :xlabel "Column" :series-title "")
    (display-png (png-from-file png)))

One of the primary uses of PCA is to reduce dimensionality of data. In this case it allows us to make the decision that first four columns are what drive the majority of the variation in the data. So knowing this we will be truncating the data set to four columns. While this is not that critical in the current dataset which has only thirteen dimensions, it can be critical when dealing with datasets with hundreds or thousands of columns.

So lets partition off the dataset so it only contains the principal columns, as we will need that later.

(defparameter truncated-standardized-wine (make-numeric-dataset  
                       (map 'list #'dimension-name (subseq (dataset-dimensions wine) 0 4))                           
                       (map 'vector (lambda (r) (subseq r 0 4)) (dataset-points standardized-wine))))

TRUNCATED-STANDARDIZED-WINE

So for wine to be judged impartially it should be served without context and evaluated solely on its own merits. This is really what we are doing we already stripped the classifications from the dataset, and we aren't even tasting the wines ourselves. Out goal is to try match the wine varietal type with the preexisting classifications.

Deciding on the clusters¶

We need to have some idea of how the data will be grouped. In some cases this will be known, in others it will not. In this case we know theere are three different classifications.

But what if we didn't know?

One method is to look at the within group of sums of squares among the contributions of the principal components of the dataset. We calculate the first value storing it in wss1 the remaining values can be calculated by taking the sum of squares of the distances returned from running k-means (as the distance-between-point-and-owner slot in the pca-result instance returned from k-means)on the selected principal components from the previous PCA analysis.

(let ((wss1 (* (- (length (components pca-result)) 1) 
               (loop for v across (subseq (contributions pca-result) 0 4) sum v)))
       (comp-ds (make-numeric-dataset '("pc1" "pc2" "pc3" "pc4")
                                      (map 'vector (lambda (r) (subseq r 0 4)) (components pca-result))))
        (png "group-sum-of-squares.png"))
    (clml.extras.eazy-gnuplot::plot-series 
      (coerce (cons wss1 
                  (loop for n from 2 upto 8 ; could be up size of dimensions the far end is generally irrelevant
                        for k-means-n = (k-means n comp-ds  :standardization nil 
                                                 :random-state (make-random-state-with-seed 100))
                        collect (loop for x across (clml.hjs.k-means::pw-distance-between-point-and-owner k-means-n) 
                                      sum (* x x)))) 
               'vector)
       :term '(:png) :output png :plot-title "Group Sum of Squares" :ylabel "In group sum of squares" 
       :xlabel "Clusters" :series-title "")
  
  (display-png (png-from-file png)))

Again we use the "elbow" method to make our selection (additionally in either selecting number of cluster or the principal components we can use the 85% precentile to make the choice point).

But what is really cool is the elbow lies right on three cluster which matches the number of clusters that had been manually classified in the wine dataset.

K-means¶

So we kind of jumped into using k-means before talking about it... Simplified description of the K-means Algorithm:

How it works¶

Initial cluster are randomly chosen.
The squared distance from each object to each cluster is computed, then the objects are assigned to the nearest cluster.
New centroid are computed for each cluster – and each cluster is by the respective cluster centroid.
The squared distance from each object to each cluster is computed, and the objects are assigned to the cluster nearest cluster with the smallest squared distance.
Based upon the new membership assignment cluster centroids are recalculated.
Steps 4 and 5 are repeated until stability is achieved.

CLML bits¶

The k in k-means is the number of clusters that k-means will be classifying.. K is required parameter in k-means but how should we decide what to set it to? Well we already know from the original dataset that there are three clusters.

So for testing or teaching you will want to set a fixed :random-state which will allow you have repeatable results (It influences the choice of the seed clusters). But keep in mind if you set it all the time you might get caught on a local optima that you might have avoided if you had allowed the state to be random.

The CLML implementation of k-means accepts a number of parameters for running k-means.

k-means arguments *
- k: [integer], number of clusters
- dataset: [numeric-dataset] | [category-dataset] | [numeric-or-category-dataset]
- :distance-fn: #'euclid-distance | #'manhattan-distance | #'cosine-distance
- :standardization: t | nil, whether to standardize the inputs
- :max-iteration: maximum number of iterations of one trial (default 1000)
- :num-of-trials: number of trials, every trial changes the initial position of the clusters. (Default 10)
- :random-state: (for testing), specify the random-state of the random number generator
- :debug: t | nil (for debugging) print out some debugging information
k-means results *
- workspacet: points, clusters, distance infomation, etc.
- table: lookup table for normalized vecs and original vecs, might be removed later.

Running k-means¶

Okay, so now that we know the number of clusters (well we've always known, but we also arrived at the same choice on our own), lets run k-means on our dataset.

(progn  (defparameter workspace nil) 
         (defparameter table nil)
         (multiple-value-setq (workspace table)
           (k-means 3 truncated-standardized-wine  :standardization nil 
                    :random-state (make-random-state-with-seed 1234))))

#<CLML.HJS.K-MEANS::PROBLEM-WORKSPACE 3 Clusters (ID size): ((0 54) (1 62)
                                                             (2 62)) {1009C9CD53}>

And we see out cluster assignment in the pretty print of the object. Lets compare roughly with the counts of the classification that came with the dataset.

(loop for c across (!! wine-with-classifications "Class") 
  when (= c 1.0) count c into one when (= c 2.0) count c into two when (= c 3.0) count c into three 
  finally (return (list (list 1  one) (list 2 two) (list 3 three))))

((1 59) (2 71) (3 48))

In comparison with the original classifications, we see our results are not too bad: (Mind you that the cluster numbers assigned are not important the grouping is however)

wine-classifications

((1 59) (2 71) (3 48))

Hierarchical Clustering¶

How it works¶

The CLML implementation of Hierarchical Clustering uses agglomerative clustering (bottom up approach) A nice concise description of the process of agglomerative clustering from improvedoutcomes.com is as follows:

Assign each object to a separate cluster.
Evaluate all pair-wise distances between clusters (distance metrics are described in Distance Metrics Overview).
Construct a distance matrix using the distance values.
Look for the pair of clusters with the shortest distance.
Remove the pair from the matrix and merge them.
Evaluate all distances from this new cluster to all other clusters, and update the matrix.
Repeat until the distance matrix is reduced to a single element.

CLML Bits¶

The distance-matrixtakes an optional argument :distance-fn which specifies the distance function used in the distance computations. The distance functions available are:

ecludian (Default)
manhattan
pearson
cosine
tanimoto
caberra

The cophonetic-matric function evaluates the distance matrix and returns the merge and cophonetic matrix. cophonetic-matrix take a optional :method parameter to specify the method used to evaluate the distance matrix. The evaluation methods are as follows.

hc-single
hc-complete
hc-average (default)
hc-centroid
hc-median
hc-ward

cuttree cuts the tree defined in the merge matrix into the specified number of pieces.

Running Hierarchical Clustering¶

(progn
  (defparameter distance-matrix (distance-matrix (numeric-matrix standardized-wine)))
  (defparameter u nil) (defparameter v nil)
  (multiple-value-setq (u v) (cophenetic-matrix distance-matrix #'hc-ward))
  (defparameter ctree (cutree 3 v))
  (format t "Cut tree: ~A ~%Class counts:~A ~%" ctree
           (loop for x across ctree 
                 when (= x 1) counting x into one 
                 when (= x 2) counting x into two 
                 when (= x 3) counting x into three 
                 finally (return (list (list 1 one) (list 2 two) (list 3 three)))))
)

Cut tree: #(1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
            1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 3 1 1 1 1 1 1 1 1 3 2 2 3 3 3 1 3 3
            2 1 3 1 3 1 3 3 3 3 1 3 3 3 3 2 3 3 3 3 3 3 3 3 3 3 3 1 2 3 3 3 3 3
            3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 2 3 3 1 3 3 3 3 3 3 3 3 2 2 2 2 2 2
            2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
            2 2 2 2 2 2 2 2) 
Class counts:((1 65) (2 54) (3 59))

NIL

In comparison with the original classifications, we see our results are not too bad: (Mind you that the cluster numbers assigned are not important the grouping is however)

wine-classifications

((1 59) (2 71) (3 48))

Conclusion¶

In general k-means is computationally more efficient. The quality of classification results of k-means vs hierarchal clustering is up for debate. In the second part of this tutorial we will explore cluster validation, which will let us evaluate our results more thoroughly.

The iPython notebook and source for this tutorial can be found in the clml.tutorials https://github.com/mmaul/clml.tutorials.git github repository.

Stay tuned to clml.tutorials blog or RSS feed for part 2 of this tutorial 'CLML Wine and cluster validation' and for more CLML tutorials..¶

CLML Datasets Tutorial¶

(C) 2015 Mike Maul -- CC-BY-SA 3.0

This document is part series of tutorials illustrating the use of CLML.

Datasets, what and why¶

CLML datasets are two dimensional tabular data structures. In CLML datasets are used for (not to sound recursive) storing datasets. Datasets may contain numerical and categorical data. Datasets also contain column metadata (dimensions) and also provide facilities for extracting columns, dataset cleaning and splitting. Datasets in CLML are similar to dataframes in R or Pandas.DataFrames in Python.

Lets get started by loading the system necessary for this tutorial and creating a namespace to work in.

(ql:quickload '(:clml.utility ; Need clml.utility.data to get data from the net
                :clml.hjs ; Need clml.hjs.read-data for dataset
                :iolib
                :clml.extras.eazy-gnuplot
                :eazy-gnuplot
            ))

To load "clml.utility":
  Load 1 ASDF system:
    clml.utility

; Loading "clml.utility"
....
To load "clml.hjs":
  Load 1 ASDF system:
    clml.hjs

; Loading "clml.hjs"

To load "iolib":
  Load 1 ASDF system:
    iolib

; Loading "iolib"
.....
To load "clml.extras.eazy-gnuplot":
  Load 1 ASDF system:
    clml.extras.eazy-gnuplot

; Loading "clml.extras.eazy-gnuplot"

To load "eazy-gnuplot":
  Load 1 ASDF system:
    eazy-gnuplot

; Loading "eazy-gnuplot"

(:CLML.UTILITY :CLML.HJS :IOLIB :CLML.EXTRAS.EAZY-GNUPLOT :EAZY-GNUPLOT)

(defpackage #:datasets-tutorial
  (:use #:cl
        #:cl-jupyter-user ; Not needed unless using iPython notebook
        #:clml.hjs.read-data
        #:clml.hjs.meta ; util function
        #:clml.extras.eazy-gnuplot))

#<PACKAGE "DATASETS-TUTORIAL">

(in-package :datasets-tutorial)

#<PACKAGE "DATASETS-TUTORIAL">

Lets load some data that we will use as we learn about datasets.

(defparameter dataset (read-data-from-file 
                       (clml.utility.data:fetch "https://mmaul.github.io/clml.data/sample/cars.csv") 
                       :type :csv :csv-type-spec '(integer integer)))

DATASET

Data and Datasets¶

CLML has a number of different specializations of datasets such as

unspecialized-dataset untyped and unspecialized data
numeric-dataset dataset containing numeric (double-float) data
category-dataset dataset for categorical (string) data
numeric-and-category-dataset dataset containing a mixture of numeric and categorical data
numeric-matrix-dataset dataset where numeric values are stored as a matrix
numeric-matrix-and-category-dataset dataset where numeric values are stored as a matrix as well as having categorical data

Data representation¶

All datasets except the matrix datasets represent data as a vector of vectors. The inner vector contains the columns of each row. For datasets with categories, numeric and category data are stored in seperate vectors.

We can see below how the data is represented.

(dataset-points dataset)

#(#(4 2) #(4 10) #(7 4) #(7 22) #(8 16) #(9 10) #(10 18) #(10 26) #(10 34)
  #(11 17) #(11 28) #(12 14) #(12 20) #(12 24) #(12 28) #(13 26) #(13 34)
  #(13 34) #(13 46) #(14 26) #(14 36) #(14 60) #(14 80) #(15 20) #(15 26)
  #(15 54) #(16 32) #(16 40) #(17 32) #(17 40) #(17 50) #(18 42) #(18 56)
  #(18 76) #(18 84) #(19 36) #(19 46) #(19 68) #(20 32) #(20 48) #(20 52)
  #(20 56) #(20 64) #(22 66) #(23 54) #(24 70) #(24 92) #(24 93) #(24 120)
  #(25 85))

It may not be convenient to display the whole dataset to take a look at is. We could have used subseq but there is a helper method called head-points.

(head-points dataset)

#(#(4 2) #(4 10) #(7 4) #(7 22) #(8 16))

Dimensions¶

All Datasets have the dimensions slot which contain the column metadata. The dimensions slot contains a list of dimension instances. Each dimension instance contains the following slots (accessor prefix is dimension):

name column name
type type of data in column (e.g. :category :numeric :unknown)
index index on column vectors of column
metadata - alist that CAN containing useful information, such as equality tests for category data

(dataset-dimensions dataset)

#(#<CLML.HJS.READ-DATA::DIMENSION NAME: speed, TYPE: UNKNOWN, INDEX: 0.>
  #<CLML.HJS.READ-DATA::DIMENSION NAME: distance, TYPE: UNKNOWN, INDEX: 1.>)

Creating datasets¶

Datasets can be created directly or can be created by reading them from a file. Supported data formats or CSV and SEXP. Earlier we used the read-data-from-file function to read a dataset from a CSV file. The file in this case is a file that is obtained with the fetch from the clml.utility system, which downloads and caches a file in a location on a local files system or a URL. Datasets can also be created programatically.

(make-numeric-and-category-dataset  
 '("cat 1" "num 1")                           ; <-- Column names 
 (vector (v2dvec #(1.0d0)) (v2dvec #(2.0d0))) ; <-- Numeric data 
          '(1)                                ; <-- Indexes of numeric column
          #(#("a") #("b"))                    ; <-- Category Data
          '(0)                                ; <-- Indexes of category data
)

#<NUMERIC-AND-CATEGORY-DATASET >
DIMENSIONS: cat 1 | num 1
TYPES:      CATEGORY | NUMERIC
NUMBER OF DIMENSIONS: 2
CATEGORY DATA POINTS: 2 POINTS
NUMERIC DATA POINTS: 2 POINTS

Specializing datasets¶

The dataset we loaded is currently unspecialized, we haven't told CLML much about it yet. We can use the pick-and-specialize-data method to fill in the details.

(pick-and-specialize-data dataset :data-types '(:numeric :numeric))

#<NUMERIC-DATASET >
DIMENSIONS: speed | distance
TYPES:      NUMERIC | NUMERIC
NUMBER OF DIMENSIONS: 2
NUMERIC DATA POINTS: 50 POINTS

We can see pick-and-specialize-data returned a numeric dataset based on the supplied :data-types specification. pick-and-specialize-data has two parameters :range and :except. Both parameters deal with column selection :range specifies a range of columns (as a list) to use in our new dataset while :except specifies a list of columns to exclude from our new dataset. We had also mentioned the matrix datasets, pick-and-specialize-data can also change the representation from a vector of vectors to a matrix.

(let ((ds (pick-and-specialize-data dataset :data-types '(:numeric :numeric) 
           :store-numeric-data-as-matrix t)))
           (print ds)
           (dataset-numeric-points ds))

#<NUMERIC-MATRIX-DATASET >
DIMENSIONS: speed | distance

TYPES:      NUMERIC | NUMERIC

NUMBER OF DIMENSIONS: 2

NUMERIC-MATRIX DATA POINTS: 50 POINTS

#2A((4.0d0 2.0d0)
    (4.0d0 10.0d0)
    (7.0d0 4.0d0)
    (7.0d0 22.0d0)
    (8.0d0 16.0d0)
    (9.0d0 10.0d0)
    (10.0d0 18.0d0)
    (10.0d0 26.0d0)
    (10.0d0 34.0d0)
    (11.0d0 17.0d0)
    (11.0d0 28.0d0)
    (12.0d0 14.0d0)
    (12.0d0 20.0d0)
    (12.0d0 24.0d0)
    (12.0d0 28.0d0)
    (13.0d0 26.0d0)
    (13.0d0 34.0d0)
    (13.0d0 34.0d0)
    (13.0d0 46.0d0)
    (14.0d0 26.0d0)
    (14.0d0 36.0d0)
    (14.0d0 60.0d0)
    (14.0d0 80.0d0)
    (15.0d0 20.0d0)
    (15.0d0 26.0d0)
    (15.0d0 54.0d0)
    (16.0d0 32.0d0)
    (16.0d0 40.0d0)
    (17.0d0 32.0d0)
    (17.0d0 40.0d0)
    (17.0d0 50.0d0)
    (18.0d0 42.0d0)
    (18.0d0 56.0d0)
    (18.0d0 76.0d0)
    (18.0d0 84.0d0)
    (19.0d0 36.0d0)
    (19.0d0 46.0d0)
    (19.0d0 68.0d0)
    (20.0d0 32.0d0)
    (20.0d0 48.0d0)
    (20.0d0 52.0d0)
    (20.0d0 56.0d0)
    (20.0d0 64.0d0)
    (22.0d0 66.0d0)
    (23.0d0 54.0d0)
    (24.0d0 70.0d0)
    (24.0d0 92.0d0)
    (24.0d0 93.0d0)
    (24.0d0 120.0d0)
    (25.0d0 85.0d0))

We should also show an example of a dataset with categories.

(pick-and-specialize-data (read-data-from-file 
                           (clml.utility.data:fetch "https://mmaul.github.io/clml.data/sample/UKgas.sexp"))
     :data-types '(:category :numeric))

#<NUMERIC-AND-CATEGORY-DATASET >
DIMENSIONS: year season | UKgas
TYPES:      CATEGORY | NUMERIC
NUMBER OF DIMENSIONS: 2
CATEGORY DATA POINTS: 108 POINTS
NUMERIC DATA POINTS: 108 POINTS

Datasets can be created and combined. Generally the dataset creation methods take the form of make-<dataset type> and either take vectors containing data or other datasets and create a new dataset.

Missing Values¶

CLML datasets support missing values. Missing values are represented as follows in the dataset-points:

category CLML.HJS.MISSING-VALUE:+C-NAN+
numeric CLML.HJS.MISSING-VALUE:+NAN+
unspecialized :NA

There are also the following predicates available to detect missing values:

CLML.HJS.MISSING-VALUE:C-NAN-P
CLML.HJS.MISSING-VALUE:NAN-P

The read-data-from-file also supports the mapping representations of missing values in data files to datasets. The missing-values-list keyword argument specifies the character sequences that will be recognized as missing values.

To illustrate missing values support lets read in a CSV file containing the follow:

a,   b,   c
1.0, 2.0, x
NA,  3.0, NA

Here missing values are represented in the CSV file by NA. For the read-data function to recognize the missing values we must set the :missing-values-list parameter as shown below:

(let ((ds (read-data-from-file 
            (clml.utility.data:fetch "https://mmaul.github.io/clml.data/sample/simple1.csv") 
            :type :csv 
            :csv-type-spec '(double-float double-float string) 
            :missing-values-list '("NA")
          )))
            (format nil "~A~%~A~%" ds (dataset-points ds)))

"#<UNSPECIALIZED-DATASET >
DIMENSIONS: a | b | c
TYPES:      UNKNOWN | UNKNOWN | UNKNOWN
NUMBER OF DIMENSIONS: 3
DATA POINTS: 2 POINTS

#(#(1.0d0 2.0d0 x) #(NA 3.0d0 NA))
"

We can also see how missing values are represented in a specialized dataset:

(let ((ds (pick-and-specialize-data
             (read-data-from-file 
               (clml.utility.data:fetch "https://mmaul.github.io/clml.data/sample/simple1.csv") 
               :type :csv 
               :csv-type-spec '(double-float double-float string) 
               :missing-values-list '("NA")
              )
              :data-types '(:numeric :numeric :category)
           )))
     (format nil "~A~%~A~%~A~%" ds  (dataset-numeric-points ds) (dataset-category-points ds))
)

"#<NUMERIC-AND-CATEGORY-DATASET >
DIMENSIONS: a | b | c
TYPES:      NUMERIC | NUMERIC | CATEGORY
NUMBER OF DIMENSIONS: 3
CATEGORY DATA POINTS: 2 POINTS
NUMERIC DATA POINTS: 2 POINTS

#(#(1.0d0 2.0d0) #(#<DOUBLE-FLOAT quiet NaN> 3.0d0))
#(#(x) #(0))
"

Dataset manipulation¶

The following operations can be preformed on datasets:

copying
splitting and sampling
cleaning
shuffling
deduplication
storing

We will use the UK Gas dataset to illustrate these operations.

(defparameter ukgas (pick-and-specialize-data (read-data-from-file 
                           (clml.utility.data:fetch "https://mmaul.github.io/clml.data/sample/UKgas.sexp"))
                           :data-types '(:category :numeric)))

UKGAS

Copying¶

The simplest operation is copying. copy-dataset makes a deep copy of the contents of a dataset.

(copy-dataset ukgas)

#<NUMERIC-AND-CATEGORY-DATASET >
DIMENSIONS: year season | UKgas
TYPES:      CATEGORY | NUMERIC
NUMBER OF DIMENSIONS: 2
CATEGORY DATA POINTS: 108 POINTS
NUMERIC DATA POINTS: 108 POINTS

Splitting and Sampling¶

Datasets can be subdivided by two similar methods make-bootstrap-sample-dataset and divide-dataset

The divide dataset returns a dataset split into two parts based upon the :divide-ratio like pick-and-specialize-data divide-dataset can limit the section values accessed with the :range and :except parameters. It can also pull values in a pseudo-random manner values in to their new datasets.

(multiple-value-list (divide-dataset ukgas :divide-ratio '(3 1) :random t))

(#<NUMERIC-AND-CATEGORY-DATASET >
DIMENSIONS: year season | UKgas

TYPES:      CATEGORY | NUMERIC

NUMBER OF DIMENSIONS: 2

CATEGORY DATA POINTS: 81 POINTS

NUMERIC DATA POINTS: 81 POINTS

 #<NUMERIC-AND-CATEGORY-DATASET >
DIMENSIONS: year season | UKgas

TYPES:      CATEGORY | NUMERIC

NUMBER OF DIMENSIONS: 2

CATEGORY DATA POINTS: 27 POINTS

NUMERIC DATA POINTS: 27 POINTS
)

make-bootstrap-sample-datasets on the other hand shuffles a dataset into a number of specified datasets of equal length to the original dataset. The :number-of-datasetsparameter defaults to 10.

(make-bootstrap-sample-datasets ukgas :number-of-datasets 3)

(#<NUMERIC-AND-CATEGORY-DATASET >
DIMENSIONS: year season | UKgas

TYPES:      CATEGORY | NUMERIC

NUMBER OF DIMENSIONS: 2

CATEGORY DATA POINTS: 108 POINTS

NUMERIC DATA POINTS: 108 POINTS

 #<NUMERIC-AND-CATEGORY-DATASET >
DIMENSIONS: year season | UKgas

TYPES:      CATEGORY | NUMERIC

NUMBER OF DIMENSIONS: 2

CATEGORY DATA POINTS: 108 POINTS

NUMERIC DATA POINTS: 108 POINTS

 #<NUMERIC-AND-CATEGORY-DATASET >
DIMENSIONS: year season | UKgas

TYPES:      CATEGORY | NUMERIC

NUMBER OF DIMENSIONS: 2

CATEGORY DATA POINTS: 108 POINTS

NUMERIC DATA POINTS: 108 POINTS
)

Cleaning¶

One nice features of CLML is the dataset cleaning capabilities. The dataset-cleaning method provides the following:

outlier detection for numeric points
- standard deviation
- mean deviation
- user provided function
- smirnov-grubbs
- outlier detection for categorical points
- frequency based
- User provided function
Outlier and missing value interpolation using:
- zero
- min
- max
- mean
- median
- mode
- spline

To illustrate we will preform dataset cleaning where outliers will be points that exceed 1 standard deviation and will be replaced by zero.

(dataset-cleaning ukgas :outlier-types-alist '(("UKgas" . :std-dev)) 
                         :outlier-values-alist '((:std-dev . 1)) 
                         :interp-types-alist '(("UKgas" . :zero)))

#<NUMERIC-AND-CATEGORY-DATASET >
DIMENSIONS: year season | UKgas
TYPES:      CATEGORY | NUMERIC
NUMBER OF DIMENSIONS: 2
CATEGORY DATA POINTS: 108 POINTS
NUMERIC DATA POINTS: 108 POINTS

Adding dimensions and Concatenating Datasets¶

Adding dimensions¶

In some cases you may want to add a computed column or add a column to a dataset to hold the product of a computation on a dataset. The add-dim method can accomplish this easily. It can add an existing column of points with the :points parameter, it can also create a column with points filled with a initial value with the :initial-value parameter. The two mandatory parameters are the dataset to add the dimension to, the name of the new dimension and the type. If the dataset is either a category or numeric only dataset add-dim will create a numeric-and-category-dataset if a column of a different type is added.

(add-dim ukgas "mpg" :numeric :initial-value 0.0d0)

#<NUMERIC-AND-CATEGORY-DATASET >
DIMENSIONS: year season | UKgas | mpg
TYPES:      CATEGORY | NUMERIC | NUMERIC
NUMBER OF DIMENSIONS: 3
CATEGORY DATA POINTS: 108 POINTS
NUMERIC DATA POINTS: 108 POINTS

Concatenating datasets¶

Two datasets with equal numbers of rows can be concatenated or glued together vertically. concatenate-datasets takes two datasets as parameters and return a dataset with the points of the first dataset stacked on top of the points of the second dataset. The dimension name names of the first one dataset are retained in the new dataset.

(concatenate-datasets ukgas ukgas)

#<NUMERIC-AND-CATEGORY-DATASET >
DIMENSIONS: year season | UKgas
TYPES:      CATEGORY | NUMERIC
NUMBER OF DIMENSIONS: 2
CATEGORY DATA POINTS: 216 POINTS
NUMERIC DATA POINTS: 216 POINTS

Deduplicating datasets¶

Datasets can be deduplicated in place with the dedup-dataset! method. This functionality is currently only implemented for numeric, category and unspecialized datasets.

Storing datasets¶

Datasets can be saved to a file in csv format with the write-dataset method.

(write-dataset ukgas "gasgas.csv")

(CLML.UTILITY.CSV::OK)

Working with dataset points¶

Columns and values can be accessed and extracted from datasets using the !! macros. This macro returns the column name or list of column names as a vectors of vectors if multiple column names are specified or as a single vector if a single column name is specified.

(!! ukgas "UKgas")

#(160.1d0 129.7d0 84.8d0 120.1d0 160.1d0 124.9d0 84.8d0 116.9d0 169.7d0 140.9d0
  89.7d0 123.3d0 187.3d0 144.1d0 92.9d0 120.1d0 176.1d0 147.3d0 89.7d0 123.3d0
  185.7d0 155.3d0 99.3d0 131.3d0 200.1d0 161.7d0 102.5d0 136.1d0 204.9d0
  176.1d0 112.1d0 140.9d0 227.3d0 195.3d0 115.3d0 142.5d0 244.9d0 214.5d0
  118.5d0 153.7d0 244.9d0 216.1d0 188.9d0 142.5d0 301.0d0 196.9d0 136.1d0
  267.3d0 317.0d0 230.5d0 152.1d0 336.2d0 371.4d0 240.1d0 158.5d0 355.4d0
  449.9d0 286.6d0 179.3d0 403.4d0 491.5d0 321.8d0 177.7d0 409.8d0 593.9d0
  329.8d0 176.1d0 483.5d0 584.3d0 395.4d0 187.3d0 485.1d0 669.2d0 421.0d0
  216.1d0 509.1d0 827.7d0 467.5d0 209.7d0 542.7d0 840.5d0 414.6d0 217.7d0
  670.8d0 848.5d0 437.0d0 209.7d0 701.2d0 925.3d0 443.4d0 214.5d0 683.6d0
  917.3d0 515.5d0 224.1d0 694.8d0 989.4d0 477.1d0 233.7d0 730.0d0 1087.0d0
  534.7d0 281.8d0 787.6d0 1163.9d0 613.1d0 347.4d0 782.8d0)

Dataset points can also be accessed with the slot accessor. Since category and numeric data are stored separately in heterogeneous datasets separate accessors are used to access the points.

The list below shows which methods are applicable to the dataset type.

dataset-points: unspecialized-dataset
dataset-numeric-points: numeric-dataset numeric-and-category-dataset numeric-matrix-dataset numeric-matrix-and-category-dataset
dataset-category-points: category-dataset numeric-and-category-dataset numeric-matrix-dataset numeric-matrix-and-category-dataset

(dataset-numeric-points ukgas)

#(#(160.1d0) #(129.7d0) #(84.8d0) #(120.1d0) #(160.1d0) #(124.9d0) #(84.8d0)
  #(116.9d0) #(169.7d0) #(140.9d0) #(89.7d0) #(123.3d0) #(187.3d0) #(144.1d0)
  #(92.9d0) #(120.1d0) #(176.1d0) #(147.3d0) #(89.7d0) #(123.3d0) #(185.7d0)
  #(155.3d0) #(99.3d0) #(131.3d0) #(200.1d0) #(161.7d0) #(102.5d0) #(136.1d0)
  #(204.9d0) #(176.1d0) #(112.1d0) #(140.9d0) #(227.3d0) #(195.3d0) #(115.3d0)
  #(142.5d0) #(244.9d0) #(214.5d0) #(118.5d0) #(153.7d0) #(244.9d0) #(216.1d0)
  #(188.9d0) #(142.5d0) #(301.0d0) #(196.9d0) #(136.1d0) #(267.3d0) #(317.0d0)
  #(230.5d0) #(152.1d0) #(336.2d0) #(371.4d0) #(240.1d0) #(158.5d0) #(355.4d0)
  #(449.9d0) #(286.6d0) #(179.3d0) #(403.4d0) #(491.5d0) #(321.8d0) #(177.7d0)
  #(409.8d0) #(593.9d0) #(329.8d0) #(176.1d0) #(483.5d0) #(584.3d0) #(395.4d0)
  #(187.3d0) #(485.1d0) #(669.2d0) #(421.0d0) #(216.1d0) #(509.1d0) #(827.7d0)
  #(467.5d0) #(209.7d0) #(542.7d0) #(840.5d0) #(414.6d0) #(217.7d0) #(670.8d0)
  #(848.5d0) #(437.0d0) #(209.7d0) #(701.2d0) #(925.3d0) #(443.4d0) #(214.5d0)
  #(683.6d0) #(917.3d0) #(515.5d0) #(224.1d0) #(694.8d0) #(989.4d0) #(477.1d0)
  #(233.7d0) #(730.0d0) #(1087.0d0) #(534.7d0) #(281.8d0) #(787.6d0)
  #(1163.9d0) #(613.1d0) #(347.4d0) #(782.8d0))

A little something extra, `R-datasets`¶

One thing that I've always found handy in R is a standard, curated, extensive and documented series of datasets. Wouldn't it be nice to have access to these directly as datasets in CLML. The R-datasets system in clml.extras provides this capability. A particularly good use case for these datasets is to be able to follow along with examples and tutorials for R in CLML.

Note¶

The clml.extras systems are not currently part of quicklisp so if you are following along with this tutorial and are expecting just to (quickload :clml.extras.Rdatasets) you can't till you clone the clml.extras repository http://github.com/mmaul/clml.extras.git into your quicklisp/local-projects directory

Using Rdatasets¶

The Rdatasets package makes datasets included with the R language distribution available as clml datasets. R datasets are obtained csv files on Vincent Centarel's github repository. More information on these datasets can be found at http://vincentarelbundock.github.com/Rdatasets

Because type information is not included it may be necessary to provide a csv-type-spec for the columns in the csv file.

(ql:quickload :clml.r-datasets)

To load "clml.r-datasets":
  Load 1 ASDF system:
    clml.r-datasets

; Loading "clml.r-datasets"

(:CLML.R-DATASETS)

(use-package :clml.r-datasets)

T

(defparameter dd (get-r-dataset-directory))

DD

(subseq (inventory dd :stream nil) 0 505)

"Package                   Item                      Title                     
------------------------- ------------------------- ------------------------- 
datasets                  AirPassengers             Monthly Airline Passenger Numbers 1949-1960 

datasets                  BJsales                   Sales Data with Leading Indicator 

datasets                  BOD                       Biochemical Oxygen Demand 

datasets                  Formaldehyde              Determination of Formaldehyde"

(subseq (dataset-documentation  dd  "datasets" "BOD" :stream nil) 0 200)

"\"
R: Biochemical Oxygen Demand


BODR Documentation

 Biochemical Oxygen Demand 

Description

The BOD data frame has 6 rows and 2 columns giving the
biochemical oxygen demand versus time in an eval"

(defparameter bod (get-dataset dd "datasets" "BOD" :csv-type-spec '(double-float double-float double-float)))

BOD

bod

#<UNSPECIALIZED-DATASET >
DIMENSIONS:  | Time | demand
TYPES:      UNKNOWN | UNKNOWN | UNKNOWN
NUMBER OF DIMENSIONS: 3
DATA POINTS: 6 POINTS

(pick-and-specialize-data bod :data-types '(:numeric :numeric :numeric))

#<NUMERIC-DATASET >
DIMENSIONS:  | Time | demand
TYPES:      NUMERIC | NUMERIC | NUMERIC
NUMBER OF DIMENSIONS: 3
NUMERIC DATA POINTS: 6 POINTS

Conclusion¶

The iPython notebook and source for this tutorial can be found in the clml.tutorials https://github.com/mmaul/clml.tutorials.git github repository.

Stay tuned to clml.tutorials blog or RSS feed for more CLML tutorials..¶

CLML Time Series Tutorial Part 1¶

(C) 2015 Mike Maul -- CC-BY-SA 3.0

This document is a the first is a series of tutorials illustrating the use of the CLML.time-series package. In fact it is a first in a series of series of tutorials illustrating the use of CLML.

Caveat¶

Anyone wishing to run this notebook or the code contained there in must take note of the following:

This time series cleaning section of this tutorial relies on the github version of CLML https://github.com/mmaul/clml.git or a quicklist-dist CLML> than 20150805
The plotting portion of this code requires the system clml.extras https://github.com/mmaul/clml.extras.git which is not currently in quicklisp.
While the above git repositories are not in quicklisp they be loaded by quickload by placing the repositories in $HOME/quicklisp/local-projects

Time Series, what and why¶

A time series is a set of data points collected over a given period of time. Examples of time series are stock ticker data, sensor data and netflow data. Generally one wants to preform some sort of analysis on a time series and or use previous performance to forecast future performance or else use past performance to detect anomalies new data.

The CLML.time-series system contains functionality to manipulate, analyze time series data. CLML.time-series has a definite opinion on what a time-series is. We will see that after we load some data.

Lets get started by loading the system necessary for this tutorial and creating a namespace to work in.

(ql:quickload '(:clml.utility ; Need clml.utility.data to get data from the net
                :clml.hjs ; Need clml.hjs.read-data to poke around the raw dataset
                :clml.time-series ; Need Time Series package obviously
                :iolib
                :clml.extras.eazy-gnuplot
                :eazy-gnuplot
            ))

To load "clml.utility":
  Load 1 ASDF system:
    clml.utility

; Loading "clml.utility"
....
To load "clml.hjs":
  Load 1 ASDF system:
    clml.hjs

; Loading "clml.hjs"

To load "clml.time-series":
  Load 1 ASDF system:
    clml.time-series

; Loading "clml.time-series"

To load "iolib":
  Load 1 ASDF system:
    iolib

; Loading "iolib"
.....
To load "clml.extras.eazy-gnuplot":
  Load 1 ASDF system:
    clml.extras.eazy-gnuplot

; Loading "clml.extras.eazy-gnuplot"

To load "eazy-gnuplot":
  Load 1 ASDF system:
    eazy-gnuplot

; Loading "eazy-gnuplot"

(:CLML.UTILITY :CLML.HJS :CLML.TIME-SERIES :IOLIB :CLML.EXTRAS.EAZY-GNUPLOT
 :EAZY-GNUPLOT)

(defpackage #:time-series-part-2
  (:use #:cl
        #:cl-jupyter-user ; Not needed unless using iPython notebook
        #:clml.time-series.read-data
        #:clml.time-series.anomaly-detection
        #:clml.time-series.exponential-smoothing
        #:clml.extras.eazy-gnuplot)
  (:import-from #:clml.hjs.read-data #:head-points #:!! #:dataset-dimensions)
  (:import-from #:clml.time-series.util #:predict)
  (:import-from #:clml.hjs.read-data #:read-data-from-file)
  )

#<PACKAGE "TIME-SERIES-PART-2">

(in-package :time-series-part-2)

#<PACKAGE "TIME-SERIES-PART-2">

Data and Datasets¶

We are going to look at how time series data can be used by CLML. First lets get some data...

(defparameter dataset (read-data-from-file 
        (clml.utility.data:fetch 
            "https://mmaul.github.io/clml.data/sample/msi-access-stat/access-log-stat.sexp")))

DATASET

CLML's main unit of currency in working with data is the dataset. The dataset is a hierarchy series of classes that contain datapoints and metadata. They are similar to dataframes in R or data-tables in Python.

dataset

#<CLML.HJS.READ-DATA:UNSPECIALIZED-DATASET >
DIMENSIONS: date/time | hits
TYPES:      UNKNOWN | UNKNOWN
NUMBER OF DIMENSIONS: 2
DATA POINTS: 9068 POINTS

CCLML has a number of different specializations of dataset such as

unspecialized-dataset untyped and unspecialized data
numeric-dataset dataset containing numeric (double-float) data
categor-dataset dataset for categorical (string) data
numeric-and-category-dataset dataset containing a mixture of numeric and categorical data

Most relevant to this tutorial

time-series-dataset dataset containing time-series data

Datasets can be created directly or can be created by reading them from a file. Supported data formats or CSV and SEXP. In this case the read-data-from-file function is reading a data set from a file. The file in this case is a file that is obtained with the fetch function, which downloads and caches a file from a location on a local files system or a URL.

Lets take a look at the data, it apparently is from a hit counter. head-points gives us the first 5 rows of a dataset ( if we wanted all the rows in a dataset we would have used dataset-points

(head-points dataset)

#(#("12/May/2008 03:00-03:59" 210) #("12/May/2008 04:00-04:59" 265)
  #("12/May/2008 05:00-05:59" 219) #("12/May/2008 06:00-06:59" 284)
  #("12/May/2008 07:00-07:59" 287))

Examining the data it looks like hits collected hourly. time-series-datasets can be created with the time-series function.

Now Lets put this in a turn this into a time-series-dataset so we can do stuff with it.

(defparameter msi-access (time-series-data dataset :range '(1) :time-label 0 :frequency 24 :start '(18 3)))

MSI-ACCESS

msi-access

#<TIME-SERIES-DATASET >
DIMENSIONS: hits
TYPES:      NUMERIC
NUMBER OF DIMENSIONS: 1
FREQUENCY:  24
START:      (18 3)
END:        (395 22)
POINTS:     9068
TIME-LABEL: date/time

This is the point where we will talk about CLML.time-series's definite opinions about time series. Time series in CLML.time-series are discrete. In CLML.time-series's opinion time series have a regular frequency. (This implies that time series data must have a reading at each period. However CLML.time-series does support missing values which will be covered in a later part of this series) The representation of frequency is a important, especially when comparing time-series points at regular intervals. The FREQUENCY slot specifies the number of datapoints per cycle. The START slot indicates the starting time index and frequency interval. The measurements are contained in the points slot and are represented as a vector of ts-point objects. Another useful thing to know is the slot accessor prefix is ts-

In fact in the dataset we just created if you look at the raw dataset above you will see there are no time specifiers in the data (there are labels however but they are not used in computations). This can actually be very important if your time-series has literally astronomical ranges. Some time-series libraries/databases encode the index as seconds or milliseconds from some fixed point in time. Doing that then constricts the ability of the time series to represent times to the range of the datatype being used to encode the time index. To be fair CLML.time-series in effect is doing the same thing however the time index is relative and the time indices can range from 0 to most-positive-fixnum (~4.6e18 in SBCL) given a datapoint is defined by the time and frequency interval (which also range from 0 to most-positive-fixnum the number of theoretically possible datapoints in a time series is most-positive-fixnum squared (in SBCL this would be greater than 2.0e35)

Lets look at the points in the dataset to see how they are represented.

(subseq(ts-points msi-access) 0 5)

#(#S(CLML.TIME-SERIES.READ-DATA::TS-POINT
     :TIME 18
     :FREQ 3
     :LABEL "12/May/2008 03:00-03:59"
     :POS #(210.0d0))
  #S(CLML.TIME-SERIES.READ-DATA::TS-POINT
     :TIME 18
     :FREQ 4
     :LABEL "12/May/2008 04:00-04:59"
     :POS #(265.0d0))
  #S(CLML.TIME-SERIES.READ-DATA::TS-POINT
     :TIME 18
     :FREQ 5
     :LABEL "12/May/2008 05:00-05:59"
     :POS #(219.0d0))
  #S(CLML.TIME-SERIES.READ-DATA::TS-POINT
     :TIME 18
     :FREQ 6
     :LABEL "12/May/2008 06:00-06:59"
     :POS #(284.0d0))
  #S(CLML.TIME-SERIES.READ-DATA::TS-POINT
     :TIME 18
     :FREQ 7
     :LABEL "12/May/2008 07:00-07:59"
     :POS #(287.0d0)))

he ts-point class encodes each measurement maintaining the time and frequency interval, a label (which is just a string, ad the actual measurements. The measurements in the pos slot are stored in a vector arbitrary length. Looking back to IN[8] you can see when we gave time-series a start time of 18 , and a start frequency interval of 3 we can see by examining the ts-pointss how this is actually represented. Another useful thing to know is that the accessor prefix of ts-point is ts-s-

time-series-datasets can also be created programattically. Some examples are:

(make-constant-time-series-data  '("a") (vector (clml.hjs.meta:make-dvec 1)))
(make-constant-time-series-data '("price") (vector (v2dvec #(43.2d0)) (v2dvec #(44.0d0)) (v2dvec #(1049.0d0))))

Plotting Datasets¶

Lets plot our data with the clml.extras.eazy-gnuplot:plot-dataset.

Some quick things to note about the plot-dataset method:

The only required arguments are dataset and y-col
The terminal defaults to :wxt :persist so for use in a notebook we specify a PNG terminal
The svg function is used to render the plot in the notebook
eazy-gnuplot is used as the plotting library all plotting arguments follow gnuplot and eazy-gnuplot's conventions
The :range argument specifies the start and end of the points to display
The :frequencies argument is a list of frequencies to plot, handy for observing behavior over specific intervals.

(progn
    (plot-dataset msi-access "hits" :terminal '(:png)
        :range '(0 40) :title "MSI Access Log - first 40 points" :ytics-font ",8" :xtics-font ",8"
        :xlabel-font ",15" :ylabel-font ",15" :output "msi_access_log_40.png")
        (display-png (png-from-file "msi_access_log_40.png")))

Now lets look at the whole dataset. Since each ts-point has a label our x axis would get overwhelmed with labels, we use the :xtic-interval to specify that we only want labels displayed every 500 points.

(progn
 (plot-dataset msi-access "hits" :terminal '(:png )
        :title "MSI Access Log" :ytics-font ",8" :xtics-font ",8"
        :xlabel-font ",15" :ylabel-font ",15" :xtic-interval 500 :output "msi_access_log.png")
 (display-png (png-from-file "msi_access_log.png")))

Cleaning Datasets¶

Sometimes data may have missing values or outliers. It is not unusual to have a broken or malfunctioning sensor generating your data. We have a way of dealing with that.

The time-series-dataset class has a ts-cleaning method which can clean missing values an outliers. Lets look at the documentation:

TS-CLEANING names a generic function:
  Lambda-list: (D &KEY)
  Derived type: (FUNCTION (T &KEY) *)
  Documentation:
    - return: <time-series-dataset>
    - arguments:
        - d : <time-series-dataset>
        - interp-types-alist: 
             a-list (key: column name, datum: interpolation(:zero :min :max :mean :median :mode :spline)) | nil
        - outlier-types-alist: 
             a-list (key: column name, datum: outlier-verification(:std-dev :mean-dev :user :smirnov-grubbs 
                          :freq)) | nil
        - outlier-values-alist : 
             a-list (key: outlier-verification datum: the value according to outlier-verification) | nil
   - comment:
       Same as /dataset-cleaning/ in read-data package.

Lets give it a try. In particular lets set the threshold for outliers to 5 standard deviations and set the interpolation method to mean.

(defparameter c-msi-access 
    (ts-cleaning msi-access :outlier-types-alist '(("hits" . :std-dev)) 
                            :outlier-values-alist '((:std-dev . 5)) 
                            :interp-types-alist '(("hits" . :mean))))

C-MSI-ACCESS

(let ((png-file "clean-msi-access-log")) 
 (plot-dataset c-msi-access "hits" :terminal '(:png)
        :title "Cleaned MSI Access Log" :ytics-font ",8" :xtics-font ",8"
        :xlabel-font ",15" :ylabel-font ",15" :xtic-interval 500
        :yrange '(0 8000) 
        :output png-file)
  (display-png (png-from-file png-file)))

Notice the datapoint near 13 July 2008 15:00 to 15:59 that previously spiked to over 7000, is now more reasonable.

Conclusion¶

I would like to thank Fredreric Peschanski the creator of fishbowl which provides common lisp support for iPython. I would also like to thank Masataro Asai the creator of eazy-gnuplot. I would like to thank the creators of iPython and project Jupyter a truly cross platform mechanisim for th presentation of code and content. Finally I would like to thank github for [providing the ability to view notebooks inside github repositories] (http://blog.jupyter.org/2015/05/07/rendering-notebooks-on-github/)

The iPython notebook and source for this tutorial can be found in the clml.tutorials https://github.com/mmaul/clml.tutorials.git github repository.

clml.tutorials

Clml Wine Pca K Means And Hierarchical Clustering

CLML, Wine, PCA, K-Means and Hierarchical Clustering¶

Caveat¶

Introduction¶

Wine¶

Take time to get to know your wine¶

Letting the wine breathe¶

Savoring the wine¶

Principal Component Analysis¶

On to the wine tasting, a blind tasting¶

Deciding on the clusters¶

K-means¶

How it works¶

CLML bits¶

Running k-means¶

Hierarchical Clustering¶

How it works¶

CLML Bits¶

Running Hierarchical Clustering¶

Conclusion¶

Stay tuned to clml.tutorials blog or RSS feed for part 2 of this tutorial 'CLML Wine and cluster validation' and for more CLML tutorials..¶

Clml Datasets

CLML Datasets Tutorial¶

Datasets, what and why¶

Data and Datasets¶

Data representation¶

Dimensions¶

Creating datasets¶

Specializing datasets¶

Missing Values¶

Dataset manipulation¶

Copying¶

Splitting and Sampling¶

Cleaning¶

Adding dimensions and Concatenating Datasets¶

Adding dimensions¶

Concatenating datasets¶

Deduplicating datasets¶

Storing datasets¶

Working with dataset points¶

A little something extra, R-datasets¶

Note¶

Using Rdatasets¶

Conclusion¶

Stay tuned to clml.tutorials blog or RSS feed for more CLML tutorials..¶

Clml Time Series Part 1

CLML Time Series Tutorial Part 1¶

Caveat¶

Time Series, what and why¶

Data and Datasets¶

Plotting Datasets¶

Cleaning Datasets¶

Conclusion¶

Stay tuned to clml.tutorials blog or RSS feed for Part II which will cover prediction, anomaly-detection.¶

A little something extra, `R-datasets`¶