---
title: "Discriminant analysis: the lumbar spine"
output: rmarkdown::html_vignette
vignette: >
%\VignetteIndexEntry{Discriminant analysis: the lumbar spine}
%\VignetteEngine{knitr::rmarkdown}
%\VignetteEncoding{UTF-8}
---
Comparing classifiers under a resampling protocol, reading the errors rather than the
accuracy, and asking whether a non-linear model earns its complexity.
The other vignettes are listed by `vignette (package = "fdm2id")`; they use the same
handful of functions on other data, and can be read in any order.
``` r
library (fdm2id)
```
# The data
A dataset collected by Henrique da Mota in Lyon, on lumbar spine conditions: 100 healthy
patients, 60 with a disc hernia and 150 with a spondylolisthesis. The last two groups can be
merged into a single "abnormal" group, which gives a second, two-class target. Each patient is
described by six biomechanical attributes of the shape and orientation of the vertebrae.
``` r
data (spine)
summary (spine)
#> V1 V2 V3 V4
#> Min. : 26.15 Min. :-6.55 Min. : 14.00 Min. : 13.37
#> 1st Qu.: 46.43 1st Qu.:10.67 1st Qu.: 37.00 1st Qu.: 33.35
#> Median : 58.69 Median :16.36 Median : 49.56 Median : 42.41
#> Mean : 60.50 Mean :17.54 Mean : 51.93 Mean : 42.95
#> 3rd Qu.: 72.88 3rd Qu.:22.12 3rd Qu.: 63.00 3rd Qu.: 52.69
#> Max. :129.83 Max. :49.43 Max. :125.74 Max. :121.43
#> V5 V6 Classif2 Classif3
#> Min. : 70.08 Min. :-11.06 AB:210 DH: 60
#> 1st Qu.:110.71 1st Qu.: 1.60 NO:100 NO:100
#> Median :118.27 Median : 11.77 SL:150
#> Mean :117.92 Mean : 26.30
#> 3rd Qu.:125.47 3rd Qu.: 41.28
#> Max. :163.07 Max. :418.54
```
``` r
plotdata (spine, k = spine [, 7])
```
``` r
plotdata (spine, k = spine [, 8])
```
# Question 1. Which method predicts the two-class problem best?
``` r
# Variable: the bootstrap draws its 100 resamples at random, so without 'seed' this table
# changes at every run -- by a few thousandths here, enough to swap two close methods.
performance (c (NB, LDA, CDA, LR), spine [, 1:6], spine [, 7], type = "evaluation",
protocol = "bootstrap", eval = "accuracy", nruns = 100, seed = 0)
#> accuracy
#> NB 0.7763503
#> LDA 0.8224087
#> CDA 0.7898095
#> LR 0.8445202
```
**Answer.** *Under a bootstrap evaluation, logistic regression is the method with the best
accuracy.*
# Question 2. And the three-class problem?
``` r
performance (c (NB, LDA, CDA, LR), spine [, 1:6], spine [, 8], type = "evaluation",
protocol = "bootstrap", eval = "accuracy", nruns = 100, seed = 0)
#> accuracy
#> NB 0.8114840
#> LDA 0.8078133
#> CDA 0.7982870
#> LR 0.8489775
```
**Answer.** *Logistic regression again.*
# Question 3. Is the three-class problem harder than the two-class one?
The two tables above already hold the answer, for the method that won both of them:
``` r
performance (LR, spine [, 1:6], spine [, 7], type = "evaluation",
protocol = "bootstrap", eval = "accuracy", nruns = 100, seed = 0)
#> accuracy
#> 0.8445202
performance (LR, spine [, 1:6], spine [, 8], type = "evaluation",
protocol = "bootstrap", eval = "accuracy", nruns = 100, seed = 0)
#> accuracy
#> 0.8489775
```
**Answer.** *The two accuracies are very close, so no. Which is worth a pause: splitting the
"abnormal" group in two adds a distinction that the six attributes apparently make quite
well.*
# Question 4. Which classes are hardest to separate?
An accuracy says how often the model is right, not what it gets wrong. The confusion matrix
does, and its rows are the truth.
``` r
performance (LR, spine [, 1:6], spine [, 8], type = "confusion",
protocol = "bootstrap", nruns = 100, seed = 0)
```
```
#> Predicted labels
#> True labels DH NO SL
#> DH 0.65764547 0.31799729 0.02435724
#> NO 0.15584764 0.80445279 0.03969957
#> SL 0.01291614 0.03074404 0.95633982
```
**Answer.** *`SL` (spondylolisthesis) is rarely confused with the other two -- it is recovered
96% of the time. Separating `NO` (healthy) from `DH` (disc hernia) is much harder: a third of
the hernias are predicted healthy.*
# Going further: are non-linear methods worth it?
The four methods above all come from linear discriminant analysis and its neighbourhood.
Adding one more of that family, a linear-kernel SVM:
``` r
performance (c (NB, LDA, CDA, LR, SVMl), spine [, 1:6], spine [, 8], type = "evaluation",
protocol = "bootstrap", eval = "accuracy", nruns = 100, seed = 0)
#> accuracy
#> NB 0.8114840
#> LDA 0.8078133
#> CDA 0.7982870
#> LR 0.8489775
#> SVMl 0.8546583
```
and then four methods that do not:
``` r
# Variable, twice over: on top of the bootstrap, KNN, MLP and the SVMs search their
# hyperparameter grid by cross-validation, so the model itself is drawn at random too.
performance (c (KNN, CART, MLP, SVMr), spine [, 1:6], spine [, 8], type = "evaluation",
protocol = "bootstrap", eval = "accuracy", nruns = 100, seed = 0)
#> accuracy
#> KNN 0.8120084
#> CART 0.8065898
#> MLP 0.7923440
#> SVMr 0.8328089
```
**Answer.** *Not one of the four beats the linear SVM, and the best of them stays two points
behind it. So the linear model is the one to prefer here -- and it has the further advantage of
being easier to interpret. A more flexible model is not a better one when the boundary it has
to find is not, in fact, curved.*
A decision tree is more explicit still about what it uses:
``` r
cartplot (CART (spine [, 1:6], spine [, 8]))
```
**Answer.** *The tree keeps only three of the six attributes, and `V6` on its own separates
`SL` from the other two classes.*
