---
title: "Feature selection: crabs"
output: rmarkdown::html_vignette
vignette: >
%\VignetteIndexEntry{Feature selection: crabs}
%\VignetteEngine{knitr::rmarkdown}
%\VignetteEncoding{UTF-8}
---
Filters against a wrapper, on five measurements that are almost perfectly correlated and
two different targets.
The other vignettes are listed by `vignette (package = "fdm2id")`; they use the same
handful of functions on other data, and can be read in any order.
``` r
library (fdm2id)
```
# The data
The `crabs` dataset of the `MASS` package, on *Leptograpsus variegatus* crabs collected at
Fremantle, Australia. Two colour forms, orange and blue, and both sexes. Five measurements were
taken on each individual (in mm): the size of the frontal lobe (`FL`), the rear width (`RW`),
the length of the carapace (`CL`), its width (`CW`) and the depth of the body (`BD`).
There are two targets rather than one: the species (`sp`, `O` or `B`) and the sex (`sex`, `F`
or `M`), with 50 males and 50 females of each species.
``` r
data (crabs, package = "MASS")
summary (crabs)
#> sp sex index FL RW CL
#> B:100 F:100 Min. : 1.0 Min. : 7.20 Min. : 6.50 Min. :14.70
#> O:100 M:100 1st Qu.:13.0 1st Qu.:12.90 1st Qu.:11.00 1st Qu.:27.27
#> Median :25.5 Median :15.55 Median :12.80 Median :32.10
#> Mean :25.5 Mean :15.58 Mean :12.74 Mean :32.11
#> 3rd Qu.:38.0 3rd Qu.:18.05 3rd Qu.:14.30 3rd Qu.:37.23
#> Max. :50.0 Max. :23.10 Max. :20.20 Max. :47.60
#> CW BD
#> Min. :17.10 Min. : 6.10
#> 1st Qu.:31.50 1st Qu.:11.40
#> Median :36.80 Median :13.90
#> Mean :36.41 Mean :14.03
#> 3rd Qu.:42.00 3rd Qu.:16.60
#> Max. :54.60 Max. :21.60
```
``` r
plotdata (crabs [, 4:8], crabs [, 1], type = "pairs")
```
``` r
plotdata (crabs [, 4:8], crabs [, 2], type = "pairs")
```
The five measurements are almost perfectly correlated with each other -- a crab is simply
bigger or smaller -- so most of what any one of them says, the others say too. That is what
makes the *selection* the interesting question here rather than the classifier.
``` r
round (cor (crabs [, 4:8]), 3)
#> FL RW CL CW BD
#> FL 1.000 0.907 0.979 0.965 0.988
#> RW 0.907 1.000 0.893 0.900 0.889
#> CL 0.979 0.893 1.000 0.995 0.983
#> CW 0.965 0.900 0.995 1.000 0.968
#> BD 0.988 0.889 0.983 0.968 1.000
```
# Question 1. Selecting for the species
Using a ranking algorithm and Fisher's index as the univariate criterion, which multivariate
criterion -- the F statistic, mRMR, or a wrapper -- gives the best predictions of the species
with a naive Bayes classifier?
``` r
# The two filters are deterministic; the wrapper is not -- it judges each subset by fitting a
# naive Bayes classifier under a bootstrap, so without 'seed' it can stop at a different
# subset from one run to the next. That is the criterion's own variance, not the data's.
s.fstat1 = selectfeatures (crabs [, 4:8], crabs [, 1], algorithm = "ranking",
unieval = "fisher", multieval = "fstat", seed = 0)
s.mrmr1 = selectfeatures (crabs [, 4:8], crabs [, 1], algorithm = "ranking",
unieval = "fisher", multieval = "mrmr", seed = 0)
s.wrap1 = selectfeatures (crabs [, 4:8], crabs [, 1], algorithm = "ranking",
unieval = "fisher", multieval = "wrapper", wrapmethod = NB,
seed = 0)
s.fstat1
#> Feature selection
#> algorithm : ranking
#> univariate criterion : fisher
#> multivariate criterion : fstat
#> features kept : 1: FL
#> score : 46.99
s.mrmr1
#> Feature selection
#> algorithm : ranking
#> univariate criterion : fisher
#> multivariate criterion : mrmr
#> features kept : 1: FL
#> score : 0.1974
s.wrap1
#> Feature selection
#> algorithm : ranking
#> univariate criterion : fisher
#> multivariate criterion : wrapper
#> features kept : 1: FL
#> score : 0.6779
```
``` r
performance (NB, crabs [, 4:8], crabs [, 1], nruns = 100, seed = 0)
#> accuracy
#> 0.6191188
performance (NB, crabs [, 4:8] [, s.fstat1$selection], crabs [, 1], nruns = 100, seed = 0)
#> accuracy
#> 0.6626326
performance (NB, crabs [, 4:8] [, s.mrmr1$selection], crabs [, 1], nruns = 100, seed = 0)
#> accuracy
#> 0.6626326
performance (NB, crabs [, 4:8] [, s.wrap1$selection], crabs [, 1], nruns = 100, seed = 0)
#> accuracy
#> 0.6626326
```
**Answer.** *All three criteria stop at the same subset and therefore give the same
performance -- which is better than using all five variables. Adding correlated measurements
to a naive Bayes classifier, whose whole assumption is that they are independent, costs
accuracy.*
# Question 2. Selecting for the sex
The same question, for the sex of the crabs.
``` r
s.fstat2 = selectfeatures (crabs [, 4:8], crabs [, 2], algorithm = "ranking",
unieval = "fisher", multieval = "fstat", seed = 0)
s.mrmr2 = selectfeatures (crabs [, 4:8], crabs [, 2], algorithm = "ranking",
unieval = "fisher", multieval = "mrmr", seed = 0)
s.wrap2 = selectfeatures (crabs [, 4:8], crabs [, 2], algorithm = "ranking",
unieval = "fisher", multieval = "wrapper", wrapmethod = NB,
seed = 0)
s.fstat2
#> Feature selection
#> algorithm : ranking
#> univariate criterion : fisher
#> multivariate criterion : fstat
#> features kept : 1: RW
#> score : 18.4
s.mrmr2
#> Feature selection
#> algorithm : ranking
#> univariate criterion : fisher
#> multivariate criterion : mrmr
#> features kept : 1: RW
#> score : 0.1264
s.wrap2
#> Feature selection
#> algorithm : ranking
#> univariate criterion : fisher
#> multivariate criterion : wrapper
#> features kept : 3: RW, CL, BD
#> score : 0.7358
```
``` r
performance (NB, crabs [, 4:8], crabs [, 2], nruns = 100, seed = 0)
#> accuracy
#> 0.6624966
performance (NB, crabs [, 4:8] [, s.fstat2$selection], crabs [, 2], nruns = 100, seed = 0)
#> accuracy
#> 0.6236062
performance (NB, crabs [, 4:8] [, s.mrmr2$selection], crabs [, 2], nruns = 100, seed = 0)
#> accuracy
#> 0.6236062
performance (NB, crabs [, 4:8] [, s.wrap2$selection], crabs [, 2], nruns = 100, seed = 0)
#> accuracy
#> 0.7129453
```
**Answer.** *Only the wrapper improves on the full set of variables. The two filter criteria
stop at a single variable and lose accuracy -- which is the trade-off of the family: a filter
judges a subset without ever fitting the model it is selecting for, and here that judgement is
wrong.*
# Question 3. Which variables, in the end?
``` r
colnames (crabs) [4:8] [s.fstat1$selection]
#> [1] "FL"
colnames (crabs) [4:8] [s.wrap2$selection]
#> [1] "RW" "CL" "BD"
```
**Answer.** *The frontal lobe alone is what distinguishes the two species; distinguishing the
sexes takes three measurements -- the rear width, the length of the carapace and the depth of
the body. Two different questions asked of the same five measurements, and two different
answers.*
