---
title: "Quality filtering"
author: "Martin Westgate"
date: '2026-11-20'
output:
  rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Quality filtering}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---



### What is 'data quality'?

Databases containing biodiversity information - such as those that can be queried using 
`galah` - contain millions of records collected over decades or centuries by thousands of 
observers. It is unsurprising, therefore, that they contain some errors.
In this context, two key questions for users of that data are:

- How large a problem are those errors for their use case? and
- What tools exist to help us solve them?

Data quality systems in galah are built on a set of fields called 'assertions'.
These are usually binary (true/false) variables that flag that something might be wrong 
with the data, generated either programmatically or by users. You can see them by calling 
`show_all_assertions()`:


```
## # A tibble: 124 × 4
##    id                                 description                        category type      
##    <chr>                              <chr>                              <chr>    <chr>     
##  1 AMBIGUOUS_COLLECTION               Ambiguous collection               Comment  assertions
##  2 AMBIGUOUS_INSTITUTION              Ambiguous institution              Comment  assertions
##  3 BASIS_OF_RECORD_INVALID            Basis of record badly formed       Warning  assertions
##  4 biosecurityIssue                   Biosecurity issue                  Error    assertions
##  5 COLLECTION_MATCH_FUZZY             Collection match fuzzy             Comment  assertions
##  6 COLLECTION_MATCH_NONE              Collection not matched             Comment  assertions
##  7 CONTINENT_COORDINATE_MISMATCH      continent coordinate mismatch      Warning  assertions
##  8 CONTINENT_COUNTRY_MISMATCH         Continent country mismatch         Warning  assertions
##  9 CONTINENT_DERIVED_FROM_COORDINATES Continent derived from coordinates Warning  assertions
## 10 CONTINENT_DERIVED_FROM_COUNTRY     continent derived from country     Warning  assertions
## # ℹ 114 more rows
```

The easiest thing to do with these is simply download them with your chosen records, and 
do your own checking on the data. This can be achived via `select()`:


``` r
galah_call() |>
  identify("Eolophus roseicapilla") |>
  filter(year >= 2020,
         basisOfRecord == "PRESERVED_SPECIMEN") |>
  select(group = c("basic", "assertions")) |>
  collect()
```

```
## Retrying in 1 seconds.
## Retrying in 2 seconds.
## Retrying in 4 seconds.
## Retrying in 8 seconds.
```

```
## # A tibble: 42 × 133
##    recordID              scientificName taxonConceptID decimalLatitude decimalLongitude eventDate           basisOfRecord occurrenceStatus dataResourceName AMBIGUOUS_COLLECTION AMBIGUOUS_INSTITUTION
##    <chr>                 <chr>          <chr>                    <dbl>            <dbl> <dttm>              <chr>         <chr>            <chr>            <lgl>                <lgl>                
##  1 03e1c487-f3be-4469-a… Eolophus rose… https://biodi…           -34.8             148. 2022-01-14 00:00:00 PRESERVED_SP… PRESENT          Australian Nati… FALSE                FALSE                
##  2 0a24639a-1387-4a53-b… Eolophus rose… https://biodi…           -34.8             148. 2022-01-14 00:00:00 PRESERVED_SP… PRESENT          Australian Nati… FALSE                FALSE                
##  3 0eaf52fc-6659-41ea-8… Eolophus rose… https://biodi…           -24.2             139. 2026-08-26 00:00:00 PRESERVED_SP… PRESENT          Australian Nati… FALSE                FALSE                
##  4 10081d36-abf8-4928-9… Eolophus rose… https://biodi…           -33.8             151. 2024-12-11 00:00:00 PRESERVED_SP… PRESENT          Australian Muse… FALSE                FALSE                
##  5 10e9d3c3-1fb2-4362-8… Eolophus rose… https://biodi…           -33.6             151. NA                  PRESERVED_SP… PRESENT          Australian Muse… FALSE                FALSE                
##  6 11663284-5f43-4cf2-a… Eolophus rose… https://biodi…           -28.2             144. 2025-09-04 00:00:00 PRESERVED_SP… PRESENT          Australian Nati… FALSE                FALSE                
##  7 1baea6d4-a2d1-4e69-8… Eolophus rose… https://biodi…           -36.4             144. 2024-09-21 00:00:00 PRESERVED_SP… PRESENT          Museums Victori… FALSE                FALSE                
##  8 1fe45f99-a245-44d7-b… Eolophus rose… https://biodi…           -28.3             136. 2025-08-02 00:00:00 PRESERVED_SP… PRESENT          Australian Nati… FALSE                FALSE                
##  9 3810878b-8ad7-4a86-a… Eolophus rose… https://biodi…           -32.6             150. 2022-05-14 00:00:00 PRESERVED_SP… PRESENT          Australian Muse… FALSE                FALSE                
## 10 3e6152db-aef1-4e32-b… Eolophus rose… https://biodi…           -30.1             138. 2025-08-01 00:00:00 PRESERVED_SP… PRESENT          Australian Nati… FALSE                FALSE                
## # ℹ 32 more rows
## # ℹ 122 more variables: BASIS_OF_RECORD_INVALID <lgl>, biosecurityIssue <lgl>, COLLECTION_MATCH_FUZZY <lgl>, COLLECTION_MATCH_NONE <lgl>, CONTINENT_COORDINATE_MISMATCH <lgl>,
## #   CONTINENT_COUNTRY_MISMATCH <lgl>, CONTINENT_DERIVED_FROM_COORDINATES <lgl>, CONTINENT_DERIVED_FROM_COUNTRY <lgl>, CONTINENT_INVALID <lgl>, COORDINATE_INVALID <lgl>,
## #   COORDINATE_OUT_OF_RANGE <lgl>, COORDINATE_PRECISION_INVALID <lgl>, COORDINATE_REPROJECTED <lgl>, COORDINATE_REPROJECTION_FAILED <lgl>, COORDINATE_REPROJECTION_SUSPICIOUS <lgl>,
## #   COORDINATE_ROUNDED <lgl>, COORDINATE_UNCERTAINTY_METERS_INVALID <lgl>, COORDINATES_CENTRE_OF_COUNTRY <lgl>, COORDINATES_CENTRE_OF_STATEPROVINCE <lgl>, COUNTRY_COORDINATE_MISMATCH <lgl>,
## #   COUNTRY_DERIVED_FROM_COORDINATES <lgl>, COUNTRY_INVALID <lgl>, COUNTRY_MISMATCH <lgl>, DEPTH_MIN_MAX_SWAPPED <lgl>, DEPTH_NON_NUMERIC <lgl>, DEPTH_NOT_METRIC <lgl>, DEPTH_UNLIKELY <lgl>,
## #   detectedOutlier <lgl>, DIFFERENT_OWNER_INSTITUTION <lgl>, ELEVATION_MIN_MAX_SWAPPED <lgl>, ELEVATION_NON_NUMERIC <lgl>, ELEVATION_NOT_METRIC <lgl>, ELEVATION_UNLIKELY <lgl>, …
```

As you can see, there are a huge number of assertions, and they will take some processing.
It is also inefficient to download data we don't need, only to filter it later. Therefore,
it is worth considering filtering out some problematic records as part of our usual filtering
strategy.

### Data quality profiles

Data quality 'profiles' are pre-populated filter statements curated by the ALA that are 
designed to remove problematic records. They provide a quick and straightforward way to 
remove these records without having to construct elaborate filter statements.


``` r
show_all_profiles()
```

```
## # A tibble: 5 × 4
##      id short_name name                                  description                                                                                                                                  
##   <int> <chr>      <chr>                                 <chr>                                                                                                                                        
## 1    92 ALA        ALA General                           "The default ALA profile filters out records based on the filter groups outlined below.  This is a moderately restricted set of data."       
## 2   124 CSDM       Species Distribution Modelling (CSDM) "Base filters for the Collaborative Species Distribution Modelling program"                                                                  
## 3   133 re-usable  Data licensed for all uses            "Data licensed for re-use, including commercial uses. This profile only filters on license, no \"quality\" filters are applied."             
## 4   224 AVH        AVH                                   "AVH data quality profile"                                                                                                                   
## 5   252 ALA-test   ALA General Test                      "A test version of the default ALA profile filters out records based on the filter groups outlined below.  This is a moderately restricted s…
```

Let's take a look at the most obvious one first:


``` r
search_all(profiles, "CSDM") |>
  show_values()
```

```
## • Showing values for 'CSDM'.
```

```
## # A tibble: 4 × 4
##      id description                                                               filter                        enabled
##   <int> <chr>                                                                     <chr>                         <lgl>  
## 1   126 "Include only records where Spatial validity is \"true\""                 "spatiallyValid:\"true\""     TRUE   
## 2   128 "Exclude potential duplicate records"                                     "-isDuplicateOf:*"            TRUE   
## 3   130 "Exclude all records that are an outlier against any environmental layer" "-outlierLayerCount:[1 TO *]" TRUE   
## 4   132 "Include only records where Year is 1970 to 2099"                         "year:[1970 TO *]"            TRUE
```

This is ok, but needs unpacking: 

 - `spatiallyValid` is an amalgam of other assertions. At the time of writing, `true` means that 
 the latitude and longitude are known and within +/- 90 and 180 degrees (respectively), but are not 
 0,0, *and* that there isn't a mismatch between the observed country and the country listed by the 
 data provider. 
- Excluding potentially duplicate records is fairly clear and sensible, though is also easily achieved
using thinning.
- Excluding points that are an outlier against any environmental layer is a high penalty, particularly 
for rare species. This is likely to falsely exclude records on a frequent basis. 
- The choice to include records since 1970 is fine, albeit clearly context-specific. 

Now let's look at the 'ALA general' approach:


``` r
 search_all(profiles, "ALA") |>
  show_values()
```

```
## ! Search returned 2 matched profiles.
```

```
## • Showing values for 'ALA'.
```

```
## # A tibble: 26 × 4
##       id description                                                                                                                                                                    filter enabled
##    <int> <chr>                                                                                                                                                                          <chr>  <lgl>  
##  1    94 "Exclude all records where spatial validity is \"false\""                                                                                                                      "-spa… TRUE   
##  2    96 "Exclude all records with an assertion that the scientific name provided does not match any of the names lists used by the ALA.  For a full explanation of the ALA name match… "-ass… TRUE   
##  3    97 "Exclude all records with an assertion that the scientific name provided is not structured as a valid scientific name. Also catches rank values or values such as \"UNKNOWN\"" "-ass… TRUE   
##  4    99 "Exclude all records with an assertion that kingdom provided doesn't match a known kingdom e.g. Animalia, Plantae"                                                             "-ass… TRUE   
##  5   100 "Exclude all records with an assertion that the scientific name provided in the record does not match the expected taxonomic scope of the resource e.g. Mammal records attrib… "-ass… TRUE   
##  6    98 "Exclude all records with an assertion that the name and classification supplied can't be used to choose between 2 homonyms"                                                   "-ass… TRUE   
##  7   103 "Exclude all records with an assertion of latitude value provided is zero"                                                                                                     "-dec… TRUE   
##  8   104 "Exclude all records with an assertion of longitude value provided is zero"                                                                                                    "-dec… TRUE   
##  9   105 "Exclude all records with an assertion of  latitude and longitude have been transposed"                                                                                        "-ass… TRUE   
## 10   106 "Exclude all records with an assertion of coordinates are the exact centre of the state or territory"                                                                          "-ass… TRUE   
## # ℹ 16 more rows
```

There is a lot here and we won't go through it all; but two points are potentially useful.
First, this makes a lot of assumptions about what is and is not useful, with 26 separate
filters. This is a lot, and while most of them are useful in some contexts, equally this 
may be too stringent a set of tests for some applications. Applying it to our earlier example,
we see it only removes two records, but your milage will vary for different queries:


``` r
galah_call() |>
  identify("Eolophus roseicapilla") |>
  filter(year >= 2020,
         basisOfRecord == "PRESERVED_SPECIMEN") |>
  apply_profile("ALA") |>
  count() |>
  collect()
```

```
## # A tibble: 1 × 1
##   count
##   <int>
## 1    40
```

Second, the `filter` column lists how the query is sent to the ALA; this can be useful 
for replicating parts of the workflow that are useful for your specific application.
We'll try this out below.

### Bespoke filters using assertions

Assertions are stored in what is called a 'multi-value field', meaning 
a column where each cell can contain multiple strings. You remove them using the 'assertions'
field name in filter:


``` r
galah_call() |>
  filter(assertions != "INVALID_SCIENTIFIC_NAME") |>
  count() |>
  collect()
```

```
## # A tibble: 1 × 1
##       count
##       <int>
## 1 195446531
```

Conversely, if you *only* want records with an invalid name, just use `==`:


``` r
galah_call() |>
  filter(assertions == "INVALID_SCIENTIFIC_NAME") |>
  count() |>
  collect()
```

```
## # A tibble: 1 × 1
##     count
##     <int>
## 1 3620270
```

Of course, we might want to filter out multiple assertions at once. Rather than 
keep writing `assertions !=`, we can use `%in%` with a leading `!` (meaning `NOT`):


``` r
excluded_assertions <- c("INVALID_SCIENTIFIC_NAME", 
                         "COORDINATES_CENTRE_OF_STATEPROVINCE")
galah_call() |>
  filter(!(assertions %in% excluded_assertions)) |>
  count() |>
  collect()
```

```
## # A tibble: 1 × 1
##       count
##       <int>
## 1 195442453
```

It is important to note, however, that a lot of things we want to filter
out are properties of the data itself, rather than results of checks the 
organisation has run. For example, the following parameters all 
involve fields (not assertions), but are included in the general filter:


``` r
galah_call() |>
  filter(basisOfRecord != "FOSSIL_SPECIMEN", # no fossils
         contentTypes != "Environmental DNA",
         decimalLatitude != 0,   # no points defaulting to zero-zero
         decimalLongitude != 0,
         occurrenceStatus != "ABSENT", # exclude observations of missingness
         year > 1700,                   # remove very early records
         -(degreeOfEstablishment %in% c("cultivated", "captive")),  # introduced species
         -(identificationVerificationStatus %in% c("verification required", "needs_id")) # uncertain cit sci records
        ) |>
  count() |>
  collect()
```

```
## # A tibble: 1 × 1
##       count
##       <int>
## 1 171820807
```


### User assertions

A complementary approach is to call 'user assertions'; flags placed by users on individual 
suspect records. You can find records flagged in this way as follows:


``` r
galah_call() |>
  count(userAssertions) |>
  collect()
```

```
## # A tibble: 4 × 3
##   userAssertions userAssertions_label count
##   <chr>          <chr>                <int>
## 1 50005          Unconfirmed          11282
## 2 50003          Corrected             3994
## 3 50002          Verified               462
## 4 50001          Open issue              95
```

The categories here are a little obscure, but cross-referencing with the ALA data quality profile,
it appears that only issues that are 'Open' (code `50001`) or 'Unconfirmed' (code `50005`) have some
likelihood of being 'true' errors. Another point to consider is *who* is making these assertions:


``` r
galah_call() |>
  filter(userAssertions %in% c("50001", "50005")) |>
  count(assertionUserId) |>
  slice_head(n = 10) |>
  collect()
```

```
## Limiting to first 10 of 1,382 rows.
## Use `atlas_counts(limit = )` to return more rows.
```

```
## # A tibble: 10 × 3
##    assertionUserId assertionUserId_label count
##    <chr>           <chr>                 <int>
##  1 57159           Thomas Mesaglio        1463
##  2 124186          Cameron Slatyer        1414
##  3 2126            Niels Klazenga         1059
##  4 3334            Penelope Mills          320
##  5 8902            ralph FOSTER            235
##  6 11936           Kay Durham              218
##  7 1108            Allen Sundholm          204
##  8 29516           Wendy Potts             187
##  9 25919           Matthew Connors         177
## 10 109408          Ben Travaglini          164
```

This isn't actually terribly informative unless we know who these people are. But it likely
that the types of errors flagged by a single user will reflect their spatial and taxonomic
knowledge of the Australian biota. To remove any records with user assertions, use `is.na(userAssertions)` within your filter 
statement:


``` r
galah_call() |>
  filter(is.na(userAssertions)) |>
  count() |>
  collect()
```

```
## # A tibble: 1 × 1
##       count
##       <int>
## 1 199050968
```

### Concluding remarks

Data quality is an ongoing topic of investigation for researchers and data infrastructures.
The tools described above are the current best available toolset implemented at the 
institutional level. For a more general assessment of how to handle data quality issues in 
R, see [Cleaning Biodiversity Data in R](https://cleaning-data-r.ala.org.au) by 
Dax Kellie et al. (2026, https://doi.org/10.54102/ala.77009).

A critical limitation of data quality assertions and their derivatives is that they focus 
exclusively on rule-based methods; there is no probabilistic component to them. Users seeking 
to draw probabilistic inference from these data will need to consider carefully whether there
are classes of outlier relevant to their use case, but which are invisible to the systems in 
question.
