NHGIS API Requests

This vignette details the options available for requesting IPUMS NHGIS data and metadata via the IPUMS API.

If you haven’t yet learned the basics of the IPUMS API workflow, you may want to start with the IPUMS API introduction. The code below assumes you have registered and set up your API key as described there.

In addition to NHGIS, the IPUMS API also supports several microdata projects. For details about obtaining IPUMS microdata using ipumsr, see the microdata-specific vignette.

Before getting started, we’ll load ipumsr and some helpful packages for this demo:

library(ipumsr)
library(dplyr)
library(purrr)

Basic IPUMS NHGIS concepts

IPUMS NHGIS supports 3 main types of data products: datasets, time series tables, and shapefiles.

IPUMS NHGIS metadata

Of course, to make a request for any of these data sources, we have to know the codes that the API uses to refer to them. Fortunately, we can browse the metadata for all available IPUMS NHGIS data sources with get_metadata_nhgis().

Users can view summary metadata for all available data sources of a given data type, or detailed metadata for a specific data source by name.

Summary metadata

To see a summary of all available sources for a given data product type, use the type argument. This returns a data frame containing the available datasets, data tables, time series tables, or shapefiles.

ds <- get_metadata_nhgis(type = "datasets")

head(ds)
#> # A tibble: 6 × 4
#>   name      group       description                              sequence
#>   <chr>     <chr>       <chr>                                       <int>
#> 1 1790_cPop 1790 Census Population Data [US, States & Counties]       101
#> 2 1800_cPop 1800 Census Population Data [US, States & Counties]       201
#> 3 1810_cPop 1810 Census Population Data [US, States & Counties]       301
#> 4 1820_cPop 1820 Census Population Data [US, States & Counties]       401
#> 5 1830_cPop 1830 Census Population Data [US, States & Counties]       501
#> 6 1840_cAg  1840 Census Agriculture Data [US, States & Counties]      601

We can use basic functions from {dplyr} to filter the metadata to those records of interest. For instance, if we wanted to find all the data sources related to agriculture from the 1900 Census, we could filter on group and description:

ds %>%
  filter(
    group == "1900 Census",
    grepl("Agriculture", description)
  )
#> # A tibble: 2 × 4
#>   name       group       description                                    sequence
#>   <chr>      <chr>       <chr>                                             <int>
#> 1 1900_cAg   1900 Census Agriculture Data [US, States & Counties]           1401
#> 2 1900_cPHAM 1900 Census Population, Housing, Agriculture & Manufactur…     1403

The values listed in the name column correspond to the code that you would use to request that dataset when creating an extract definition to be submitted to the IPUMS API.

Similarly, for time series tables:

tst <- get_metadata_nhgis("time_series_tables")

While some of the metadata fields are consistent across different data types, some, like geographic_integration, are specific to time series tables:

head(tst)
#> # A tibble: 6 × 7
#>   name  description         geographic_integration sequence time_series years   
#>   <chr> <chr>               <chr>                     <dbl> <list>      <list>  
#> 1 A00   Total Population    Nominal                    100. <tibble>    <tibble>
#> 2 AV0   Total Population    Nominal                    100. <tibble>    <tibble>
#> 3 B78   Total Population    Nominal                    100. <tibble>    <tibble>
#> 4 CL8   Total Population    Standardized to 2010       100. <tibble>    <tibble>
#> 5 A57   Persons by Urban/R… Nominal                    101. <tibble>    <tibble>
#> 6 A59   Persons by Urban/R… Nominal                    101. <tibble>    <tibble>
#> # ℹ 1 more variable: geog_levels <list>

Note that for time series tables, some metadata fields are stored in list columns, where each entry is itself a data frame:

tst$years[[1]]
#> # A tibble: 24 × 3
#>    name  description sequence
#>    <chr> <chr>          <int>
#>  1 1790  1790               1
#>  2 1800  1800               2
#>  3 1810  1810               3
#>  4 1820  1820               4
#>  5 1830  1830               5
#>  6 1840  1840               6
#>  7 1850  1850               7
#>  8 1860  1860               8
#>  9 1870  1870              12
#> 10 1880  1880              22
#> # ℹ 14 more rows
tst$geog_levels[[1]]
#> # A tibble: 2 × 3
#>   name   description   sequence
#>   <chr>  <chr>            <int>
#> 1 state  State                4
#> 2 county State--County       25

To filter on these columns, we can use map_lgl() from {purrr}. For instance, to find all time series tables that include data from a particular year:

# Iterate over each `years` entry, identifying whether that entry
# contains "1840" in its `name` column.
tst %>%
  filter(map_lgl(years, ~ "1840" %in% .x$name))
#> # A tibble: 2 × 7
#>   name  description        geographic_integration sequence time_series years   
#>   <chr> <chr>              <chr>                     <dbl> <list>      <list>  
#> 1 A00   Total Population   Nominal                    100. <tibble>    <tibble>
#> 2 A08   Persons by Sex [2] Nominal                    102. <tibble>    <tibble>
#> # ℹ 1 more variable: geog_levels <list>

For more details on working with nested data frames, see this tidyr article.

Detailed metadata

Once we have identified a data source of interest, we can find out more about its detailed options by providing its name to the corresponding argument of get_metadata_nhgis():

cAg_meta <- get_metadata_nhgis(dataset = "1900_cAg")

This provides a comprehensive list of the possible specifications for the input data source. For instance, for the 1900_cAg dataset, we have 66 tables to choose from, and 3 possible geographic levels:

cAg_meta$data_tables
#> # A tibble: 66 × 7
#>    name  description       universe nhgis_code sequence dataset_name n_variables
#>    <chr> <chr>             <chr>    <chr>         <int> <chr>              <int>
#>  1 NT1   Total Population  Persons  AWS               1 1900_cAg               1
#>  2 NT2   Number of Farms   Farms    AW3               2 1900_cAg               1
#>  3 NT3   Average Farm Size Farms    AXE               3 1900_cAg               1
#>  4 NT4   Farm Acreage      Farms    AXP               4 1900_cAg              10
#>  5 NT5   Farm Management   Farms    AXZ               5 1900_cAg               3
#>  6 NT6   Race of Farmer    Farms    AYA               6 1900_cAg               2
#>  7 NT7   Race of Farmer b… Farms    AYJ               7 1900_cAg              12
#>  8 NT8   Number of Farms   Farms    AYK               8 1900_cAg               1
#>  9 NT9   Farms with Build… Farms w… AYL               9 1900_cAg               1
#> 10 NT10  Acres of Farmland Farms    AWT              10 1900_cAg               1
#> # ℹ 56 more rows
cAg_meta$geog_levels
#> # A tibble: 3 × 4
#>   name   description   has_geog_extent_selection sequence
#>   <chr>  <chr>         <lgl>                        <int>
#> 1 nation Nation        FALSE                            1
#> 2 state  State         FALSE                            4
#> 3 county State--County FALSE                           25

You can also get detailed metadata for an individual data table. Since data tables belong to specific datasets, both need to be specified to identify a data table:

get_metadata_nhgis(dataset = "1900_cAg", data_table = "NT2")
#> $name
#> [1] "NT2"
#> 
#> $description
#> [1] "Number of Farms"
#> 
#> $universe
#> [1] "Farms"
#> 
#> $nhgis_code
#> [1] "AW3"
#> 
#> $sequence
#> [1] 2
#> 
#> $dataset_name
#> [1] "1900_cAg"
#> 
#> $variables
#> # A tibble: 1 × 2
#>   description nhgis_code
#>   <chr>       <chr>     
#> 1 Total       AW3001

Note that the name element is the one that contains the codes used for interacting with the IPUMS API. The nhgis_code element refers to the prefix attached to individual variables in the output data, and the API will throw an error if you use it in an extract definition. For more details on interpreting each of the provided metadata elements, see the documentation for get_metadata_nhgis().

Now that we have identified some of our options, we can go ahead and define an extract request to submit to the IPUMS API.

Defining an IPUMS NHGIS extract request

To create an extract definition containing the specifications for a specific set of IPUMS NHGIS data, use define_extract_nhgis().

When you define an extract request, you can specify the data to be included in the extract and indicate the desired format and layout.

Basic extract definitions

Let’s say we’re interested in getting state-level data on the number of farms and their average size from the 1900_cAg dataset that we identified above. As we can see in the metadata, these data are contained in tables NT2 and NT3:

cAg_meta$data_tables
#> # A tibble: 66 × 7
#>    name  description       universe nhgis_code sequence dataset_name n_variables
#>    <chr> <chr>             <chr>    <chr>         <int> <chr>              <int>
#>  1 NT1   Total Population  Persons  AWS               1 1900_cAg               1
#>  2 NT2   Number of Farms   Farms    AW3               2 1900_cAg               1
#>  3 NT3   Average Farm Size Farms    AXE               3 1900_cAg               1
#>  4 NT4   Farm Acreage      Farms    AXP               4 1900_cAg              10
#>  5 NT5   Farm Management   Farms    AXZ               5 1900_cAg               3
#>  6 NT6   Race of Farmer    Farms    AYA               6 1900_cAg               2
#>  7 NT7   Race of Farmer b… Farms    AYJ               7 1900_cAg              12
#>  8 NT8   Number of Farms   Farms    AYK               8 1900_cAg               1
#>  9 NT9   Farms with Build… Farms w… AYL               9 1900_cAg               1
#> 10 NT10  Acres of Farmland Farms    AWT              10 1900_cAg               1
#> # ℹ 56 more rows

Dataset specifications

To request these data, we need to make an explicit dataset specification. All datasets must be associated with a selection of data tables and geographic levels. We can use the ds_spec() helper function to specify our selections for these parameters. ds_spec() bundles all the selections for a given dataset together into a single object (in this case, a ds_spec object):

dataset <- ds_spec(
  "1900_cAg",
  data_tables = c("NT1", "NT2"),
  geog_levels = "state"
)

str(dataset)
#> List of 3
#>  $ name       : chr "1900_cAg"
#>  $ data_tables: chr [1:2] "NT1" "NT2"
#>  $ geog_levels: chr "state"
#>  - attr(*, "class")= chr [1:3] "ds_spec" "ipums_spec" "list"

This dataset specification can then be provided to the extract definition:

nhgis_ext <- define_extract_nhgis(
  description = "Example farm data in 1900",
  datasets = dataset
)

nhgis_ext
#> Unsubmitted IPUMS NHGIS extract 
#> Description: Example farm data in 1900
#> 
#> Dataset: 1900_cAg
#>   Tables: NT1, NT2
#>   Geog Levels: state

Dataset specifications can also include selections for years and breakdown_values, but these are not available for all datasets.

Time series table specifications

Similarly, to make a request for time series tables, use the tst_spec() helper. This makes a tst_spec object containing a time series table specification.

Time series tables do not contain individual data tables, but do require a geographic level selection, and allow an optional selection of years:

define_extract_nhgis(
  description = "Example time series table request",
  time_series_tables = tst_spec(
    "CW3",
    geog_levels = c("county", "tract"),
    years = c("1990", "2000")
  )
)
#> Unsubmitted IPUMS NHGIS extract 
#> Description: Example time series table request
#> 
#> Time Series Table: CW3
#>   Geog Levels: county, tract
#>   Years: 1990, 2000

Shapefile specifications

Shapefiles don’t have any additional specification options, and therefore can be requested simply by providing their names:

define_extract_nhgis(
  description = "Example shapefiles request",
  shapefiles = c("us_county_2021_tl2021", "us_county_2020_tl2020")
)
#> Unsubmitted IPUMS NHGIS extract 
#> Description: Example shapefiles request
#> 
#> Shapefiles: us_county_2021_tl2021, us_county_2020_tl2020

Invalid specifications

An attempt to define an extract that does not have all the required specifications for a given dataset or time series table will throw an error:

define_extract_nhgis(
  description = "Invalid extract",
  datasets = ds_spec("1900_STF1", data_tables = "NP1")
)
#> Error in `validate_ipums_extract()`:
#> ! Invalid `ds_spec` specification:
#> ✖ `geog_levels` must not contain missing values.

Note that it is still possible to make invalid extract requests (for instance, by requesting a dataset or data table that doesn’t exist). This kind of issue will be caught upon submission to the API, not upon the creation of the extract definition.

More complicated extract definitions

It’s possible to request data for multiple datasets (or time series tables) in a single extract definition. To do so, pass a list of ds_spec or tst_spec objects in define_extract_nhgis():

define_extract_nhgis(
  description = "Slightly more complicated extract request",
  datasets = list(
    ds_spec("2018_ACS1", "B01001", "state"),
    ds_spec("2019_ACS1", "B01001", "state")
  ),
  shapefiles = c("us_state_2018_tl2018", "us_state_2019_tl2019")
)
#> Unsubmitted IPUMS NHGIS extract 
#> Description: Slightly more complicated extract request
#> 
#> Dataset: 2018_ACS1
#>   Tables: B01001
#>   Geog Levels: state
#> 
#> Dataset: 2019_ACS1
#>   Tables: B01001
#>   Geog Levels: state
#> 
#> Shapefiles: us_state_2018_tl2018, us_state_2019_tl2019

For extracts with multiple datasets or time series tables, it may be easier to generate the specifications independently before creating your extract request object. You can quickly create multiple ds_spec objects by iterating across the specifications you want to include. Here, we use {purrr} to do so, but you could also use a for loop:

ds_names <- c("2019_ACS1", "2018_ACS1")
tables <- c("B01001", "B01002")
geogs <- c("county", "state")

# For each dataset to include, create a specification with the
# data tabels and geog levels indicated above
datasets <- purrr::map(
  ds_names,
  ~ ds_spec(name = .x, data_tables = tables, geog_levels = geogs)
)

nhgis_ext <- define_extract_nhgis(
  description = "Slightly more complicated extract request",
  datasets = datasets
)

nhgis_ext
#> Unsubmitted IPUMS NHGIS extract 
#> Description: Slightly more complicated extract request
#> 
#> Dataset: 2019_ACS1
#>   Tables: B01001, B01002
#>   Geog Levels: county, state
#> 
#> Dataset: 2018_ACS1
#>   Tables: B01001, B01002
#>   Geog Levels: county, state

This workflow also makes it easy to quickly update the specifications in the future. For instance, to add the 2017 ACS 1-year data to the extract definition above, you’d only need to add "2017_ACS1" to the ds_names variable. The iteration would automatically add the selected tables and geog levels for the new dataset. (This workflow works particularly well for ACS datasets, which often have the same data table names across datasets.)

Data layout and file format

IPUMS NHGIS extract definitions also support additional options to modify the layout and format of the extract’s resulting data files.

For extracts that contain time series tables, the tst_layout argument indicates how the longitudinal data should be organized.

For extracts that contain datasets with multiple breakdowns or data types, use the breakdown_and_data_type_layout argument to specify a layout . This is most common for data sources that contain both estimates and margins of error, like the ACS.

File formats can be specified with the data_format argument. IPUMS NHGIS currently distributes files in csv and fixed-width format.

See the documentation for define_extract_nhgis() for more details on these options.

Next steps

Once you have defined an extract request, you can submit the extract for processing:

nhgis_ext_submitted <- submit_extract(nhgis_ext)

The workflow for submitting and monitoring an extract request and downloading its files when complete is described in the IPUMS API introduction.