How do I import data in R using HTTP requests from publicly accessible API that returns JSON data, when the server doesn't have an SSL certificate

Question

I was hoping someone could help me with an issue I'm facing with data collection.

CONTEXT

The U.S. Cluster Mapping Project has built an API to provide public, developer-friendly access to the curated and continuously updated data archive on the U.S. Cluster Mapping website. The API is publicly accessible using HTTP requests and returns JSON data. The base URL of the API is: http://clustermapping.us/data

Then you can narrow the specificity of the data by adding /TypeOfRegion/RegionID. Here is an example of the JSON data for a specific county. For example: https://clustermapping.us/data/region/county/48321

I would like to import their data into R.

NOTE: However this server (USCMP Website) could not prove that it is clustermapping.us; its security certificate expired 87 days ago.

This seems to be preventing me from importing the data in R.

I've used the following code to try import the data into R.

First I tried:

lnk <- readLines('https://clustermapping.us/data/region/county/48321')
jsonlite::fromJSON(sprintf('[%s]', gsub('}{', '},{', lnk, fixed = TRUE)))

Which always returned the following error:

Error in file(con, "r") : 
  cannot open the connection to 'https://clustermapping.us/data/region/county/48321'
In addition: Warning message:
In file(con, "r") :
  URL 'https://clustermapping.us/data/region/county/48321': status was 'SSL connect error'

Then I tried:

base_url <- "http://clustermapping.us/data/region/county/48321"
response <- GET(base_url, config = list(ssl_verifypeer = FALSE, ssl_verifyhost= FALSE ))

Which always returned the following error:

Error in curl::curl_fetch_memory(url, handle = handle) : 
  schannel: next InitializeSecurityContext failed: SEC_E_CERT_EXPIRED (0x80090328) - The received certificate has expired.

I had added ssl_verifypeer = FALSE,ssl_verifyhost= FALSE because I kept getting this error message due to the SSL certificate associated with the web site being expired, which is causing the HTTPS request to fail. From what I gather, adding this is supposed to allow me to make the request using HTTP instead of HTTPS. However, it made no difference.

I am unsure how to proceed. Would greatly appreciate your input on how I might address this issue.

P.S. I am very much a beginner when it comes to this.

Answer 1

You want to use the config() function to pass those values. Your call should look like this:

base_url <- "https://clustermapping.us/data/region/county/48321"
response <- GET(base_url, config(ssl_verifyhost=0, ssl_verifypeer=0))
httr::content(response)

Notice how we pass in the results of the config() function rather than passing a config= parameter with a list of values.

If you wanted to pass a list to config=, you need to nest the the values in the config options. For example

response <- GET(base_url, config=list(options=list(
  ssl_verifyhost=0, ssl_verifypeer=0
)))