Downloading large chunks of Pubmed abstracts using easyPubMed in R

Question

I am trying to download large chunks of abstracts via the easyPubMed package in R.

Here is my code:

out.A <- easyPubMed::batch_pubmed_download(pubmed_query_string = "ineichen bv", 
                               format = "medline", 
                               api_key = "myAPIkey", 
                               batch_size = 50, 
                               res_cn = 1, 
                               dest_file_prefix = "Test3", 
                               encoding = "ASCII")

From a quick pubmed search in the gui, I know that this search should return around 71'000 references. I tried changing the parameters with smaller and larger batch sizes and starting from different batches by changing the argument res_cn but I keep getting the same error:

Killing the request! Something is not working. Please, try again later [1] "PubMed data batch 1 / 71 downloaded..." Data retrieval error. Retrying...

I did not find anything about that in the documentation. I have two questions:

1. Does anyone know what the problem is? Is it related to the download limit set by PubMed?
2. If the problem is caused by the download limit of Pubmed, how can I download large chunks of abstracts? (few mio).

Answer 1

The current CRAN version of easyPubMed has a 10,000 record limit from Entrez. I have experienced this a lot and the workaround is to cut the queries into small chunks until you get under the 10,000 limit.

The old package readme on Github stated the following:

At this moment, easyPubMed only supports retrieving 10,000 records per query. This is due to some recent changes in the NCBI E-utilities (see: https://ncbiinsights.ncbi.nlm.nih.gov/2022/09/13/updated-pubmed-eutilities/). This is a known issue/limitation. I'll try to re-write the R library to account for the changes in the E-utilities as soon as possible (likely, within a few months). Thanks for your patience.

The new readme states,

New features of easyPubMed version 3.1.3

Automatic Job splitting into Sub-Queries. The Entrez server imposes a strict n=10,000 limit to the number of records that can be programmatically retrieved from a single query. Whenever possible, the easyPubMed library automatically attempts to split queries returning large number of records into lists of smaller, manageable queries.

So either change your query to ask for less than 10k records, or download the development version of easyPubMed.