4

I've got a library of 1200 references I'm using for a systematic review. Now I need to download the PDFs of all these references, which will take days if I do it manually. Is there a simple way to automatically download as many as possible from pubmed / Google Scholar / (maybe Scihub)? I have institutional access.

Edit: My solution was to load the reference list into multiple reference managers and run the PDF import function in all of them. Some managers succeeded where others failed. I had to do the rest manually.

Nereus
  • 143
  • 1
  • 5
  • Did you try to ask your librarian? – EarlGrey May 24 '23 at 11:56
  • 2
    If you want to do a systematic review of these 1200 works I suspect you would need to read at the very least the abstract but presumably significant parts of the text in each of them. Relative to that effort, the downloading is completely trivial. – quarague May 24 '23 at 12:43

2 Answers2

4

For systematic review, if you're following PRISMA, you'll typically do some preliminary 'checks' before getting to the lists for full text review.

I'm assuming the 1200 odd references are your final list after the duplicate removal and screening, and perhaps your forward-and-reverse literature chaining.

In Zotero, you can enable automatic PDF download in preference. For your purpose, you then bulk import reference using doi or bibtex or ris.

The trick with Zotero is that Zotero is able to download pdf link to an entry from multiple sources. There are limitations though.
Beware that most academic database would lock you out if performing large bulk downloads at fast rate. Always a good idea to use proxy. In your case, you already having institutional access which might assist.

With Endnote, you can bulk import PDF files or like you have in Mendeley, you can set a 'watch' on a folder from which Endnote will automatically import entries for PDF files added to the folder. Unfortunately, that does not address your challenge.
With Endnote, similar to Zotero, you can import reference list to populate tour Endnote database.

  • simply export reference list from your search (Google, Scopus ...) to a RIS file.

[Technical approach beyond the scope of Academia forum]
For other technical solutions beyond the scope of Academia, you can work directly with API of academic database.

  • Science direct and Scopus provide API access, which you register for for free. You'll still need access to perform low-level tasks and download.
  • you can leverage Python to work with academic database API
  • for Google Scholar, use Scholarly: Scholarly pypi, GitHub
  • for Scopus, pybliometrics is well used.
  • Pyscopus claims to be more friendly. I'm yet to use Pyscopus unlike others. More so, it's inactive since 2019!

There's one I've used recently, just can't recall the name offhand. It allows robust analysis and topic search. I'll update in due course.

[Scientific PDF download]

I'm busy with a fork of Automated Search Helper. A research project by Lech Madeyski team at Wroclaw University of Science and Technology, Poland. I'm yet to upload latest revision which has the

  • pdf downloader working with JSGlue, Jinja2
  • I have it working locally but need some code clean-up and documentation.

NB: with automatic downloaders, beware of captcha and blocking/ban by academic database

SciPDFParser comes across as a good parser of downloaded articles PDF.

semmyk-research
  • 3,560
  • 1
  • 3
  • 23
1

I don't know of any tool to specifically scrape academic databases for pdf's. There may be some obscure program out there on GitHub or a web-crawler that could be repurposed. This question is a bit dated but addresses your problem in more detail and proposes some interesting solutions in that vein.

The easiest off the shelf solution is Endnote. It has a feature that allows for automatic search and retrieval of pdf's. If you have access, it works fairly well. Though it doesn't capture everything. I suspect that there are other reference managers with a similar feature. I don't know of free ones specifically, if that is a concern.

If none of those options are workable for you, consider if it is necessary to download all those pdf's. I'm assuming that you are just beginning to conduct your screening and so you likely don't need the full texts right away. I have conducted a handful of systematic reviews and I have always relied on title and abstract for the initial screen. If I could not make a decision from that info, simply navigating to the original online version was sufficient. Since you already have institutional access, why store them on your local device from the get-go? It would be significantly easier to download and store the papers you flag for further review or inclusion. This may not be right in your case, but it's something to consider if all else fails.

sErISaNo
  • 7,362
  • 13
  • 38