Academic writing with Markdown, Visual Studio Code and Zotero

Academic researchers devote a significant amount of time to writing papers to report their findings. As this is an important aspect of their job, they need to use a document preparation system (a word processor) that they are comfortable working with. Another important aspect of scientific writing is to contextualize their work within the current body of knowledge. This involves citing other relevant scientific papers. To manage and organize all these references effectively, they use a reference manager system. In this post, I will detail my current setup for academic writing that revolves around two pieces of software (with ad hoc extensions): Zotero and Visual Studio Code (VS Code).

Zotero

I’ve been using a reference manager for 12 years now, starting with Mendeley. I later decided to switch to Zotero. That was about 5 years ago. The decision to switch was long overdue, as I wanted to switch since 2013 when Elsevier bought Mendeley (Jordan, 2019).

In 2013, Elsevier bought Mendeley, primarily a bibliographic tool but also an ASNS [Academic Social Network Sites — ed.] (Shaw, 2013) as part of their strategy to acquire a wide range of scholarly online tools and establish themselves as a platform (Robertson, 2018). However, the business model of ASNS and their intentions to monetise the networks remains opaque.

With hindsight, I am glad I opted for Zotero and here are the reasons why:

• Zotero is free;
• Zotero is open source https://github.com/zotero;
• Zotero is led by a non-profit organization: (the Corporation for Digital Scholarship);
• Zotero is developed by a global community;
• Zotero is thoroughly documented for users (e.g. the quick start guide) and for developers (see the documentation);
• Zotero is easy to use and has all the features that make a great reference manager system:
  • it stores your PDF in a dedicated folder (/storage) and stores all metadata (author, title, publisher, etc.) in a sqlite database (see ‘Zotero data directory’) that are both easy to backup,
  • it can retrieve documents metadata from identifiers (DOI ISBN, DOI, PubMed ID, or arXiv ID, see ‘Adding Items to Zotero’) and directly from PDF metadata,
  • it makes the organization of references easy with collections and tags,
  • it has great interfaces to search items and to handle duplicates,
  • it makes it easy to cite specific papers (basically CTRL+SHIFT+C) or to create/export bibliographies (e.g. using the context menu),
  • it supports a large array of citation format (available as CSL files).
  • it offers neat plugins to seamlessly add reference directly from word processors (e.g the Zotero LibreOffice Plugin),
  • it has great add-ons (such as Better Bibtex that I introduce below).

The only Mendeley’s feature I have missed so far is the search in body text of the entire collection of PDF (very useful for non-native english speakers). That said, as all PDF files are actually stored in one place, I simply created an alias using the grep command line:

1

$ alias szot='grep -rnw ~/Zotero/storage/ -e'

which fulfills my need! This command should work on most of Unix-like operating system and I am pretty sure there are simple ways to do this on Windows.

To conclude this first section, I’d like to clarify which version of Zotero I am referring to here. As I am writing this post, I am using Zotero v6.0.20 (installed on my Ubuntu 22.04 via flatpak), but the latest version available is v6.0.23 (see the changelog for more information).

Better Bibtex

I briefly mentioned Better BibTeX earlier, it is most definitely a must-have add-on for Zotero! As explained on the extension’s website:

Better BibTeX (BBT) is an extension for Zotero and Juris-M [a variant of Zotero — ed.] that makes it easier to manage bibliographic data, especially for people authoring documents using text-based toolchains (e.g. based on LaTeX / Markdown).

As someone who frequently writes documents in Markdown, I find Better BibTeX particularly useful! Among the great features of Better BibTeX, the automatic export function is essential to my workflow. I have configured it in order to update a .bib file (for example, myref.bib) every time I add a new item to my reference list. I have set my citation key formula to: auth.lower + "_"+ year + "_" + shorttitle(2,2) which uses the author’s last name in lower case, followed by the publication year and the first two words of the title that will be capitalized (seeCitation Keys), the three elements being separated by two underscores. As an example, let’s take the citation Jordan (2019) I mentioned earlier; with my citation key configuration, the key is jordan_2019_FromSocial.

If you want to learn more about Better BibTeX, simply visit the documentation website and if you want to dig deeper, the source code is available on , where you can also access the latest release. I personally used wget to do that:

1

$ wget https://github.com/retorquere/zotero-better-bibtex/releases/download/v6.7.54/zotero-better-bibtex-6.7.54.xpi

then see Plugins for Zotero to install it.

Visual Studio Code

To write articles there are various valid options on the table that can be split into two broad categories:

1. WYSIWYG (“What You See Is What You Get”) editors, represented by the popular word processors such as LibreOffice, Google Document, and Microsoft Word (and there are Zotero plugins for these three options)
2. WYSIWYM (“What You See Is What You Mean”) editors, which include general purpose code editors – such as Emacs or Sublime Text – and specialized ones, e.g. Overleaf and Remarkable.

At first glance, it may seem weird to use a code editor to write paper. However, it allows for the separation of content and presentation which I love. This approach helps me focus on the content and avoid wasting time tweaking the format while I am developing my ideas.

The first time I wrote a report using a code editor was when I switched to LaTeX, which allowed me to efficiently write good-looking equations compared to LibreOffice. About 10 years ago, I switched to Markdown which I now use for all kind of documents, including reports, README files, emails, blog posts, etc. (see my post Markdown everywhere!). For several years, I used Atom as my code editor to write papers in Markdown, but Atom has been recently sunset. For the past two years, I have been working with Visual Code Studio (VS Code) which owes Atom big time since the Electron framework that VS Code uses was originally built to develop Atom.

VS Code includes a good default Markdown editing feature and there are additional extensions available to make it even better (e.g. see the following posts: ‘How to Turn VSCode Into the Ultimate Markdown Editor’ and ‘Top 5 VSCode Extensions for Markdown’). By the way, when I write long-form document with VS Code, I often use the Zen mode to be more focused. That being said, the only thing missing to write articles is a connection to Zotero and for this, there are two extensions I recommend:

1. VS Code Citation Picker for Zotero,

2. Pandoc Citer Extension for Visual Studio Code.

The first option requires Better BibTeX up and running (so Zotero should remain open) and directly connects to the Zotero database. Once installed, SHIFT + ALT + Z triggers a small pop-up window from which you can select the references you want to cite. Note that all the references should also be included in a bibliography file (.bib, .yaml or .json) that will be used to generate the document in a different format (see below). The second option – the one I use – does not directly require Better BibTeX to be running, but it does require a .bib file from which the citation can be picked. By default, the path to this file will be taken from the field bibliography in the YAML front-matter, but a default file can be configured. In my case, I have configured the extension so it points to the file Better BibTeX keeps updated, which is why I need Better BibteX. Therefore, if I want to add a reference while I am working on my document, Zotero must be up and running. With this option, I simply press @ and a menu with all my references is prompted, from which I select the one I want (see the video below). Note that irrespective of the extension chosen, the path to the bibliography file needs to be specified so that references can actually be added to the document and this will be done with Pandoc.

1
2
3
4

---
title: My new paper
bibliography: [~/Documents/nextcloud/kevcaz.bib]
---

Pandoc

In the process of writing your article, you may need to convert it in a different format, either because you want to preview what your article will look like in the final format, to share it with collaborators who do not use Markdown, or to use a different format for submission. In this case, you will use a conversion tool that will most likely be or be based on Pandoc. Pandoc is an amazing tool to “convert files from one markup format into another”; it is an essential part of many authoring frameworks designed to generate dynamic documents that combine code and text, such as Rmarkdown and Quarto in the R ecosystem.

In the section that follows, I will use the latest Pandoc release, currently 3.1.2, to show how to convert a Markdown document to PDF and HTML. With the recent release of Pandoc, one simply uses the --citeproc flag to specify that references need to be parsed and -o to specify the output format. Based on the file extension, Pandoc will automatically determine what it needs to do.

1
2
3
4

# conversion to html
$ pandoc --citeproc paper.md -o paper.html  
# conversion to PDF
$ pandoc --citeproc paper.md -o paper.pdf

This is the raw way of doing the conversion, using the default (and super light) Pandoc’s HTML and PDF templates. Note that Pandoc has tens of options (flags) and filters to further customize documents. For instance there are filters cross-reference sections, figures, tables and equations (see pandoc-xnos). Furthermore there are higher level tools leveraging Pandoc that make the rendering easier with better-looking (and heavier) templates. For instance, in the R ecosystem, R users could use R Markdown and something like this to produce both documents at once:

1
2

R> library(rmarkdown)
R> render("paper.md", c("html_document", "pdf_document"))

Plus, there are dedicated packages to use specific templates; one good example is rticles, which offers more than 40 LaTeX templates to facilitate submission to various scientific journals (see this post for further details). Note that for many journals, you can always do a submission with a Word document that can be generated by Pandoc:

1

$ pandoc --citeproc paper.md -o paper.docx

or a tool that leverages Pandoc, e.g. R Markdown:

1
2

R> library(rmarkdown)
R> render("paper.md", "word_document")

Workflow (tl;dr)

To sum up, I use the following tools for academic writing:

• I write my articles in Markdown
• I use VS Code as my editor
• I manage my references with Zotero
• I have configured a BibTeX file with all my references updated by Better BibTeX.
• I use Pandoc Citer to generate a context menu of my references that I can easily access when I hit @
• When I need to convert my file to another format, I use Pandoc
• I share my Markdown files with collaborators via GitHub

References