Tags: howtos, research

« Previous post: Watching the Watchmen: Pitfalls in Graph … — Next post: Open Source and Academia »

I have already written about the benefits of keeping a lab book for doing research. Here, I want to briefly expand on how I take all my notes now and how I configured vim to support me.

Why vim?

First, let us get the perennial question out of the way: ‘why vim instead of $EDITOR?’ The answer is simple: because I am used to it. I am sure that the overall gist of what I will describe in this article will translate just as well to another editor, but I do not know their plugin system sufficiently well to make a precise statement here. So vim it is.¹

The setup

I store all my notes in various Markdown files. This enables me to write text with some nice embellishments while having the opportunity to export it to other formats for pretty-printing. At the same time, these documents are also eminently searchable via ripgrep and under version control.

Since I spend quite a lot of time editing these documents, I tuned my vim configuration for more productivity. Among some of the plugins I use, UltiSnips is arguably the most critical part of my daily workflow. Let me briefly extol its virtues: UltiSnips allows you to define small snippets for specific file types such as Markdown files, Python files, and so on. Small key phrases can be used to trigger their insertion. So far, so good—but the best feature of UltiSnips is that it permits calling external programs to modify the trigger text. As we shall later see, this opens up quite a few interesting directions.

Setting up UltiSnips

Install UltiSnips using your favourite plugin manager. For my setup, this boils down to a single line in my init.vim:

Plug 'SirVer/ultisnips'

Next, we need to define the key that triggers the search for UltiSnips snippets. I use Tab for initial expansion and jumping forward in the suggestion list, as well as Shift + Tab to jump back in the suggestion list. This behaviour can be set by adding the following lines to init.vim:

let g:UltiSnipsExpandTrigger="<tab>"
let g:UltiSnipsJumpForwardTrigger="<tab>"
let g:UltiSnipsJumpBackwardTrigger="<s-tab>"

Simple snippets

Unless configured otherwise, UltiSnips expects snippets to be in the runtime path under an UltiSnips directory. When using Neovim, for instance, the default directory would be ~/.config/nvim/UltiSnips. The type of snippet, and thus their applicability, is defined by its filename. Hence, to create snippets for Markdown, we need to create a markdown.snippets file. As a simple start, here are two of the snippets I regularly use:

snippet ornament
❦
endsnippet

snippet today
`date +%F`
----------
endsnippet

The first one is called ornament and inserts a floral ornament into the text. The second one is called today and inserts a second-level heading² into a Markdown document that contains the current date, formatted following ISO 8601, i.e. YYYY-MM-DD.³

When you encounter these snippets for the first time, you may wonder whether they do not impede regular writing—after all, both ‘ornament’ and ‘today’ are words that might crop up in normal writing quite often. UltiSnips is however quite ingenious in that it offers the snippets as a suggestion, but only ever inserts them upon pressing the aforementioned key combination.

Hence, I can write a text consisting of nothing but ‘Today I bought ornament after ornament’ without UltiSnips being triggered. If I want snippet completion, I need to type its name and press the trigger key: typing today Tab inserts the current date into the file.

More complicated snippets

So far, this seems like a fun way to save a few keystrokes, nothing less. However, UltiSnips really shines when more complex transformations have to be transformed by a given snippet. In the following example, we will handle a case that often occurs in my workflow:

1. Find a nice set of papers.
2. Put their URLs in my notes.
3. Write a brief note about them.

Previously, I would have to format the corresponding links by hand; a somewhat tedious task since I first have to open the link (thus switching away from vim), find the paper title, copy it, switch back to vim, insert it, and make a Markdown link for it. I found this to be extremely dull, tedious, and repetitive, so I decided to create a snippet that solves this task for me. Before we discuss how it works, here is the result of my labours:

snippet l "Create link to paper"
`!p
import bs4
import requests

# Better to set a fake user agent because some sites block requests if
# they contain the default user agent.
headers = {
	'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36'
}

url = snip.v.text.strip()

# Ensure that we always get the abstract for arXiv papers
# even if the URL contains a link to the PDF only.
if 'arxiv.org' in url:
    url = url.replace('.pdf', '')
    url = url.replace('pdf', 'abs')

# Ditto for OpenReview
elif 'openreview.net' in url:
    url = url.replace('/pdf', '/forum')

page = requests.get(url, headers=headers)
soup = bs4.BeautifulSoup(page.text, 'html.parser')

# Set this to be undefined at first in order to decide whether we were
# able to extract the proper information.
name = None

title_tags = ['citation_title', 'DC.Title', 'og:title']

for tag in soup.find_all('meta'):
    # The title of the cited element is stored in the 'citation_title'
    # tag or a related one.
    if tag.get('name') in title_tags or tag.get('property') in title_tags:
        name = tag.get('content')

if name is not None:
    name.replace('\n', ' ').replace('\r', '')
    snip.rv = f'[*{name}*]({url})'
else:
    snip.rv = url`
endsnippet

The magic happens by assigning something to snip.rv, the return value of the snippet transformation. UltiSnips supplies the Python script directly with a global variable called snip, with the value, i.e. the text that triggered the snippet, being encoded by snip.v. The Python code you are seeing above just takes snip.v, queries the corresponding URL while pretending to be a proper browser, and extracts information about a paper from the result. For the last part, I am making use of the marvellous Beautiful Soup project, which makes parsing markup so much easier.

The cool features of this script are that, depending on the type of URL, automated transformations are performed. For instance, even if someone sends you a link to a cool PDF on arXiv, the script will change the URL so that the link will point to the abstract page of the preprint instead. Something similar happens for OpenReview links. If the markup cannot be properly understood, the input URL is just returned as-is.

This snippet has saved me so much time over the past few years! I now merely have to select a URL in visual mode, type Tab + l, and the link will be nicely formatted.

Technical details

For the Python integration to work nicely, you need to have a Python interpreter on your system. With Neovim, integrating Python is super easy; I personally use a virtual environment so that the packages I require in my snippets do not clutter up my global Python environment. This can be achieved by setting the python3_host_prog variable in your init.vim. For example, this is what I use on all my systems:

let g:python3_host_prog = expand('$HOME/.virtualenvs/nvim/bin/python3')

I strongly recommend following something similar in your own setup in order to prevent any package-related headaches.

Conclusion

This post demonstrated how I improved my lab book note-taking setup by writing my own snippets for the UltiSnips plugin. We have only scratched the surface here; UltiSnips can do much more and help you be more productive when writing software as well. To get the most out of that plugin, you should put all your snippets under version control so that you can keep track of changes and replicate them to other machines under your control. This should be the subject of another post, though.

May your snippets be useful, until next time!