by Matt Brennan

How to cache data from an external API in Meteor

This is the blog post I wish had existed months ago.

If you’ve been using Meteor for a while, chances are you’ve come across the fantastic blog Meteor Capture and, in particular, their two-post series How to Publish Anything. If not, I strongly recommend going and reading them both, because this is the unofficial third post in that series.

Let’s say you want to get data from an external service by id. You’ve probably set up your publish something like this:

import {Meteor} from 'meteor/meteor';
import {HTTP} from 'meteor/http';

const getData = id => HTTP.get('https://example.com/things/' + id).data;

Meteor.publish('things', function(id) {
  this.added('things', id, getData(id));
  this.ready();
});

It’s great, and it works, but you notice that when loading several things at once, they’re all fetched serially, and pop in one at a time, and they’re not available when your app first runs. Unlike you might have expected, this.added('collection') only makes the data available in the collection with that name on the client’s Minimongo, and doesn’t persist it to the server’s Mongo.

Meteor publications are code. There’s nothing stopping you inserting documents as you publish from the external API. You can also check if an _id exists in Mongo, and only perform the expensive HTTP request if it doesn’t. Your publish might look like this when you’re done:

import {Meteor} from 'meteor/meteor';
import {HTTP} from 'meteor/http';
import {Things} from '../shared/collections';

const getData = id => HTTP.get('https://example.com/things/' + id).data;

Meteor.publish('things', function(id) {
  const cached = Things.get({_id: id});
  if(cached.count()) {
    return cached;
  }
  
  const thing = getData(id);
  thing._id = id;
  this.added('things', id, thing);
  Things.insert(thing);
  this.ready();
});

Let’s break it down. Usually to get a single document by id, you want findOne, but here we need a cursor. We check if there’s any matches, and return the cursor if so. Publish does the rest. Otherwise, we get the data and publish it as before, but also insert it into the collection. We need to add _id to the document so it’s the same one we queried with. That way, next time you subscribe, it’s already there and it’s nice and fast.

There’s a couple of caveats here. Once you’ve fetched the data, you only ever get it from Mongo afterwards. This works great for data that’s unlikely to change: the Google Books API from the Meteor Capture post is a good example. Otherwise, you’ll want to have some way of updating the documents in the background. There might be a way to do that after making the cached fetch, but I’m not sure.