Memoize in Meteor to Speed Up Aggregate and Complex Functions


by Marcelo Reyna

Some functions take a long time to run, this can be especially true with Mongo Aggregation functions. To improve this, we can save the results of the function into a database (server side or client side) and read the result instead of running the function again. This method is known as MEMOIZE and is particularly easy to implement with lodash.

Client-Side

Using memoize on the client-side is easy, we can simply call _.memoize() on our function and we are done. Here’s an example:

function slow() { ... }
const memoizedSlow = _.memoize(slow);
// Now run memoizedSlow() multiple times, it will only take a while to calculate the first time it runs, then it will be quick

Server-Side

For server-side memoization we need to customize the way memoize works. To do this we need to define an ECMA Map that will describe how memoize should interact with the database.

Let’s build one for Mongo in Meteor.

const Memos = new Mongo.Collection('memos');
class MongoMapper {
  clear() {
    Memos.remove({});
  }
  delete(key) {
    Memos.remove({key});
    return true;
  }
  get(key) {
    const memo = Memos.findOne({key});
    return memo && memo.value;
  }
  has(key) {
    const memo = Memos.findOne({key});
    return memo ? true : false; // The spec requires true||false not truey||falsy
  }
  set(key, value) {
    Memos.upsert({key}, {
      $set: {
        key,
        value,
      },
    });

    return this; // Required
  }
}

Notice how there are 5 required functions in the MongoMapper prototype: clear, delete, get, has, and set. memoize uses these prototype functions to basically set and get the results that your time consuming function would produce. In practice, we can convert our time consuming aggregation functions and reduce the load on the database and server by wrapping them with a memoize.

Let’s have a look at what that would look like:

// Build a complex function
function calculateResults() {
  const complexPipeline = [...]
  return Data.aggregate(complexPipeline);
};

// Tell memoize what map to use
_.memoize.Cache = MongoMapper;

// Memoize the function
const calculateResultsMemo = _.memoize(calculateResults);

// Now run calculateResultsMemo() multiple times

Now the calculateResultsMemo function is using the Memos collection as its cache by following the rules we set on MongoMapper.

But what if now we want the memos to have a time limit? Maybe we want the memos to update after one week of being created, how do we handle that? It’s very easy! We just need to set up that logic on MongoMapper. Let’s give it a time limit using momentjs.

const Memos = new Mongo.Collection('memos');
class MongoCache {
  clear() {
    Memos.remove({});
  }
  delete(key) {
    Memos.remove({key});
    return true;
  }
  get(key) {
    const memo = Memos.findOne({key});
    return memo && memo.value;
  }
  has(key) {
    let memo = Memos.findOne({key});

    // Check whether the memo has expired, if it has, remove it
    if (memo && (moment().unix() > memo.expires)) {
      Memos.remove({key});
      memo = undefined;
    }

    return memo ? true : false;
  }
  set(key, value) {
    Memos.upsert({key}, {
      $set: {
        key,
        value,
        // Add an expiration date to the memo
        expires: moment().startOf('day').add(3, 'days').unix(),
      },
    });

    return this;
  }
}

Sweet! Now our memos can expire. You can modify the way your cache works by working with the MongoMapper prototype functions for more complex rules. When should I use this? The memoize pattern is an excellent way to quickly increase the server’s response times on complex computations. In the real world, this has applications to Big Data calculations (that don’t change often) and any function that runs multiple times and cannot be throttled.

Here is the full working example of what we saw:

const Memos = new Mongo.Collection('memos');
class MongoCache {
  clear() {
    Memos.remove({});
  }
  delete(key) {
    Memos.remove({key});
    return true;
  }
  get(key) {
    const memo = Memos.findOne({key});
    return memo && memo.value;
  }
  has(key) {
    let memo = Memos.findOne({key});

    // Check whether the memo has expired, if it has, remove it
    if (memo && (moment().unix() > memo.expires)) {
      Memos.remove({key});
      memo = undefined;
    }

    return memo ? true : false;
  }
  set(key, value) {
    Memos.upsert({key}, {
      $set: {
        key,
        value,
        // Add an expiration date to the memo
        expires: moment().startOf('day').add(3, 'days').unix(),
      },
    });

    return this;
  }
}

// Build a complex function
function calculateResults() {
  const complexPipeline = [...]
  return Data.aggregate(complexPipeline);
};

// Tell memoize what map to use
_.memoize.Cache = MongoMapper;

// Memoize the function
const calculateResultsMemo = _.memoize(calculateResults);

// Now run calculateResultsMemo() multiple times
Follow Me

Share Share on Twitter Share on Facebook Share on LinkedIn

How Can We Help?

Reaching out doesn’t mean you’re ready to start a project, but we’d love to learn more about the challenge you’re facing, answer any questions, and see if we might be a good fit for working together.

Contact Us