Mongo, Node, Express - API Responses with metadata
I have been playing more and more with the Ember-CLI as of late and I was having some issues getting Ember-Data to play nicely with API's that I had already created. This is mostly because my API's weren't following the syntax that was being expected by Ember-Data. After some head scratching I decided that this particular section of this super-opinionated (IMO) framework was actually (at least partially) correct.
The Situation
Here was the situation. I had a small application that had really only a few MongoDB collections with the main one being called 'Inventory'. Here was phase one of my pain. Ember-Data didn't like that my api was already pluralized so instead of my api endpoints being http://some-ip-address/api/inventory, I had to rename the endpoint to be http://some-ip-address/api/inventories. The programmer in me understands this idiom. The grammar freak in me hates it, however I'll work with it regardless.
Secondly, my api's weren't always super awesome and refined. I have often created services that just returned a JSON array with absolutely no metadata, but as my applications have been growing and demands for front-end features increasing, I've started adding in meta and I essentially recycle the same code over and over again to make it happen. Eventually I upgraded from returning my JSON with just an array of objects to including a meta section and a data section
{
"meta":{
"page":1,
"pages":10,
"limit":50,
"totalResults":500
},
"data":[
...
]
}
#likeaboss right? -- Not so much.
I've learned recently that this COULD have sufficed for my EmberJS if you did some customization of Ember-Data (something I didn't feel like doing -- and won't be going over in this post as I'm not married to Ember as of yet).
In order to make this work properly for me, I needed to return the exact same object but instead of using the term "data" I need "inventories"
"meta":{
"page":1,
"pages":10,
"limit":50,
"totalResults":500
},
"inventories":[
...
]
Now, the real point for making this post -- the metadata.
This is essentially the code that I recycle consistently and (one day) should probably make some sort of a module to do it for me automatically.
The Model
So it makes a little more sense later on, here's a simplified example of the model that I'm working with.
//models/inventory.js
var mongoose = require('mongoose');
var inventorySchema = mongoose.Schema({
category:String,
subCategory:String,
description:String,
costs:{
purchase:Number,
sale:Number
},
added:{type:Date, default: new IsoDate(),
updated:{type:Date, default: new IsoDate()
});
module.exports=mongoose.model('Inventory', inventorySchema);
The REST Endpoint
To get a list of items, the endpoint is obviously going to be the pluralized one. For brevity, I'm assuming you all know how mongoose models work etc.
//routes/inventories.js
var express=requre('express'),
router=express.Router(),
mongoose=require('mongoose'),
Inventory=mongoose.model('Inventory');
router.get('/', getInventory);
...
function getInventory(req, res, next){
//Build the Query to provide to Mongo. This separates code that could be recycled and used potentially in other endpoints
buildQuery(req, function(err, query){
if(err){
return res.status(400).send({message:'Error Generating Query', error:err});
}
//Execute the query generated in the previous function
executeQuery(query, function(err, result){
if(err){
return res.status(400).send({message:'Error Generating Result Set', error:err});
}
else{
return res.jsonp(result);
}
}
}
}
...
This is all pretty straight forward I think. Plain English and descriptive names almost always takes away the need for excessive commenting.
The Extraction
^ Most Epic Section Name Ever.
Next we get into the buildQuery(req) function. This does exactly what it implies. Builds a Mongoose query object then callsback when it's done.
//routes/inventory.js
...
function buildQuery(req, callback){
var params = req.query;
var query = {},
projection = {},
modifiers = {},
validParameters=true,
invalidParamaters=[];
Object.keys(params).forEach(function(p){
switch(p){
case 'limit': //Special case for pagination only. Not invalid query parameters but Not useful
break;
case 'page': //Special case for pagination only. Not invalid query parameters but Not useful
break;
case 'category':
query.category = params[p];
break;
case 'subCategory':
query.subCategory = params[p];
break;
default:
invalidParameters.push(p);
validParameters = false;
break;
}
});
if(!validParameters){
callback({error:'INVALID PARAMETERS: 'invalidParameters.toString()}, null);
}
else{
if(params.limit){
if(parseInt(params.limit) !== 0)){ //unless the user specifically called for no limit
modifiers.limit = parseInt(params.limit);
}
else{
modifiers.limit = 100; //Then by default set the limit to 100
}
//Note: If the user entered 0 as the limit, then the results wont be truncated.
}
//Here we will be determining how many results we skip from the aforementioned query.
if(params.page && (params.limit && (parseInt(params.limit) !== 0))){
modifiers.skip = params.limit * params.page;
}
//Return the results
callback(null, {query:query, projection:projection, modifiers:modifiers});
}
}
You will notice that I haven not touched the projection variable here. For this blurb the projection option isn't useful but I like to stay in the habit of remembering what the three options MongoDB takes in for their queries. This allows this section of code to be recycled and the only part that you really need to modify for your different models is the switch section which is all model dependent things that you an play around with. The next section is where we will get our results and build up our metadata. This is pretty straight forward -- some simple arithmetic.
...
function executeQuery(parsedQuery, callback){
var results = {};
Inventory.find(parsedQuery.query, parsedQuery.projection, parsedQuery.modifiers, function(err, docs){
if(err){
callback({message:'Error Getting Results', error:err}, null);
}
else{
Inventory.count(parsedQuery.query, function(err, c){
if(err){
callback({message:'Error getting metadata', error:err}, null);
}
results.meta = {};
results.meta.totalResults = c;
results.meta.pages = parsedQuery.modifiers.limit?Math.ceil(c/parsedQuery.modifiers.limit):1;
results.meta.limit = parsedQuery.modifiers.limit?parsedQuery.modifiers.limit:c;
results.meta.page = parsedQuery.modifiers.skip?parsedQuery.modifiers.skip/parsedQuery.modifiers.limit:0;
results.inventories = docs;
callback(null, results);
});
}
});
}
...
There we have it! Simple pagination using Mongoose, Node and Express. As you can see, there are definitely ways that this code could be optimized to be reusable. Especially the executeQuery part. For each new model all you change is the model (Inventory in this case) to match the specific route.
Let me know your thoughts!
Cheers until next time,
James