For anybody that hasn’t heard of NOSQL databases, there’s a lot of information out there about them. The most important thing to remember is that they are not for everyone. Just like relational databases are not perfect all the time, NOSQL databases are not perfect either. That being said, we’ve decided to try out a new project with a NOSQL database at work. We decided to use MongoDB. MongoDB is an open source NOSQL database that seems to be quite popular. There’s a really good short course on it if you go over to Pluralsight that will only take you 2 and a half hours. You can also look at the MongoDB website at mongodb.org, or if you’re looking for some fancy marketing stuff/training etc. you can head over to mongodb.com. I’ve personally never used any of the resources at mogodb.com, but there are resources out there that exist. MongoDB has a feature that I find particularly cool. When you create an index on a collection, you can create an index that provides objects with a time to live (TTL). This allows you to put things into a collection, without ever having to worry about writing a process to clean things up every once in a while (or manually cleaning things up). Mongo runs a TTL thread every 60 seconds to clean up any documents that have expired. This does make it so that something can be expired without yet having been removed, but in most cases I don’t think the 60 second delay is that critical. If it is, you could always check the lifespan of the document in code at runtime to ensure that the document hasn’t expired yet, but like I said, I don’t think that’s necessary. If you have a large amount of data in your collections, it is possible that the TTL thread will take longer than 60 seconds to run. I have not done any studies on how the performance would be affected by the workload, but I imagine that the TTL thread will be just as fast as a manual cleanup process. There’s more information in the Mongo documentation here.
Procedure
In order to give a document a TTL, you must do so with an Index. You can not just tell a single document when you want it to be removed from a collection, you must use an Index. The index must be on a field in the collection that is a BSON Date, or an array of BSON Dates. I do not know how it works if you use an array of BSON dates because I’ve never had to worry about it. To create a TTL index on a collection, use ensureIndex() with the expireAfterSeconds option. This can be done in one of two ways:
- Delete a document a specific number of seconds after the document was inserted in the collection. This allows you to use the index to query a document’s creation time.
- Specify a time to delete on the document. This allows you to use the index to query the document’s expiration time.
Keep in mind that if you set the expireAfterSeconds property to something less than 60 seconds (you know, for testing purposes) that the TTL thread only runs every 60 seconds, so you may find yourself confused as to why the documents aren’t expiring when you expect them to.
Expire after a Certain Amount of Time
If you want all objects that are put into a collection to expire x seconds after being put into the collection, then this is the way to go. All objects have the same lifespan, and if you need to change the lifespan you need to recreate the TTL index.
All we need to do is create an index on a BSON Date property in your collection, and set the expireAfterSeconds option. The date should be the time when the object was created, and the expireAfterSeconds should (as the name implies) be the number of seconds that you want before the document expires.indexedProperty + expireAfterSeconds. So if you don’t set the indexed field to be the date the document was created, you may have some interesting results." rel="footnote">1
|
1 2 3 4 5 |
db.<Collection Name>.ensureIndex( { "<property on objects in the collection>" : 1 }, { expireAfterSeconds: <some number of seconds> }) |
This could be useful if for example, we wanted to have a collection of server logs. We don’t want the logs to stick around forever, perhaps we only want the logs to be kept for 1 hour. In this case we could do something like this
|
1 |
db.server_log.ensureIndex({"createdAt":1}, {expireAfterSeconds:3600}) |
Then we add something to the collection
|
1 2 3 4 |
db.server_log.insert({ "createdAt": new Date(), "message" : "Something super awesome happened, and I want everybody to know" }) |
This document that we added to the collection will be deleted from the collection in 1 hour.
Expire at a Set Time
This is great for if you want to allow your objects in the collection to expire after different amounts of time. For example, maybe we want our verbose server logs to expire after an hour, but any debug logs we want for 1 week, and any error logs, we want for 1 month. This relies on us setting the time to expire a document in code when we insert the document into the collection, so it’s a little more work on our part, but it’s also a little more flexible.
All we need to do is create an index on a BSON Date property in your collection and set the expireAfterSeconds option to 0. The date should be the time when the object should expire.
|
1 2 3 4 5 |
db.<Collection Name>.ensureIndex( { "<property on objects in the collection>" : 1 }, { expireAfterSeconds:0 }) |
So for our server logs example, we could do this
|
1 |
db.server_log.ensureIndex({"expireAt":1}, {expireAfterSeconds:0}) |
Then we add something to the collection
|
1 2 3 4 5 |
db.server_log.insert({ "expireAt": new Date('July 19, 2014, 13:00:00'), "level" : "verbose" "message" : "Something super awesome happened, and I want everybody to know" }) |
And this would cause our server log to expire on July 19th, 2014 at 1:00 PM
And now I don’t need to write some kind of cleanup service!
I haven’t done any experiments to know exactly how the TTL index is implemented, but from reading the documentation, It appears that a document expires at the date [crayon-6736aa268fa15318281849-i/]. So if you don’t set the indexed field to be the date the document was created, you may have some interesting results. ↩