mercredi 31 décembre 2014

Node.js MongoDB returning 0 documents while write lock is in place


I have a simple node server running that queries mongoDB and returns documents to the client for processing. In the background, I have a java application that's capturing data from various sources and writing to the mongoDB instance as a batch process. The java app also updates records in the mongoDB instance as appropriate. The java app does this approximately once per minute, and usually includes about 500 inserts and maybe 1000 updates.


Node.JS:



var MongoClient = require('mongodb').MongoClient;
var collection = '';
MongoClient.connect('pathToMongo', function(err, db){
collection = db.collection('mycollection');
});
...
app.get('/getData', function(req, res){
collection.find(queryString, {_id: 0, createdAt: 0}).sort({EVENTTIME:-1}).toArray(function(err,docs){
if (err){
console.log(err);
res.send(500,err);
} else {
res.json(docs);
}
});
});


Java snippet (assuming I already have all my data to insert/update in result and a valid connection to mongoDB):



public static void doWork(DBCollection coll){
while(result.next()){
BasicDBObject tempDoc = new BasicDBObject();
//Add values to tempDoc based on result iteration
DBObject unchanged = coll.findOne(tempDoc);
if (unchanged != null){
//update existing record
DBObject modifiedObject = new BasicDBObject();
modifiedObject.put("$inc", new BasicDBObject().append("cyclesUnchanged", 1));
coll.update(unchanged, modifiedObject, false, false);
} else {
//insert record
coll.insert(tempDoc);
}
}
}


Initialization of the java Mongo instance:



MongoClient mongoClient;
DB db;
DBCollection coll = null;
try{
mongoClient = new MongoClient('localhost", 27017);
db = mongoClient.getDB("test");
coll = db.getCollection("mycollection");
} catch (Exception e){
System.out.println("Mongo Error.");
System.exit(0);
}
final DBCollection newColl = coll;
doWork(newColl); //call to method above


My node server returns the exact number of documents I'm expecting most of the time (so it's not a query issue). However, occasionally, the query will run into a write lock (verified with db.currentOp() on mongod AND db.collection('$cmd.sys.inprog').findOne on the Node.js server), and return 0 documents. This is an issue, as I'm using this count for metrics, and need some sort of a method to ensure integrity of my query (which could be 0 documents). Is there something in Node.js's mongoDB package that attempts to retry if a write lock is in place?


I'm running MongoDB v2.6, mongo java driver v2.12.3, and Node.js v0.10.29.


UPDATE 12/31/2014:


The mongoDB query I'm using (referenced as 'queryString' above):



[{"$or", [{"PRI":{"$gte":"1","$lte":"9"}}]},{"EVENTTIME":{"$gte":"2014:12:31::16:59:18"}}]


I should note that this query, 99% of the time, works flawlessly. I'll have instances where I return 12 documents, as expected, every minute, for hours. And then, for some reason (I'm assuming a write lock is preventing reads), I'll get 0 documents returned.


The "sometimes appearing" currentOp (which is just an insert):



"opid" : 71715202,
"active" : true,
"secs_running" : 0,
"microsecs_running" : NumberLong(251),
"op" : "insert",
"ns" : "test.mycollection",
"insert" : {
"_id" : ObjectId("54a429a2880cc7fb7d2236465"),
"altID" : "999999",
"PRI" : "9",
"EVENTTIME" : "2014:12:31::17:55:12",
"LAT" : "0",
"LON" : "0",
"createdAt" : ISODate("2014-12-31T16:51:46.250Z")
},
"client" : "127.0.0.1:60750",
"desc" : "conn171",
"connectionId" : 171,
"locks" : {
"^" : "w",
"^test" : "W"
},
"waitingForLock" : false,
"numYields" : 0,
"lockStats" : {
"timeLockedMicros" : {

},
"timeAcquiringMicros" : {
"r" : NumberLong(0),
"w" : NumberLong(3)
}
}




Aucun commentaire:

Enregistrer un commentaire