2014년 12월 16일 화요일

find_and_modify brought down my Replica Set

I ran into an issue where an errant find_and_modify brought down my replica set (3-node RS running 2.6.1).

I had a document where I previously had been storing date information as a Date object.  I decided to try storing the Timestamp type instead.  I used find_and_modify to write the new timestamp in the new format 

db.feeds.find_and_modify(
query = {'relativePath':relativePath}, 
update = {'$currentDate':{'readReceipt':{ '$type': 'timestamp' }}}
)

and then I experienced a failure that I didn't think was possible using a 3-node replica set.  2 of my 3 nodes went down with the assertion below.  The remaining healthy node was stuck as a SECONDARY and my entire backend was down.  What was worse is that the offending operation seemed to be included in a dirty journal file and mongodb wouldn't repair or reboot.  I had to nuke all of the data in /var/lib/mongodb to get the servers to relaunch and trigger the remaining healthy node to switch to PRIMARY.  Is this expected behavior or a bug?

2014-12-14T06:09:57.075+0000 [repl writer worker 1] Assertion: 13111:wrong type for field (readReceipts.key) 17 != 9
2014-12-14T06:09:57.083+0000 [repl writer worker 1] cricket.feeds 0xe93423 0xe3fe29 0xe26a01 0xe26aac 0x7c7ef3 0x7cf2cb 0x7cb88e 0xad26e2 0xae010b 0xadb4d4 0xae2335 0xad6e0e 0xc256b5 0xc5df3d 0xc5d9c0 0xe36338 0xedf8eb 0x7ff5e6a00e9a 0x7ff5e58e32ed
 /usr/local/bin/mongod(_ZN5mongo15printStackTraceERSo+0x23) [0xe93423]
 /usr/local/bin/mongod(_ZN5mongo10logContextEPKc+0x1a9) [0xe3fe29]
 /usr/local/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x91) [0xe26a01]
 /usr/local/bin/mongod() [0xe26aac]
 /usr/local/bin/mongod() [0x7c7ef3]
 /usr/local/bin/mongod(_ZN5mongo20compareElementValuesERKNS_11BSONElementES2_+0x4cb) [0x7cf2cb]
 /usr/local/bin/mongod(_ZNK5mongo11mutablebson7Element22compareWithBSONElementERKNS_11BSONElementEb+0x34e) [0x7cb88e]
 /usr/local/bin/mongod(_ZN5mongo11ModifierSet7prepareENS_11mutablebson7ElementERKNS_10StringDataEPNS_17ModifierInterface8ExecInfoE+0x412) [0xad26e2]
 /usr/local/bin/mongod(_ZN5mongo12UpdateDriver6updateERKNS_10StringDataEPNS_11mutablebson8DocumentEPNS_7BSONObjEPNS_11FieldRefSetE+0x20b) [0xae010b]
 /usr/local/bin/mongod(_ZN5mongo6updateERKNS_13UpdateRequestEPNS_7OpDebugEPNS_12UpdateDriverEPNS_14CanonicalQueryE+0x724) [0xadb4d4]
 /usr/local/bin/mongod(_ZN5mongo14UpdateExecutor7executeEv+0x75) [0xae2335]
 /usr/local/bin/mongod(_ZN5mongo6updateERKNS_13UpdateRequestEPNS_7OpDebugE+0x1e) [0xad6e0e]
 /usr/local/bin/mongod(_ZN5mongo21applyOperation_inlockERKNS_7BSONObjEbb+0xdd5) [0xc256b5]
 /usr/local/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x29d) [0xc5df3d]
 /usr/local/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x50) [0xc5d9c0]
 /usr/local/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x548) [0xe36338]
 /usr/local/bin/mongod() [0xedf8eb]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7ff5e6a00e9a]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7ff5e58e32ed]
2014-12-14T06:09:57.083+0000 [repl writer worker 1] ERROR: writer worker caught exception:  :: caused by :: 13111 wrong type for field (readReceipts.key) 17 != 9 on: { ts: Timestamp 1418522384000|2, h: -1705807359185394341, v: 2, op: "u", ns: "cricket.feeds", o2: { _id: ObjectId('5479863209472593bb242a9e') }, o: { $set: { readReceipts.key: Timestamp 1418522384000|1 } } }
2014-12-14T06:09:57.083+0000 [repl writer worker 1] Fatal Assertion 16360
2014-12-14T06:09:57.083+0000 [repl writer worker 1]

***aborting after fassert() failure


댓글 없음:

댓글 쓰기