2014년 11월 29일 토요일

Cant import csv file with about 6 milion documents to MongoDB 2.8.0-rc0 with wiredtiger

Im struggling with new version fo MongoDB 2.8.0-rc0 (x64) with storage engine called WiredTiger. To start server I use command: mongod --storageEngine wiredtiger --dbpath path and I'm trying to import csv file via this command: mongoimport --type csv -c collectionname --file filePath --headerline
When import is in progress (after about 670k documents) it stuck suddenly with err "2014-11-18T16:52:33.074+0100 error inserting documents: WiredTigerRecordStore ::insertRecord 12: Not enough space" and then server crashes.
I've got 250gb free disk memory, while importing over 5GB ram free.
Is there anyone who know how to solve this problem.



You say you have available disk space - but are there quotas?  Or possibly the directory on which the datapath is has run out of space?
Its hard to know without seeing what's in your mongod log file though.
Can you provide that information please?



There is no quotas and there is no possibility to run out of space in dbpath directory.
In this version (2.8.0-rc0) but without using wiredtiger engine, import works good.

Here are server logs:



Hey,
the problem has been resolved with 2.8.0-rc1-pre- version.



Can you give us a link to the bug report.



I Havent reported it. Ive just  tried  to do import with the newest version of mongo.



I've been inserting significantly larger loads into wiredtiger.

In order to understand exactly what happens, please try this again and this time include the entire error (including the few lines leading up to it and all of the subsequent stack trace).  This will allow me to see exactly where in the code the problem is being flagged.



Dear Asya,

I would love to send to you this data, but im neewby in monodb and i just dont know where i can find entire error logs :(. Can you instruct me how to get those things ?
Here is my mongod.conf:
"systemLog:
   destination: file
   path: "C:/Users/Marek/Desktop/mongo/data/log/mongod.log"
   logAppend: true"

Just let me know how to do it and I would do import again.



Your log file is here:

 "C:/Users/Marek/Desktop/mongo/data/log/mongod.log"

Since you are using append option, it's probably pretty large by now.   I'm only interested in the last part, immediately before and after the error happens.



I'll retry import as soon as possible and show logs here, ofcourse ill clear it before.



Ok, I did import again.
One more time to start server i use command:
mongod --config path --dbpath path --storageEngine wiredtiger

mongodb.conf looks like that:
systemLog:
   destination: file
   path: "C:/Users/Marek/Desktop/mongo/data/log/mongod.log"
   logAppend: true

To start import i run command:
mongoimport --type csv -c colname --file filepath --headerline


I have cleared the log file before starting server, and it looks like it now after server crash :

In addition im using mongodb v. 2.8.0-rc0 for Windows.


To be clear ill provide you exact steps to reproduce
(use 2.8.0-rc0)
2. Unzip it.
3. Prepare it with this command(deleting "\n" signs): cat Train.csv | tr "\n"" " | tr "\r" "\n" | head -n -1 > Train2.csv
4. start server with command:  mongod --config path --dbpath path --storageEngine wiredtiger
5. start import with command:  mongoimport --type csv -c colname --file filepath --headerline
6. Wait for error/crash




I looked at the pastebin - I don't see any error, the log file ends with:

2014-11-21T12:12:43.122+0100 I WRITES   [conn1] insert test.Train ninserted:10000 keyUpdates:0 numYields:0  242ms
2014-11-21T12:12:43.768+0100 I WRITES   [conn1] insert test.Train ninserted:10000 keyUpdates:0 numYields:0  251ms
2014-11-21T12:12:45.376+0100 I WRITES   [conn1] insert test.Train ninserted:10000 keyUpdates:0 numYields:0  214ms

Last line is line 84.   I'd expect that there should be an error and a stack trace in the logs...



Ive copied logs right after server crash - I dont realy know why there is nothing about. 



Could it be something else is crashing on the server/computer other than Mongo? We are talking about Windows here. LOL!



Interesting...

So the only way you know mongod crashed is ... that suddenly the process is no longer running?

What do you see in the logs when you restart mongod after the crash?


댓글 없음:

댓글 쓰기