mongod startup problem - CentOS 6.5

The mongod process fails to start up automatically on reboot. It starts up without problem when done so manually.

1. The daemon has been properly configured to start up automatically:

adsx-131: /sbin/chkconfig --list mongod

mongod          0:off 1:off 2:off 3:on 4:off 5:on 6:off

2. There is one entry in 'boot.log' saying the startup failed

  Starting mongod: [FAILED]

3. When started up manually, mongod.log shows

2014-08-13T15:40:18.568-0400 ***** SERVER RESTARTED *****

2014-08-13T15:40:18.617-0400 [initandlisten] MongoDB starting : pid=8113 port=27017 dbpath=/proj/adsx/mongodb/data 64-bit host=<host name>

2014-08-13T15:40:18.618-0400 [initandlisten] db version v2.6.4

2014-08-13T15:40:18.618-0400 [initandlisten] git version: 3a830be0eb92d772aa855ebb711ac91d658ee910

2014-08-13T15:40:18.618-0400 [initandlisten] build info: Linux build7.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49

2014-08-13T15:40:18.618-0400 [initandlisten] allocator: tcmalloc

2014-08-13T15:40:18.618-0400 [initandlisten] options: { config: "/etc/mongod.conf", processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, security: { authorization: "enabled" }, storage: { dbPath: "/proj/adsx/mongodb/data" }, systemLog: { destination: "file", logAppend: true, path: "/proj/adsx/mongodb/logs/mongod.log" } }

2014-08-13T15:40:18.834-0400 [initandlisten] journal dir=/proj/adsx/mongodb/data/journal

2014-08-13T15:40:18.836-0400 [initandlisten] recover : no journal files present, no recovery needed

2014-08-13T15:40:21.024-0400 [initandlisten] waiting for connections on port 27017

Any ideas what's going on here?

Can you please post the output of "bash -x /etc/init.d/mongod start"? 

That may help see the difference between what's happening when manually started vs. started by the system at boot time. 

I have attached a file with the output. Seems like something goes boink with both changing the 'open files' setting and the 'runuser' execution.

bash_test.txt

It looks like it wasn't running as root. Can you re-run it with sudo or in a root shell (for example - "sudo bash -x /etc/init.d/mongod start")?

I ran it with sudo now. See attached file for output.

mongod.bash.output.txt

We have also this permanently when rebooting the server without shutting down properly the Mongo before. As written in a couple of parallel topics on this forum, the following doesn't work:

sudo service mongod stop.

This is what the server is trying to make when shutting down, but it fails. The server still reboots.

In the background, the /var/lib/mongod/mongod.lock, containing the PID is still present. When the server restarts, it sends a "service start", but the script reads the lock file and aborts the stop because there is a PID in the file, so it thinks mongo is already running.

Solve the problem:

Check the file (or similar place depending on your conf), check it contains an integer (the old process id), delete the file (no risk !) and mongod will start :-)

If you shutdown properly the mongo daemon (use admin -> db.shutdownServer()) before the reboot, mongod will start normally after the reboot

The init script log looks normal. Does anything show up in mongod.log when the startup fails? (For example anything about the lock file?)

That's the thing. There's nothing in the mongod.log file! And the thing that confuses me is that mongod starts up fine when started manually (with 'service mongod start'). Just for the heck of it I'll remove the space before the PID file path in the mongod.conf and see what happens at the next reboot.

Did that help at all? 

When you run "service mongod start" it should just be calling the init script, so if they're showing different behavior it may be a sign of worse system problems. Can you verify that?

I just saw there was no answer here. It case you didn't figured it out yourself, removing the space does help and fix the problem !