MongoDB Replica Set

Say I have the following architecture 

The application uses "primary.mydomain.com" to talk to the mongodb replica set.  What happens when primary.mydomain.com goes completely down, meaning it was powered off or something that makes the hostname "primary.mydomain.com" go completely unresponsive?

I know that the replica set will do an election and assign one of the secondary as the new primary, but the application is still sending request to "primary.mydomain.com".  My question, is how do I handle this situation?

The application uses "primary.mydomain.com" to talk to the mongodb replica set.  What happens when primary.mydomain.com goes completely down, meaning it was powered off or something that makes the hostname "primary.mydomain.com" go completely unresponsive? 

I know that the replica set will do an election and assign one of the secondary as the new primary, but the application is still sending request to "primary.mydomain.com".  My question, is how do I handle this situation?

Hi John,

In order to handle this situation you should always provide a seed list of two or more nodes to connect to in your application code rather than a single node. Your MongoDB driver will connect to the first available node and work out what the current replica set configuration is (including the primary). In the event of a replica set reconfiguration or failover, the driver will automatically reconnect. There will be a brief period where you may get exceptions during the replica set election and then your application should resume with the new primary.

There are some examples of connection strings in the MongoDB manual: http://docs.mongodb.org/manual/reference/connection-string/

Just the answer I needed.  Thanks Stephen!!!

If you don't mind some extra processes and also want some simplicity to application setup opposed to complex setup for databases, you may configure a single shard of your replica set and connect your application via 'mongos'.
I've set this configuration on three database servers + about ten application servers with connectors as simple as "localhost:27099" and all logic left to mongos'.

Hi, this is question ,(not reply: could not find where to ask, so poster it here)

Problem in mongo clustering :secondary replicaset members not joining:

rsProd:PRIMARY> rs.add("egnarodev-VirtualBox5:27017");
{
    "errmsg" : "exception: need most members up to reconfigure, not ok : egnarodev-VirtualBox5:27017",
    "code" : 13144,
    "ok" : 0
}

while,

1. egnarodev-VirtualBox5 is up (by using "sudo service mongod start")
2. egnarodev-VirtualBox5 is reachable from host (by using "mongo --host IPofegnarodev-VirtualBox5")
3. in /etc/momgod.conf file following changes are done: auth=true bind ip is deleted port=27017 replSet=rsProd

Any help ?

PS: pinged and used telnet host port . all reply are positive

You're using IP of host in testing connectivity with mongo shell.   Test what the replica is configured with, I.e. the name.

I'm guessing that won't work.

I have tested addin in primary with the hostname of the replica set members also . It doesn't work 

You're saying that when you are on the machine the primary node is running on you can do:

mongo egnarodev-VirtualBox5:27017/test

and you get a shell prompt?

If that's the case, would you please type the following at the mongo prompt and let us know what the output is:

> hostname()

On egnarodev-VirtualBox5 what is in the mongod logs around the time you try to add this host to your replica set?

What is in the primary mongod logs at the same time?

Primary mongodb log:

2014-12-30T10:45:55.278+0530 ***** SERVER RESTARTED *****
2014-12-30T10:45:55.282+0530 [initandlisten] MongoDB starting : pid=2437 port=27017 dbpath=/var/lib/mongodb 64-bit host=egnarodev-VirtualBox3
2014-12-30T10:45:55.282+0530 [initandlisten] db version v2.6.6
2014-12-30T10:45:55.282+0530 [initandlisten] git version: 608e8bc319627693b04cc7da29ecc300a5f45a1f
2014-12-30T10:45:55.282+0530 [initandlisten] build info: Linuxbuild10.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2014-12-30T10:45:55.282+0530 [initandlisten] allocator: tcmalloc
2014-12-30T10:45:55.282+0530 [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1" }, storage: { dbPath: "/var/lib/mongodb" }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2014-12-30T10:45:55.351+0530 [initandlisten] journal dir=/var/lib/mongodb/journal
2014-12-30T10:45:55.351+0530 [initandlisten] recover : no journal files present, no recovery needed
2014-12-30T10:45:55.364+0530 [initandlisten] waiting for connections on port 27017
2014-12-30T10:46:57.953+0530 [clientcursormon] mem (MB) res:37 virt:348
2014-12-30T10:46:57.953+0530 [clientcursormon]  mapped (incl journal view):160
2014-12-30T10:46:57.953+0530 [clientcursormon]  connections:0
2014-12-30T10:49:33.230+0530 [initandlisten] connection accepted from 127.0.0.1:39697 #1 (1 connection now open)
2014-12-30T10:49:36.434+0530 [conn1] end connection127.0.0.1:39697 (0 connections now open)
2014-12-30T10:51:57.993+0530 [clientcursormon] mem (MB) res:37 virt:349
2014-12-30T10:51:57.993+0530 [clientcursormon]  mapped (incl journal view):160
2014-12-30T10:51:57.993+0530 [clientcursormon]  connections:0
2014-12-30T10:52:30.154+0530 [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2014-12-30T10:52:30.176+0530 [signalProcessingThread] now exiting
2014-12-30T10:52:30.176+0530 [signalProcessingThread] dbexit:
2014-12-30T10:52:30.176+0530 [signalProcessingThread] shutdown: going to close listening sockets...
2014-12-30T10:52:30.176+0530 [signalProcessingThread] closing listening socket: 8
2014-12-30T10:52:30.176+0530 [signalProcessingThread] closing listening socket: 9
2014-12-30T10:52:30.176+0530 [signalProcessingThread] removing socket file: /tmp/mongodb-27017.sock
2014-12-30T10:52:30.176+0530 [signalProcessingThread] shutdown: going to flush diaglog...
2014-12-30T10:52:30.176+0530 [signalProcessingThread] shutdown: going to close sockets...
2014-12-30T10:52:30.176+0530 [signalProcessingThread] shutdown: waiting for fs preallocator...
2014-12-30T10:52:30.177+0530 [signalProcessingThread] shutdown: lock for final commit...
2014-12-30T10:52:30.177+0530 [signalProcessingThread] shutdown: final commit...
2014-12-30T10:52:30.184+0530 [signalProcessingThread] shutdown: closing all files...
2014-12-30T10:52:30.184+0530 [signalProcessingThread] closeAllFiles() finished
2014-12-30T10:52:30.184+0530 [signalProcessingThread] journalCleanup...
2014-12-30T10:52:30.184+0530 [signalProcessingThread] removeJournalFiles
2014-12-30T10:52:30.195+0530 [signalProcessingThread] shutdown: removing fs lock...
2014-12-30T10:52:30.195+0530 [signalProcessingThread] dbexit: really exiting now

VirtualBox5 Mongodb log :

local.system.replset config from self or any seed (EMPTYCONFIG)
2014-12-30T11:02:24.077+0530 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
2014-12-30T11:02:25.077+0530 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
2014-12-30T11:02:26.077+0530 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
2014-12-30T11:02:27.078+0530 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
2014-12-30T11:02:28.078+0530 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
2014-12-30T11:02:29.079+0530 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
2014-12-30T11:02:30.079+0530 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)

So you can't connect to it on 27017.

You can connect on the port it's actually running on: 27019.

Why aren't you adding the node to replica set with port 27019?

I am adding the node to replica set with port 27019 !

Please refer to my last mail .I have attached the screenshots in it 

First of all, your original question said:

rsProd:PRIMARY> rs.add("egnarodev-VirtualBox5:27017");
{
    "errmsg" : "exception: need most members up to reconfigure, not ok : egnarodev-VirtualBox5:27017",
    "code" : 13144,
    "ok" : 0
}

Second of all, your screenshot showed unsuccessful attempts to connect to 27017 (and successful attempts to 27019).

Third, your primary is running with "net: { bindIp: "127.0.0.1" }" which means no one can connect to it so you won't be able to configure other nodes as they won't be able to talk to the primary.

refer to only last mail screen shots .
In VirtualBox5 , in mongod.conf file , I have set port to 27019 and commented out bind_ip option .

Same true for  VirtualBox3 mongod.conf file , only port=28017 .


While keeping mongod service running on VirtualBox5 port 27019 , I am trying to add "egnarodev-VirtualBox5:27019" in VirtualBox3 by the following command :
rsProd:PRIMARY> rs.add("egnarodev-VirtualBox5:27019");
{
    "down" : "egnarodev-VirtualBox5:27019","ok" : 1
}






You have to be able to communicate to each host from the other host.   So you have to be able to reach 3 from 5 and 5 from 3.



Try both ways from the command line using the shell.







How ?
Can u tell me the commands?






Same way you did before:



$ mongo hostname:port/dbname

> 



If you get to the shell prompt then you connected.  If not then you didn't and replica set cannot possibly communicate.  It must work both directions between two nodes in a replica set.







I am getting the shell prompt . But , if I type hostname() there
, it is VirtualBox3 (which should  be VirtualBox , in case of connecting to 5 from 3)







Sounds like your hostnames are messed up in the VMs...







problem solved :)







Great!


What was the solution?



Detailing it here may help someone else in the future who may see similar errors.







k.I'll share once i'll properly note down the steps