PHP MongoDB failover detection

Hello experts,

Here's my testing environment:

PHP MongoDB driver version: 1.5.8 stable

Webserver: Apache 2.2

Application stack: PHP with Code Igniter. Code Igniter version: 2.0.3

PHP version: PHP 5.4.6-1ubuntu1.8 (cli) (built: Apr 4 2014 01:28:36)

Here's the mongodb.php configuration (modified from codeigniter). The replica sets are configured using hostnames

<?php  if ( ! defined('BASEPATH')) exit('No direct script access allowed');

/* -------------------------------------------------------------------

 * EXPLANATION OF VARIABLES

 * -------------------------------------------------------------------

 *

 * ['mongo_hostbase'] The hostname (and port number) of your mongod or mongos instances. Comma delimited list if connecting to a replica set.

 * ['mongo_database'] The name of the database you want to connect to

 * ['mongo_username'] The username used to connect to the database (if auth mode is enabled)

 * ['mongo_password'] The password used to connect to the database (if auth mode is enabled)

 * ['mongo_persist']  Persist the connection. Highly recommend you don't set to FALSE

 * ['mongo_persist_key'] The persistant connection key

 * ['mongo_replica_set'] If connecting to a replica set, the name of the set. FALSE if not.

 * ['mongo_query_safety'] Safety level of writequeries. "safe" = committed in memory, "fsync" = committed to harddisk

 * ['mongo_suppress_connect_error'] If the driver can't connect by default it will throw an error which dislays the username and password used to connect. Set to TRUE to hide these details.

 * ['mongo_host_db_flag']   If running in auth mode and the user does not have global read/write then set this to true

 */

$config['default']['mongo_hostbase'] = '<MONGO-1-HOSTNAME>:27017,<MONGO-2-HOSTNAME>:27017';

$config['default']['mongo_database'] = '<DB_NAME>';

$config['default']['mongo_username'] = '';

$config['default']['mongo_password'] = '';

# https://github.com/alexbilbie/codeigniter-mongodb-library/issues/93

$config['default']['mongo_persist']  = FALSE;

/*$config['default']['mongo_persist_key']  = 'ci_persist';*/

$config['default']['mongo_persist_key']  = 'ci_persist';

$config['default']['mongo_replica_set']  = 'rs-1';

$config['default']['mongo_query_safety'] = 'safe';

$config['default']['mongo_suppress_connect_error'] = FALSE;

$config['default']['mongo_host_db_flag']   = FALSE;

$config['default']['readPreference']   = 'primaryPreferred';

# Add support for configuring the socket and connect timeouts

$config['default']['socketTimeoutMS'] = '5000';

$config['default']['connectTimeoutMS'] = '20000';

?>

I also enabled PHP MongoDB driver debug.

I am testing how quickly the PHP Mongo driver is able to detect the Mongo failover.

How am I testing the failover?

One the Primary Mongo I simply set up aiptables rule to DROP all inbound and outbound packets.

iptables -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT;iptables -A OUTPUT -p tcp --sport 22 -m state --state ESTABLISHED,RELATED -j ACCEPT;iptables -P INPUT DROP;iptables -P OUTPUT DROP

The Secondary Mongo and Arbiter are successfully able to detect Primary Mongo is no longer available and therefore the Arbiter participates in electing Secondary Mongo to become Primary - So far so good.

Now switch focus to the PHP Mongo driver. Whilst the PHP Mongo driver does 'blacklist' the Mongo for which connection is unavailable, accessing the web application is simply very, very sluggish.

2014-12-23 19:03:46 | PARSE (INFO): Parsingmongodb://<MONGO-1-HOSTNAME>:27017,<MONGO-2-HOSTNAME>:27017

2014-12-23 19:03:46 | PARSE (INFO): - Found node: <MONGO-1-HOSTNAME>:27017

2014-12-23 19:03:46 | PARSE (INFO): - Found node: <MONGO-2-HOSTNAME>:27017

2014-12-23 19:03:46 | PARSE (INFO): - Connection type: MULTIPLE

2014-12-23 19:03:46 | PARSE (INFO): - Found option 'replicaSet': 'rs-1'

2014-12-23 19:03:46 | PARSE (INFO): - Switching connection type: REPLSET

2014-12-23 19:03:46 | PARSE (INFO): - Found option 'socketTimeoutMS': 5000

2014-12-23 19:03:46 | PARSE (INFO): - Found option 'connectTimeoutMS': 20000

2014-12-23 19:03:46 | PARSE (INFO): - Found option 'readPreference': 'primaryPreferred'

2014-12-23 19:03:46 | CON (INFO): mongo_get_read_write_connection: finding a REPLSET connection (read)

2014-12-23 19:03:46 | CON (INFO): connection_create: creating new connection for <MONGO-1-HOSTNAME>:27017

2014-12-23 19:03:46 | CON (FINE): Connecting totcp://<MONGO-1-HOSTNAME>:27017 (<MONGO-1-HOSTNAME>:27017;rs-1;.;7369) with connection timeout: 20.000000

2014-12-23 19:04:06 | CON (WARN): connection_create: error while creating connection for <MONGO-1-HOSTNAME>:27017: Connection timed out

2014-12-23 19:04:06 | CON (WARN): Couldn't connect to <MONGO-1-HOSTNAME>:27017': Connection timed out

2014-12-23 19:04:06 | CON (INFO): connection_create: creating new connection for <MONGO-2-HOSTNAME>:27017

2014-12-23 19:04:06 | CON (FINE): Connecting totcp://<MONGO-2-HOSTNAME>:27017 (<MONGO-2-HOSTNAME>:27017;rs-1;.;7369) with connection timeout: 20.000000

2014-12-23 19:04:06 | CON (INFO): stream_connect: Not establishing SSL for <MONGO-2-HOSTNAME>:27017

2014-12-23 19:04:06 | CON (WARN): discover_topology: could not connect to new host: <MONGO-1-HOSTNAME>:27017: Previous connection attempts failed, server blacklisted

...

...

...

Why are new connections being created - shouldn't the connections be persistent?

2014-12-23 19:20:37 | CON (INFO): connection_create: creating new connection for <MONGO-2-HOSTNAME>:27017

2014-12-23 19:20:37 | CON (FINE): Connecting totcp://<MONGO-2-HOSTNAME>:27017 (<MONGO-2-HOSTNAME>:27017;rs-1;.;7921) with connection timeout: 20.000000

So what I would like to know is:

1. What are the best practices when in comes to configuring the PHP Mongo driver?

2. And how to ensure persistent connections are maintained?