Blacklist server from a replica set, within a specific client

We have a special replica set, distributed on two hostings.

We-re using the php-driver to connect.

Problem: clients from hosting A (driver for php-fpm workers)  establish persistent connections to discover the servers from the replicaset.

because the ping is big (50ms)  between hosting A and B we have huge performance problems.

The result: we cannot enable the replicaSet option, we must tell exactly on which server to connect.

In case of a primary reelection we must manually change it.

Is there a way to bypass this normal behaviour?

A more specific question is

Based on Read Preference or Set Tags, can I stop the php driver to connect to all replica instances? In the above example the read preference was PRIMARY, the seed list contains only the servers from hosting A, but still the php made persistent connections to hosting B servers.

Could you share exactly how you are connecting to the replica set, including read preference and tags? The drivers connect to all members of the replica set are part of handling failover and managing read preferences and tags, but it shouldn't be delaying normal operations to have the persistent connections.

Thanks for the answer, the params were

no tags

seedlist - only the mongod from datacenter A

                'connectTimeoutMS'     => 500,
                'socketTimeoutMS'    => 500,
                'readPreference'     =>  MongoClient::RP_PRIMARY,

                'db'                => $con_params['db'],
                'replicaSet'        => isset($con_params['rs']) ? $con_params['rs'] : null,
                'secondaryAcceptableLatencyMS'    => 30,

So the drivers connects to all members of the replicaset and exchange some data, at every connect() or query ?

If this is the case would explain the lag, because even with persistent connection would be a 40-50ms latency (the same value that rs.status() responds at (member.pingMs).

I thought the drivers stores the data about the members in the connection somewhere, and it doesn't need to contact it every time (especially when specifying Primary or nearest, because these flags doesn't change so often).

I the answer is yes I need to find a way to block the client to connect to them, even if the failover would not work and even if a primary from data-center B is chosen and it cannot be reached by clients from datacenter A, performance is more important in this case.

The driver connects to all members of the replica set (discovering those not included on the seed list) and periodically checks their health. Do you see 40-50ms latency in every query? I don't believe it should be communicating with more servers than what is needed for the read preference for each query, though the PHP driver developers would have to weigh in with the final word on the internals. You can try raising mongo.ping_interval, which determines how often the state of secondaries is checked and defaults to 5 seconds.

Not at each query, but at instantiate of the MongoClient and connect().

> The driver connects to all members of the replica set (discovering
> those not included on the seed list) and periodically checks their
> health. Do you see 40-50ms latency in every query? I don't believe it
> should be communicating with more servers than what is needed for the
> read preference for each query, though the PHP driver developers would
> have to weigh in with the final word on the internals. You can try
> raising mongo.ping_interval> <http://php.net/manual/en/mongo.configuration.php#ini.mongo.ping-interval>,
> which determines how often the state of secondaries is checked and
> defaults to 5 seconds.
Raising the ping won't help for the initial "new MongoClient", as we
need to do the discovery in any case. Ping is only for further health
checks, which will also suffer from the latency of course. It's unlikely
that a PHP script *lasts* 5 seconds though.

We need to do the discovery, otherwise we don't know the tags in the
first place.

However, you have set your connectTimeoutMS to 500 ms, which means that
you probably never succeed connecting to the "Hosting B" servers. And
because there is a connection timeout, the driver blacklists these nodes
that it can't connect to. That's expected behaviour.

> > I thought the drivers stores the data about the members in the
> > connection somewhere, and it doesn't need to contact it every time
> > (especially when specifying Primary or nearest, because these flags
> > doesn't change so often).
We store the connections, but not the state information, which we
resurvey on every "new MongoClient".

> > I the answer is yes I need to find a way to block the client to
> > connect to them, even if the failover would not work and even if a
> > primary from data-center B is chosen and it cannot be reached by
> > clients from datacenter A, performance is more important in this
> > case.
If "Hosting B" is only for backup, then you could mark those nodes as
"hidden" and the PHP driver will never connect to them because MongoDB
won't disclose those nodes on an "ismaster" call. But setting nodes as
"hidden" also means the MongoDB won't every automatically failover to
those nodes as "hidden" implies "priority=0".

Thanks a lot for the answer, now I understand better how the driver works.

Hosting B is not for backup, other app reads from it locally. 

We made a hack, we set the priority 0 on hosting B slaves and we blocked the ip access, all the clients from hosting A cannot connect with the replicas from B, so the driver sees them as dead-not responding.

This way we could activated the replicaset w/o performance loss.

If we mark the replicase from B as hidden, I think the clients from hosting B will not connect to them.