warning in log file

suddenly we started having a lot of messages on log file of master on one of shards:

Wed Dec 24 11:05:11.477 [migrateThread] warning: secondaries having hard time keeping up with migrate
Wed Dec 24 11:05:11.497 [migrateThread] warning: secondaries having hard time keeping up with migrate
Wed Dec 24 11:05:11.517 [migrateThread] warning: secondaries having hard time keeping up with migrate
Wed Dec 24 11:05:11.537 [migrateThread] warning: secondaries having hard time keeping up with migrate
Wed Dec 24 11:05:11.558 [migrateThread] warning: secondaries having hard time keeping up with migrate
~~~~~~~~~~~~~~~~

Wed Dec 24 11:05:11.920 [migrateThread] warning: secondaries having hard time keeping up with migrate

Wed Dec 24 11:05:11.940 [migrateThread] warning: secondaries having hard time keeping up with migrate

it continues to storm log file with those messages, but things look working ok

what is that? how to fix that?

When you say things look okay, did you check secondary state?  In particular their replication lag?



it was because of addition of hidden secondaries with priority 0 , and they were doing initial sync




Ah, so now you increased the majority of the cluster, yes?

If you had 3 nodes before, then 2 are majority for voting and for
writeConcern majority.  But if you added a 4th node your writeConcern
majority is now three.   If you make sure you set the votes for hidden
secondaries to 0 then voting majority at least will go back to 2,
however, for writeConcern it will still be 3.   Since migrations
always wait for replication to majority of the cluster before
committing the move, this can manifest as those messages you see (they
should stop when the nodes are all caught up).



thanks for reply Asya,

the picture was so:

1) we have 3 replicas in sharded cluster

2) each replica is 3 members replica set: 1 primary, 1 secondary and 1 arbiter

3) to each replica set it was added hidden member with zero priority

4) after 3 - I started seeing those errors messages

Finally I assumed that this happened because of adding new members so I removed them with rs.remove("...") but the warning still continued to storm the log file

just after I did stepdown on the replica set on which it was happening things became ok and chunks migrated successfully

In the future, consider doing this when you want to add a third data
holding (hidden) member:

seed that member with data files from a current secondary (follow the
process here: http://docs.mongodb.org/manual/tutorial/resync-replica-set-member/#sync-by-copying-data-files-from-another-member)

When you add this member, at the same time *remove the arbiter*.
That way you preserve three members in this replica set instead of
making it have four members.