[tokumx] mongo2toku is not catching up

I am in an effort to try and migrate mongo to tokumx.

The mongo replica set include 3 members,

I created a dump then restore it to toku and initiate the toku replica set.

It looks like the tools is working (I do see quries on the toku logs), but for some reason the lag keep increasing:

ed Dec 24 13:39:45.854 trying to add new host toku-20001-001-prod.eu1.appsflyer.com:27017 to replica set rep001

Wed Dec 24 13:39:45.855 successfully connected to new host toku-20001-001-prod.eu1.appsflyer.com:27017 in replica set rep001

Wed Dec 24 13:39:45.856 Primary for replica set rep001 changed totoku-20001-001-prod.eu1.appsflyer.com:27017

Wed Dec 24 13:39:45.856 replica set monitor for replica set rep001 started, address is rep001/toku-20001-001-prod.eu1.appsflyer.com:27017

connected to: rep001/toku-20001:27017

Wed Dec 24 13:39:45.856 [ReplicaSetMonitorWatcher] starting

Wed Dec 24 13:39:45.856 [mongo2toku] starting new replica set monitor for replica set afreports with seed of mongo-20006:27017,mongo-20007:27017,mongo-20008:27017

Wed Dec 24 13:39:45.856 [mongo2toku] successfully connected to seed mongo-20006:27017 for replica set afreports

Wed Dec 24 13:39:45.857 [mongo2toku] changing hosts to { 0: "mongo-20006-001-prod.eu1.appsflyer.com:27017", 1: "mongo-20008-001-prod.eu1.appsflyer.com:27017", 2: "mongo-20007-001-prod.eu1.appsflyer.com:27017" } from afreports/

Wed Dec 24 13:39:45.857 [mongo2toku] trying to add new host mongo-20006-001-prod.eu1.appsflyer.com:27017 to replica set afreports

Wed Dec 24 13:39:45.857 [mongo2toku] successfully connected to new host mongo-20006-001-prod.eu1.appsflyer.com:27017 in replica set afreports

Wed Dec 24 13:39:45.857 [mongo2toku] trying to add new host mongo-20007-001-prod.eu1.appsflyer.com:27017 to replica set afreports

Wed Dec 24 13:39:45.858 [mongo2toku] successfully connected to new host mongo-20007-001-prod.eu1.appsflyer.com:27017 in replica set afreports

Wed Dec 24 13:39:45.858 [mongo2toku] trying to add new host mongo-20008-001-prod.eu1.appsflyer.com:27017 to replica set afreports

Wed Dec 24 13:39:45.859 [mongo2toku] successfully connected to new host mongo-20008-001-prod.eu1.appsflyer.com:27017 in replica set afreports

Wed Dec 24 13:39:45.859 [mongo2toku] successfully connected to seed mongo-20007:27017 for replica set afreports

Wed Dec 24 13:39:45.861 [mongo2toku] successfully connected to seed mongo-20008:27017 for replica set afreports

Wed Dec 24 13:39:45.865 [mongo2toku] Primary for replica set afreports changed to mongo-20006-001-prod.eu1.appsflyer.com:27017

Wed Dec 24 13:39:45.867 [mongo2toku] replica set monitor for replica set afreports started, address is afreports/mongo-20006-001-prod.eu1.appsflyer.com:27017,mongo-20007-001-prod.eu1.appsflyer.com:27017,mongo-20008-001-prod.eu1.appsflyer.com:27017

Wed Dec 24 13:39:50.991 [mongo2toku] synced up to 1419411394:952 (Dec 24 08:56:34:3b8), source has up to 1419428390:798 (Dec 24 13:39:50:31e), 16996 seconds behind source.

Wed Dec 24 13:42:56.597 [mongo2toku] synced up to 1419411413:838 (Dec 24 08:56:53:346), source has up to 1419428576:1163 (Dec 24 13:42:56:48b), 17163 seconds behind source.

Wed Dec 24 13:45:59.948 [mongo2toku] synced up to 1419411430:496 (Dec 24 08:57:10:1f0), source has up to 1419428759:1796 (Dec 24 13:45:59:704), 17329 seconds behind source.

Wed Dec 24 13:49:07.999 [mongo2toku] synced up to 1419411448:1062 (Dec 24 08:57:28:426), source has up to 1419428947:1528 (Dec 24 13:49:07:5f8), 17499 seconds behind source.

Wed Dec 24 13:52:03.678 [mongo2toku] synced up to 1419411465:768 (Dec 24 08:57:45:300), source has up to 1419429123:1399 (Dec 24 13:52:03:577), 17658 seconds behind source.

Wed Dec 24 13:55:12.759 [mongo2toku] synced up to 1419411484:1155 (Dec 24 08:58:04:483), source has up to 1419429312:1458 (Dec 24 13:55:12:5b2), 17828 seconds behind source.

Wed Dec 24 13:58:17.879 [mongo2toku] synced up to 1419411503:284 (Dec 24 08:58:23:11c), source has up to 1419429497:1155 (Dec 24 13:58:17:483), 17994 seconds behind source.

.

.

.

.

It doesn't look like any system metric is reaching limits:

CPU utilization ~30%

IOwait ~0.5

LoadAVG ~1.5 (instance with 8 cores)

Memory usage i fairly low:

ariel@toku-20001-001-prod:~# free -g

             total       used       free     shared    buffers     cached

Mem:            60          3         56          0          0          0

-/+ buffers/cache:          3         56

Swap:            0          0          0 

The instance has provision IOPS disks (4000 iops)

All the instances (mongo and toku) are the same region, latency is  few ms.

more info:

root@toku-20001-001-prod:~# mongo

TokuMX mongo shell v2.0.0-mongodb-2.4.10

Any advice will be appreciated.

What version of TOKUMX are you using?
Make sure to adjust cacheSize to avoid swapping, especially if you are running multiple instances of tokumx on the same box

Please privde us with the followings:

1. All complete log files.  Not snippets or just logs from one machine, especially if it might be a problem with replication or sharding, which necessarily involve multiple machines.  If worried about size, gzip should work wonders on these logs.
2. All server configurations, either from config files or from db.runCommand('getCmdLineOpts') (similar to "show variables" in MySQL) on each server.
3. CPU, memory, and disk I/O.  dstat, vmstat and iostat are very important, and should probably be logged (e.g. iostat -x -k 60) all the time.