Mongo doesn't sort in a compound text index?

I have the following text index:
{ name : 'text', modifiedDate: -1 }
in the getIndexes it looks like this:
        {
                "v" : 1,
                "key" : {
                        "_fts" : "text",
                        "_ftsx" : 1,
                        "modifiedDate" : -1
                },
                "name" : "name_text_modifiedDate_-1",
                "ns" : "testdb.testcol",
                "background" : true,
                "weights" : {
                        "name" : 1
                },
                "default_language" : "english",
                "language_override" : "language",
                "textIndexVersion" : 2
        },

When performing the following query:
db.testcol.find({ $text : { $search : 'text' } }).sort({modifiedDate:-1}).limit(20).explain(true)
I get the following:
{
        "cursor" : "TextCursor",
        "n" : 20,
        "nscannedObjects" : 14056,
        "nscanned" : 14056,
        "nscannedObjectsAllPlans" : 14056,
        "nscannedAllPlans" : 14056,
        "scanAndOrder" : true,
        "nYields" : 287,
        "nChunkSkips" : 0,
        "millis" : 12697,
        "allPlans" : [
                {
                        "cursor" : "TextCursor",
                        "n" : 20,
                        "nscannedObjects" : 14056,
                        "nscanned" : 14056,
                        "scanAndOrder" : true,
                        "nChunkSkips" : 0
                }
        ],
        "server" : "MONGOTEST:27017",
        "filterSet" : false,
        "stats" : {
                "type" : "SORT",
                "works" : 28077,
                "yields" : 287,
                "unyields" : 287,
                "invalidates" : 2351,
                "advanced" : 20,
                "needTime" : 28056,
                "needFetch" : 0,
                "isEOF" : 1,
                "forcedFetches" : 1,
                "memUsage" : 59431,
                "memLimit" : 33554432,
                "children" : [
                        {
                                "type" : "TEXT",
                                "works" : 28056,
                                "yields" : 287,
                                "unyields" : 287,
                                "invalidates" : 2351,
                                "advanced" : 13997,
                                "needTime" : 14058,
                                "needFetch" : 0,
                                "isEOF" : 1,
                                "keysExamined" : 14056,
                                "fetches" : 14056,
                                "parsedTextQuery" : {
                                        "terms" : [
                                                "text"
                                        ],
                                        "negatedTerms" : [ ],
                                        "phrases" : [ ],
                                        "negatedPhrases" : [ ]
                                },
                                "children" : [ ]
                        }
                ]
        }
}

Why is it doing a scan & order query instead of using the modifiedDate:-1 part of the index to sort the results?