MongoDB $sort array

MongoDB provides a very useful $sort modifier to use with the $push operation, but unfortunately it looks like I can only ask it to sort my elements by a named key - which I don't have.
Another option that could be the alternative is the sort function: sort({'array.0': 1})^source, but what if I have an array inside an array?

My documents structure in the collection is as follows:

{
"_id" : 123456,
"v" : [ 
    [ 
        some id, 
        some score
    ], 
    [ 
        "456456", 
        1.5
    ], 
    [ 
        "654645", 
        43
    ],
    ...
]
}

And I want to sort the v array by the score (second element) of each child array.

So using $sort: I can't use a key name

'$push': {
    'v': {
        '$each': [[id, score]],
        '$sort': {???: -1},
    }
}

and using sort(): I can't use an array of array:

sort({'v.???.0': -1})

Is there something I'm missing, or is there another way to do this kind of sorting without fetching the document, sorting, and replacing? That would be a very time-consuming opeartion. Thanks!