How do I find objects with a property inside another object in JavaScript

You should use Object.keys, Array.prototype.filter and Array.prototype.map:

// This will turn users object properties into a string array
// of user names
var userNames = Object.keys(users);

// #1 You need to filter which users aren't playing. So, you
// filter accessing users object by user name and you check that
// user.isPlaying is false
//
// #2 Using Array.prototype.map, you turn user names into user objects
// by projecting each user name into the user object!
var usersNotPlaying = userNames.filter(function(userName) { 
   return !users[userName].isPlaying; 
}).map(function(userName) {
   return users[userName];
});

If it would be done using ECMA-Script 6, you could do using arrow functions:

// Compact and nicer!
var usersNotPlaying = Object.keys(users)
                       .filter(userName => users[userName].isPlaying)
                       .map(userName => users[userName]);

Using Array.prototype.reduce

As @RobG has pointed out, you can also use Array.prototype.reduce.

While I don't want to overlap his new and own answer, I believe that reduce approach is more practical if it returns an array of user objects not playing.

Basically, if you return an object instead of an array, the issue is that another caller (i.e. a function which calls the one doing the so-called reduce) may need to call reduce again to perform a new operation, while an array is already prepared to fluently call other Array.prototype functions like map, filter, forEach...

The code would look this way:

// #1 We turn user properties into an array of property names
// #2 Then we call "reduce" on the user property name array. Reduce
//    takes a callback that will be called for every array item and it receives
//    the array reference given as second parameter of "reduce" after 
//    the callback.
// #3 If the user is not playing, we add the user object to the resulting array
// #4 Finally, "reduce" returns the array that was passed as second argument
//    and contains user objects not playing ;)
var usersNotPlaying = Object.keys(users).reduce(function (result, userName) {
    if (!users[userName].isPlaying) 
        result.push(users[userName]);
    return result;
}, []); // <-- [] is the new array which will accumulate each user not playing

Clearly using Array.prototype.reduce concentrates both map and filter in a single loop and, in large array, reducing should outperform "filter+map" approach, because looping a large array twice once to filter users not playing and looping again to map them into objects again can be heavy...

Summary: I would still use filter+map over reduce when we talk about few items because sometimes readability/productivity is more important than optimization, and in our case, it seems like filter+map approach requires less explanations (self-documented code!) than reduce.

Anyway, readability/productivity is subjective to who does the actual coding...