It's scanning through the index scanner without regarding keys that didn't come back- therefore when the intersection is done it's only taking into account those keys which were returned from the scanner and not all the keys that were asked for.
When a key doesn't exist at all, there should be 0 shards to scan through. Instead, the key that doesn't exist is basically just ignored from the query.