I think from the tests I did here with 2 people (from different instances) the downvotes are from any instance that federates with mine (though still not 100% sure).

I asked above to downvote me from their non-mander.xyz instance and I can see 2 downvotes.