The house sparrows are overtaking my house?

So, there is a bird house on my house. And sparrows have lived there ever since we moved in, 3 years ago. But what I've noticed is that they scare off all the other birds. And sparrows are apparently non-native to the US, and they murder other bird babies. But, at the same time, they lived at my house before I did. What right do I have to try and get rid of them? Sparrows are birds too! Just because they are mean and not pretty-- is that any reason to shoo them away? What do you think?