It's your server, you can kick whoever you want. If you need an excuse, just say they're ban evading and you don't want to get in trouble for having them in your server.