How did you configure it? There are different connectors: ssh, jnlp, swarm, ...
- SSH one is master based, so if you remove the jenkins agent from the Jenkins UI then it should be gone.
- JNLP is client side, so, if you installed as a service then you need to remove that particular service.