Hello Allard,
Are you sure it’s jitter and not just packet loss?
In any case, there’s a lot of stuff usually involved. Network quality between you and your SIP server, between the SIP server and the gateway to the analog network (if the other party is a normal phone). Current load on the SIP server or gateway. Network quality between the server and the phone on the other side, if the other party is VoIP as well.
If you’re using a sever on the internet, I would start with checking the upload speed of your internet connection. Often it is much slower than the download speed.
Alexey