Hello Chris,
The correct RTSP URL (to use with MistServer) should be:
rtsp://user:password@domain:port/streaming/channels/1
The second URL: rtsp://user:password@domain:port/hls/live/stream.m3u8
Could possibly work if you change it to: http://user:password@domain:port/hls/live/stream.m3u8 as that should use MistServers HLS input, as far as I'm aware you cannot send a HLS stream over RTSP, so it probably should've started with HTTP.
Though we have noticed (and fixed) an issue with some RTSP cameras not being able to handle url decoding on special characters. Which is fixed in our development branches, I'll send our stable development branch your way, that will probably help with the first RTSP url.
If that doesn't work please let me know, we'd be happy to help you get the stream working.