There are many compile errors in your design. But I would think the reason why you were getting no waveforms would be because you specified the top entity as "encoder" instead of "encoder_tb".
Other errors:
- it's '0' not "0"
- rst should be driven by a "signal assignment operator" <=, not a "variable assignment operator" :=
- your simulation runs forever because of the "clocking" process. You're going to need some way of preventing that (eg "while some_signal loop" instead of "while true loop" OR the VHDL2-008 stop/finish procedures).