Yes, it will be released. The stack is unwound with an exception, so monitors are released and finally blocks are executed. (If a finally block calls into an STM function an exception is thrown again.)
There is a potential for deadlock if you mix transactions with synchronized. ScalaSTM is just a library, so it isn't aware of scheduling contraints imposed by the monitors, and the underlying algorithm is not lock free. It is based on SwissTM, which eagerly acquires write locks. Under contention it also uses read locks.