That's quite bad that RSB cannot do transactional processing correctly. Fine if you can live with that but imho it should be safe by default.
And imho 10 minutes is way too long for a single transaction, especially a distributed transaction. If your processing takes so long you should probably use some long-running task for storing intermediate status, for example a saga.
And I confirm, there is a difference between SQL 2005 and 2008 in how they handle distributed transactions - in SQL 2005 you will be promoted to a distributed transaction if you open a second connection to same database inside a transaction scope. SQL 2008 will not use DTC in such case.
I've been fighting with transactional processing of messages for quite a long time and have even built an SQL-only message bus just to be able to guarantee ACID without resorting to DTC. It's nice to have it working but there are many pitfalls and if you can, it's better to avoid using DTC at all.
Best regards
RG