The error you're getting doesn't have anything to do with using the property in an order_by. It's being triggered just by accessing "Product.Rating". WIth hybrid properties, when you access them via the class as you've done here, the "self" parameter is set to the Product class itself. So on the second line of the function, you are actually saying "reviewsCount = len(Product.Reviews)", which doesn't make any sense.
You need to define a separate implementation of the hybrid property that returns an SQL construct so that it can be embedded in an SQL query, using the "<propertyname>.expression" decorator. Since you need to access a separate table to calculate your Rating, you probably need a correlated subquery, something like this:
That example uses "func.sum" to calculate a sum over the related rows. For your purposes, you probably want "func.avg" to calculate the mean.
(Note that the performance of this query is going to get worse as the number of products and ratings increases)
Hope that helps,
Simon