Hi Mike,
Yes, Shipping Destination. That's been my experience. If you're running into Sales Tax issues you might want to double check it's working correctly with a quick test by inputting a phone order and vary the billing/shipping destinations. Knowing you, I'm thinking you've already done this though.
Keep in mind your cart is calculating sales tax independently of SEOM. Are you just having the issue with local SEOM phone orders or cart orders that have been downloaded into SEOM?
-Travis