I think you are right. Last July we had a challenge session at the CSTA conference to develop an App Inventor app that picked a random contact from the phone's contact list and call it (where the user could decide to call or not). But there's no way in App Inventor to just get a list of the phone's contacts. Instead, the contacts are opened in a Android list view and you then can select one manually.
So I ended up having to write an Android activity to return a list of the phone's contacts to App Inventor via the ActivityStarted component. App Inventor starts the activity and handles the interaction with the user.
The attached zip file contains:
* RandomContactCaller.apk -- the whole app package
* RandomContact.apk -- The Android activity that returns a list of contacts
* RandomContact.java -- Java source code for the Android activity
* RandomContactCaller.aia -- The App Inventor source code
I thought I had written up a web doc describing this for the conference but I can't find it now??