Just thought I'd post an update here as I solved this problem The problem was not at all that I couldn't pass data to the client side. You can indeed create a regular RPC Service which makes calls to the Force.com api instead of the datastore, then pass this data to the client as a list or whatever you like.
My mistake was not putting my security token after the password while trying to login. Doh!
public List<Account> getStuff()
{
List<Account> list = new ArrayList<Account>();
try
{
// config
ConnectorConfig config = new ConnectorConfig();
config.setTransport(GaeHttpTransport.class);
config.setUsername("usernamehere");
config.setPassword("pasword+TOKEN");
config.setTraceMessage(true);
// initialise connection
PartnerConnection connection = Connector.newConnection(config);
// give it a query
QueryResult qr = connection.query("SELECT Id, Name FROM Account ORDER BY Name ASC");
// put records in array
SObject[] records = qr.getRecords();
// add to list as Account
for(int i = 0; i < qr.getSize() ; i++)
{
String id = records[i].getId();
String name = (String) records[i].getField("Name");
Account account = new Account(id, name);
list.add(account);
}
return list;
}
catch(ConnectionException e)
{
System.out.print(e.getMessage());
System.out.print(e.getStackTrace());
//System.out.print(e.getMessage());
return null; // empty
}
}
It converts the SObjects to Account objects and returns them in a list, as I prefer this to returning an array or SObjects.