Lazy Loading or Android Cursor
kennymac
Can anybody recommend a way to get lazy loading type functionality
with ormlite on android? I know that the conventional android cursors
provide this, but they do not support ORM. If the worst came to the
worst I could use raw sql queries to populate a cursor for the queries
that demanded lazy loading, just wondering if anybody could recommend
a better method?
Cheers,
Kenny
Gray
> Can anybody recommend a way to get lazy loading type functionality
> with ormlite on android?
point me at the documents? Google searches for 'android cursor lazy'
don't get me far.
Are you talking about the Cursor that allows you to page through a
large database? That is handled by the Dao.iterator() methods.
gray
kennymac
> I don't know about this feature Kenny. What does it do? Can you
> point me at the documents? Google searches for 'android cursor lazy'
> don't get me far.
hibernate where objects members are not queried from the db until they
are actually read. Meaning for example I could queryForAll() and
cache the result as a list of objects, but the objects themselves
would actually not be populated with their data until a call to read a
specific object/field was requested. http://en.wikipedia.org/wiki/Lazy_loading
I guess you know this and are confused by me comparing it to an
android cursor. I don't know if they are equivalent but they could
both be use to solve the problem I have.
>
> Are you talking about the Cursor that allows you to page through a
> large database? That is handled by the Dao.iterator() methods.
> gray
http://developer.android.com/reference/android/database/Cursor.html
It can be used to page through a database, but it can also be used to
back an ListAdapter object in android. This allows users to see a list
on screen and when they scroll down the list the new objects are
loaded from the db when required and not all at once (this is why I
was comparing this functionality to lazy loading). My problem is that
I am querying about 2000 objects to be displayed in a list like this,
but I don't want to cache 2000 objects all at once... rather load them
as the user scrolls down the list. I am pretty sure this is one of
the reasons Cursors exist but to use them I would have to write a
native android style Dao object to return a Cursor. This would however
detract from the usefulness of using ormlite. I would rather use orm
if possible, this is why I was asking if anybody could recommend a
way.
Thanks for the reply, and if you have any suggestions!
Kenny
Gray Watson
> I was refering to the kind of lazy loading that is implemented in
> hibernate where objects members are not queried from the db until they
> are actually read.
Yeah, ORMLite will never support lazy loading. What hibernate is doing to support this is outside any possible definition of the word "lite". However, read on.
> This is the cursor I speak of:
> http://developer.android.com/reference/android/database/Cursor.html
Right. I know about this and ORMLite uses it with the iterator() methods from the DAO. These allow you to page through the items in a big table. It won't do it automatically but by using the Iterator<T> you can get the next X results and put them into your list.
Here's the documentation on the iterator:
http://ormlite.sourceforge.net/javadoc/ormlite-core/doc-files/ormlite_2.html#IDX111
You can do it as a for loop or you can get the iterator directly.
gray
kennymac
> Yeah, ORMLite will never support lazy loading. What hibernate is doing to support this is outside any possible definition of the word "lite". However, read on.
>
to be considered "lite".
> Right. I know about this and ORMLite uses it with the iterator() methods from the DAO. These allow you to page through the items in a big table. It won't do it automatically but by using the Iterator<T> you can get the next X results and put them into your list.
>
value for the data which iterators do not provide. I have got around
this problem now by calling getHelper().getReadableDatabase() and
implementing a query on the database directly. I then use the cursor
that is returned directly. This obviously bypasses any ORM but it was
the simplest option.
As a feature suggestion for future releases of ormlite for android,
maybe a method in the dao class to return an android cursor directly
from a query? Not essential but might make life slightly simpler for
some people.
Thanks for the help!
Kenny
Gray Watson
> The BaseAdapter android class needed random access as well as a "size"
> value for the data which iterators do not provide.
Yeah. Sounds like that violates the idea of a cursor at least according to JDBC.
> I have got around
> this problem now by calling getHelper().getReadableDatabase() and
> implementing a query on the database directly.
Sure. Could the DOA help you convert a row into a T type? How about a method that takes a DatabaseResults (which you get in Android land from a Cursor) and returns the current result T? Would that have helped?
> As a feature suggestion for future releases of ormlite for android,
> maybe a method in the dao class to return an android cursor directly
> from a query? Not essential but might make life slightly simpler for
> some people.
We can get close but not quite.
- QueryBuilder.prepare() returns a PreparedQuery<T>
- PreparedQuery.compile(DatabaseConnection conn) returns a CompiledStatement.
- AndroidCompiledStatement has the cursor but it is private. I've made the getCursor() public here but it won't be on the CompiledStatement interface.
All of the levels of iterface and stuff are [unfortunately] necessary to protect the OMRLite core code from the implementation details of JDBC and Android calls.
gray
Gray Watson
>> Sure. Could the DOA help you convert a row into a T type? How about a method that takes a DatabaseResults (which you get in Android land from a Cursor) and returns the current result T? Would that have helped?
>
> This would be really helpful for my purposes.
So I added this in v4.3 and refactored it in v4.10.
http://ormlite.com/raw-queries
What it does now is give you a String[] and you return the T. What is still missing is the same thing but as an Object[]. Do you need that functionality?
> After a quick look at the documentation, I couldn't find it.
What sorts of things were you looking under? What terms or concepts?
> I plan to override java.util.List and move the cursor when necessary
> and load the data only if needed. So all the internal stuff is nicely
> hidden in a single class.
This functionality is not already included in ORMLite or are you talking about if you roll your own?
gray
p.s. I'm CC'ing the list because this sounds like stuff other's might be interested in.
Jorim Jaggi
I didn't notice that I sent the answer to you instead of Google
Groups. Sorry for that ;-)
> So I added this in v4.3 and refactored it in v4.10.
> http://ormlite.com/raw-queries
> What it does now is give you a String[] and you return the T. What is still missing is the same thing but as an Object[]. Do you need that functionality?
unclear way. The main problem is that Android needs a functionality to
jump to an arbitrary location in the list of the results (because I'm
displaying the results from the database in a ListView). Because
ORMLite supports only ordered access to the results of a query (over
the Iterator-interface), the Iterator is no option for me because I
need random access. My plans to get around this are the following:
- I make a query to the database with PreparedQuery.compile(...)
- Get the underlying cursor object over
AndroidCompiledStatement.getCursor()
- Implement a new java.util.List, lets call it CursorList
- When the user of CursorList asks for an item, jump to that location
(with the cursor), and load the data from row into the actual object,
let's call it T.
For the last step, it would be really helpful if there were a method
to convert a row which is accessed by an Android-Cursor to T. I know
that your library does this internally, but I couldn't find a method/
mechanism to do this from outside. Maybe I didn't find it or I didn't
understand all the concepts with RowMapper and so on, so it would be
great if you could help me :-)
Jorim
Gray Watson
> The main problem is that Android needs a functionality to
> jump to an arbitrary location in the list of the results (because I'm
> displaying the results from the database in a ListView). Because
> ORMLite supports only ordered access to the results of a query (over
> the Iterator-interface), the Iterator is no option for me because I
> need random access.
The GenericRawResults can also get the results as a List. This isn't efficient for large numbers of results but I suspect that the ListView isn't made for large numbers.
http://ormlite.com/javadoc/ormlite-core/com/j256/ormlite/dao/GenericRawResults.html#getResults%28%29
> For the last step, it would be really helpful if there were a method
> to convert a row which is accessed by an Android-Cursor to T.
Hrm. So the PreparedQuery is also a RowMapper. It has a mapRow(DatabaseResults results) method on it. You should be able to move the cursor, and then pass the results object to mapRow to get the T.
Let me know if that works. A small test program demonstrating that would be a great addition to the Android examples.
gray
jc
>
> The main problem is that Android needs a functionality to
> jump to an arbitrary location in the list of the results (because I'm
> displaying the results from the database in a ListView). Because
> ORMLite supports only ordered access to the results of a query (over
> the Iterator-interface), the Iterator is no option for me because I
> need random access.
>
you tried using the SimpleCursorAdapter backed by the Cursor you get
from the AndroidCompiledStatement? The class has it's own limitations
mainly with the fact that it expects an integer _id column, but it's
possible to work around that.
If that doesn't do what you need, can you provide a bit more detail
about what exactly you are trying to do in Android? It would help if
you could provide a better description about the number and types of
columns you are querying and what you are trying to display in the
list view.
Luke Meyer
> The GenericRawResults can also get the results as a List. This isn't efficient for large numbers of results but I suspect that the ListView isn't made for large numbers.
well, though I don't suppose it often does. The random access is so
that only the entries actually on the screen have to be kept in
memory.
SimpleCursorAdapter is fine for handling the raw cursor as mentioned
here, but it would be neat to have some kind of SimpleQueryAdapter
that could take the results of a query and map them into a DTO which
would then be used in each entry.
Since I'm up against exactly this problem now, I'll give it a go and
let you know if I come up with anything useful.
Gray Watson
>>
> Actually, ListView is intended to handle large numbers of entries
> well, though I don't suppose it often does. The random access is so
> that only the entries actually on the screen have to be kept in
> memory.
The problem that I see is that very few database cursors allow you to go backwards. If they do then I assume that the results are in memory, not being re-read from disk.
> Since I'm up against exactly this problem now, I'll give it a go and
> let you know if I come up with anything useful.
Please do. Best of luck.
gray
Luke Meyer
before adding it to the package, but I think it'll be really useful
when complete. I modified the ClickCounter example:
https://github.com/sosiouxme/Android-examples/tree/master/ClickCounter
My POC adapter is here:
https://github.com/sosiouxme/Android-examples/blob/master/ClickCounter/src/com/kagii/clickcounter/data/SimpleQueryAdapter.java
And the modified code to use it (instead of the original
implementation which filled the Listview with a ListAdapter) is here:
https://github.com/sosiouxme/Android-examples/blob/master/ClickCounter/src/com/kagii/clickcounter/ClickConfig.java
Some notes:
* Since I extended ResourceCursorAdapter (and thus CursorAdapter) I'm
stuck with the curious restriction that all id fields need to be named
"_id" to work properly. Debating whether this is grounds for making a
completely separate branch of BaseAdapter or if the other bits in
ResourceCursorAdapter make it worth keeping as a base, or whether
there's some better way to work around this (column alias?).
* In order to keep Android from complaining about finalizing the
Cursor I created as part of the adapter initialization, I require an
Activity as an argument instead of the usual Context, so I can have
the Activity "manage" the cursor. Hard to think of a situation where
you'd be using this adapter in a non-Activity context, but maybe I
haven't thought hard enough.
* It might be worthwhile to follow in the footsteps of
SimpleCursorAdapter and, rather than just hand the ListItemBinder an
object and a list item view, actually break out the binding by the
various sub-views that need to be populated, allowing some
optimization by finding them up front as SimpleCursorAdapter does.
... and I'll stop there and see if anyone is interested enough to give
feedback.
On Feb 20, 12:58 pm, Gray Watson <256....@gmail.com> wrote:
> The problem that I see is that very few database cursors allow you to go backwards. If they do then I assume that the results are in memory, not being re-read from disk.
least, you have a random access cursor, and as the rest of Android is
pretty obsessive about memory use, I imagine there are some
optimizations to reduce the footprint when flying through a long
Listview. But I can't say I've ever built a list long enough to really
notice.
jc
>
> Some notes:
> * Since I extended ResourceCursorAdapter (and thus CursorAdapter) I'm
> stuck with the curious restriction that all id fields need to be named
> "_id" to work properly. Debating whether this is grounds for making a
> completely separate branch of BaseAdapter or if the other bits in
> ResourceCursorAdapter make it worth keeping as a base, or whether
> there's some better way to work around this (column alias?).
>
around it without completely rewriting all the android components.
Create a CursorWrapper that handles the alias for "_id" and return the
column index that you want instead. The one side-effect of this is
that the Adapter#getItemId returns an int so if you use *that* method
you will always get "0" and will need to stick with the position and
getItem methods. Since you are already extending CursorAdapter, you
can easily add a method like getItemIdObject() to do what getItemId
but don't call Cursor.getInt().
Since you are passing the helper and query to your adapter, just wrap
the cursor in the constructor. My only feedback on your
implementation so far is that the Adapter is now specific to OrmLite.
You can make it more generic by keeping the Cursor inputs on your
Adapter and instead extend Dao/DaoImpl to add a
getCursor(PreparedQuery<T>) there. If you do that, your Dao can wrap
the Cursor and alias the _id when needed and then you also have a
Cursor that will work with the SimpleCursorAdapter as well and not
just yours.
I was planning to update and post modified samples showing this
approach, but you beat me to it.
Here is my CursorWrapper that I am using now in case so you can try
it.
/**
* A {@link CursorWrapper} implementation that allows a {@link Cursor}
* without a field named "_id" to be used with various Android {@link
ListView}
* classes that expect a column named "_id". This is done by
specifying an
* alias field name to be used in place of "_id".
*
* @author j_...@users.sourceforge.net
*/
public class NoIdCursorWrapper extends CursorWrapper {
private int idColumnIndex;
/**
* Create a NoIdCursorWrapper using the alias column index.
* @param c the cursor to wrap
* @param idColumnIndex the column index to use as the _id column
alias
*/
public NoIdCursorWrapper(Cursor c, int idColumnIndex) {
super(c);
this.idColumnIndex = idColumnIndex;
}
/**
* Create a NoIdCursorWrapper using the alias column name.
* @param c the cursor to wrap
* @param idColumnName the column name to use as the _id column alias
*/
public NoIdCursorWrapper(Cursor c, String idColumnName) {
super(c);
idColumnIndex = c.getColumnIndex(idColumnName);
}
@Override
public int getColumnIndex(String columnName) {
int index = super.getColumnIndex(columnName);
if (index < 0 && "_id".equals(columnName)) {
index = idColumnIndex;
}
return index;
};
@Override
public int getColumnIndexOrThrow(String columnName) throws
IllegalArgumentException {
int index = getColumnIndex(columnName);
if (index>=0) {
return index;
}
// let the AbstractCursor generate the exception
return super.getColumnIndexOrThrow(columnName);
};
}
Luke Meyer
SimpleCursorAdapter as well. I'm looking into creating a
AndroidBaseDaoImpl extends BaseDaoImpl with a
getCursor(PreparedQuery<T>) method to do this.
While we're at it, might as well give the wrapper a field to keep the
query and an OrmLite-specific method using that to translate itself
into a DTO; this would simplify usage of my example adapter. But I
don't think you can really remove the OrmLite-specificity of the
adapter because I want it to give you an OrmLite DTO to work with and
not require the user to work via DB columns.
Something to keep in mind with the SimpleCursorAdapter is that it's
not going to do any translation from the value in the database; it
pretty much just treats everything as a string unless you supply a
ViewBinder. It could be useful to have a corresponding OrmLite adapter
that does the mapping from field names to view names.
jc
> Great idea to wrap the cursor and make it useful to
> SimpleCursorAdapter as well. I'm looking into creating a
> AndroidBaseDaoImpl extends BaseDaoImpl with a
> getCursor(PreparedQuery<T>) method to do this.
>
> While we're at it, might as well give the wrapper a field to keep the
> query and an OrmLite-specific method using that to translate itself
> into a DTO; this would simplify usage of my example adapter. But I
> don't think you can really remove the OrmLite-specificity of the
> adapter because I want it to give you an OrmLite DTO to work with and
> not require the user to work via DB columns.
>
getCursor method
would be feasible and useful for more than just Android? We
previously left this as
updating the samples, but if other people would find these useful then
maybe we can
talk him into including an AndroidBaseDaoImpl in the android library
along with a
CursorWrapper and some extra ormlite specific adapters? opinions?
> Something to keep in mind with the SimpleCursorAdapter is that it's
> not going to do any translation from the value in the database; it
> pretty much just treats everything as a string unless you supply a
> ViewBinder. It could be useful to have a corresponding OrmLite adapter
> that does the mapping from field names to view names.
FormattingCursorAdapter
that takes the columns and a String format pattern. I did this so
that I could
use it to format data and display multiple columns in the default
ListView
layouts that have a single text view. Just throwing that out there
for for
anyone who thinks that reformatting database data maybe difficult.
Luke Meyer
On Feb 22, 12:06 pm, jc <jroma...@gmail.com> wrote:
> I had posed a question to Gray if adding a
> getCursor method would be feasible and useful for more than just Android?
generalize as the Android one has to return the Android-specific
Cursor. Which is what makes all this kinda ugly... OrmLite is doing
everything through nice, clean interfaces, and we have to cast
everything to get the Android-specific functionality.
> We previously left this as updating the samples, but if other people would find these useful then
> maybe we can talk him into including an AndroidBaseDaoImpl in the android library
> along with a CursorWrapper and some extra ormlite specific adapters? opinions?
something like this is essential for OrmLite to be generally
applicable on Android... but it needs to be done well.
jc
> On Feb 22, 12:06 pm, jc <jroma...@gmail.com> wrote:
>
> Maybe, but unfortunately it would probably not be possible to
> generalize as the Android one has to return the Android-specific
> Cursor. Which is what makes all this kinda ugly... OrmLite is doing
> everything through nice, clean interfaces, and we have to cast
> everything to get the Android-specific functionality.
>
interface that is not compatible with other databases. I said
"useful" because Android is probably one of the few platforms with
components that would want that specific Cursor interface frequently.
Once you figure it out, it's not hard to get to but it's also not
perfectly clear to an Android developer who looks at the library for
the first time.
> I doubt any coercion is needed if it's a good idea :-) I think
> something like this is essential for OrmLite to be generally
> applicable on Android... but it needs to be done well.
useful for an AndroidBaseDao/Impl first. Right now, the single method
would be nice but that can easily get added to the Android docs since
not everyone may use it.
Gray Watson
> We should figure out what else besides a getCursor() method would be
> useful for an AndroidBaseDao/Impl first. Right now, the single method
> would be nice but that can easily get added to the Android docs since
> not everyone may use it.
I've been following along with this but don't have much to add so I've not been joining in. I'm very happy to add some more base classes that provide a richer feature set for Android folks.
gray
Jorim Jaggi
so I couldn't respond. Adding this kind of functionality ORMLite
sounds wise because I bet a lot of people run into performance issues
like I did on greater lists of data.