Touchdb Android Attachments ?

[Deepwinter]

I'm seeing something that I think is strange when trying to use attachments in TouchDb Android.

I create the attachment using Ektorp as

byte[] thumb = values.getAsByteArray("thumb");

ByteArrayInputStream thumbStream =  new ByteArrayInputStream(thumb);

AttachmentInputStream a = new AttachmentInputStream("thumb.jpg",

    thumbStream,

    "image/jpeg");

String _rev = couchDbConnector.createAttachment(id, documentNode.get("_rev").getTextValue(), a);

But then, when I check in the database (actually I'm checking in a replicated database, not looking directly in the database on the phone) I see that the VALUE of the attachment is

{"id":"medium.jpg","contentType":"image/jpeg","contentLength":-1}

I'm getting this by doing a wget on the replicated database, i.e. 

http://url.something.net/BD4AF2DB-8C51-47B1-9F53-7541E88353AF/thumb.jpg

Going through this same process using TouchDB on the iphone gives me a binary file (what I expected).   Is there something wrong with with my code for creating an attachment?  

[Marty Schoch]

Please go ahead and file an issue on github. My guess is replication
of attachments is not working correctly. Its not something I know
I've tried, and I just took a glance at the unit tests, and while I
test attachments and replication, I don't see anything to test
replication of documents with attachments.

marty

[Deepwinter]

Marty - before filing the issue, I wanted to check your hypothesis that the problem I'm seeing is due to replication.    I don't have sqlite3 installed on my phone, but I'm not able to view the sqlite database via adb shell.   I did verify that my app can display images replicated FROM a remote database (though that doesn't say anything about this issue).  These attachments work completely as expected, in spite of the fact that calls to couchDbInstance.getAttachment return an InputStream reporting a content length = -1.    I also checked what the value of the contents an attachment that has NOT gone through replication are, and reading as a string I also get {"id":"thumb.jpg","contentType":"image/jpeg","contentLength":-1} from 

try {

        data = couchDbConnector.getAttachment(documentId,

                        "thumb.jpg");

        Log.v(TAG, data.toString());

   

        } catch (DocumentNotFoundException e) {

        return null;

        }

I've filed the ticket, but I'm uneasy since you say the unit tests for attachment saving are functional.  From what I wrote above either there's a bug in saving of attachments (rather than their replication), or I've made a mistake I'm not seeing in my attachment creation code below.

[Marty Schoch]

Thanks,

Theres no harm in filing a ticket, it does help ensure I won't forget about it.

While the attachment code is unit tested, the Ektorp adapter is not
unit tested at all, and the TDRouter class has spotty coverage. So
there are a couple of paths your sample code will go through that are
not well tested.

I think I have enough information to try and reproduce it, hopefully
I'll find time tomorrow.

marty

[Marty Schoch]

On Tue, May 29, 2012 at 8:21 PM, Marty Schoch <marty....@gmail.com> wrote:
> Thanks,
>
> Theres no harm in filing a ticket, it does help ensure I won't forget about it.
>
> While the attachment code is unit tested, the Ektorp adapter is not
> unit tested at all, and the TDRouter class has spotty coverage.  So
> there are a couple of paths your sample code will go through that are
> not well tested.
>
> I think I have enough information to try and reproduce it, hopefully
> I'll find time tomorrow.
>
> marty

This was a bug in the TouchDB Ektorp adapter. The code incorrectly
processed the body of a POST/PUT. Anyone affected by this bug should
try the latest code.

I also added a few test cases for the Ektorp Adapter.

marty

[Deepwinter]

Marty,

This functionality still isn't working correctly for me. I've updated the ticket with a gist containing details.

m.

[Matt Martin]

I'm seeing similar behavior when replicating from a remote database and retrieving an attachment from one of the documents.

1. CouchDbConnector connect = Global.getLocalTouchConnector(mContext);
2. AttachmentInputStream data = connect.getAttachment("real_photo", "image.jpeg");
3. byte[] buffer = new byte[]{};
4. int i = 10;
5.  
6. try
7. {
8.         while( (i=data.read(buffer)) > 0);
9. }
10. catch (IOException e)
11. {
12.         // TODO Auto-generated catch block
13.         e.printStackTrace();
14. }
    
    
    'data' shows a contentLength of -1 and zero bytes are returned from data.read().

[Marty Schoch]

On Tue, Aug 14, 2012 at 12:30 AM, Matt Martin <matt....@gmail.com> wrote:
> I'm seeing similar behavior when replicating from a remote database and
> retrieving an attachment from one of the documents.
>
> CouchDbConnector connect = Global.getLocalTouchConnector(mContext);
> AttachmentInputStream data = connect.getAttachment("real_photo",
> "image.jpeg");
> byte[] buffer = new byte[]{};
> int i = 10;
>
> try
> {
> while( (i=data.read(buffer)) > 0);
> }
> catch (IOException e)
> {
> // TODO Auto-generated catch block
> e.printStackTrace();
> }
>
> 'data' shows a contentLength of -1 and zero bytes are returned from
> data.read().

All of the problems in that thread were resolved.
https://github.com/couchbaselabs/TouchDB-Android/issues/26

Can you run the Attachments test case (from the Ektorp test package
com.couchbase.touchdb.testapp.ektorp.tests) and see if that works?
Have you verified that the document replicated correctly in the first
place? There are several problems with replicator right now,
particularly if you have large attachments.

marty

[Matt Martin]

The attachments unit test passes. I tried a small 16-byte string attachment of text/plain with the same results. The following code has provided me with more information:

1. Photo p = connect.get(Photo.class, "real_photo");
2. Attachment a = p.getAttachments().get("image.jpeg");

Examining 'a': 

• The attachment length of 68431 is correct.
  
• a.getDataBase64() returns null.
  
• a.digest() returns a SHA-1 digest.

It appears that the attachment content is not being replicated but its metadata is. The same results were found when examining the text/plain attachment.

[Marty Schoch]

Actually this doesn't tell us as much as you think. getDataBase64()
won't return the attachment data in most cases (see the javadocs, its
only useful when passing attachment data in-line)

I suspect if you call a.isStub() it will return true.

Further, the fact that you say the digest is SHA-1 increases the
chance that we have (or at one-time had) the data. Because I believe
CouchDB will return an MD5 hash, so if we just got the meta-data from
CouchDB and stored it, this would be MD5 not SHA-1.

I don't know what else to suggest, I'll try to reproduce what you're
seeing this afternoon.

marty

[Marty Schoch]

I looked into this further. At first I thought it was related to me
not setting the Content-Length header on the replies, this got lost
when I changed the code to stream attachments back. But it turned out
that wasn't related.

The problem is your code to read from the inputstream does not
allocate any space in the buffer. So the read always returns -1.

In the first sample code you set where you read from
AttachmentInputStream, change the 2 lines before you read data to be:

byte[] buffer = new byte[1024];
int i = -1;

Reading the value of getContentLength() from the AttachmentInputStream
will still return -1 (until I merge this patch in the future to set
Content-Length correctly). But you will be able to read the data.

marty

[Matt Martin]

Rookie mistake! Resolved. Thanks.