CardDAV Failing to update on IOS devices

[Jake Garrison]

Hey Guys and Gals,

Forgive me If I've failed to find this question somewhere where it has already been answered... But I'm pretty sure my situation is unique to those found when googling this problem.

I've been working on integrating SabreDAV into our custom CRM system. Let me start with what's working:

• User Auth - Employees are able to use their CRM username and password to access the CardDAV server. 
• Shared Address Book - Everyone sees the same addresses. This was accomplished by removing "WHERE addressbookid = ?" from the getCard/s function.
• Connecting with OSX Contacts - Desktop contacts app connects without a problem and downloads all vCards.
• Updating vCards from CRM - Any changes made in the custom CRM modifies the vCard and updates the etag for that row as well as the synctoken for EACH users address book AND creates an entry in addressbookchanges table for EACH address book.
• vCards update in OSX Contacts - Any changes in the CRM are reflected on the next fetch by OSX Contacts.

So, here is where I'm having problems...

On my iOS devices, I am able to add the CardDAV server WITHOUT any problems. All the cards download as expected. However, any changes made in the CRM (new contacts, contact modifications, contact deletions) never make it over to the iOS device. 

I had all of this working once upon a time with iOS 6 and SabreDav 1.8. I feel like this has something to do with my lack of understanding of the new SyncTokens. 

I just thought it was weird that it works fine on OSX contacts but iOS contacts can't shake it. 

This sound familiar to anyone? Is there something obvious that I'm missing?

Thanks for your help! 

-Jake

[Evert Pot]

Hi Jake,

If you're not receiving any updates, that would indeed be a strong indicator that there's something wrong with your sync-tokens.
So you should either share code, or otherwise demonstrate the problem.

Cheers,
Evert

[Jake Garrison]

Here is (what i think are) the relevant chunks of code.

These are my modifications to /CardDAV/Backend/PDO.php

   public function getCards($addressbookId) {

        $stmt = $this->pdo->prepare('SELECT id, uri, lastmodified, etag, size FROM ' . $this->cardsTableName . ' WHERE addressbookid = 1 AND status != \'deleted\'');

        $stmt->execute(array());

        $result = [];

        while($row = $stmt->fetch(\PDO::FETCH_ASSOC)) {

            $row['etag'] = '"' . $row['etag'] . '"';

            $result[] = $row;

        }

        return $result;

    }

    public function getCard($addressBookId, $cardUri) {

        $stmt = $this->pdo->prepare('SELECT id, carddata, uri, lastmodified, etag, size FROM ' . $this->cardsTableName . ' WHERE addressbookid = 1 AND uri = ? LIMIT 1');

        $stmt->execute(array($cardUri));

        $result = $stmt->fetch(\PDO::FETCH_ASSOC);

        if (!$result) return false;

        $result['etag'] = '"' . $result['etag'] . '"';

        return $result;

    }

    public function getMultipleCards($addressBookId, array $uris) {

        return array_map(function($uri) use ($addressBookId) {

            return $this->getCard($addressBookId, $uri);

        }, $uris);

        $query = 'SELECT id, uri, lastmodified, etag, size FROM ' . $this->cardsTableName . ' WHERE addressbookid = 1 AND uri = IN (';

        // Inserting a whole bunch of question marks

        $query.=implode(',', array_fill(0, count($uris), '?'));

        $query.=')';

        $stmt = $this->pdo->prepare($query);

        $stmt->execute(array($uris));

        $result = [];

        while($row = $stmt->fetch(\PDO::FETCH_ASSOC)) {

            $row['etag'] = '"' . $row['etag'] . '"';

            $result[] = $row;

        }

        return $result;

    }

    public function createCard($addressBookId, $cardUri, $cardData) {

        $stmt = $this->pdo->prepare('INSERT INTO ' . $this->cardsTableName . ' (carddata, uri, lastmodified, addressbookid, size, etag) VALUES (?, ?, ?, 1, ?, ?)');

        $etag = md5($cardData);

        $result = $stmt->execute([

            $cardData,

            $cardUri,

            time(),

            strlen($cardData),

            $etag,

        ]);

        $this->addChange($addressBookId, $cardUri, 1);

        return '"' . $etag . '"';

    }

    public function updateCard($addressBookId, $cardUri, $cardData) {

        $stmt = $this->pdo->prepare('UPDATE ' . $this->cardsTableName . ' SET carddata = ?, lastmodified = ?, size = ?, etag = ? WHERE uri = ?');

        $etag = md5($cardData);

        $stmt->execute([

            $cardData,

            time(),

            strlen($cardData),

            $etag,

            $cardUri

        ]);

        $this->addChange($addressBookId, $cardUri, 2);

        return '"' . $etag . '"';

    }

public function getChangesForAddressBook($addressBookId, $syncToken, $syncLevel, $limit = null) {

        // Current synctoken

        $stmt = $this->pdo->prepare('SELECT synctoken FROM ' . $this->addressBooksTableName . ' WHERE id = 1');

        $stmt->execute();

        $currentToken = $stmt->fetchColumn(0);

        if (is_null($currentToken)) return null;

        $result = [

            'syncToken' => $currentToken,

            'added'     => [],

            'modified'  => [],

            'deleted'   => [],

        ];

        if ($syncToken) {

            $query = "SELECT uri, operation FROM " . $this->addressBookChangesTableName . " WHERE synctoken >= ? AND synctoken < ? AND addressbookid = 1 ORDER BY synctoken";

            if ($limit>0) $query.= " LIMIT " . (int)$limit;

            // Fetching all changes

            $stmt = $this->pdo->prepare($query);

            $stmt->execute([$syncToken, $currentToken]);

            $changes = [];

            // This loop ensures that any duplicates are overwritten, only the

            // last change on a node is relevant.

            while($row = $stmt->fetch(\PDO::FETCH_ASSOC)) {

                $changes[$row['uri']] = $row['operation'];

            }

            foreach($changes as $uri => $operation) {

                switch($operation) {

                    case 1:

                        $result['added'][] = $uri;

                        break;

                    case 2:

                        $result['modified'][] = $uri;

                        break;

                    case 3:

                        $result['deleted'][] = $uri;

                        break;

                }

            }

        } else {

            // No synctoken supplied, this is the initial sync.

            $query = "SELECT uri FROM " . $this->cardsTableName . " WHERE addressbookid = ?";

            $stmt = $this->pdo->prepare($query);

            $stmt->execute([$addressBookId]);

            $result['added'] = $stmt->fetchAll(\PDO::FETCH_COLUMN);

        }

        return $result;

    }

    protected function addChange($addressBookId, $objectUri, $operation) {

        $stmt = $this->pdo->prepare('INSERT INTO ' . $this->addressBookChangesTableName .' (uri, synctoken, addressbookid, operation) SELECT ?, synctoken, id, ? FROM ' . $this->addressBooksTableName);

        $stmt->execute([

            $objectUri,

            $operation

        ]);

        $stmt = $this->pdo->prepare('UPDATE ' . $this->addressBooksTableName . ' SET synctoken = synctoken + 1');

        $stmt->execute();

    }

And this is from my CRM 

$sql = "UPDATE contacts SET `uri`='$filename', `carddata`='$vCard', `lastmodified`='$timenow', `addressbookid`='1', `size`='$size', `etag`='$etag' WHERE id = '$contactId'";

if (!mysqli_query($con, $sql))  

{

echo "error updating contact vcard";

echo $sql." ";

die('Error: ' . ((is_object($GLOBALS["___mysqli_ston"])) ? mysqli_error($GLOBALS["___mysqli_ston"]) : (($___mysqli_res = mysqli_connect_error()) ? $___mysqli_res : false)));  

} else {

$sql = 'UPDATE addressbooks SET synctoken = synctoken + 1';

mysqli_query($con, $sql);

$sql = 'INSERT INTO addressbookchanges (uri, synctoken, addressbookid, operation) SELECT \''.$filename.'\', synctoken, id, 2 FROM addressbooks';

if (!mysqli_query($con, $sql)){

echo "Error! filename: $filename $sql";

}

}

Anything else you might need? 

Thanks for your help!

-Jake

[Jake Garrison]

Ok, I failed to notice all your guidelines for posting in this group. I had no idea what a Charles dump is. But after looking it up, Im so excited to find it. I can see how that would really help with debuging SabreDAV!

I was able to route my iPhone traffic through Charles and I can see where there are some issues (still not sure how to fix them though). Here is the gist.

https://gist.github.com/karpadiem/c4afc1c4477f6313bf76

Thanks again!

-Jake 

[Evert Pot]

On Mon Jul 14 19:45:33 2014, Jake Garrison wrote:
> Here is (what i think are) the relevant chunks of code.
>
>
> These are my modifications to /CardDAV/Backend/PDO.php
>

[snip]

It looks like addChange is called with the original addressbookId, and
not set to 1.

>
>
>
> And this is from my CRM
> |
> $sql = "UPDATE contacts SET `uri`='$filename', `carddata`='$vCard',
> `lastmodified`='$timenow', `addressbookid`='1', `size`='$size',
> `etag`='$etag' WHERE id = '$contactId'";
> if (!mysqli_query($con, $sql))
> {
> echo "error updating contact vcard";
> echo $sql." ";
> die('Error: ' . ((is_object($GLOBALS["___mysqli_ston"])) ?
> mysqli_error($GLOBALS["___mysqli_ston"]) : (($___mysqli_res =
> mysqli_connect_error()) ? $___mysqli_res : false)));
> } else {
> $sql = 'UPDATE addressbooks SET synctoken = synctoken + 1';
> mysqli_query($con, $sql);
> $sql = 'INSERT INTO addressbookchanges (uri, synctoken, addressbookid,
> operation) SELECT \''.$filename.'\', synctoken, id, 2 FROM addressbooks';
> if (!mysqli_query($con, $sql)){
> echo "Error! filename: $filename $sql";
> }
> }
> |

Here it looks like you are first increasing the sync-token, and then
using that to insert into the addressbookchanges table.

The original function does the opposite, which means that the sync-token
in this table are 1 too high.

Instead of fixing this, what I would actually recommend though is that
your backend class and your CRM use the same code. So ideally your CRM
just creates an instance of your PDO backend and calls createCard on it,
instead of re-doing all the work again. This is the most fool-proof way
to fix this, and keep things correct in the future when stuff changes again.

Hope that fixed it!

Evert

[Jake Garrison]

Ok, I think I have properly implemented these changes. However, I'm thinking this problem might actually be originating somewhere else. As I've been playing to Charles, I keep finding two constant errors. 

http://DOMAIN/principals/jgarrison/

REQUEST:

OPTIONS /principals/jgarrison/ HTTP/1.1

Host: DOMAIN

Pragma: no-cache

Connection: keep-alive

Accept: */*

User-Agent: Mac OS X/10.9.2 (13C1021) AddressBook/1369

Accept-Language: en-us

Content-Length: 0

Accept-Encoding: gzip, deflate

RESPONSE:

HTTP/1.1 401 Unauthorized

Date: Tue, 15 Jul 2014 15:14:23 GMT

Server: Apache/2.2.26 (Unix) mod_wsgi/3.3 Python/2.7.5 mod_fastcgi/2.4.6 mod_ssl/2.2.26 OpenSSL/0.9.8y DAV/2 PHP/5.4.24

X-Powered-By: PHP/5.4.24

Set-Cookie: PHPSESSID=6j5oba09uj41fascc71f0kj63dfofmgk7bpif21l5qh620l6imuicdii9j8mav5s544l3935l4ufhqd9nvovu8vujl4qegsgorrl0d1; path=/

Expires: Thu, 19 Nov 1981 08:52:00 GMT

Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0

Pragma: no-cache

WWW-Authenticate: Digest realm="SabreDAV",qop="auth",nonce="**************",opaque="**************"

MS-Author-Via: DAV

Content-Length: 292

Keep-Alive: timeout=15, max=100

Connection: Keep-Alive

Content-Type: application/xml; charset=utf-8

<?xml version="1.0" encoding="utf-8"?>

<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">

  <s:sabredav-version>2.0.1</s:sabredav-version>

  <s:exception>Sabre\DAV\Exception\NotAuthenticated</s:exception>

  <s:message>No digest authentication headers were found</s:message>

</d:error>

http://DOMAIN/addressbooks/jgarrison/

REQUEST:

PROPFIND /addressbooks/jgarrison/ HTTP/1.1

Host: DOMAIN

Pragma: no-cache

Accept: */*

Brief: t

Accept-Language: en-us

Accept-Encoding: gzip, deflate

Content-Type: text/xml

Content-Length: 717

Depth: 1

Connection: keep-alive

User-Agent: Mac OS X/10.9.2 (13C1021) AddressBook/1369

Prefer: return-minimal

<?xml version="1.0" encoding="UTF-8"?>

<A:propfind xmlns:A="DAV:">

  <A:prop>

    <A:add-member/>

    <D:bulk-requests xmlns:D="http://me.com/_namespace/"/>

    <A:current-user-privilege-set/>

    <A:displayname/>

    <B:max-image-size xmlns:B="urn:ietf:params:xml:ns:carddav"/>

    <B:max-resource-size xmlns:B="urn:ietf:params:xml:ns:carddav"/>

    <C:me-card xmlns:C="http://calendarserver.org/ns/"/>

    <A:owner/>

    <C:push-transports xmlns:C="http://calendarserver.org/ns/"/>

    <C:pushkey xmlns:C="http://calendarserver.org/ns/"/>

    <A:quota-available-bytes/>

    <A:quota-used-bytes/>

    <A:resource-id/>

    <A:resourcetype/>

    <A:supported-report-set/>

    <A:sync-token/>

  </A:prop>

</A:propfind>

RESPONSE:

HTTP/1.1 401 Unauthorized

Date: Tue, 15 Jul 2014 15:14:23 GMT

Server: Apache/2.2.26 (Unix) mod_wsgi/3.3 Python/2.7.5 mod_fastcgi/2.4.6 mod_ssl/2.2.26 OpenSSL/0.9.8y DAV/2 PHP/5.4.24

X-Powered-By: PHP/5.4.24

Set-Cookie: PHPSESSID=kdjpvgkll5gh5bpbvucruvf8vtqo00qsanveb3bnm9n0m1kqqqsi4begblu97sv534l75s90m68r5sh17kb077br8nar02g3ca4mu12; path=/

Expires: Thu, 19 Nov 1981 08:52:00 GMT

Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0

Pragma: no-cache

WWW-Authenticate: Digest realm="SabreDAV",qop="auth",nonce="*********",opaque="*************"

MS-Author-Via: DAV

Content-Length: 292

Keep-Alive: timeout=15, max=98

Connection: Keep-Alive

Content-Type: application/xml; charset=utf-8

<?xml version="1.0" encoding="utf-8"?>

<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">

  <s:sabredav-version>2.0.1</s:sabredav-version>

  <s:exception>Sabre\DAV\Exception\NotAuthenticated</s:exception>

  <s:message>No digest authentication headers were found</s:message>

</d:error>

Could this be the actual root of my problem? Im no longer getting any errors about sync tokens.

Here is the only modification I've made to /lib/dav/auth/backend/PDO.php

    public function getDigestHash($realm,$username) {

        $stmt = $this->pdo->prepare('SELECT digesta1 FROM '.$this->tableName.' WHERE username = ?');

        $stmt->execute(array($username));

        return $stmt->fetchColumn() ?: null;

    }

Please let me know if I'm missing any relevant information.

Thanks so much!

-Jake

[Jake Garrison]

Welllllll... not sure what I did... but I fiddled with this... and tinkered with that.... and now its working. Thanks for your help! 

I am still getting the errors from my last post. Is that a big deal? Or to be expected?

Thanks again!

-Jake

[Evert Pot]

On Tuesday, July 15, 2014 11:25:49 AM UTC-4, Jake Garrison wrote:

Ok, I think I have properly implemented these changes. However, I'm thinking this problem might actually be originating somewhere else. As I've been playing to Charles, I keep finding two constant errors. 

http://DOMAIN/principals/jgarrison/

The 401 errors are expected. iOS keeps re-negotiating the authentication information.
After every one of those 401's, the same request should be done again with an Authorization header, which succeeds.

Evert

[Thomas Tanghus]

On Tuesday 15 July 2014 11:06 Evert Pot wrote:
> The 401 errors are expected. iOS keeps re-negotiating the authentication
> information.
> After every one of those 401's, the same request should be done again with
> an Authorization header, which succeeds.

I've never really found an answer to this: Is it mandatory for the client to
re-negotiate the authentication information, or is it just a choice of
implementation; for added security maybe?

I've been asked about the 401's many times without being able to give a proper
answer.

--
Med venlig hilsen / Best Regards

Thomas Tanghus

[Evert Pot]

Hey Thomas,

It's been years since I wrote the code for digest auth, but basically my understanding is that there are two nonces, a server-generated nonce and a client-generated nonce.
If the server-generated nonce does not change (all the time) the client can cache it, and just make another request without having to first fetch it.

If that request would fail, the server would still send back its updated server nonce.

I haven't really seen many clients that successfully cache the server nonce, most clients just forget it immediately and have to do two requests where one would suffice.
So I think it's usually just laziness on the client's side, but I'm also not a 100% certain.

Evert