Can't sync to gc anymore

[Julien Cubizolles]

After upgrading master today, syncing bbdb to gc fails with:

> Applying base config from file /home/wilk/info/emacs/ASynK/config/config_v7.json...
> Applying base config from file /home/wilk/info/emacs/ASynK/config/config_v7.json...done
> Applying user customizations from file /home/wilk/.asynk/config.py...
> Applying user customizations from file /home/wilk/.asynk/config.py...done
> [09:54:44.850 INFO] Debug logging to file: /home/wilk/.asynk/logs/asynk_logs.2015-09-23.09-54-44.850389
> [09:54:44.850 INFO] Deleting log files older than 7 days, if any...
> [09:54:44.851 INFO] Deleting log files older than 7 days, if any...done
> [09:54:44.851 INFO] Parsing BBDB file /home/wilk/.bbdb...
> [09:54:44.851 INFO] Parsing BBDB Store with encoding utf-8...
> [09:54:44.959 INFO] Parsing BBDB Store with encoding utf-8...Success
> [09:54:44.959 INFO] Successfully parsed 540 entries.
> [09:54:44.960 INFO] Attempting to log into Google...
> [09:54:44.962 INFO] access_token is expired. Now: 2015-09-23 07:54:44.962221, token_expiry: 2015-09-22 15:05:42
> Traceback (most recent call last):
> File "/home/wilk/info/emacs/ASynK/asynk.py", line 321, in <module>
> main()
> File "/home/wilk/info/emacs/ASynK/asynk.py", line 318, in main
> asynk.dispatch()
> File "/home/wilk/info/emacs/ASynK/asynk/asynk_core.py", line 97, in dispatch
> res = getattr(self, self.get_op())()
> File "/home/wilk/info/emacs/ASynK/asynk/asynk_core.py", line 366, in op_sync
> pname = self._load_profile()
> File "/home/wilk/info/emacs/ASynK/asynk/asynk_core.py", line 590, in _load_profile
> self._login()
> File "/home/wilk/info/emacs/ASynK/asynk/asynk_core.py", line 85, in _login
> coll.login()
> File "/home/wilk/info/emacs/ASynK/asynk/state_collection.py", line 336, in login
> self.get_username(), self.get_pwd())
> File "/home/wilk/info/emacs/ASynK/asynk/pimdb_gc.py", line 71, in __init__
> self.gc_init()
> File "/home/wilk/info/emacs/ASynK/asynk/pimdb_gc.py", line 286, in gc_init
> self.credentials.refresh(http=self._new_http())
> File "/home/wilk/info/emacs/ASynK/asynk/pimdb_gc.py", line 244, in _new_http
> debug = self.get_config().get_gc_logging()
> File "/home/wilk/info/emacs/ASynK/asynk/state.py", line 355, in get_gc_logging
> return self.get_db_config('gc')['log']
> KeyError: 'log'
> Exception AttributeError: "'GCPIMDB' object has no attribute 'server'" in <bound method GCPIMDB.__del__ of <pimdb_gc.GCPIMDB object at 0x7f39b3ce3d50>> ignored


If I remove the dat file (because of the access_token expired message),
I get the authorization window and, after I accept, I get the following:

> Applying base config from file /home/wilk/info/emacs/ASynK/config/config_v7.json...
> Applying base config from file /home/wilk/info/emacs/ASynK/config/config_v7.json...done
> Applying user customizations from file /home/wilk/.asynk/config.py...
> Applying user customizations from file /home/wilk/.asynk/config.py...done
> [09:57:00.694 INFO] Debug logging to file: /home/wilk/.asynk/logs/asynk_logs.2015-09-23.09-57-00.694450
> [09:57:00.694 INFO] Deleting log files older than 7 days, if any...
> [09:57:00.695 INFO] Deleting log files older than 7 days, if any...done
> [09:57:00.695 INFO] Parsing BBDB file /home/wilk/.bbdb...
> [09:57:00.695 INFO] Parsing BBDB Store with encoding utf-8...
> [09:57:00.807 INFO] Parsing BBDB Store with encoding utf-8...Success
> [09:57:00.807 INFO] Successfully parsed 540 entries.
> [09:57:00.807 INFO] Attempting to log into Google...
> [09:57:00.807 INFO] Starting to listen on port 1977...
> [09:57:00.808 INFO] Staring the wonderful oAuth dance...
> Using PPAPI flash.
> [6666:6666:0923/095702:ERROR:nss_util.cc(856)] After loading Root Certs, loaded==false: NSS error code: -8018
> Created new window in existing browser session.
> [09:57:05.456 INFO] Successfully retrieved access token
> [09:57:05.634 INFO] Failed to retrieve access token: {
> "error" : "invalid_grant",
> "error_description" : "Code was already redeemed."
> }
> ----------------------------------------
> Exception happened during processing of request from ('127.0.0.1', 39693)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
> self.process_request(request, client_address)
> File "/usr/lib/python2.7/SocketServer.py", line 321, in process_request
> self.finish_request(request, client_address)
> File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
> self.RequestHandlerClass(request, client_address, self)
> File "/usr/lib/python2.7/SocketServer.py", line 655, in __init__
> self.handle()
> File "/usr/lib/python2.7/BaseHTTPServer.py", line 340, in handle
> self.handle_one_request()
> File "/usr/lib/python2.7/BaseHTTPServer.py", line 328, in handle_one_request
> method()
> File "/home/wilk/info/emacs/ASynK/asynk/pimdb_gc.py", line 231, in do_GET
> self.credentials = self.flow.step2_exchange(params)
> File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 142, in positional_wrapper
> return wrapped(*args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 2165, in step2_exchange
> raise FlowExchangeError(error_msg)
> FlowExchangeError: invalid_grantCode was already redeemed.
> ----------------------------------------
> [09:57:05.866 INFO] Failed to retrieve access token: {
> "error" : "invalid_grant",
> "error_description" : "Code was already redeemed."
> }
> ----------------------------------------
> Exception happened during processing of request from ('127.0.0.1', 39697)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
> self.process_request(request, client_address)
> File "/usr/lib/python2.7/SocketServer.py", line 321, in process_request
> self.finish_request(request, client_address)
> File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
> self.RequestHandlerClass(request, client_address, self)
> File "/usr/lib/python2.7/SocketServer.py", line 655, in __init__
> self.handle()
> File "/usr/lib/python2.7/BaseHTTPServer.py", line 340, in handle
> self.handle_one_request()
> File "/usr/lib/python2.7/BaseHTTPServer.py", line 328, in handle_one_request
> method()
> File "/home/wilk/info/emacs/ASynK/asynk/pimdb_gc.py", line 231, in do_GET
> self.credentials = self.flow.step2_exchange(params)
> File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 142, in positional_wrapper
> return wrapped(*args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 2165, in step2_exchange
> raise FlowExchangeError(error_msg)
> FlowExchangeError: invalid_grantCode was already redeemed.
> ----------------------------------------
> Traceback (most recent call last):
> File "/home/wilk/info/emacs/ASynK/asynk.py", line 321, in <module>
> main()
> File "/home/wilk/info/emacs/ASynK/asynk.py", line 318, in main
> asynk.dispatch()
> File "/home/wilk/info/emacs/ASynK/asynk/asynk_core.py", line 97, in dispatch
> res = getattr(self, self.get_op())()
> File "/home/wilk/info/emacs/ASynK/asynk/asynk_core.py", line 366, in op_sync
> pname = self._load_profile()
> File "/home/wilk/info/emacs/ASynK/asynk/asynk_core.py", line 590, in _load_profile
> self._login()
> File "/home/wilk/info/emacs/ASynK/asynk/asynk_core.py", line 85, in _login
> coll.login()
> File "/home/wilk/info/emacs/ASynK/asynk/state_collection.py", line 336, in login
> self.get_username(), self.get_pwd())
> File "/home/wilk/info/emacs/ASynK/asynk/pimdb_gc.py", line 71, in __init__
> self.gc_init()
> File "/home/wilk/info/emacs/ASynK/asynk/pimdb_gc.py", line 279, in gc_init
> self.credentials = self._oauth_dance(storage)
> File "/home/wilk/info/emacs/ASynK/asynk/pimdb_gc.py", line 266, in _oauth_dance
> http = self.credentials.authorize(http=self._new_http())
> File "/home/wilk/info/emacs/ASynK/asynk/pimdb_gc.py", line 244, in _new_http
> debug = self.get_config().get_gc_logging()
> File "/home/wilk/info/emacs/ASynK/asynk/state.py", line 355, in get_gc_logging
> return self.get_db_config('gc')['log']
> KeyError: 'log'


Any idea what's going wrong ?

Julien.

[Sriram Karra]

I am not sure why your tokens are failing, but I did push out a fix for the log traceback you are seeing. Can you pull to tip of master (421a3e9)

Julien.

--
You received this message because you are subscribed to the Google Groups "ASynK Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to asynk+un...@googlegroups.com.
To post to this group, send an email to as...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/asynk/871tdp4ldx.fsf%40free.fr.
For more options, visit https://groups.google.com/d/optout.

[Julien Cubizolles]

Sriram Karra <karra.etc-Re5JQE...@public.gmane.org> writes:

> I am not sure why your tokens are failing, but I did push out a fix for the
> log traceback you are seeing. Can you pull to tip of master (421a3e9)

I did, and it's working again. Thanks.

Two remarks though:

+ asynk.py is now a link to asynk_cmdline.py but it doesn't have
execution rights

+ I now need to run asynk.py with ./asynk.py, I now need to use python
./asynk.py (I guess that's related to the previous remark)


Julien.

[Sriram Karra]

On Tue, Sep 29, 2015 at 3:56 PM, Julien Cubizolles <j.cubi...@free.fr> wrote:

Two remarks though:

+ asynk.py is now a link to asynk_cmdline.py but it doesn't have
  execution rights

+ I now need to run asynk.py with ./asynk.py, I now need to use python
  ./asynk.py (I guess that's related to the previous remark)

Yes, I made the change as I felt it was important to allow the user to choose the specific version of python installed in their set up. For e.g. by make asynk_cmdline.py a #! executable will require me to hard code a path to the interpreter, and this will not work very well with a virtualenv set up for e.g. This becomes more important now as many required packages are now mentioned as external dependencies that users have to install, and I fully anticipate many people to use pip in a virtualenv. At least I do :) So it is a conscious choice, and I intend to keep it this way. 

[Julien Cubizolles]

Julien Cubizolles <j.cubi...@free.fr> writes:

> Sriram Karra <karra.etc-Re5JQE...@public.gmane.org> writes:
>
>> I am not sure why your tokens are failing, but I did push out a fix for the
>> log traceback you are seeing. Can you pull to tip of master (421a3e9)
>
> I did, and it's working again. Thanks.

In the process, I got many duplicates both on the gmail and bbdb
site. Merging them results in multiple entries in the following field.

asynk:bbgcsynk:bb
asynk:bbgcsynk:bb
gids
bbdb-id

I'd like to remove the duplicates on both sides and then merge the two
databases. What would you advise to clean the databases ?

Deleting the asynk:bbgcsynk on the gmail results in the corresponding
contacts being deleted from bbdb at the next sync. Should I remove all
asynk:*** fields and bbdb-id on both sides before syncing ?

Julien.

[Sriram Karra]

+asynk mailing list in the hope it may be relevant to a wider audience.

On Tue, Oct 6, 2015 at 9:02 PM, Julien Cubizolles <j.cubi...@free.fr> wrote:

 

>
> I did, and it's working again. Thanks.

In the process, I got many duplicates both on the gmail and bbdb
site. Merging them results in multiple entries in the following field.

Unfortunately the sync is not transactional, and in case of some bugs this is bound to happen. Defending against such dups is too much work for this side project.

 

asynk:bbgcsynk:bb
asynk:bbgcsynk:bb
gids
bbdb-id

I'd like to remove the duplicates on both sides and then merge the two
databases. What would you advise to clean the databases ?

The best way to recover is to restore from backup - without running ASynK look for the backup directory and restore the BBDB from your local dir. Gmail offers a roll back based on timestamp, which should give you back your contact store.

If you don't have changes on both ends you care to sync to the other side, this is what you should do:

- Delete your BBDB file

- Restore your gmail contacts to a known safe time

- Delete your asynk profile profile 

- Clear sync artefacts (see http://asynk.io/doc/asynk/asynk_6.html#Misc-Housekeeping-Activities for examples of this and above)

- Create the profile afresh

- Do a sync to recreate your BBDB as a mirror of your gmail contacts

 

[Julien Cubizolles]

Sriram Karra <karr...@gmail.com> writes:


> The best way to recover is to restore from backup - without running ASynK
> look for the backup directory and restore the BBDB from your local dir.
> Gmail offers a roll back based on timestamp, which should give you back
> your contact store.
>
> If you don't have changes on both ends you care to sync to the other side,
> this is what you should do:
>
> - Delete your BBDB file
> - Restore your gmail contacts to a known safe time
> - Delete your asynk profile profile
> - Clear sync artefacts (see
> http://asynk.io/doc/asynk/asynk_6.html#Misc-Housekeeping-Activities for
> examples of this and above)
> - Create the profile afresh
> - Do a sync to recreate your BBDB as a mirror of your gmail contacts

The problem is that some of my bbdb contacts weren't synchronized to
gmail. I'm thinking I'll

- exporting all my gmail contacts that used to be synchronized to vcard
- delete them from gmail
- import them in bbdb
- clean duplicates in bbdb and the remaining gmail contacts
- delete the asynk profile
- clear sync artefacts
- create a new profile
- sync this new profile.

Any reason why it shouldn't work ?

Thanks for your help.

Julien.

[Sriram Karra]

On Wed, Oct 7, 2015 at 1:43 PM, Julien Cubizolles <j.cubi...@free.fr> wrote:

 

The problem is that some of my bbdb contacts weren't synchronized to
gmail. I'm thinking I'll

- exporting all my gmail contacts that used to be synchronized to vcard
- delete them from gmail
- import them in bbdb
- clean duplicates in bbdb and the remaining gmail contacts
- delete the asynk profile
- clear sync artefacts
- create a new profile
- sync this new profile.

Any reason why it shouldn't work ?

It should work; but if you access your backups from the most recent BBDB backup and turn the clock back in gmail, you can avoid most of the above pain, right?  

[Julien Cubizolles]

Sriram Karra <karr...@gmail.com> writes:


> It should work; but if you access your backups from the most recent BBDB
> backup and turn the clock back in gmail, you can avoid most of the above
> pain, right?

Sure, but I'll still have duplicates and I'd like to clean the mess I
made.

Julien.

[Sriram Karra]

In theory you should be able to roll back to a version without the duplicates. 

[Julien Cubizolles]

It turned out to be quite easy, since bbdb-delete-field-or-record can
act on all records and makes it straightforward to clear a single field
in the whole database.

I:
* exported all the google contacts synced by asynk to a vcard
* imported it in bbdb
* merged the duplicates
* cleared all the gids fields (from google) with C-* C-u C-k
* cleared all the notes fields (from the importation of the vcard) with C-* C-u C-k

However Asynk won't delete the profile, complains with:

--8<---------------cut here---------------start------------->8---
Due to errors in clearing sync tags, profile bbgcsynk is not being deleted from your Asynk configuration
--8<---------------cut here---------------end--------------->8---

Since all my contacts are now in a clean bbdb file, I guess I can force
it to clear it. Is it enough to remove the state.json file in ~/.asynk ?

Thanks again for your help.

Julien.

[Julien Cubizolles]

Julien Cubizolles <j.cubi...@free.fr> writes:


> It turned out to be quite easy

not so fast...

When first syncing to populate the gmail contacts, I get many errors
like the following:

--8<---------------cut here---------------start------------->8---
Upload to Google failed for: Someone: Reason: Can only have one primary phone. Code: SYNC_BAD_REQUEST
--8<---------------cut here---------------end--------------->8---

The corresponding entry in bbdb is:

--8<---------------cut here---------------start------------->8---
phone (Mobile): some number
phone (work): some other number
mail: some email
bbdb-id: 085be125-79a2-11e5-b66c-80ee73838ff9
folder: asynk
mail-alias: some alias
--8<---------------cut here---------------end--------------->8---

How does gmail distinguishes the primary and other phone numbers ?

Julien.

[Sriram Karra]

If you want to start afresh, just delete or rensme your ~/.asynk dir

--
Find ASynK useful? Consider leaving a tip at this Bitcoin address: 1KarraEGixRyScZ9qFHdekxNkgpPTuzWBc
---

You received this message because you are subscribed to the Google Groups "ASynK Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to asynk+un...@googlegroups.com.

[Julien Cubizolles]

Julien Cubizolles <j.cubi...@free.fr> writes:

> Julien Cubizolles <j.cubi...@free.fr> writes:
>
>
>> It turned out to be quite easy
>
> not so fast...
>
> When first syncing to populate the gmail contacts, I get many errors
> like the following:
>

> Upload to Google failed for: Someone: Reason: Can only have one primary phone. Code: SYNC_BAD_REQUEST
>
>

> The corresponding entry in bbdb is:
>

> phone (Mobile): some number
> phone (work): some other number
> mail: some email
> bbdb-id: 085be125-79a2-11e5-b66c-80ee73838ff9
> folder: asynk
> mail-alias: some alias
>

> How does gmail distinguishes the primary and other phone numbers ?

I can send you examples of an entry for which the the upload works and
another one where it fails if you need.

Julien.

[Sriram Karra]

On Sun, Oct 25, 2015 at 2:28 AM, Julien Cubizolles <j.cubi...@free.fr> wrote:

I can send you examples of an entry for which the the upload works and
another one where it fails if you need.

Yes, that would help 

[Julien Cubizolles]

I just sent two examples to your private address.

Regards,

Julien.