AOSP repo sync fails on Mac OSX/MacOS with UnicodeDecodeError: 'ascii' codec can't decode byte

[Bence Szalai]

This is not really a question any more, since I've managed to solve it, just posting here so if anyone in the future may face similar issue can find the solution.

I've been facing an issue when doing `repo sync` on Mac, like:

error: Cannot fetch platform/external/libxml2 (UnicodeDecodeError: 'ascii' codec can't decode byte 0xd5 in position 578: ordinal not in range(128))
Exception in thread Thread-158:
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/Volumes/.../AOSP/Nougat_7.0/android/.repo/repo/subcmds/sync.py", line 270, in _FetchProjectList
    success = self._FetchHelper(opt, project, *args, **kwargs)
  File "/Volumes/.../AOSP/Nougat_7.0/android/.repo/repo/subcmds/sync.py", line 314, in _FetchHelper
    prune=opt.prune)
  File "/Volumes/.../AOSP/Nougat_7.0/android/.repo/repo/project.py", line 1271, in Sync_NetworkHalf
    self._InitMRef()
  File "/Volumes/.../AOSP/Nougat_7.0/android/.repo/repo/project.py", line 2364, in _InitMRef
    self._InitAnyMRef(R_M + self.manifest.branch)
  File "/Volumes/.../AOSP/Nougat_7.0/android/.repo/repo/project.py", line 2370, in _InitAnyMRef
    cur = self.bare_ref.symref(ref)
  File "/Volumes/.../AOSP/Nougat_7.0/android/.repo/repo/git_refs.py", line 57, in symref
    self._EnsureLoaded()
  File "/Volumes/.../AOSP/Nougat_7.0/android/.repo/repo/git_refs.py", line 64, in _EnsureLoaded
    self._LoadAll()
  File "/Volumes/.../AOSP/Nougat_7.0/android/.repo/repo/git_refs.py", line 85, in _LoadAll
    self._ReadLoose('refs/')
  File "/Volumes/.../AOSP/Nougat_7.0/android/.repo/repo/git_refs.py", line 137, in _ReadLoose
    self._ReadLoose1(p, prefix + name)
  File "/Volumes/.../AOSP/Nougat_7.0/android/.repo/repo/git_refs.py", line 155, in _ReadLoose1
    ref_id = ref_id.decode()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd5 in position 578: ordinal not in range(128)

Actually after modifying `git_refs.py` around line 155 from:
    try:
      ref_id = ref_id.decode()
    except AttributeError:
      pass
    if not ref_id:
      return

to:

    try:
      ref_id = ref_id.decode()
    except AttributeError:
      pass
    except:
      print('ref_id cannot be decoded', file=sys.stderr)
      print(' under path: %s with name: %s' % (path, name), file=sys.stderr)
      print(' ref_id as HEX: "%s"' % ref_id.encode("hex"), file=sys.stderr) #just in case text print would fail
      print(' ref_id as text: %s' % (ref_id), file=sys.stderr)
      raise
    if not ref_id:
      return

and around line 16 from:

import os
from trace import Trace

to:

from __future__ import print_function
import os
import sys
from trace import Trace

I was able to detect the offending file, which turned out to be `/Volumes/.../AOSP/Nougat_7.0/android/.repo/projects/external/libxml2.git/refs/.DS_Store`.

So i looks like while i was browsing the downloaded files out of curiosity Finder added it's junk .DS_Store to one of the folders and in turn that file was picked up by `repo sync` as a reference, which indeed couldn't be handled as such.

Solution was to just get rid of those files, e.g. by executing `find {path to your repo fiolder} -name ".DS_Store" -delete`.

Indeed similar UnicodeDecodeError may happen for many reason, but all my internet search didn't point out the possibility of unwanted .DS_Store files as the reason.

Conclusions:
- you can use the above modification to your `git_refs.py` to find out exactly what is the problematic file causing the issue
- you may want to get rid of all .DS_Store files in your AOSP folders when using the repo tool.

[Bence Szalai]

Probably you also want to get rid of `._.DS_Store` files, which may appear sometimes as well.

[xs9627]

Saved my day, thanks!

在 2017年9月17日星期日 UTC+8上午3:54:30，Bence Szalai写道：