How to test alternate sys-img.xml files?

[Duane Sand]

Is there a way to add another sys-img.xml file to the list of Add-On Sites listed by my instance of SDK Manager?
I would like to test my zipped Mips system image, that was packaged for use with Google's standard sys-img.xml file.
I tried to test it by creating my own sys-img.xml file, and inserting that into the Add-On Sites list via the User Defined Sites screen.
(That test method worked fine, for the older style of addon.xml file like Intel used for their ABI 10 system image.)

When I try using the User Defined Sites screen with a sys-img.xml file containing a  <sdk:sdk-sys-img> tag,
the Android SDK Manager Log screen shows

Fetching URL //ubuntu-duanes/intel/sdk-img.xml

but no following Validate XML or Parse XML steps, and no discovery of the system image zip.
Instead, if the server happens to also have an addon.xml file at that URL, the log shows Validate XML and Parse XML steps happening on file //ubuntu-duanes/intel/addon.xml .
 

[Xavier Ducrohet]

Create an addon list (see this one for instance:
http://dl-ssl.google.com/android/repository/addons_list-2.xml )

and then point to it before launching the SDK Manager with:

$ export SDK_TEST_BASE_URL=http://www.mips.com/folder/containing/the/list/
$ tools/android

--
Xavier Ducrohet
Android SDK Tech Lead
Google Inc.
http://developer.android.com | http://tools.android.com

Please do not send me questions directly. Thanks!

[Duane Sand]

That didn't work for me. 
I cloned the standard addson_list-2.xml, added one entry for my local sys-img.xml.
I also cloned the standard repository-7.xml file, without changes.
The log looks okay (but short) with

Fetching https://ubuntu-duanes/repo8/addons_list-2.xml
Fetched Add-ons List successfully
Fetching URL: https://ubuntu-duanes/repo8/repository-7.xml
Done loading packages

But the 'Official Add-on Sites' page is empty (not even containing the standard things I did not remove) and the main SDK Manager screen shows none of these things.

From the source code, it looks to me like the SDK_TEST_BASE_URL switch is intended to be used with a cloning of the entire contents of the //dl-ssl.google.com/android/repository directory.
If SDK_TEST_BASE_URL is nonempty and some url begins with //dl-ssl.google.com/android/repository, it unconditionally replaces that url with one with the test prefix.
If the resulting file is missing, it gets tossed rather than reverting to the non-test url.
I don't know the names of all objects at the //dl-ssl.google.com/android/repository/* location, so I can't use wget to clone it all.

SDK_TEST_BASE_URL is an effective way to run unit test suites, but I don't yet see how to use it to try out a sys-img addon in a production environment.

[Raphaël Moll]

Exact, SDK_TEST_BASE_URL is designed to replace the whole thing for internal testing, not to override specific items (testing should fail if it's not proper, not silently revert and make us test something else).

For what you want, ideally you should be using SDK Manager > Tools > Manager Add-on Sites > User Defined Sites and add your sys-img.xml there. Unfortunately it only supports addons there at that point so I made a little patch to work around this:

https://android-review.googlesource.com/40240

Simply apply this patch, "make sdkmanager" and use the resulting android tool; then use the user defined sites dialog to add a url ending with "sys-img.xml". It can be a file:/// or http(s) URL.

HTH

R/

[Duane Sand]

That worked; thanks!

[Duane Sand]

This method of using sys-img.xml files via the User Defined Sites screen is not working in Release 21 preview 2.
(The older method using addon.xml does work, as before.)
Preview2 is acting as if change  https://android-review.googlesource.com/#/c/40240/  is not included.

I do see that change in aosp master, and in the change log for SDK changeset 79146c389bcf8fb8f1252683738c02cf4adf0ddd.
Is Prevew2 built from the aosp master branch, or from some internal repo?

[Raphaël Moll]

Duane, is that from ADT or from the standalone "android" AVD Manager tool?

Tor, do you know at which point you cut r21 preview 2?
R/

 

[Tor Norbye]

I grabbed the latest finished build from AOSP master yesterday around 1pm PDT and the build manifest says that it was build from "platform/sdk 79146c389bcf8fb8f1252683738c02cf4adf0ddd".

That CL ( https://android-review.googlesource.com/#/c/40240/ ) should definitely be in it since it was integrated 3 weeks ago, and more recent fixes have been verified by users to be fixed in preview 2. Maybe there's some other bug in play?

-- Tor

[Duane Sand]

On Thursday, August 23, 2012 1:33:37 PM UTC-7, Raphaël wrote:

Duane, is that from ADT or from the standalone "android" AVD Manager tool?

I was using the User-Defined Add-on screen of ADT's instance of the AVM Manager.
I get the same result from running AVD Manager stand-alone, by executing the (linux) tools/android program via command line.
Standalone's About is showing "Android SDK Manager, revision 21, Add-on XML Schema #5, Repository XML Schema #7".

The symptom I'm seeing is that the "Android SDK Manager Log" is showing
    Fetching URL: http://www.mips.com/global/sdk-sys-img.xml
    Done loading packages.
without showing any Validate XML or Parse XML step, and not finding any images to download.

There are several ways to get to that particular silent behavior:
1. Not having change 40240 in the build, or
2. Misspelling the URL to a nonexistant file at an existing host site.
3. Garbled xml that doesn't fit template for sys-img.xml

I've rechecked the spelling of the URL.  The file opens properly in a browser. 
SDK Manager works fine if the URL is changed to http://www.mips.com/global/sdk-addon.xml for the older method.

[Raphaël Moll]

I'll check on my side using  that URL, sounds like a bug on my side.

R/

[Duane Sand]

I just tried with my current copy of R20, and it behaved similarly, not proceeding to a Validate XML step.
This was working for me 3 weeks ago, with a different build of R20.  I suspect there is something wrong with my xml file.

[Duane Sand]

I copied https://dl-ssl.google.com/android/repository/sys-img.xml to my system and served it from my /var/www folder, as filename sys-img.xml
And named it on my User Defined Addons screen.  It too was skipped (by my enhanced R20) without a Validate XML step.  :(

[Duane Sand]

I tried to debug this by adding monitor.logError() calls to SdkSource.java.
Those changes were effective in the copy of SdkManager invoked at command line as tools/android.
But none of those code changes show up in the copy of SdkManager accessed from Eclipse ADT.
Even when I start over with an entirely fresh install of Eclipse, after killing adb and deleting all the known folders of Eclipse & ADT state.
Is there an entirely separate copy of these SdkManager modules, embedded in the ADT plugin files?

[Tor Norbye]

Yes, the plugins bundle their own copies of the libs (since the plugins get copied into the eclipse install, and they can't reference the SDK area which typically lives somewhere else).

$ jar tvf com.android.ide.eclipse.base_20.0.0.v201206260941-0626.jar | grep sdklib

405634 Tue Jun 26 09:42:00 PDT 2012 libs/sdklib.jar

-- Tor

[Raphaël Moll]

Yes ADT has its own copy of the jar embedded. If you build it yourself, you need to do something like this to update ADT:

$ . build/envsetup.sh && lunch sdk-eng && ./sdk/eclipse/scripts/create_all_symlinks.sh

This will build the sdk manager jars and copy/symlink them in the ADT libs folder. If you then use Eclipse RCP to map the ADT plugin project and run Eclipse in debug mode from there, you get the updated jar files in ADT. For the purpose of this debugging, using the standalone version is just a lot simpler ;-)

FYI earlier I tried using your link and I get the system images just fine in the manager (using "android sdk"). Of course I'm running from tip-of-tree in master AOSP. I'll get the preview from Tor to compare.

R/

[Duane Sand]

In a tools_r20 SDK with cl 40240 added, using the stand-alone SDK Manager, I am getting the skipped sys-img.xml file behavior for both
http://www.mips.com/global/sdk-sys-img.xml and a local copy.  But the sys-img.xml file for Intel at dl-ssl.google.com is getting XML verified etc.

By adding trace code, I found that the test
     if (url.endsWith("sys-img.xml"))
in loadUserAddons() is NOT triggering on the failing URLs for Mips (on my runs), and IS triggering on the sys-img.xml file for Intel.
The erroneous call on SdkAddonSource() rather than on SdkSysImgSource() then leads to getXmlSchemaVersion() returning a failure of 0,
when its root-element search loop gets a mismatch on
   if (getRootElementName().equals(name)) .

I noticed that the endsWith() check does not trim the url first, and does not have the //$NON-NLS 1$ comment that it likely should.
But changing those did not help the check.  Nor did replacing the endsWith() check by a contains() check.
I don't see what the problem is.  I'm not familiar with Java or with how it handles Unicode.

[Duane Sand]

How does one view the log.error() messages created from an ISdkLog in module SdkSources?

In Java or in this part of SDK, how does one format a string to show all characters' actual representations, perhaps like Python's %r format?

[Raphaël Moll]

FYI I pushed preview 3 of Tools 21 yesterday. The SDK Manager in 21rc3 does properly display the MIPS sys-img site for me when configured in Tools>Add-on sites> User defined sites..

Details at https://plus.google.com/109739758904815427948/posts/F9UX8peXjDQ

Hope this helps

R/

[Duane Sand]

The add-on-site method is still not working for me.

I installed the 21rc3 ADT into Indigo Eclipse on linux , then from ADT I installed 21rc3 Android SDK Tools.
Using ADT's SDK Manager, I entered 'http://www.mips.com/global/sdk-sys-img.xml' into the User Defined add-on sites page.
The fetching and validate steps took a very long time, nearly a minute.  Maybe due to added logging in this build?
The Android SDK Manager log showed:

Fetching URL: http://www.mips.com/global/sdk-sys-img.xml
Validate XML: http://www.mips.com/global/sdk-sys-img.xml/addon.xml
Done loading packages

but no system image packages were found or loaded.
And note the /addon.xml suffix added to the URL in the Validate XML step.  There is no such filename.

On Friday, August 31, 2012 3:51:25 PM UTC-7, Raphaël wrote:

FYI I pushed preview 3 of Tools 21 yesterday. The SDK Manager in 21rc3 does properly display the MIPS sys-img site for me when configured in Tools>Add-on sites> User defined sites..

[Duane Sand]

On my second attempt, I ran the tools/android program directly rather than via ADT's copy of SDK Manager.
In its rescanning of the existing list of sites, it did sucessfully fetch, validate, and parse http://www.mips.com/global/sdk-sys-img.xml,
and it did find the two system image files (api 15 and api 16) described inside.

On my third attempt, I ran ADT's copy of SDK Manager, and its log showed success in re-fetching, validating, and parsing sdk-sys-img.xml, and finding the two image files.

[Raphaël Moll]

On Wed, Sep 5, 2012 at 2:25 PM, Duane Sand <duane....@gmail.com> wrote:

On my second attempt, I ran the tools/android program directly rather than via ADT's copy of SDK Manager.
In its rescanning of the existing list of sites, it did sucessfully fetch, validate, and parse http://www.mips.com/global/sdk-sys-img.xml,
and it did find the two system image files (api 15 and api 16) described inside.

On my third attempt, I ran ADT's copy of SDK Manager, and its log showed success in re-fetching, validating, and parsing sdk-sys-img.xml, and finding the two image files.

Note that starting with ADT 21, launching the sdk manager from ADT will run the *external* sdk manager, it's spawning an external app process just as if you had started it from the command-line. You should not get a different behavior depending on how you start it. I have been running it that way with your sys-img url for a the last couple weeks or so and each time I do see your system images. 

It sounds like it works for you now, which is nice to know. Let me know if you see that odd behavior again.

R/

[Duane Sand]

On Wednesday, September 5, 2012 2:49:15 PM UTC-7, Raphaël wrote:

It sounds like it works for you now, which is nice to know. Let me know if you see that odd behavior again.

I repeated the experiment, with a fresh install of Eclipse and 21rc3, and invoking tools/android for every step.
In SDK Manager's 1st scan of the sys-img.xml addon, the fetch step took 2 minutes,
the validate step took another 2 minutes and wrote the garbled URL into the log,
and no parsing occurred.  The two  2-minute delays look like thread timeouts.

In the second invocation of SDK Manager, its re-scan of the whole list did the new sys-img.xml item successfully with parse in less than a second, and found the two image file URLs.


In my third experiment, SDK's 1st scan choked on the sys-img.xml addon for two minutes in fetch and validate steps.
I then did a second scan without leaving the SDK Manager process, and it again choked without getting these system image files.
I then cleared SDK Manager's cache and did a third scan without leaving the SDK Manager process, and it again choked.
I then exited the SDK Manager, restarted it, and did a fourth scan which promptly succeeded.

I think the problem is hidden garbage characters in the initial URL string.  (I've been using cut/paste to enter this text from elsewhere.)
When SDK Manager shuts down entirely, it relies on a textfile version of its list of addon sites.  That file has cleaned-up names in it.
When merely rescanning from within the same process, it instead relies on some other internal form of this addon sites list, that has junk characters that are interfering with the *sys-img.xml pattern check and with the way of appending a default 'addon.xml' name.
I haven't been able to view the actual UTF contents of this URL.

I want to add calls to log.error() from within module SdkSources.  Where is that log written to?  Is it turned on by default?

How can I print out the binary content of the utf bytes in this string?  I'm looking for something like repr(s) or "%r" in python.

[Duane Sand]

Hand-typing the URL did not help; still choked.

Killing the SDK Manager process while choked, and starting a new one with the revised addon list file, was successful.

[Raphaël Moll]

On Wed, Sep 5, 2012 at 5:04 PM, Duane Sand <duane....@gmail.com> wrote:

On Wednesday, September 5, 2012 2:49:15 PM UTC-7, Raphaël wrote:

It sounds like it works for you now, which is nice to know. Let me know if you see that odd behavior again.

I repeated the experiment, with a fresh install of Eclipse and 21rc3, and invoking tools/android for every step.
In SDK Manager's 1st scan of the sys-img.xml addon, the fetch step took 2 minutes,
the validate step took another 2 minutes and wrote the garbled URL into the log,
and no parsing occurred.  The two  2-minute delays look like thread timeouts.

I don't remember having any thread timeout in the xml parser/validator.

There isn't much that should take 2 minutes in there. If that's really how long it takes, run this via the Eclipse debugger and pause the VM during the pause to see where the main thread is stuck.

If you don't have the Eclipse source setup for debugging, you can either use something like the Eclipse remote debugger to connect to a live process.

Can you give an example of that "garbled" URL you see in the log? I don't see it in your earlier emails.

The code that takes the URL from the sites dialog is in com.android.sdkuilib.internal.repository.ui.AddonSitesDialog.userNewOrEdit(boolean) ; it does a url.trim() to remove leading/trailing whitespace.

 

In the second invocation of SDK Manager, its re-scan of the whole list did the new sys-img.xml item successfully with parse in less than a second, and found the two image file URLs.


In my third experiment, SDK's 1st scan choked on the sys-img.xml addon for two minutes in fetch and validate steps.
I then did a second scan without leaving the SDK Manager process, and it again choked without getting these system image files.
I then cleared SDK Manager's cache and did a third scan without leaving the SDK Manager process, and it again choked.
I then exited the SDK Manager, restarted it, and did a fourth scan which promptly succeeded.

I think the problem is hidden garbage characters in the initial URL string.  (I've been using cut/paste to enter this text from elsewhere.)
When SDK Manager shuts down entirely, it relies on a textfile version of its list of addon sites.  That file has cleaned-up names in it.
When merely rescanning from within the same process, it instead relies on some other internal form of this addon sites list, that has junk characters that are interfering with the *sys-img.xml pattern check and with the way of appending a default 'addon.xml' name.
I haven't been able to view the actual UTF contents of this URL.

Java is internally all UTF-16 so if you step through it, you see the actual utf characters.

The java.utils.Properties.store() methods escape any non-Latin-1 characters, so if there was "garbled" characters you'd see them in the properties file in the form of escapes.

http://docs.oracle.com/javase/6/docs/api/java/util/Properties.html#store(java.io.OutputStream, java.lang.String)

 

I want to add calls to log.error() from within module SdkSources.  Where is that log written to?  Is it turned on by default?

Only to the log window.

 

How can I print out the binary content of the utf bytes in this string?  I'm looking for something like repr(s) or "%r" in python.

 

There's no such direct equivalent afaik.

You could use String.getBytes("UTF-8") for example to get utf-8 bytes out.

http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/String.html#getBytes(java.lang.String)

For debugging I'd use something like 

log.error(null/*throwable*/, "%s", Arrays.toString(s.getBytes("UTF-8")));

R/

[Duane Sand]

Found the problem via System.out.printf() calls.

See https://android-review.googlesource.com/#/c/42695/ for the suggested fix.

There are two places urls for addon.xml and sys-img.xml files are mixed and must be distinguished.
Raphael fixed the place dealing with filenames reprocessed from the local repository.cfg file.
But the user addon-site url dialog was committing the initial file validate attempt to be as an add-on only.
This caused the other symptoms.
Please include this fix in the next release candidate.

I was unable to find to where the ISdkLog logger writes its stuff; it might be going to null.

[Duane Sand]

On Friday, September 21, 2012 8:39:41 AM UTC-7, David Kellogg wrote:

Hey guys, hate to hijack the conversation, but I'm also trying to host a local sysimg.xml. I'd be interested in trying the utility you seem to be working on, but I'm not familiar with git and I'm behind a corporate proxy. How can I grab this & try it out?

You can do this without SDK source changes.
Upgrade an installation of SDK to use SDK release 21 release candidate 4, using the new slick way to install previews/release candidates:

http://tools.android.com/download/adt-21-preview
http://tools.android.com/preview-channel

Create a sys-img.xml file via manually editing an existing example.  Have it point (via URLs) to the system image zips that you want to use.
You can find an example xml file at

http://www.mips.com/global/sdk-sys-img.xml

The zipped file needs to have a certain structure of folders and files.  Follow the example of

http://download-software.intel.com/sites/landingpage/android/sysimg_x86-16_r01.zip

This will involve manually gathering files built from both standard android makes and makes of sdk.  (For both, you'll need a full Android src repo)

To get the sys-img.xml file used by SDK, go to SDK Manager, Tools, Manage Add-on Sites, User-Defined Sites, New.  Ignore the warning about system images.  Name the xml as either a http-hosted file on your intranet, or as a local file.  The spelling of the last part of the name is important; it must end with 'sys-img.xml'.  Hit Close.  On SDK Manager's main panel, lower right corner, is an icon for seeing SDK Manager's log window.  Click on it.  You'll see that SDK Manager attempted to open and scan your new xml file, and found it, but didn't get anything useful from it in its first attempt.  That's a bug in the release candidate.  Exit SDK Manager (and Eclipse, if you are using it).  Re-enter SDK Manager, and you'll see in the log window that it is now parsing the xml file and noting the name of your system image zip file.  If it liked it, the system image should now be showing on SDK Manager's main panel as an available (but not yet installed) system image, for whatever Android ABI level you claimed.  You can then install it.

This will only work for cpu types and library abi conventions that SDK is already expecting, ie one of arm, armv7, x86 atom, or mips.

There is an older way to access system image files, that works with older releases of SDK also.  This was and is still being used for Intel's first system image for API 10,  Follow the example at

http://software.intel.com/sites/landingpage/android/addon.xml

This involves a larger zipped folder structure than is used with sys-img.xml links.