Latest dev update won't boot on my Cr-48

[Steve Pirk]

I unlocked my Cr-48 this morning and saw that there was a new version available. I clicked restart to update, and the system hangs at the Chrome splash screen. I am in hardware dev mode, and I am also on the dev channel, so this would be the latest version of dev that was just pushed out.

Much to my relief, I was able to get to the console with ctrl-alt-f2, and I logged in, so the system is up, but it looks like the window manager failed to start.

I am going to try to scp all of my local downloads to another machine in case I have to toggle back to stock mode. In the meantime, are there any processes the Chromium developers would like me to look at?

I should also probably post this on Chromebook Central. Let me know if I should move the discussion there.

-- steve

[Richard Barnette]

On Jan 31, 2013, at 11:01 AM, Steve Pirk <pirk...@gmail.com> wrote:

> I unlocked my Cr-48 this morning and saw that there was a new version available. I clicked restart to update, and the system hangs at the Chrome splash screen. I am in hardware dev mode, and I am also on the dev channel, so this would be the latest version of dev that was just pushed out.
>
> Much to my relief, I was able to get to the console with ctrl-alt-f2, and I logged in, so the system is up, but it looks like the window manager failed to start.
>
> I am going to try to scp all of my local downloads to another machine in case I have to toggle back to stock mode. In the meantime, are there any processes the Chromium developers would like me to look at?
>

A few files that would be of interest:
/etc/lsb-release (to show what version you've got, exactly)
/var/log/ui/ui.*
/var/log/chrome/*
/var/log/messages

Also, reporting the dates on these files could help:
/var/log/update_engine/*
The *content* of the update engine logs isn't necessarily
going to be useful, although one of those files will show
the before and after version numbers.

Commands to try to get status:
sudo status system-services
sudo status ui

You may be able to get back w/o toggling dev mode. Based on
your symptoms, it's likely you can trigger a rollback to the previous
version by rebooting 5 times (or so).

I recommend you poke around, and grab a few things, then try to
force the rollback: if the rollback works, debug will be easier, because
you should have a more functional system.

> I should also probably post this on Chromebook Central. Let me know if I should move the discussion there.
>

Yeah, a query on Chromebook Central along the lines of "anybody
else seen this?" might be helpful.


> -- steve
>
> --
> Chromium OS discuss mailing list: chromium-...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-os-discuss?hl=en

-- jrb

[Steve Pirk]

Thanks Richard! I will post the contents of those files shortly. There was an update announcement on the Chromebook-Central list and 2 other cr-48 users are reporting the same thing. Hanging at boot. I will update that thread with what I find out from the tests you mentioned.

[Dylan Reid]

+Benson, he saw something similar on his Alex today. He had a cursor
that would move over the chrome logo and managed to click the
invisible "browse as guest" button in the lower left corner.

[Steve Pirk]

Hmmm... I may have to try that. Not sure I have a cursor...

-- steve

[Steve Pirk]

On Thu, Jan 31, 2013 at 11:23 AM, Richard Barnette <jrbar...@chromium.org> wrote:

  /etc/lsb-release (to show what version you've got, exactly)
    /var/log/ui/ui.*
    /var/log/chrome/*
    /var/log/messages

/etc/lsb-release - 

CHROMEOS_RELEASE_TRACK=dev-channel

CHROMEOS_RELEASE=3536.0.0

GOOGLE_RELEASE=3635.0.0 

/var/log/ui/ui.LATEST complains that org.freedesktop.DBus.Error.NameHasNoOwner: Could not get owner of name 'org.chromium.CrosDisks': no such name

org.chromium.UpdateEngine was not provided by any .service files

same error for

org.chromium.Mtpd

org.bluez

Then it complains that VidModeEtensions are missing on display 0:0

Last line is

ERROR:gpu-watchdog_thread.cc(209) The GPU process hung. Terminating after 10000 ms.

Last errors in /var/log/messages are that the Liveness checker did not receive a reply.

system-services show start/running

ui start/running, process 760

Previous version was 3428.65.0

I have never tried a rollback of versions. Let me google the commands.

Thanks!

-- steve

[Steve Pirk]

Holy cow. I clicked a bit further to the right hoping to find the invisible guest mode button, and what I clicked on brought up a regular login screen. It took my gmail user/pass and it is slowly reloading all the pages I have open before the upgrade.

The *first* thing I am going to do is upload all of the files in my Downloads folder.

Hat/tip to Dylan Reid for mentioning the invisible links in the lower left. Try to find the "Login" one instead of the "Browse as Guest" link.

-- steve

[Richard Barnette]

On Jan 31, 2013, at 12:06 PM, Steve Pirk <pirk...@gmail.com> wrote:

>
> On Thu, Jan 31, 2013 at 11:23 AM, Richard Barnette <jrbar...@chromium.org> wrote:
> /etc/lsb-release (to show what version you've got, exactly)
> /var/log/ui/ui.*
> /var/log/chrome/*
> /var/log/messages
>
> /etc/lsb-release -
> CHROMEOS_RELEASE_TRACK=dev-channel
> CHROMEOS_RELEASE=3536.0.0
> GOOGLE_RELEASE=3635.0.0
>
> /var/log/ui/ui.LATEST complains that org.freedesktop.DBus.Error.NameHasNoOwner: Could not get owner of name 'org.chromium.CrosDisks': no such name
>
> org.chromium.UpdateEngine was not provided by any .service files
> same error for
> org.chromium.Mtpd
> org.bluez
>

The DBus error messages are actually normal. :-(

> Then it complains that VidModeEtensions are missing on display 0:0
> Last line is
> ERROR:gpu-watchdog_thread.cc(209) The GPU process hung. Terminating after 10000 ms.
>

This isn't normal, and is probably very much at the root of the problem.

> Last errors in /var/log/messages are that the Liveness checker did not receive a reply.

Any errors in /var/log/messages would be buried deep; the last
few lines aren't likely to help. That said, the GPU process complaint
seems likely to be more informative than anything from /var/log/messages.

> system-services show start/running
> ui start/running, process 760
>
> Previous version was 3428.65.0
> I have never tried a rollback of versions. Let me google the commands.
>

I'm not sure if it's been discussed enough for a Google search to turn
up useful information. Here's the summary:
* When a new update is downloaded and installed, a counter is set
for the new image, and the image is marked as "not yet successful".
The information is stored in the boot disk. The initial counter is 5.
* On boot, about 45 seconds after Chrome shows the login screen,
the update_engine daemon goes and marks the image as "booted
successfully".
* On every boot, if the image is still marked "not yet successful",
the firmware decrements the counter. If the counter goes to zero,
the firmware falls back to the previous OS version on the boot disk.

So, if you have an update where Chrome fails to start or the system
can't stay running for at least 45 seconds, then repeatedly rebooting
the system will roll you back to the previous version.

In the case of this bug, the rollback won't work: The fact that
'system-services' was running means that update_engine would
have started and marked the image as good. :-(


> Thanks!
>
> -- steve

-- jrb

[Richard Barnette]

On Jan 31, 2013, at 12:23 PM, Steve Pirk <pirk...@gmail.com> wrote:

> Holy cow. I clicked a bit further to the right hoping to find the invisible guest mode button, and what I clicked on brought up a regular login screen. It took my gmail user/pass and it is slowly reloading all the pages I have open before the upgrade.
> The *first* thing I am going to do is upload all of the files in my Downloads folder.
>
> Hat/tip to Dylan Reid for mentioning the invisible links in the lower left. Try to find the "Login" one instead of the "Browse as Guest" link.
>

... has anyone filed a bug in the tracker for this? Seems like we're
well past the time where we have enough data.

-- jrb

[Richard Barnette]

On Jan 31, 2013, at 1:14 PM, Richard Barnette <jrbar...@chromium.org> wrote:

> On Jan 31, 2013, at 12:23 PM, Steve Pirk <pirk...@gmail.com> wrote:
>
>> Holy cow. I clicked a bit further to the right hoping to find the invisible guest mode button, and what I clicked on brought up a regular login screen. It took my gmail user/pass and it is slowly reloading all the pages I have open before the upgrade.
>> The *first* thing I am going to do is upload all of the files in my Downloads folder.
>>
>> Hat/tip to Dylan Reid for mentioning the invisible links in the lower left. Try to find the "Login" one instead of the "Browse as Guest" link.
>>
> ... has anyone filed a bug in the tracker for this? Seems like we're
> well past the time where we have enough data.
>

http://crosbug.com/38429

-- jrb

[jay...@chromium.org]

Any idea why these two version numbers are different in the last two lines below from /etc/lsb-release from what Steve quoted earlier (or Steve, did you make a typo when you quoted it :-)) ?

CHROMEOS_RELEASE_TRACK=dev-channel

CHROMEOS_RELEASE=3536.0.0

GOOGLE_RELEASE=3635.0.0 

[Aka Arnold]

I have the same problem with the todays update for the CR 48!

It downloads completely and then the   RESTART brings up the Chrome icon and the title Chrome and then hangs!  

It's my  first  problem with the CR 48 ever and I have the machine for the last 1 1/2 years!

--- Tony

[Steve Pirk]

On Thu, Jan 31, 2013 at 1:44 PM, <jay...@chromium.org> wrote:

/etc/lsb-release

Yes, Steve entered a typo... Doh!

CHROMEOS_RELEASE_APPID={87efface-864d-49a5-9bb3-4b050a7c227a}

CHROMEOS_RELEASE_BOARD=x86-mario-signed-mp-v2keys

CHROMEOS_RELEASE_DESCRIPTION=3635.0.0 (Official Build) dev-channel x86-mario 

CHROMEOS_RELEASE_NAME=Chrome OS

CHROMEOS_RELEASE_TRACK=dev-channel

CHROMEOS_RELEASE_VERSION=3635.0.0

Now that I am in, I should be able to copy/paste log data instead of typing it in manually. I starred issue 38429, so will be able to post data there also.

The watchdog error is not present in /var/log/ui/ui.LATEST, but was present when the system was hung.

-- steve

[Steve Pirk]

Tony,

Wait for 10-15 seconds, and then carefully move the cursor over the lower left of the screen. The cursor will change to a select icon when you are over a link. About 1" from the left is the log in as guest link. Try clicking that. When I tried it, I was presented with a username/password log in screen. I entered my gmail user/pass and it logged me back into my session. It might be worth a shot.

-- steve

--

[Benson Leung]

Posting a few more findings from crosbug.com/38429. comment #33

from sh...@chromium.org :

There are two orthogonal bugs here. The first one is a GPU process
crash on alex (not confirmed yet whether this happens on other
platforms) when the machine is first booted. This should be split off
into another bug, if confirmed.

The second bug seems to be related to wallpaper. On the machine we've
done the most repro on, it was installed at R21, took an update to
R23, and then from there to R26. A user ("User 21") is created on
R21, with one of the default wallpapers set, and migrated up R26. A
second user ("User 26") is then created.

Depending on which user was last logged-in:

* If "User 21" is last logged-in, on boot-up, the user account
selection pods will not show up. The underlying Chrome instance is
still alive though; if the machine is logged-in to guest mode
(clicking on an invisible button), and then logged-out again, the
user-selection pods will show, but with a black background.
* If "User 26" is last logged-in, on boot-up, everything is fine.

--
Benson Leung
Software Engineer, Chrom* OS
ble...@chromium.org

[Benson Leung]

The suspicion at this point is that if you created your user account
before R23, and never touched the new wallpaper settings during R23 or
R25, the old wallpaper settings may be causing the login screen to not
refresh properly after the splash screen, if the user account in
question was the last to log in.

You can bring yourself out of this state by clicking on "Add User"
which is the second button from the bottom left (the first being Shut
Down). You'll have to do this blind, but the mouse cursor will change
to the button selection icon. This will bring the UI back. Log in as
yourself, and change the wallpaper to something (anything really).

The next time you reboot, you should see the screen refresh and come
up properly.

You'll need to do this to all of your user accounts, if you have more than one.

[Steve Pirk]

This was indeed my situation, default R23 wallpaper, and doing all updates as they occurred.

I will try rebooting to see if it comes up properly (now that I have copied my Downloads folder to another machine :)

Once logged in as a "new" user using Add user, I ended up with the default R26 wallpaper instead of the Golden Gate bridge one.

Thanks for the quick workaround everyone.  

-- steve

[DennisLfromGA]

@Steve

As far as a rollback to the previous version, I think you should be able to issue a 'cgpt ...' command to change the boot priority back to either the A[2] or B[4] kernel partition. I use an alias called cgshow that does: sudo cgpt show /dev/sda|grep -i -e Content -e Label -e Attr

That lets your see what priority the partitions are set at so you can do a: cgpt add -i [2] or [4] -P 3 -T 3 -S 1 /dev/sda

to roll-back to the inactive partition. Once there, you can change your wallpaper or do whatever else is needed before rebooting with the new update.

Just some suggestions that hopefully may help, you're probably aware of all this but it may help some others...

[Steve Pirk]

Thanks Dennis! I totally forgot that was possible.

It looks like they have identified the issue and are testing fixes. I have been able to reboot and get back into the machine on 3 different occasions, so rolling back shouldn't be needed this time. Clicking Add user in the lower left works great.

-- steve

--

[gerst...@gmail.com]

Thanks  for the "Hidden" log-in tip!

With it my  good old CR 48 boots ups again and its usable!

Thanks!

[Benson Leung]

Thanks everyone for your help debugging this issue. It's this sort of
issue that Dev channel is around to help catch before it goes out to
more users.

[Johnny LeHane]

I am experiencing this issue but cant find the invisible login as guest button nor get to console with alt-ctrl-2.

Help appreciated.

[Steve Pirk]

When you are on the screen showing the Chrome splash image (say 20 seconds after boot), move your mouse down to the lower right corner of the screen. The mouse icon should change to a pointing finger. If this does not happen, then you might be seeing another problem. The invisible links are:

Shut down, Add user, and Browse as guest. The links cover a width of about 2.5" (length of my little finger), and start about 1/8" up from the bottom edge.

I had the best results using + Add user, as it allowed me to "add" myself as a new user and get into my old session.

The update to fix this issues has been pushed out, so you might also try clicking on the system menu in the lower right. Once that pops up (you probably won't see it), you can click on the "Restart to update" option about 1" up from the bottom edge, and that should install the update.

Strange that ctrl-alt-f2 (right arrow) does not bring up the console. I a pretty sure most people who were affected were able to do this.

-- steve

[Johnny LeHane]

It may be a different issue then. To be clear, i am not on the white screen with the chrome logo, but the wall paper screen, but there are no buttons, icons or anything.

I just retried the console key combo without success.

[Steve Pirk]

Ack! Yes, that does sound like a different issue. How long ago did it happen? 

-- steve

--

[Johnny LeHane]

I believe it happen with an update from last week. I know i had an update notice for a day or so, and ran it on friday or saturday. This morning was when I first couldn't boot successfully.

Thanks for the fast responses.

Johnny

Ps will be offline until 2 or 3pm ET, due to appointment.

[Steve Pirk]

I just applied last week's update to my Cr-48 after I saw your initial email. Everything looked great except for one time when the login box disappeared, and I could got get it to show up. It seems to be working ok after a restart, but I will keep an eye on it today. Check the Dev Channel section of the group  "chromebook-central" and see if anyone else with a Cr-48 is reporting issues. The version they released late last week was 26.0.1401.4.

-- steve