fosm.org API down

[Tim Sheerman-Chase]

Looks like the fosm.org API is down, and I can't ssh in.

Any thoughts?

Tim

[80n]

I'll take a look. 

[80n]

Tim
Hmm, not sure what's wrong.

http to the static pages seems to be working.

ssh responds with a login prompt but closes the connection after a valid password is entered.

My guess is that we are out of disk space or something.

There is a control panel for managing the server at https://panel.bytemark.co.uk/ but the credentials I have don't work.

If anyone has credentials for the Bytemark control panel please speak up.  Otherwise I think we need to put in a support call to Bytemark.

80n

[80n]

Ok, I reset the password for my control panel access.  This doesn't give me a console connection, which what I was hoping for, but it did confirm that I am the technical contact so I will
raise a support ticket with Bytemark asking for a reboot.

80n

[Ross Scanlon]

I have the same problem.

Can't ssh in and get 503 error on attempting to create a changeset or attempt to download data.

And there are no functions from the control panel to reset or anything with the server.

Cheers

Ross

--
You received this message because you are subscribed to the Google Groups "OSM Fork" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osm-fork+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[80n]

Here's the response from Bytemark support, I've replied that Monday morning would be fine.

I had a look on the Serial Console and there are filesystem/drive related errors, so it looks like one of the hard drives may have reached the end of its life and likely to need replacing. A reboot probably won't help as I suspect the filesystem has become read-only. I copied the errors below.

So, what I suggest needs doing is for the bad drive (/dev/sda) to be swapped out and replaced with a new drive, and then the RAID array rebuilt from the other drive (/dev/sdb). That's something we can do for you.

Depending on how urgent this is for you, I can either arrange a visit to the data centre today (ie, ASAP) to do the work, or for us to do it first thing on Monday morning (ie, 8am). What would you like us to do?


==========
[80181084.638270] EXT3-fs error (device md1): ext3_find_entry: reading directory #5922827 offset 0
[80181112.817661] end_request: I/O error, dev sda, sector 984202550
[80181112.826072] end_request: I/O error, dev sda, sector 984202550
[80181112.832359] end_request: I/O error, dev sda, sector 984202550
[80181143.240468] end_request: I/O error, dev sda, sector 1264908654
[80181203.023178] end_request: I/O error, dev sda, sector 1264908654
[80181205.039296] end_request: I/O error, dev sda, sector 191428406
[80181205.085638] EXT3-fs error (device md1): ext3_find_entry: reading directory #5922827 offset 0
[80181230.467204] end_request: I/O error, dev sda, sector 1898086766
[80181230.471366] end_request: I/O error, dev sda, sector 1898086766
[80181230.480590] end_request: I/O error, dev sda, sector 1898086766
[80181239.339193] end_request: I/O error, dev sda, sector 854280766
[80181239.348675] end_request: I/O error, dev sda, sector 854280766
[80181239.354973] end_request: I/O error, dev sda, sector 854280766
[80181262.782666] end_request: I/O error, dev sda, sector 1264908654
[80181321.411731] end_request: I/O error, dev sda, sector 103289414
[80181321.418210] end_request: I/O error, dev sda, sector 103289414
[80181323.252598] end_request: I/O error, dev sda, sector 1264908654
[80181324.733779] end_request: I/O error, dev sda, sector 191428406
[80181324.740103] EXT3-fs error (device md1): ext3_find_entry: reading directory #5922827 offset 0
[80181329.090055] end_request: I/O error, dev sda, sector 1898085894
[80181329.097127] end_request: I/O error, dev sda, sector 1898085894
[80181329.103528] end_request: I/O error, dev sda, sector 1898085894
==========


Kind regards,

--
Jamie Nguyen

Service Manager
Bytemark Hosting
https://www.bytemark.co.uk/
+44 1904 890 890

To unsubscribe from this group and stop receiving emails from it, send an email to osm-fork+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OSM Fork" group.

To unsubscribe from this group and stop receiving emails from it, send an email to osm-fork+unsubscribe@googlegroups.com.

[80n]

Update from Bytemark this morning:

Many thanks for your patience. We seem to be facing more of a hardware problem in that the servers are unable to read the disks any longer.

Due to this, it might unfortunately take a little bit more time to resolve - however we'll de-rack the server and investigate.

I'll give you an update as soon as I can.


Kind regards,

--
Timothy Frew

Bytemark Hosting
https://www.bytemark.co.uk/
+44 1904 890 890

[80n]

Another update:

We think there's two possibilities at the moment for what's happened. Either:

1. the motherboard or PSU (or some other component of the server) has neared the end of its life and causing the server to be unable to see both of the hard drives.

2. both hard drives have died.


We sincerely hope it is the first of those two possibilities. At the moment we are transferring the hard drives to separate hardware to try and identify where the problem lies. We'll keep you updated.

I might suggest you start to think your disaster recovery options and get your backups ready to restore from, just in case both hard drives have died.

[80n]

And now some bad news:

Many thanks for your patience this morning.

I'm afraid to say that unfortunately the motherboard and PSU etc. have proven to be in a healthy state, which means that the disks are dead. Both disks are currently undetectable (even in the bios) by the machine. We also tried detecting the drives in alternative hardware, which didn't work.

I've placed your server back in the rack with brand new (manufacturer sealed) 1TB drives and placed your old drives in an anti-static bag. If you would like the drives sending to a designated data recovery service, please do let me know.

If you could please let me know what OS you would like your server to be re-imaged with, I'll progress with that immediately so you can start your disaster recovery process.

I'm really sorry I couldn't deliver better news.