raid1 mystery and work around

[Paul Johnson]

Hi guys:

I've found a workaround for a problem with a raid 1
array. I'm posting to share the "solution" and to
ask why this went wrong in the first place.

On an old test machine, I have 2 4-year-old
Seagate IDE drives in raid1 mirror for my home
partition. One failed and Seagate was very very
pleasant to replace it. I didn't even need a receipt!
They just went by the serial number to conclude I
was in warranty.

I followed "the usual" procedure for failed
drives. mdadm marked the drive as a failure and
it was removed from the array. Then I tried to
add it back into the raid 1.

One of the HOWTOs I relied on was this one:
http://www.howtoforge.com/replacing_hard_disks_in_a_raid1_array

Here's what went wrong. After being added, the new drive
went through a long "recovery" process--2 hours--but when
it finished, the new drive was marked as "spare" and the
raid 1 array continued to show only one drive was active.

Every time the system restarts, the new drive tries
to resync itself, it copies for 2 hours, but it never
enters the array. It is always spare.

In the end, gave up trying to fix /dev/md0.
I "guessed" a solution--create a new /dev/md1
device and refit the system to use that. I explain that
fix below, in case the same problem hits other
people.

But I'm still curious to know why it did not work.

Now the details:

The raid1 array was /dev/md0 and it used disks sdb1
and sdc1 and the one that failed was sdb1.

Here's what I saw while the new drive was being added:

# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdc1[1] sdb1[2]
244195904 blocks [2/1] [_U]
[==================>..] recovery = 94.4%
(230658240/244195904) finish=6.9min speed=32396K/sec

unused devices: <none>

# mdadm --examine /dev/sdb1
/dev/sdb1:
Magic : a92b4efc
Version : 0.90.00
UUID : 37e6e9b6:34cdfcb2:63afba50:8b88d6fc
Creation Time : Sat Aug 18 19:10:40 2007
Raid Level : raid1
Used Dev Size : 244195904 (232.88 GiB 250.06 GB)
Array Size : 244195904 (232.88 GiB 250.06 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 0

Update Time : Thu Oct 29 00:35:50 2009
State : clean
Active Devices : 1
Working Devices : 2
Failed Devices : 0
Spare Devices : 1
Checksum : a557d3b3 - correct
Events : 6874

Number Major Minor RaidDevice State
this 2 8 17 2 spare /dev/sdb1

0 0 0 0 0 removed
1 1 8 33 1 active sync /dev/sdc1
2 2 8 17 2 spare /dev/sdb1

After the rebuild was done, here's the situation: the
new drive is a spare:

# mdadm --examine /dev/sdc1
/dev/sdc1:
Magic : a92b4efc
Version : 0.90.00
UUID : 37e6e9b6:34cdfcb2:63afba50:8b88d6fc
Creation Time : Sat Aug 18 19:10:40 2007
Raid Level : raid1
Used Dev Size : 244195904 (232.88 GiB 250.06 GB)
Array Size : 244195904 (232.88 GiB 250.06 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 0

Update Time : Thu Oct 29 00:35:50 2009
State : clean
Active Devices : 1
Working Devices : 2
Failed Devices : 0
Spare Devices : 1
Checksum : a557d3c7 - correct
Events : 6874

Number Major Minor RaidDevice State
this 1 8 33 1 active sync /dev/sdc1

0 0 0 0 0 removed
1 1 8 33 1 active sync /dev/sdc1
2 2 8 17 2 spare /dev/sdb1

# mdadm --query /dev/md0
/dev/md0: 232.88GiB raid1 2 devices, 1 spare. Use mdadm --detail for
more detail.

# mdadm --detail /dev/md0
/dev/md0:
Version : 0.90
Creation Time : Sat Aug 18 19:10:40 2007
Raid Level : raid1
Array Size : 244195904 (232.88 GiB 250.06 GB)
Used Dev Size : 244195904 (232.88 GiB 250.06 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Thu Oct 29 00:35:50 2009
State : clean, degraded, recovering
Active Devices : 1
Working Devices : 2
Failed Devices : 0
Spare Devices : 1

Rebuild Status : 97% complete

UUID : 37e6e9b6:34cdfcb2:63afba50:8b88d6fc
Events : 0.6874

Number Major Minor RaidDevice State
2 8 17 0 spare rebuilding /dev/sdb1
1 8 33 1 active sync /dev/sdc1

After that, rebuilding seems finished:

# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdc1[1] sdb1[2]
244195904 blocks [2/1] [_U]

But I have only 1 drive in the active array:

# mdadm --detail /dev/md0
/dev/md0:
Version : 0.90
Creation Time : Sat Aug 18 19:10:40 2007
Raid Level : raid1
Array Size : 244195904 (232.88 GiB 250.06 GB)
Used Dev Size : 244195904 (232.88 GiB 250.06 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Thu Oct 29 00:43:21 2009
State : clean, degraded
Active Devices : 1
Working Devices : 2
Failed Devices : 0
Spare Devices : 1

UUID : 37e6e9b6:34cdfcb2:63afba50:8b88d6fc
Events : 0.6880

Number Major Minor RaidDevice State
2 8 17 0 spare rebuilding /dev/sdb1
1 8 33 1 active sync /dev/sdc1

# mdadm --examine /dev/sdb1
/dev/sdb1:
Magic : a92b4efc
Version : 0.90.00
UUID : 37e6e9b6:34cdfcb2:63afba50:8b88d6fc
Creation Time : Sat Aug 18 19:10:40 2007
Raid Level : raid1
Used Dev Size : 244195904 (232.88 GiB 250.06 GB)
Array Size : 244195904 (232.88 GiB 250.06 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 0

Update Time : Thu Oct 29 00:44:02 2009
State : clean
Active Devices : 1
Working Devices : 2
Failed Devices : 0
Spare Devices : 1
Checksum : a557d5af - correct
Events : 6882

Number Major Minor RaidDevice State
this 2 8 17 2 spare /dev/sdb1

0 0 0 0 0 removed
1 1 8 33 1 active sync /dev/sdc1
2 2 8 17 2 spare /dev/sdb1

I tried a lot of ways to set this right.
I tried "grow" the array, set the number of spares
to 0, and so forth. No success.

After a lot of tries, I gave up trying to get /dev/md0 to work.
So I stopped it, and the used the "--assume-clean" option to
create a new array on md1. I found that suggestion here

http://neverusethisfont.com/blog/tags/mdadm/

# mdadm -S /dev/md0

# mdadm --create --assume-clean --level=1 --raid-devices=2 /dev/md1
/dev/sdc1 /dev/sdb1

That works! So I just needed to reset the configuration
to use that. First, grab the metadata

# mdadm --detail --scan
ARRAY /dev/md1 metadata=0.90 UUID=6a408f8b:515f605f:bfe78010:bc810f04

And revise the mdadm.conf file

# cat /etc/mdadm.conf

DEVICE /dev/sdb1 /dev/sdc1
ARRAY /dev/md1 level=raid1 num-devices=2
UUID=6a408f8b:515f605f:bfe78010:bc810f04 devices=/dev/sdc1,/dev/sdb1

And I changed /etc/fstab to point at md1, not md0.

But why did /dev/md0 hate me in the first place?

I wonder if it was personal :(

--
Paul E. Johnson email: paul...@ku.edu
Professor, Political Science http://pj.freefaculty.org
1541 Lilac Lane, Rm 504
University of Kansas Office: (785) 864-9086
Lawrence, Kansas 66044-3177 FAX: (785) 864-5700

[heartofamer...@yahoo.com]

If anyone is familiar with Drupal or Joomla (or MODx or Mambo, or any CMS you prefer), let me know. I would like to discuss your results/thoughts, and get some tips on getting started with them.

----------------------
I am doing less internal hosting, and have a client who's wanting me to rework there site. I currently love hostmonster/bluehost, and they use godaddy. Both unix/linux servers.

The top rated ones on both sites are Drupal & Joomla. They both look great. Drupal seems to have more mods, but Joomla seems easier to use.

Seems is the keyword. They are both probably easy to maintain, but to start from a clean install I am almost totally lost.
Both sites-hostmonster&godaddy-also offer MODx and Mambo

I started with a Wiki, but the goal now is the whole shibang, and a clean easily updated site that is easy for the public as well as internal users to use/navigate/update.

http://en.wikipedia.org/wiki/Drupal
http://en.wikipedia.org/wiki/Joomla
http://en.wikipedia.org/wiki/List_of_content_management_systems

Part of the goal is not just a website they can update, but is:

Clean front end.... (unlike a wiki)
with the following more Integrated than a bunch of perl/php/ssi's
Dynamic to draw people back,
Uses an open calendar (iCal/iCalendar/vCalendar/CalDAV: Linkable with Google/Apple/exchange software)
offers a todo list
project management system (more so than a simple todo list)
Ticket system (more so than a simple todo list)
Collaborative work environment (kinda like a wiki, but also allows for spreadsheet/word/presentation management)
the great thing is they also offer plugins for eCommerce, photogallery's, and it may even offer adaptive front ends i.e. printer friendly, Mobile/wap, IE/Firefox/Safari

[Steve Morgan]

Why don't you take a look at CMS Made Simple. I built a couple of sites
with it and was pleased with the results and ease of use. Much easier to
work with than Joomla, which I have also used.

http://www.cmsmadesimple.org/

--
Steve Morgan <st...@notiones.com>


On Sun, 2009-11-01 at 07:36 -0800, heartofamer...@yahoo.com
wrote:

[Tom Currie]

I have built web sites with drupal, typo3, cake, wordpress, joomla, alfresco, and probably others that I forget right now.  The choice is not "better or worse", but "which ones are most compatible with what you are trying to do".  Try this page:

http://www.cmsmatrix.org/matrix/cms-matrix?func=search

I like writing lots of backend code to do extra stuff.  some folks don't. I like having very fine grained access controls for end users.  maybe you find that to be a pain in the neck. I like having administrative access (backend access) completely and totally separate from front end (end user) access.   my criteria will be different from yours.  the cms matrix page loads slow and takes a while, but I think it's pretty correct for what I've looked at.

tc

--
... waiting for the deus ex machina ...

[Jeffrey Watts]

Paul, it's been a while since I've messed with the MD driver, but did you --remove the failed drive before putting the new drive in and doing an --add?  To remove a drive you must always --fail then --remove it (the --fail is option when it has actually failed).

I didn't have time to look at your output too closely, but if you didn't remove the prior device then MD will probably think it's still around.  Given that a mirror only has two pieces, if you add a new drive without removing the failed one, the new one will be a spare.  I can't remember if there's a way to force MD to make the spare the mirror, my guess is that the easiest way to fix it would be to --remove the failed mirror and it should automatically promote the spare.

Jeffrey.

--

"He that would make his own liberty secure must guard even his enemy from oppression; for if he violates this duty he establishes a precedent that will reach to himself." -- Thomas Paine

[Jeffrey Watts]

Sorry, I should have said that the --fail option isn't necessary when the drive is already in a failed state.  It comes in handy however when you are upgrading your disks to a larger size.

--fail one.
--remove it.
Power down.  Install new, larger hard drive.
--add new drive.  Let it sync.
--fail other drive.
--remove it.
Extend the partition on the new drive.
resize2fs
Power down.  Install new, larger hard drive.
--add new drive.  Let it sync.
Done.

J.

[Dejan Čabrilo]

On Sun, Nov 1, 2009 at 4:36 PM, <heartofamer...@yahoo.com> wrote:
> If anyone is familiar with Drupal or Joomla (or MODx or Mambo, or any CMS you prefer), let me know. I > would like to discuss your results/thoughts, and get some tips on getting started with them.

That's pretty much what I do for living and I'd advise you against
Joomla. It's very popular and many clients request it, but it is hell
to administer. You can count on (even popular) third-party
plugins/modules to break when updating, it forces you to output
ridiculously bad HTML (using tables for layout and such), etc.

When looking at CMS, I advise you to use these parameters:
1) How hard or easy will it be to write your own plugins/modules.
2) How hard or easy will it be for you to implement your user's wishes
without rewriting most of the CMS.
3) Separation of content and presentation (e.g. you don't want your
users to login from the frontend). Tom mentioned this.

I second Steve's recommendation of CMSMS. It gets a lot of points for
using Smarty as template engine, it's simple to use for end users, and
modules can be developed quickly.

Also, I found Typolight to be a great CMS, but with several
limitations, such as URL's which are bad for SEO and steep learning
curve for end user. Also, a HUGE downside of it is that its creators
will try to force you into buying their support (e.g. non-trivial
updates without official support).

Also, if you are acquainted with PHP, you should look into a framework
instead of CMS. There is a plethora of these, and you can quickly
build nice sites with CodeIgniter or CakePHP.

Feel free to contact me off the list if you want to discuss this more,
Dejan

[Marshal Graham]

I will third CMS Made Simple, I've used it for a couple of projects and it is very good. Also, I recently came across concrete5, http://www.concrete5.org/.  It seems to be a newer project that looks like it has a lot of potential. Especially when it comes to training non-technical users for content entry and maintenance.

2009/11/2 Dejan Čabrilo <dcab...@gmail.com>

[nick....@gmail.com]

I like CMSMS. My story about why I like it and don't like Joomla/
Mambo: http://groups.google.com/group/kulua-l/browse_thread/thread/7a49949e6c0d8128/c22c01eee68b02e9?lnk=gst&q=CMSMS#c22c01eee68b02e9

-Nick

On Nov 2, 10:01 am, Marshal Graham <marshal.gra...@gmail.com> wrote:
> I will third CMS Made Simple, I've used it for a couple of projects and it

> is very good. Also, I recently came across concrete5,http://www.concrete5.org/. It seems to be a newer project that looks like

> it has a lot of potential. Especially when it comes to training
> non-technical users for content entry and maintenance.
>

> 2009/11/2 Dejan Čabrilo <dcabr...@gmail.com>

[heartofamer...@yahoo.com]

A few people mentioned Drupal, so I took a second look, and its coming along. But I do not see anywhere how to install a module. When I search, I find other modules to manage modules, but even they say...

" Simply extract it into place (as you would
any other module) and enable it on the modules page."

I am extracting things here via tar:
~/public_html/inside/sites/all/

On the .../administer/modules page, I tried clicking on update, but it says there are no updates.

[Marshal Graham]

Extract the downloaded module tarball to <drupal root>/sites/all/modules.  You may need to create this directory. Next you need to enable and configure the module in the web interface under Administer->Site Building->Modules. Some modules may have dependencies, these are usually explained on this page as well.

--

You received this message because you are subscribed to the Google Groups "kulua-l" group.
To post to this group, send email to kul...@googlegroups.com.
To unsubscribe from this group, send email to kulua-l+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/kulua-l?hl=en.