Drobo forgot partition type, hence free space worries

[Nicolas Will]

Hello,

I have a Drobo S full of 2 TB disks, 1x 8TB EXT3 partition, under Linux.

It worked well, including free space reclaim, for a long while.

I must have done one bad thing at some point, as now the Drobo does not know what file system is in use. It does not report the proper used/free space or adapt it to current values. It keeps warning that I reached a threshold with a yellow drive, even after a big data deletion effort.

Operations are running OK, AFAIK. (maybe slower than normal?)

I tried to see if plugging it into a windows machine with official software would solve this, but it also reports "unknown filesystem", with no option to manually set it.

Is there a way to make the device see the partition as an EXT3 type again, without starting from scratch? (I do have a lot of data there, and no temporary place large enough to put it on...).

Thanks!

Nico





nico@eeebox:~$ sudo drobom -V
0.6.2.2


nico@eeebox:~$ sudo drobom info
---------------------------------------------------------
Drobo Name: Drobo S Devices: /dev/sdb
Time: Mon Apr 27 14:47:59 2015
---------------------------------------------------------
Configuration maxima: slots: 5, luns: 16, lunsize: 36028805 TB
Capacity (in GB): used: 7086, free: 882, total: 7969
protocol version: 0.11
SCSI emulation information:
Vendor: Drobo Model: Drobo3 Revision: 3.00
/dev/sdb: bus=scsi2 channel=0 id=0 lun=0 (vendor=Drobo , version=30527)
query slotinfo result: number of slots: 5
slot GB Model Status
0 2000 WDC WD20EARS-00S8B1 yellow
1 2000 WDC WD20EARS-00S8B1 green
2 2000 WDC WD20EARS-00S8B1 green
3 2000 WDC WD20EARS-00S8B1 green
4 2000 WDC WD20EARS-00S8B1 green
Firmware: 2.1.5
Revision: 5.41 ( 62910 ) built: Apr 15 2013,12:53:35
Features: NO_AUTO_REBOOT,NO_FAT32_FORMAT,USED_CAPACITY_FROM_HOST,DISKPACKSTATUS,ENCRYPT_NOHEADER,CMD_STATUS_QUERIABLE,VARIABLE_LUN_SIZE_1_16,PARTITION_LUN_GPT_MBR,FAT32_FORMAT_VOLNAME,SUPPORTS_NEW_LUNINFO2,feature x0800,LUN_MANAGEMENT,feature x2000 ,SUPPORTS_OPTIONS2,SUPPORTS_SHUTDOWN,leftovers (0x5f0000)
/dev/sdb /mnt/drobo Drobo S 88% full - (['Yellow alert'], 0)
query options result: 
UseStaticIPAddress False
UseManualVolumeManagement False
YellowThreshold 85
SpinDownDelayMinutes 15
RedThreshold 95
SpinDownDelay True
NetMask 64.3.232.3
DualDiskRedundancy False
IPAddress 0.0.3.1
hoho! multiple partition types! Brave are we not?
lun size (GB) used PTFmt FStype:
0 8796 7086 GPT [] 
---------------------------------------------------------





nico@eeebox:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 7.9T 5.0T 2.6T 67% /mnt/drobo


/dev/sdb1 on /mnt/drobo type ext3 (rw)





nico@eeebox:~$ sudo fdisk -l /dev/sdb

Disk /dev/sdb: 8 TiB, 8796093022208 bytes, 17179869184 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 5ADF8D47-28B1-EA49-9278-07D246108CA8

Device Start End Sectors Size Type
/dev/sdb1 2048 17179867135 17179865088 8T Linux filesystem






nico@eeebox:~$ sudo gdisk -l /dev/sdb
GPT fdisk (gdisk) version 0.8.8

Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /dev/sdb: 17179869184 sectors, 8.0 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): 5ADF8D47-28B1-EA49-9278-07D246108CA8
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 17179869150
Partitions will be aligned on 2048-sector boundaries
Total free space is 4029 sectors (2.0 MiB)

Number Start (sector) End (sector) Size Code Name
1 2048 17179867135 8.0 TiB 8300 






nico@eeebox:~$ sudo parted /dev/sdb
GNU Parted 3.2
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print 
Model: Drobo Drobo3 (scsi)
Disk /dev/sdb: 8796GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number Start End Size File system Name Flags
1 1049kB 8796GB 8796GB ext3 

[Peter Silva]

I would not trust it at this point.  it's fishy.

I would cobble together some other place to store the data, offload it.

Then write lots of fake data to it to test that it works after the re-build.

if, after a lot of beating, it seems OK, then I would re-load the data onto it.

--
You received this message because you are subscribed to the Google Groups "drobo-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to drobo-talk+...@googlegroups.com.
To post to this group, send email to drobo...@googlegroups.com.
Visit this group at http://groups.google.com/group/drobo-talk.
For more options, visit https://groups.google.com/d/optout.

[Temple]

Agreed, GET YOUR DATA BACKED UP!!!!!

Then start fiddling around.

Temple

Sent from my BlackBerry 10 smartphone.

From: Peter Silva Sent: Tuesday, 28 April 2015 1:23 AM To: drobo...@googlegroups.com Reply To: drobo...@googlegroups.com Subject: Re: Drobo forgot partition type, hence free space worries

[Nicolas Will]

OK, thanks for the warnings, which I took to the letter!

It took 5 days, but I have my data off the Drobo.

I reset the LUN and reformatted, mounted and copied data

But drobom still indicates that the drobo apparently still has no clue about the EXT3 filesystem! Space reclaiming appears to be dead.

I've started rsyncing some data back, I don't know yet if the drobo does any reclaiming. It doesn't look like its doing any, but I have a small size test only at hand. I'll report back later on that.

I've put my detailed process below in the hope that one of you will point at a blatant error I've made, or to some other stuff I didn't know. I have a windows partition somewhere, I can use it for the official Drobo software if needed.

Again, this setup has been working for years. I've never looked closery about the FS knowledge in drobom, but I know that space utilization reporting worked, going up and down, before.

Thanks for your help!

Nico

Ubuntu 15.04 (Vivid Vervet) 32 bits on an ASUS EeeeBox

Linux eeebox 3.19.0-16-generic #16-Ubuntu SMP Thu Apr 30 16:13:00 UTC 2015 i686 i686 i686 GNU/Linux

I used the CLI exclusively:

nico@eeebox:/mnt/toto$ drobom -V
0.6.2.2

I've first reset the drobo by seting up 2TB (4 LUNs) LUNs.

nico@eeebox:~$ sudo drobom set lunsize 2
/dev/sdb - Drobo S 00% full - ([], 0)
You asked to set the lunsize to 2
WARNING: Ready to destroy all you data. Continue? (y/n) y
set lunsize to 2 TiB
Done... Drobo is likely now rebooting.  
                 In a few minutes, it will come back with the new LUN size.

I then went back to a single 8TB LUN.

nico@eeebox:~$ sudo drobom set lunsize 8
/dev/sdb:/dev/sdd:/dev/sde:/dev/sdf - Drobo S 00% full - ([], 0)
WARNING: lun size > 2 TiB known not work in many cases under Linux
You asked to set the lunsize to 8
WARNING: Ready to destroy all you data. Continue? (y/n) y
set lunsize to 8 TiB
Done... Drobo is likely now rebooting.  
                 In a few minutes, it will come back with the new LUN size.

I then reformated to ext3 using the drobom script. I just changed the parted command from "0 100%" to "0% 100%", which gives proper alignment (at least parted does not complain.

nico@eeebox:/mnt/drobo$ sudo drobom -d /dev/sdb format ext3
/dev/sdb /mnt/drobo Drobo S 00% full - ([], 0)
preparing a format script for a ext3 file system as you requested
OK, I built the script but nothing is erased yet...
You can have a look at it with: cat /tmp/fmtscript
If you are really sure, go ahead and do: sh /tmp/fmtscript
WARNING: Ready to destroy all your data. Continue? (y/n) n
Phew... You stopped just in time!

Which gives the modified script

#!/bin/sh
parted -s /dev/sdb mklabel gpt
parted -s /dev/sdb mkpart primary ext3 0 100%
parted -s /dev/sdb print; sleep 5
mke2fs -j -i 262144 -L Drobo01 -m 0 -O sparse_super,^resize_inode /dev/sdb1

And when executed

nico@eeebox:~$ sudo sh /tmp/fmtscript

Model: Drobo Drobo3 (scsi)
Disk /dev/sdb: 8796GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:


Number  Start   End     Size    File system  Name     Flags

 1      1049kB  8796GB  8796GB               primary


mke2fs 1.42.12 (29-Aug-2014)
Creating filesystem with 2147483136 4k blocks and 33554432 inodes
Filesystem UUID: 740d5465-10b6-4f86-93c3-2a9212a3f06a
Superblock backups stored on blocks:
 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
 102400000, 214990848, 512000000, 550731776, 644972544, 1934917632


Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

The whole thing looks good in the logs when mounting

[511404.652694] usb 1-3: USB disconnect, device number 15
[511520.592149] usb 1-3: new high-speed USB device number 16 using ehci-pci
[511520.725158] usb 1-3: New USB device found, idVendor=19b9, idProduct=5d10
[511520.725177] usb 1-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[511520.725191] usb 1-3: Product: Drobo3
[511520.725203] usb 1-3: Manufacturer: Data Robotics Inc.
[511520.725215] usb 1-3: SerialNumber: 0DB101950208
[511525.354206] usb-storage 1-3:1.0: USB Mass Storage device detected
[511525.355018] scsi host14: usb-storage 1-3:1.0
[511526.379070] scsi 14:0:0:0: Direct-Access     Drobo    Drobo3           3.00 PQ: 0 ANSI: 5
[511526.383082] sd 14:0:0:0: Attached scsi generic sg1 type 0
[511526.385500] sd 14:0:0:0: [sdb] Very big device. Trying to use READ CAPACITY(16).
[511526.386000] sd 14:0:0:0: [sdb] 17179869184 512-byte logical blocks: (8.79 TB/8.00 TiB)
[511526.387116] sd 14:0:0:0: [sdb] Write Protect is off
[511526.387132] sd 14:0:0:0: [sdb] Mode Sense: 03 00 00 00
[511526.390881] sd 14:0:0:0: [sdb] No Caching mode page found
[511526.391074] sd 14:0:0:0: [sdb] Assuming drive cache: write through
[511526.394127] sd 14:0:0:0: [sdb] Very big device. Trying to use READ CAPACITY(16).
[511526.402069]  sdb: unknown partition table
[511526.403480] sd 14:0:0:0: [sdb] Very big device. Trying to use READ CAPACITY(16).
[511526.407635] sd 14:0:0:0: [sdb] Attached SCSI disk
[511570.025745] sd 14:0:0:0: [sdb] Very big device. Trying to use READ CAPACITY(16).
[511570.031898]  sdb: unknown partition table
[511579.923386] sd 14:0:0:0: [sdb] Very big device. Trying to use READ CAPACITY(16).
[511579.929223]  sdb: unknown partition table
[511580.058283] sd 14:0:0:0: [sdb] Very big device. Trying to use READ CAPACITY(16).
[511580.078608]  sdb:
[511581.608843] sd 14:0:0:0: [sdb] Very big device. Trying to use READ CAPACITY(16).
[511581.623037]  sdb: sdb1
[518560.413873] EXT4-fs (sdb1): mounting ext3 file system using the ext4 subsystem
[518561.068932] EXT4-fs (sdb1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro

I have it mounted by UUID

nico@eeebox:~$ ls -lah /dev/disk/by-uuid/
total 0
drwxr-xr-x 2 root root 100 May 12 09:33 .
drwxr-xr-x 8 root root 160 May 12 09:33 ..
lrwxrwxrwx 1 root root  10 May 12 09:33 740d5465-10b6-4f86-93c3-2a9212a3f06a -> ../../sdb1

and I use this UUID in fstab

# Drobo
UUID=740d5465-10b6-4f86-93c3-2a9212a3f06a /mnt/drobo ext3 noatime,nodiratime,user_xattr,errors=remount-ro 0 0

mount shows:

dev/sdb1 on /mnt/drobo type ext3 (rw,noatime,nodiratime,errors=remount-ro,data=ordered)

fdisk shows

nico@eeebox:/mnt/toto$ sudo fdisk -l /dev/sdb

Disk /dev/sdb: 8 TiB, 8796093022208 bytes, 17179869184 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt

Disk identifier: 146A3B85-5347-4E2E-ADC3-5EC421B4D60C

Device     Start         End     Sectors Size Type
/dev/sdb1   2048 17179867135 17179865088   8T Linux filesystem

parted shows

nico@eeebox:/mnt/toto$ sudo parted /dev/sdb print

Model: Drobo Drobo3 (scsi)
Disk /dev/sdb: 8796GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:


Number  Start   End     Size    File system  Name     Flags

 1      1049kB  8796GB  8796GB  ext3         primary

but drobom still shows no clue about the partition type!!!

nico@eeebox:/mnt/toto$ sudo drobom info

---------------------------------------------------------
Drobo Name: Drobo S    Devices: /dev/

sdb
      Time: Thu Jan  1 03:24:13 1970

---------------------------------------------------------
Configuration maxima: slots: 5, luns: 16, lunsize: 36028805 TB

Capacity (in GB):  used: 32, free: 7936, total: 7969

protocol version:  0.11
SCSI emulation information:
Vendor: Drobo     Model: Drobo3           Revision: 3.00

/dev/sdb: bus=scsi14 channel=0 id=0 lun=0 (vendor=Drobo   , version=30527)

query slotinfo result:  number of slots: 5
slot   GB                Model               Status

   0 2000  WDC WD20EARS-00S8B1                green
   1 2000  WDC WD20EARS-00S8B1                green
   2 2000  WDC WD20EARS-00S8B1                green
   3 2000  WDC WD20EARS-00S8B1                green
   4 2000  WDC WD20EARS-00S8B1                green

Firmware: 2.1.5
Revision: 5.41 ( 62910 ) built: Apr 15 2013,12:53:35
Features: NO_AUTO_REBOOT,NO_FAT32_FORMAT,USED_CAPACITY_FROM_HOST,DISKPACKSTATUS,ENCRYPT_NOHEADER,CMD_STATUS_QUERIABLE,VARIABLE_LUN_SIZE_1_16,PARTITION_LUN_GPT_MBR,FAT32_FORMAT_VOLNAME,SUPPORTS_NEW_LUNINFO2,feature x0800,LUN_MANAGEMENT,feature x2000 ,SUPPORTS_OPTIONS2,SUPPORTS_SHUTDOWN,leftovers (0x5f0000)

/dev/sdb /mnt/drobo Drobo S 00% full - ([], 0)

query options result:
 UseStaticIPAddress False
 UseManualVolumeManagement False
 YellowThreshold 85
 SpinDownDelayMinutes 15
 RedThreshold 95
 SpinDownDelay True

 NetMask 0.0.0.0
 DualDiskRedundancy False
 IPAddress 0.2.131.157
hoho! multiple partition types! Brave are we not?

lun     size (GB) used    PTFmt  FStype:

  0     8796        32      GPT []
---------------------------------------------------------

I then started copying data on the storage.

df gives

Filesystem      Size  Used Avail Use% Mounted on

/dev/sdb1       8.0T   38G  8.0T   1% /mnt/drobo

drobom info gives

Capacity (in GB):  used: 50, free: 7919, total: 7969

/dev/sdb /mnt/drobo Drobo S 00% full - ([], 0)

lun     size (GB) used    PTFmt  FStype:

  0     8796        50      GPT []

If I delete the data I've put on the drobo, I get

df

Filesystem      Size  Used Avail Use% Mounted on

/dev/sdb1       8.0T  152K  8.0T   1% /mnt/drobo

drobom info, after 30 minutes of wait with not activity

Capacity (in GB):  used: 50, free: 7919, total: 7969



/dev/sdb /mnt/drobo Drobo S 00% full - ([], 0)

hoho! multiple partition types! Brave are we not?
lun     size (GB) used    PTFmt  FStype:

  0     8796        50      GPT []

It doesn't appear to be reclaiming. I'll test with way more data, it just takes time with USB2 on all ends...

[Nicolas Will]

Reclaim is really fishy...

I put data in( 50GB), removed it all, than put new data in again (about 9GB at the time of test).

Drobom shows space use that adds the values of the first and second data dumps (59GB)!

Aaaah! o_0

Help!

Pretty please!

Nico

On Monday, 27 April 2015 15:36:16 UTC+1, Nicolas Will wrote:

[Nicolas Will]

Would this be an lead to follow?

http://www.drobospace.com/forums/showthread.php?tid=142931

Nico

[Nicolas Will]

Right...

I've tried old Linux versions, in case Ext3 was changed in a way that my Drobo S could not understand anymore.

I've individually checked my 5 disks, they all work on their own.

I've zeroed my disks with dd, forcing the Drobo to reinitialise them on the next restart.

All with the same result...

This used to work well, 8 TB LUN, Ext3 FS and reclaim.

I'm wondering about firmware issues, now, the last part that I can play with,

I've downgraded 2.1.5 to 2.1.3 with the same result.

I'm now looking for older firmware files, not easy! Please share if you can!

Thanks!

Nico

[Nicolas Will]

Sooooo....

I have tried all the firmwares...

I have tried the pinhole reset...

My disks are good.

I've done my tests with 1Tb or 2 TB LUNs (faster to format, and a "known to work" setup).

There is no way for the Drobo S to see the partition as Ext3, and for it to reclaim freed space...

It works for FAT32 or NTFS. Just not Ext3.

Either there is a mysterious part of the electonics that had died, or Ext3 has changed at some point, enough for the Drobo to not understand it anymore...

I have a door stop that I need to replace...

I'm a sad puppy...

Nico

[Nicolas Will]

So, Isn't there anyone with hints, opinions, recommendations, advice, wisdom... commiserations?

Please, I would so love to be told that I'm an idiot and forgot the very basic of thing...

Nico

[Peter]

I just noticed you said you changed 0 to 0% to get rid of some warning from parted. 

It is possible that the offset confuses the firmware.  You might want to retry your tests with 0, rather than 0%.

Let parted complain.