HELP!!! New to ESOS - Setting Up with an ESX vCenter VM to Create a FC SAN Target For Powervault MD1000

[X-Savior]

Hi There,

I have been searching like crazy trying to find something to do the following:

I currently have several Dell Poweredge Servers (a PE2950, PE6850) and the PE6850 has a PERC H800 SAS Card that is connected to a Powervault MD1000 (That has a 21TB Array of SATA Drives) along with several home PC's. The Poweredge Servers are virtualized with VMware ESX /w vCenter.

The Powervault is setup in VMware vCenter as a local Host Datastore that all the VM's on that machine have access to.

I am looking to build a FC SAN for backend high speed file transfers between all MY systems to the Powervault Datastore (Thus creating a "local" drive for all my PC connected to the FC SAN) and freeing up the Ethernet traffic bandwidth.

So I have purchased a handful of QLogic 2462 (Dual Port) and 2464 (Quad Port) 4Gb  Cards (Quad for Servers and Dual for PC's) as well as a QLogic 5602 Fiber Switch for all the connections.

I have found VERY LITTLE information available for setting up a FC SAN and creating a target. I am experienced with Raid arrays (SATA, SCSI, SAS) as well a iSCSI within VMware (Also Using ZFSGuru for creating iSCSI Targets).

I hope I have been clear on what I want to do and I hope ESOS will be my solution. It looks like I will need a dedicated OS (ESOS) to create a FC SAN Target that will share out the Datastore on the SAN (As it appears it is impossible within Server 2012R2 to do anything like this).

So My IDEA is that I would create a VM on the PE6850 with ESOS that would be able to share out the Powervault to the the FC SAN switch as a target. I might still not be fully understanding of all his FC stuff (This is my first attempt at ever working with Fiber). From what I read it would make for the most cost effective high speed solution for file transfers between systems. Now if I am correct so far how would I go about just creating the local drive as the USB (Since I am virtualized I do not want to have a USB plugged in... so a virtualized USB of ESOS?)

Ultimately I do have one more small server that I could turn into a dedicated ESOS machine and then I could put the PERC H800 card in it instead (with the Powervault also) to share out as a target. But I am trying to limit the number of Servers in the rack if possible.

PLEASE help me out here :)

Thank You!

[Steve Jones]

Sounds like you have a large scale version of what I do at home..  I built a similar, but smaller lab with ESOS, but using generic white box hardware.  I have an ESOS server that just has six various size SATA hard drives in it, and I boot from USB to load ESOS, and then I have a similar quad port FC card, with 3 of the ports going straight point to point to 3 separate ESXi boxes.

I think what you're planning to do will work except that I'm not sure about virtualizing the ESOS box..  It depends on whether you can pass the FC through from the physical card to the VM.  I've not tried that.  I know ESXi has special FC virtualization, but I've never played with that.

If it were me, I'd want to keep the ESOS physical, and share out the FC from a physical server, if only for performance, but otherwise it sounds like it would at least work.  I'm also not terribly well versed in FC but have muddled through it a few times in building ESOS machines with good luck.  One thing you might want to try to keep it simple is to create your targets and share them out via iSCSI first, just to prove ESOS is working and the ESXi machines are seeing it.  you can add FC later, and present the same LUNs through the FC.  I've done that in cases where I needed to add another ESXi machine but didn't have another FC card, at least temporarily.

Ironically, just today, I was going through the ESOS docs again because I'm building a new test box now (like here: https://github.com/astersmith/esos/wiki/12_Installation) and it has done me good to refresh on the terminology..

Marcc has been awesome on this list, helping me through every problem I've had, but if I can be any help, I'd love to try to pay it forward and try to lighten the load on Marcc, although he will correct me when I give bad advice!  :-)

One more thing..  You definitely CAN install ESOS as a VM, instead of on a USB stick, or even on a real hard disk on a PC..  I've installed it in VMWare to allow me some flexibility to recover some data from some damaged hard drives after a house fire, but in that case, I used iSCSI not FC..

As you get started, ask any questions here and I'll answer if I can.. and if I can't, I'm sure someone more knowledgeable will..

-Steve

--
You received this message because you are subscribed to the Google Groups "esos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to esos-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[X-Savior]

Thank you so much for the re-assurance.

I have been less then overjoyed with my FC experience so far and it even had me contemplating just investing in 10Gb Ethernet instead... but I do not give up that easy ;)

I am going to give this a try. I think I agree it would be best for me to install this on a little Dell Poweredge 860 server and have it handle the powervault.

Now as for my Fiber Switch I have been able to log into it but I have no clue what I am doing. Can I expect to need to do settings there or will it function out of the box with factory defaults for basic SAN sharing?

I am going to start messing around tonight and see what I can make happen. Rest assured I WILL have questions incoming.

Thank you very much!

[Steve Jones]

I'm not sure..  I've actually just bought another  FC switch..  I had 2 before I lost them in the fire, and they defaulted to working, but from the little I know about FC, I've learned that FC isn't much like Ethernet - You may have to "zone" the switch, which can mean that you configure the specific WWNs (think MACs) are allowed to connect, or you can configure port based zoning, where (for example) port 1 can be a host for ports 2,3, and 4, no matter what is connected to the ports..  I think that's the old way, and most people prefer to use WWN zoning.  In a corporate environment, I see how the WWN zoning would be more secure, but at home, it didn't matter to me.  Once the zoning is done, then you ALSO have to tell ESOS what initiators can talk to which targets.   This part is very similar to iSCSI. 

What kind of switch did you get?  I just got an EMC (broacade) DS-5000B.  I haven't yet gotten into it, although I haven't had much time to try yet. 

-Steve

[X-Savior]

Hey Steve,

Ok, I have Frankensteined my old Dell 860 and R200 servers into a single unit ready to roll. Put my H800 card in /w the Powervault MD1000 and also a QLogic 2464 Quad Port FC Card in.

So I got a USB stick made for now (Just so you know it does NOT work in Windows 10... Had to dig out my laptop and do it from Win 7 to get ESOS To create the USB).

So I have followed the information on initial login and changes my PW and also set the time. I am going to edit the Menu system and explanations in the instructions leave a lot to be desired. What I would LOVE is for a detailed step-by-step guide to getting a simple single FC Target created. Right now I do not have it working yet but I think I might be close. I am really good with being shown something once and then I can backtrack my steps and repeat it and learn from there.

Here is what I have done so far fiddling around in the menus.

1) Went to HOSTS and added a group on the single FC Port and called it "MD1000"
2) Went into HOSTS and added an Initiator selecting the Single FC Ports and then Choosing my MD1000 Group and then called it "Powervault"
3) Went to DEVICES and added a Device "DEV_DISK" and selected my PERC H800 Card for it (Only option) - This is the card connecting to the Powervault with External SAS Cables
4) Went to DEVICES and selected "MAP_To_Host_Group" and then selected my MD1000 and then selected the single FC Port. It asked me for a LUN ID (left it default at 0) and then chose NO for Read-Only
5) Went to TARGET and Set Relative Target ID to 1 and confirmed the FC Port was enabled (Main screen shows Enabled and Link Speed 4Gbps) - Other 3 Ports are disabled
6) Went to AULA and Created a Group Called "MD1000" and then Added Target Group for MD1000 and Called it Powervault and set ID = 1
7) Went to AULA and Selected ADD DEVICE TO GROUP and selected my DEV_DISK then Finally Add Target to Group and selected my Group and called it "Test" and set ID to 1

That pretty much concludes any configuration I can do.

I have no clue what I am doing but it all seems ok on ESOS end (no errors that I can see).

Now I go to my VMware vCenter and go to the host also plugged in to the FC Switch (By the way it is a QLogic 5602 16 Port Switch with 4Gbps Mini-GBIC's installed)

I try and refresh my HBA's looking for a LUN to pop up in ESX but I still see nothing. on ESOS at the bottom of the screen I see a single session with the Target ID of the Active FC Port and then LUN's=0, Cmds=0, Read IO (KB)=153, Write IO (KB)=0

So I guess now the question is... what did I do wrong, what can I do better, and how can we get this working? For my test I have just installed 2 x 1TB drives striped before I use my large drives for the final deployment.

Thanks for the help guys and really look forward to your assistance to solve this/

Thank You!
Cam

[X-Savior]

Ok as an Update, when I noticed ESOS showed LUN=0 I figured I would dig a little deeper.

This time I did Back-end setup and created a new volume. It showed the H800 card and it created a LUN using xfs format or something like that (2TB). ESOS showed it as LUN=1 and when I rescanned on ESX vCenter up it popped on one of the FC Ports.

So I am making progress.

now when I tried to add the new LUN in ESX as a new datastore I get an error and it is not able to add it.

so instead I decided it might be a better idea to assign a VM directly to it by "adding hard drive" and then selecting the FC LUN being presented. Once Server 2012 Loads I go to Computer Management and try and assign a drive letter. I am unable to do anything other then delete the volume. Once I did that then it let me try and create the new NTFS volume. It warned me that the volume was created by some other OS. I allowed it to delete it and create a new volume but when I try and format it... it fails saying it was unable to format it.

So right now I am unsure how I can get this LUN presented to multiple VM's (on multiple hosts as well as some standalone PC). I want all machines to be able to read and write to the LUN freely.

If you can help point me in the right direction I would appreciate it.

This worked (Server 2012R2 VM) but when I go into

[X-Savior]

Well I am at a deadend here.

I can get the LUN to show up on the host ESX machine so then I edit an individual VM and add another hard disk and select the LUN giving it RAW access to the entire LUN (Powervault). It adds ok and I go into Windows Server 2012R2 and then it pops up as a new drives and you can give it a GPT and then assign a drive letter. The problem comes when it tries to format the new drive (NTFS). It is unable to format the drive It seems to hang for about 10 minutes then says it could not format it. So now I have a Drive letter with RAW partition.

I could try and create a Virtual Hard drive on the LUN but then this might not work for any other VM's trying to access to use the same storage.

Is this even going to work? Does anyone know if it is even possible for me to get multiple windows machines to access the same filesystem storage? I am just trying to create a hyper speed FC SAN for all windows machines to access central storage (This is going to replace my File Server). If I am approaching this all wrong please tell me so I can start re-evaluating my options for high speed.

My primary reason for doing this is Gigabit Ethernet is too slow. I need MUCH MUCH faster file transfers then 150mb/sec (I was hoping I could get 400 - 500MB/sec using 4Gb Fiber).

Should I be considering using the dedicated host in a NAS setup and explore 10Gb Ethernet instead?

I am getting pretty frustrated with this silly FC stuff.

(rant over)

Thank You :)

On Saturday, March 5, 2016 at 3:43:35 PM UTC-7, X-Savior wrote:

[Marc Smith]

Can you post your /etc/scst.conf and /var/logs/kern.log files?

Marc

[Steve Jones]

I had a similar problem using ESOS with ESXi, when I tried to increase the blocksize in the ESOS device.  ESXi couldn't use it unless I left it at the default of 512.  Did you change that?  I'm just curious if it could be the same with Windows.  

-Steve

[Steve Jones]

Windows 10 definitely can run the ESOS installer..  I've done it at least 3 times with no problems, so not sure what's going on there..

I haven't had to use the ALUA section yet - That wasn't there when I first started playing with ESOS, and I haven't found a good place to read about what it is or what I might need it for.  

This does sound like what happens to me when I thought I'd increase block size to improve performance on ESXi though..

-Steve

[X-Savior]

Hi Marc,

How would I go about getting that information for you. If I plug the USB into my machine it can not read the files (to copy and paste it).

Steve:

Last night I decided to recreate the setup again but less co p,ex just to see if I can get it working. I have a lone 500g hard drive in the system that is attached by sata to the motherboard. I created a fresh new ESOS USB stick and started over. This time I did not bother with the ALUA stuff (as it looks like I do not need to use this from what I read). So I did the steps above accept no ALUA and used XFS with option "create file system" on backend storage options. It was quick and I got it to produce a LUN (showing LUN=1) and there is a small amount of data showing for read and write.

I went into ESX on another machine attached to the FC San and up the ESOS target popped (just like last time). I try and format it using VMFS and it failed exactly the same way again. Next I directly mapped the LUN inside of a server 2012r2 VM by adding a hard drive and selecting RAW mapping to the LUN.

I need to note that my hardware is a little older so I am unable to PCI pass through the FC card to a VM (not sure if this makes a difference).

Now can either of you confirm for me that I will be able (or should be able to get working) this system of being able to share a single datastore LUN to the FC SAN so that multiple Windows Machine (virtualized server 2012r2 as well as standalone Windows 7 and Windows 10) can access the same data at the same time. If so what file system will support that? VMFS is only good for storing VM's and NFS does not really work on Windows 10 unless you have Enterprise from what I read.

I only have a couple days to get this working before I need t get a working solution (I have borrowed hard drives holding my 15T of data while in limbo). Am I chasing a pipe dream here? I am considering looking at FreeNAS instead and dumping the FC stuff (backup plan). If you tell me that absolutely we can get this working with FC then I will keep at it. I know NAS really does fit what I am trying to do but I was hoping to tap into the speeds of 4Gbps FC. I wish that 10g STP Ethernet switches were not so much money.... ;)

[Marc Smith]

On Tue, Mar 8, 2016 at 9:15 AM, X-Savior <cam...@cc-comp.net> wrote:
> Hi Marc,
>
> How would I go about getting that information for you. If I plug the USB into my machine it can not read the files (to copy and paste it).

Exit to the shell in ESOS, and type "less /etc/scst.conf" and
copy/paste each page, or use SCP to retrieve the file (eg, using
WinSCP connect to the ESOS host and browse for /etc/scst.conf).

>
> Steve:
>
> Last night I decided to recreate the setup again but less co p,ex just to see if I can get it working. I have a lone 500g hard drive in the system that is attached by sata to the motherboard. I created a fresh new ESOS USB stick and started over. This time I did not bother with the ALUA stuff (as it looks like I do not need to use this from what I read). So I did the steps above accept no ALUA and used XFS with option "create file system" on backend storage options. It was quick and I got it to produce a LUN (showing LUN=1) and there is a small amount of data showing for read and write.
>
> I went into ESX on another machine attached to the FC San and up the ESOS target popped (just like last time). I try and format it using VMFS and it failed exactly the same way again. Next I directly mapped the LUN inside of a server 2012r2 VM by adding a hard drive and selecting RAW mapping to the LUN.

When you add the SCST device, what block size are you choosing? If its
anything other than the default, don't do that -- ESXi only supports
512. This is why I was asking for your /etc/scst.conf so I could see
how you have it configured.

>
> I need to note that my hardware is a little older so I am unable to PCI pass through the FC card to a VM (not sure if this makes a difference).
>
> Now can either of you confirm for me that I will be able (or should be able to get working) this system of being able to share a single datastore LUN to the FC SAN so that multiple Windows Machine (virtualized server 2012r2 as well as standalone Windows 7 and Windows 10) can access the same data at the same time. If so what file system will support that? VMFS is only good for storing VM's and NFS does not really work on Windows 10 unless you have Enterprise from what I read.

For multiple hosts/servers to access the same block-storage device
simultaneously, you'll need a cluster-aware file system -- a file
system that supports concurrent reads/writes from multiple hosts. For
Microsoft Windows, I'm 99% sure you can do this with NTFS IF you use
the Microsoft Cluster Services on all Windows hosts (the initiators).
I believe this is a requirement as the MS cluster stack will prevent
more than one hosting writing to a file / block at the same time.
Again, I'm going off what I remember, and not for sure that is how it
is. But you do need something special, using NTFS by itself is not
enough.

If you were doing this with Linux, examples of clustered file systems
are OCFS and GFS.

>
> I only have a couple days to get this working before I need t get a working solution (I have borrowed hard drives holding my 15T of data while in limbo). Am I chasing a pipe dream here? I am considering looking at FreeNAS instead and dumping the FC stuff (backup plan). If you tell me that absolutely we can get this working with FC then I will keep at it. I know NAS really does fit what I am trying to do but I was hoping to tap into the speeds of 4Gbps FC. I wish that 10g STP Ethernet switches were not so much money.... ;)

This should be do-able, provided you understand the part above about
the clustered file system. What is the application that will be
running/using the data concurrently (the shared ESOS block device)?


--Marc

[X-Savior]

Hi Mark,

Yes tonight when I get home from work I can WinSCP to the host and get the file info.

When I add it I am not changing the block size. from default AFAIK. I will go and double check it.

Now for Clustering this might be an issue. For Windows 2012 it is easy to cluster multiple servers but how you you cluster in Windows 7 and WIndows 10 PC's also....

What I am doing is centralized File Server storage basically. I will be placing all my TV Shows, Movies and Apps on the storage making it available for and PC's accessing it (Like a NAS Basically). HTPC's will play the content right off the storage as well. My entire goal is more bandwidth for file transfers between my PC and Servers. My original idea was to offload my file trasnfers into a back-end SAN for high speed transfers to then my Gigabit ethernet traffic load is lightened.

The more I look into this the more I believe this is just the wrong overall approach to do this. I believe I should be looking into a dedicate NAS Setup (FreeNAS on a Dell Server attached to the Powervault) and then plan on implementing 10GoE with Copper to my network (only expensive part right now is getting a 10GoE Switch so I might need to wait on that and settle for 6 x 1Gb Teamed Ethernet Ports). It will not give me the total single PC throughput as 4Gbps Fiber initially but then 10Gb will offset it eventually.

I believe ESOS would be better in my situation making single transfers between 2 PC. Good for dumping large data to a backup or something.

I look at the complexity of what will be required to try and get this working and I think it will far outweigh the benefits... it is like taking an orange and wish really hard to make it an apple. You may end up with a red orange but it will never be an apple ;)

I really wanted this to work but the infrastructure overhead and costs with the fiber is extensive. The cost of 1000' of Cat 6e is the same as 4 x 10' 10Gb OM4 Fiber. What I save in used SAN infrastructure over 10Gb Ethernet will be used up in Fiber costs.

[Marc Smith]

Yeah, I normally don't like to steer people away from ESOS, but in
your case, it sounds like a file server (eg, NAS, network/file level
storage) is probably the best solution to your problem.

Now, if you really wanted to keep the SAN components and use ESOS, the
recommended way would be to have an ESOS storage server that contains
all of your storage centrally, and then you can split up and provision
storage to other hosts. So in this case, provision some storage from
ESOS for a new file server. The file server runs on the block-level
storage and you run NFS, CIFS or whatever on the file server.

Might be a bit too complex for a home setup though. =)


--Marc

[Todd Hunter]

Little late chiming in, and sorry if this has been covered.  Long tread so I many have missed it. 

If you are presenting LUNs to Win12 I believe they need to be BlockIO, not FileIO.  I don't recall the details but I do remember having to stick to BlockIO when using Win12k. 

This was some time ago so it may have changed with the newer versions of ESOS but if you are using FileIO I would try BlockIO. 

Todd

[X-Savior]

Hi Guys,

Thanks for the help. I agree with all of you. This just is not the right application for me.

I might keep messing around with my fiber stuff down the road when I have time. I have to see if I can find some useful application for me (or else I just sell it all off). I have already started work on a dedicated NAS Server and it is running much smoother.

I really appreciate your time. Good work on the project and best wishes for all of you.

On Saturday, March 5, 2016 at 3:43:35 PM UTC-7, X-Savior wrote: