cookbook: Linux + SDS + iSCSI

[Cesar Inacio Martins]

Cool Stuffs: Linux/Desktop + SDS/MACH11 + iSCSI

This is a little cookbook/step-by-step what I was tested successfully here and want to share with everybody.
Is more focused how configure the linux ... the IDS part is to easy to do a cookbook :)
but! if you need information how configure informix SDS, look for the Redbook "Informix Dynamic Server 11:Extending Availability and Replication" on IBM support site.

Why this?:
  I don't like to execute and study Informix over limited vmwares machines where take a lot disk, memory space and processor, I prefer real machines...how actually I already use Linux on my work desktop and my notebook... make all this more easy.

Objective:
  Configure SDS on IDS 11.50 UC3DE to work with Linux/Desktop at differ machines over Ethernet using iSCSI Software .
  About iSCSI : http://en.wikipedia.org/wiki/Iscsi

Environment:
  - IDS 11.50 UC3 Developer
  - 3 Desktops machines
  - OpenSUSE 11 x86 32bits on server (disk storage) and node machines (IDS Primary/Secondary)
  - Server OpenSuse + Target iSCSI 0.4.16 ( http://iscsitarget.sourceforge.net )
  - Node   OpenSuse + Open-iSCSI 2.0.869 (initiator only) ( http://www.open-iscsi.org )

  - linux_Sun : The Server machine where the disks are shared 
  - lnxEarth  : The client/node machine where run the SDS Primary
  - lnx_Moon  : The client/node machine where run the SDS Secondary

Considerations and limitations:
  - This example is just a development enviroment, NOT PRODUCTION.
  - I don't know if this environment are supported by IBM to a production,
    and I not recommend this to a production environment
  - Do not use the Target iSCSI version 0.4.15, have lot bugs.
  - At Server machine, the Target iSCSI (version 0.4.16) do not support access the physical disks with RAW,
    the access are made via block device ( /dev/sda )
  - To node machines, the I/O throughput on disk access are limited by Ethernet speed
    eg.: over 100 Mbits Ethernet you will have the I/O transfer at 10 MBytes/Sec.
  - In this example , no tunning and no security configuration at iSCSI are made .
    I use the default configuration.
  - At node machines the IDS can access the remote disks with RAW (Open-iSCSI + RAW).
    but, remember, this RAW access is local to iSCSI initiator, still exists the gap over ethernet.
  - I'm not executed any performance tests yet to discover what is better, KAIO or AIO vp

==============================================
Configuring Server Disk Machine:

  - My server machine information:
    |  linux_Sun:~ # hostname
    |  linux_Sun
    |
    |  linux_Sun:~ # ifconfig eth1 | egrep "eth1|inet addr"
    |  eth1      Link encap:Ethernet  HWaddr 00:02:B3:xx:xx:xx
    |            inet addr:172.30.252.222  Bcast:172.30.255.255  Mask:255.255.0.0
    |
    |  linux_Sun:~ # lsb_release  -dr
    |  Description:    openSUSE 11.0 (i586)
    |  Release:        11.0
    |
    |  linux_Sun:~ # ietd --version
    |  iscsid version 0.4.16


  - Here, all steps must be executed with root.

  - Create the partitions or cooked file to work like storage area.
    Here I will use the disk partition sda10 (1GB), sda11 (1GB), sda12 (3GB), sda13 (3GB)

    |  linux_Sun:~ # fdisk -l /dev/sda
    | 
    |  Disk /dev/sda: 80.0 GB, 80026361856 bytes
    |  255 heads, 63 sectors/track, 9729 cylinders
    |  Units = cylinders of 16065 * 512 = 8225280 bytes
    |  Disk identifier: 0x0000fb3f
    | 
    |  Device Boot      Start         End      Blocks   Id  System
    |  /dev/sda1               1         262     2104483+  82  Linux swap / Solaris
    |  /dev/sda2   *         263         288      208845   83  Linux
    |  /dev/sda3             289        8265    64075252+   f  W95 Ext'd (LBA)
    |  /dev/sda5             289         941     5245191   83  Linux
    |  /dev/sda6             942        1594     5245191   83  Linux
    |  /dev/sda7            1595        2247     5245191   83  Linux
    |  /dev/sda8            2248        4859    20980858+  83  Linux
    |  /dev/sda9            4860        7291    19535008+  83  Linux
    |  /dev/sda10           7292        7413      979933+  da  Non-FS data
    |  /dev/sda11           7414        7535      979933+  da  Non-FS data
    |  /dev/sda12           7536        7900     2931831   da  Non-FS data
    |  /dev/sda13           7901        8265     2931831   da  Non-FS data

  - Configure the Target iSCSI daemon on file /etc/ietd.conf.
    To more information about the syntax on ietd.conf, see the "man ietd.conf"
    This is look like my ietd.conf:
    |  Target iqn.2008-10.galaxy.solar_system:informix.disks
    |     Lun 0 Sectors=1000000,Type=nullio
    |     Lun 1 Path=/dev/sda10,Type=blockio,ScsiId=0
    |     Lun 2 Path=/dev/sda11,Type=blockio,ScsiId=1
    |     Lun 3 Path=/dev/sda12,Type=blockio,ScsiId=2
    |     Lun 4 Path=/dev/sda13,Type=blockio,ScsiId=3
    |     Alias Disks_Informix

    The "Lun 0" are used only for test and tunning propose.

  - Start the Target iSCSI Daemon, and check if they are running :

    |  linux_Sun:/etc # /etc/init.d/iscsi-target  start
    |
    |  linux_Sun:/etc # netstat -nltp | grep iet
    |  tcp        0      0 0.0.0.0:3260            0.0.0.0:*               LISTEN      12586/ietd
    |  tcp        0      0 :::3260                 :::*                    LISTEN      12586/ietd
    |
    |  linux_Sun:/etc # tail /var/log/messages
    |  .
    |  .
    |  Nov  7 13:56:15 linux_Sun kernel: iSCSI Enterprise Target Software - version 0.4.16
    |  Nov  7 13:56:15 linux_Sun kernel: iscsi_trgt: Registered io type fileio
    |  Nov  7 13:56:15 linux_Sun kernel: iscsi_trgt: Registered io type blockio
    |  Nov  7 13:56:15 linux_Sun kernel: iscsi_trgt: Registered io type nullio

  - Finish: At this point, the disks are shared over Ethernet

==============================================
Configuring Node Machine (primary/secondary):


  - My node machine information:
    |  lnxEarth:/etc/iscsi # hostname
    |  lnxEarth
    | 
    |  lnxEarth:/etc/iscsi # ifconfig eth0 | egrep "eth0|inet addr"
    |  eth0      Link encap:Ethernet  HWaddr 00:11:D8:xx:xx:xx
    |            inet addr:172.30.252.224  Bcast:172.30.255.255  Mask:255.255.0
    | 
    |  lnxEarth:/etc/iscsi # iscsid --version
    |  iscsid version 2.0-869

  - Do you will need change a litle the config on /etc/iscsi/iscsi.conf :
    ATTENTION: this configuration are important to do before discovery the disks.
    Comment this line:
     |  node.startup = manual
    and active this :
     | node.startup = automatic

  - Start the Open-iSCSI Daemon:
    |  lnxEarth:/etc/iscsi # /etc/init.d/open-iscsi start
    |  Starting iSCSI initiator service:                                     done
    |  iscsiadm: no records found!
    |  Setting up iSCSI targets:                                             unused

  - Detect remote disks :
    Don't worry with theses logins erros
    |  lnxEarth:/etc/init.d # iscsi_discovery  172.30.252.222
    |  iscsiadm: No active sessions.
    |  iscsiadm: Cannot modify iface.transport_name. Use iface mode to update this value.
    |  Cannot login over tcp to portal 172.30.252.222:3260
    |  iscsiadm: Cannot modify iface.transport_name. Use iface mode to update this value.
    |  Cannot login over tcp to portal 172.30.252.222:3260
    |  iscsiadm: Cannot modify iface.transport_name. Use iface mode to update this value.
    |  Cannot login over tcp to portal 172.30.252.222:3260
    |  iscsiadm: Cannot modify iface.transport_name. Use iface mode to update this value.
    |  Cannot login over tcp to portal 172.30.252.222:3260
    |  discovered 2 targets at 172.30.252.222
    |
    |  lnxEarth:/etc/init.d # iscsiadm -m node
    |  172.30.252.222:3260,1 iqn.2008-10.galaxy.solar_system:informix.disks
    |  172.30.252.222:3260,1 iqn.2008-10.galaxy.solar_system:informix.disks
    |
    
   - There is, the 5 disks shared on another machine: sdb ... sdf
    |  lnxEarth:/etc/init.d # fdisk -l 2>/dev/null |grep Disk
    |  Disk /dev/sda: 80.0 GB, 80060424192 bytes
    |  Disk identifier: 0x9fa99fa9
    |  Disk /dev/sdb: 512 MB, 512000000 bytes
    |  Disk identifier: 0x00000000
    |  Disk /dev/sdc: 1003 MB, 1003451904 bytes
    |  Disk identifier: 0x00000000
    |  Disk /dev/sdd: 1003 MB, 1003451904 bytes
    |  Disk identifier: 0x00000000
    |  Disk /dev/sde: 3002 MB, 3002194944 bytes
    |  Disk identifier: 0x00000000
    |  Disk /dev/sdf: 3002 MB, 3002194944 bytes
    |  Disk identifier: 0x00000000

   - Repeat all steps over others machines/nodes (Secondary).

   
==============================================
Linux basic tests....

  - CAUTION: be careful! make sure you are writing over the right disk!

  - First test the I/O throughput over ethernet, without disk writes using the nullio
    configured in the server machine.
    |  lnx_Moon:/etc # dd if=/dev/sdb of=/dev/null bs=2k count=100000
    |  100000+0 records in
    |  100000+0 records out
    |  204800000 bytes (205 MB) copied, 17.7421 s, 11.5 MB/s

    |  lnx_Moon:/etc # dd if=/dev/zero of=/dev/sdb bs=2k count=100000
    |  100000+0 records in
    |  100000+0 records out
    |  204800000 bytes (205 MB) copied, 77.9 s, 2.6 MB/s


  - On the primary machine , clear the first 2k on the first disk of 1GB, in my case is sdc device.
    |  lnxEarth:/etc/iscsi # dd if=/dev/zero of=/dev/sdc bs=2k count=1
    |  1+0 records in
    |  1+0 records out
    |  2048 bytes (2.0 kB) copied, 0.00122305 s, 1.7 MB/s

  - Read the first 2k on the Primary machine and will see, nothing there:
    |  lnxEarth:/etc/iscsi # dd if=/dev/sdc bs=1k count=1
    |  1+0 records in
    |  1+0 records out
    |  1024 bytes (1.0 kB) copied, 0.0176213 s, 58.1 kB/s

  - Read the first 2k on the Secondary machine and will see, nothing there too:
    Obs.: coincidentally the first disk ith 1GB are sdc on this machine too.
    |  lnx_Moon:/etc/iscsi # dd if=/dev/sdc bs=1k count=1
    |  1+0 records in
    |  1+0 records out
    |  1024 bytes (1.0 kB) copied, 0.0029615 s, 346 kB/s

  - Now, write something on this disk and read this information in other machine:
    |  lnx_Moon:/etc/iscsi # dd if=/etc/services of=/dev/sdc  bs=2k
    |  373+1 records in
    |  373+1 records out
    |  764358 bytes (764 kB) copied, 0.08247 s, 9.3 MB/s

    |  lnxEarth:/etc/iscsi # dd if=/dev/sdc bs=1k count=1
    |  #
    |  # Network services, Internet style
    |  #
    |  # Note that it is presently the policy of IANA to assign a single well-known
    |  # port number for both TCP and UDP; hence, most entries here have two entries
    |  # even if the protocol doesn't support UDP operations.
    |  #
    |  # This list could be found on:
    |  #            http://www.iana.org/assignments/port-numbers
    |  #
    |  # See also: services(5), http://www.sethwklein.net/projects/iana-etc/
    |  #
    |  # PORT NUMBERS
    |  #
    |  #

  - There is , the disks already configured!
==============================================
Configure the RAW devices:

  - Here, I was configured manually:
    |  lnx_Moon:/etc # mknod /dev/raw/raw1 c 162 1
    |  lnx_Moon:/etc # mknod /dev/raw/raw2 c 162 2
    |  lnx_Moon:/etc # mknod /dev/raw/raw3 c 162 3
    |  lnx_Moon:/etc # mknod /dev/raw/raw4 c 162 4
    |  lnx_Moon:/etc # raw /dev/raw/raw1 /dev/sdc
    |  /dev/raw/raw1:  bound to major 8, minor 32
    |  lnx_Moon:/etc # raw /dev/raw/raw2 /dev/sdd
    |  /dev/raw/raw2:  bound to major 8, minor 48
    |  lnx_Moon:/etc # raw /dev/raw/raw3 /dev/sde
    |  /dev/raw/raw3:  bound to major 8, minor 64
    |  lnx_Moon:/etc # raw /dev/raw/raw4 /dev/sdf
    |  /dev/raw/raw4:  bound to major 8, minor 80

  - Test the read with the first raw
    |  lnx_Moon:/etc # dd if=/dev/raw/raw1 bs=2k count=1
    |  #                                               
    |  # Network services, Internet style              
    |  #                                               
    |  # Note that it is presently the policy of IANA to assign a single well-known
    |  # port number for both TCP and UDP; hence, most entries here have two entries
    |  # even if the protocol doesn't support UDP operations.      

  - To configure the raw device to start on boot, see the file /etc/raw and /etc/init.d/raw
    On this version of OpenSuse have a bug where the raw/nodes are not automaticly created
    so, I made a litle change on my /etc/init.d/raw :
    |  lnxEarth:/etc/init.d # diff raw.old raw
    |  39c39
    |  <
    |  ---
    |  >       x=0
    |  40a41
    |  >         x=$(( x + 1 ))
    |  43a45
    |  >         [ ! -e /dev/raw/$rawdev ] && mknod /dev/raw/$rawdev c 162 $x

==============================================
Configure to RAW devices automaticly have informix as owner after any reboot

  - Create a new rule to udev daemon at /etc/udev/rules.d:
    |  lnxEarth:/etc/udev/rules.d # echo 'SUBSYSTEM=="raw", KERNEL=="raw[0-9]*", OWNER="informix", GROUP="informix"' > 99-informix.rules
   
  - Reload the rule:
    |  lnxEarth:/etc/udev/rules.d # udevadm control --reload_rules

  - Apply the new rule:
    |  lnxEarth:/etc/udev/rules.d # udevadm trigger

  - Check if the owner change
    |  lnxEarth:/etc/udev/rules.d # ls -l /dev/raw
    |  total 0
    |  crw-rw---- 1 informix informix 162, 1 2008-11-18 16:34 raw1
    |  crw-rw---- 1 informix informix 162, 2 2008-11-18 16:34 raw2
    |  crw-rw---- 1 informix informix 162, 3 2008-11-18 16:34 raw3
    |  crw-rw---- 1 informix informix 162, 4 2008-11-18 16:34 raw4
    |  crw-rw---- 1 root     disk     162, 0 2008-11-18 16:34 rawctl

  - Configure to all services start automaticly on boot:
    |  lnxEarth:/ # chkconfig raw on open-iscsi on

==============================================
Now, the easy part, just install Informix and configure the SDS

  Here I execute the test with IDS 11.50 UC3 Developer , with Primary+Secondary SDS.
  All tests works fine, I don't execute any heavy process (yet) , but I don't see any
problem to use this configuration on test or development environment.
  And to finish, I use the BTS 2.00 (Basic Text Search) with Smart Blob Spaces in this configuration and work too!
  very cool!

---

Veja quais são os assuntos do momento no Yahoo! + Buscados: Top 10 - Celebridades - Música - Esportes