IScsi installation on CentOS 6
This server has a Intel i5 processor with 8G memory. 3ware data raid adapter with 14 - 1TB sata hard drives. The Iscsi server is running VMware Virtual Windows 2008 r2 and it is performing well.
Do a minimal Centos 6 server installation.
Do all updates on Server
# yum update
Other needed applications
# yum install sysstat sg3_utils
3ware Storage Adapter Software
Download software from http://www.lsi.com/channel/support/pages/downloads.aspx?k=*
$ unzip 3DM2_CLI-Linux_10.2.1_9.5.4.zip
$ sudo sh ./install.sh choose default options.
When you are finished with your installation you should see /usr/sbin/3dm2 running 3 times.
To connect go to https://domainname:888
Installation of Iscsi Target software
# yum install scsi-target-utils
Disk partition
Make sure you have GPT Kernel Support. (CentOS and Redhat Enterprise has this option enabled by default)
File Systems > Partition Types > [*] Advanced partition selection and [*] EFI GUID Partition support
Use parted and add LVM Partitions of the size you would like.
Example:
# parted /dev/sdb
mklabel gpt (Creates a new GPT disklabel, only need to do once on new disk)
mkpart primary 0 3001G (Create 3TB partition size)
set 1 lvm on (Change number as for the partition number you are setting up)
quit
# mkfs.ext4 -j /dev/sda1 (Use mkfs to format file system for a local partition)
Partition Problems
Server boots up into disk repair mode and you want to remount / rw
mount -n -o remount,rw /
Find UUID of a partition
# blkid
Network Setup
# vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE="eth0"
IPADDR="10.0.0.2"
HWADDR="mac address"
NETWORK="10.0.0.0"
NETMASK="255.255.255.0"
USERCTL="no"
BOOTPROTO="none"
GATEWAY="10.0.0.1"
ONBOOT="yes"
IPV6INIT="no"
Create a Bond0 Configuration File
# vi /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE="bond0"
IPADDR="192.168.0.4"
NETWORK="192.168.0.0"
NETMASK="255.255.255.0"
USERCTL="no"
BOOTPROTO="none"
ONBOOT="yes"
IPV6INIT="no"
MTU="9000"
# ifconfig bond0 mtu 9000 (This worked)
To confirm the MTU used between two specific devices.
# ip route get {IP-address}
# vi /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE="eth1"
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
# vi /etc/sysconfig/network-scripts/ifcfg-eth2
DEVICE="eth2"
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
# vi /etc/modprobe.d/bonding.conf
alias bond0 bonding
options bond0 mode=balance-alb miimon=100
# modprobe bonding
# service network restart
# cat /proc/net/bonding/bond0 Check output
Configuration of Iscsi Target software
# vi /etc/tgt/targets.conf add similar to below, Lots of examples is file.
# This is a sample config file for tgt-admin.
# By default, tgt-admin looks for its config file in /etc/tgt/targets.conf
#
# The "#" symbol disables the processing of a line.
# This one includes other config files:
#include /etc/tgt/temp/*.conf
# Set the driver. If not specified, defaults to "iscsi".
default-driver iscsi
## For Server1
<target iqn.2008-05.domainname.3ware.vlm2>
direct-store /dev/sda2
initiator-address 192.168.0.8
</target>
## For VMware Host Servers
<target iqn.2008-05.domainname.3ware.vmware1>
direct-store /dev/sda3
initiator-address 192.168.0.10
initiator-address 192.168.0.11
initiator-address 192.168.0.12
initiator-address 192.168.0.13
initiator-address 192.168.0.14
initiator-address 192.168.0.15
</target>
# Parameters below are only global. They can't be configured per LUN.
# Tuning parameters (global, per target)
MaxRecvDataSegmentLength 262144
MaxXmitDataSegmentLength 262144
#HeaderDigest None
#DataDigest None
InitialR2T No
MaxOutstandingR2T 8
ImmediateData Yes
#FirstBurstLength 65536
#MaxBurstLength 262144
#DataPDUInOrder Yes
#DataSequenceInOrder Yes
#ErrorRecoveryLevel 0
#IFMarker No
#OFMarker No
#DefaultTime2Wait 2
#DefaultTime2Retain 20
#OFMarkInt Reject
#IFMarkInt Reject
#MaxConnections 1
Start and Stop Iscsi Target Services
# /etc/init.d/tgtd start
# /etc/init.d/tgtd stop
# /etc/init.d/tgtd reload (Use this after changes to targets.conf)
# /etc/init.d/tgtd force-restart (Sometimes I had to run this to force changes from targets.conf)
# /etc/init.d/tgtd status
Reload may not reload ACL information for connecting more servers.
tgtadm --lld iscsi --op bind --mode target --tid 3 --initiator-address 192.168.0.10
--tid point to Target 3 and you can change bind to unbind so you can remove a address.
Show Target Information
# tgtadm --lld iscsi --mode target --op show
Setup IscsiTarget server to start with server
# chkconfig tgtd on
Performance Adjustments
Edit # vi sysctl.conf and add theses changes
# 3Ware Performance Adjustments
vm.dirty_background_ratio = 20
vm.dirty_ratio = 60
# Network Performance Adjustments
net.core.rmem_max = 1048576
net.core.rmem_default = 1048576
net.core.wmem_max = 1048576
net.core.wmem_default = 1048576
net.ipv4.tcp_mem = 1048576 1048576 1048576
net.ipv4.tcp_wmem = 1048576 1048576 1048576
net.ipv4.tcp_rmem = 1048576 1048576 1048576
For Server with more memory
# Network Performance Adjustments
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
net.core.rmem_default = 65536
net.core.wmem_default = 65536
net.ipv4.tcp_rmem = 4096 87380 8388608
net.ipv4.tcp_wmem = 4096 65536 8388608
net.ipv4.tcp_mem = 8388608 8388608 8388608
net.ipv4.route.flush = 1
Special Tweaks
# vi /etc/rc.local
echo "deadline" > /sys/block/sda/queue/scheduler
echo "deadline" > /sys/block/sdb/queue/scheduler
echo "64" > /sys/block/sda/queue/max_sectors_kb
echo "512" > /sys/block/sda/queue/nr_requests