| Installation Notes | |
| Version 1.2.00 | |
| 08/17/2009 | |
| Broadcom's Linux iSCSI Offload Driver | |
| Broadcom Corporation | |
| 5300 California Avenue, | |
| Irvine, CA 92617 | |
| Copyright (c) 2006-2009 Broadcom Corporation | |
| All rights reserved | |
| Table of Contents | |
| ================= | |
| Introduction | |
| Limitations | |
| Packaging | |
| Installing Source RPM Package | |
| Building Driver From TAR File | |
| Load and Run Necessary iSCSI Software Components | |
| BNX2I Driver Dependencies | |
| BNX2I Driver Parameters | |
| Unloading and Removing Driver | |
| Driver Messages | |
| User Application - 'bnx2id' | |
| Open-iSCSI User applications | |
| Bind iSCSI target to Broadcom NX2 iSCSI tranport name | |
| Making connections to iSCSI Targets | |
| Maximize Offload iSCSI Connections | |
| Caveats | |
| Introduction | |
| ============ | |
| This file describes the bnx2i Linux drivers for the Broadcom's | |
| NetXtreme II BCM5706/BCM5708/5709/57710 10/100/1000/2500/10G Mbps PCI/PCI-X/PCIE | |
| CNIC Network Controller. The bnx2i driver enables iSCSI offload on NetXtreme II | |
| family of devices. | |
| Limitations | |
| =========== | |
| The current version of the driver has been tested on 2.6.x kernels starting | |
| from 2.6.18 kernel. The driver may not compile on older kernels, one exception | |
| is SLES10 SP1 which run 2.6.16.46 kernel. SUSE upgraded the iscsi_transport | |
| kernel module in SLES10 SP1 and Broadcom iSCSI offload Initiators is supported | |
| on SLES10 SP1. Broadcom QA validation is limited to i386 and x86_64 | |
| architectures, Redhat EL5 and SUSE 10 SP1 distributions | |
| Recently Broadcom added support for SLES11U0 running 2.6.27 kernel. | |
| Packaging | |
| ========= | |
| The driver is released in two packaging formats: source RPM and compressed tar | |
| formats. The file names for the two packages are netxtreme2-<version>.src.rpm and | |
| netxtreme2-<version>.tar.bz2 respectively. Identical source files to build the | |
| driver are included in both packages. Updated Open-iSCSI components are releases | |
| in source RPM format. Following is a list of files included - | |
| a) netxtreme2-<version>.src.rpm - NetXtreme II L2/cnic/bnx2i driver source RPM | |
| b) bnx2i-<version>.tar.bz2 - tar zipped NetXtreme II L2/cnic/bnx2i driver source | |
| c) README.TXT - This file | |
| d) RELEASE.TXT - release information text providing list of bug fixes and | |
| d) RELEASE_SLES11U0.TXT - release information text providing list of bug fixes and | |
| enhancement for driver related to SLES11U0 | |
| e) LICENSE - Licensing information | |
| f) iscsi-initiator-utils-6.2.0.868-0.7c.src.rpm - OPTIONAL - source RPM, | |
| updated open-iscsi utils for Redhat EL5.0/5.1/5.2 distribution | |
| g) open-iscsi-2.0.707-0.25b.src.rpm - source RPM, updated open-iscsi components | |
| for SLES10 SP1 distribution | |
| h) open-iscsi-2.0.707-0.44c.src.rpm - source RPM, updated open-iscsi components | |
| h) open-iscsi-2.0-870.25b.src.rpm - source RPM, updated open-iscsi components | |
| for SLES10 SP2 distribution | |
| Installing Source RPM Package | |
| ============================= | |
| The following are general guidelines for installing the driver. | |
| 1. Install the source RPM package: | |
| # rpm -ivh netxtreme2-<version>.src.rpm | |
| 2. change directory to the RPM path and build the binary driver for your kernel: | |
| # cd /usr/src/{redhat,OpenLinux,turbo,packages,rpm ..} | |
| # rpmbuild -bb SPECS/netxtreme2.spec | |
| Note that the RPM path is different for different Linux distributions. | |
| The driver will be compiled for the running kernel by default. To build | |
| the driver for a kernel different than the running one, specify the | |
| kernel by defining it in KVER: | |
| # rpmbuild -bb SPECS/netxtreme2.spec --define "KVER <kernel version>" | |
| where <kernel version> in the form of 2.x.y-z is the version of another | |
| kernel that is installed on the system. | |
| 3. Install the newly built package (driver and man page): | |
| # rpm -ivh RPMS/<arch>/netxtreme2-<version>.<arch>.rpm | |
| where <arch> is the machine architecture such as i386: | |
| # rpm -ivh RPMS/i386/netxtreme2-<version>.i386.rpm | |
| Note that the --force option may be needed on some Linux distributions | |
| if conflicts are reported. | |
| The driver will be installed in the following path: | |
| 2.6.16 and newer kernels: | |
| /lib/modules/<kernel_version>/kernel/drivers/scsi/bnx2i.ko | |
| OR | |
| /lib/modules/<kernel_version>/updates/bnx2i.ko | |
| 4. Please refer to section, "Load and Run Necessary iSCSI Software Components", | |
| on how to load necessary software components required to use Broadcom iSCSI | |
| Building Driver From TAR File | |
| ============================= | |
| The following are general guidelines for installing the driver. | |
| 1. Create a directory and extract the files: | |
| tar xvfz netxtreme2-<version>.tar.gz | |
| 2. Build the driver bnx2i.ko along with bnx2/bnx2x/cnic drivers as a loadable modules | |
| for the running kernel: | |
| # cd netxtreme2-<version>/ | |
| # make clean; make | |
| The driver will be compiled for the running kernel by default. To build | |
| the driver for a kernel different than the running one, specify the | |
| kernel by defining it in KVER: | |
| # make KVER=<kernel version> | |
| where <kernel version> in the form of 2.x.y-z is the version of another | |
| kernel that is installed on the system. | |
| 3. Test the driver by loading it (unload existing driver first if necessary): | |
| # rmmod bnx2i | |
| # insmod bnx2i/driver/bnx2i.ko | |
| 4. Install the driver and man page: | |
| make install | |
| See RPM instructions above for the location of the installed driver. | |
| 5. install user daemon (bnx2id) | |
| # cd ${DRV_BASE}/bnx2i/driver | |
| # make install_usr | |
| This command will install bnx2id binary under '/sbin' | |
| 6. Please refer to section, "Load and Run Necessary iSCSI Software Components", | |
| on how to load necessary software components required to use Broadcom iSCSI | |
| Offload feature. | |
| Load and Run Necessary iSCSI Software Components | |
| ================================================ | |
| Broadcom iSCSI Offload software suite consists of 3 kernel modules and | |
| a user daemon. Required software components can be loaded either manually or | |
| through system services - | |
| 1. Unload existing driver & kill user daemon if necessary: | |
| Manual: | |
| ------- | |
| # rmmod bnx2i | |
| # pkill -9 bnx2id | |
| Using service: | |
| -------------- | |
| # service bnx2id stop | |
| 2. Load iscsi driver & the user daemon: | |
| Manual: | |
| ------- | |
| # bnx2id | |
| # insmod bnx2i.ko | |
| or | |
| # modprobe bnx2i | |
| Using service: | |
| -------------- | |
| # service bnx2id start | |
| BNX2I Driver Dependencies | |
| ========================= | |
| The driver uses library functions in the scsi_transport_iscsi, bnx2, bnx2x, | |
| cnic and ipv6.ko. It is required to load these library modules either as | |
| loadable module or as kernel built-in component before attempting to load | |
| the driver or unresolved symbol errors will appear. Using modprobe will | |
| resolve the dependencies automatically. | |
| BNX2I Module Parameters | |
| ====================== | |
| optional parameters "en_tcp_dack", "error_mask1", and "error_mask2" | |
| can be supplied as a command line arguments to the insmod or modprobe | |
| command for bnx2i. | |
| ---------------------------- | |
| error_mask1 and error_mask2: | |
| ---------------------------- | |
| Description: "Config FW iSCSI Error Mask #", use to configure | |
| certain iSCSI protocol violation to treated either as warning | |
| or fatal error. All fatal iSCSI protocol voilations will result | |
| in session recovery (ERL 0). These are bit masks | |
| Defaults: All violation will be treated as errors. | |
| CAUTION: Do not meddle with 'error_mask' if you are not sure about the | |
| consequences. These values are to be discussed with Broadcom | |
| development team on case by case basic. This is just a mechanism | |
| to work around iSCSI implementation issues on the target side and | |
| without proper knowledge of iSCSI protocol details, user are advised | |
| not to experiment with these parameters | |
| ------------ | |
| en_tcp_dack: | |
| ------------ | |
| Description: "Enable TCP Delayed ACK", enables/disables TCP delayed ACK | |
| feature on offloaded iSCSI connections | |
| Defaults: TCP delayed ACK is ENABLED | |
| e.g. | |
| # insmod bnx2i.ko en_tcp_dack=0 | |
| or | |
| # modprobe bnx2i en_tcp_dack=0 | |
| -------- | |
| sq_size: | |
| -------- | |
| Description: "Configure SQ size", used to choose send queue size for offloaded | |
| connections and SQ size determines maximum SCSI commands that can be | |
| queued. SQ size also has a bearing on the number of connections that can | |
| be offloaded, as QP size increases, number of connections supported will | |
| decrease. With default values, 5706/5708 can offload 28 connections. | |
| Defaults: 128 | |
| Range: 32 to 128 | |
| Note: Broadcom validation is limited to power of 2, e.g. 32, 64, 128 | |
| -------- | |
| rq_size: | |
| -------- | |
| Description: "Configure RQ size", used to choose size of asynchronous buffer | |
| queue size per offloaded connections and RQ size is not required be | |
| greater than 16 as it is used to place iSCSI ASYNC/NOP/REJECT messages | |
| and SCSI sense data. | |
| Defaults: 16 | |
| Range: 16 to 32 | |
| Note: Broadcom validation is limited to power of 2, e.g. 16, 32 | |
| ---------------- | |
| event_coal_div : | |
| ---------------- | |
| Description: "Event Coalescing Divide Factor", performance tuning parameter | |
| used to moderate the rate of interrupt generation by the iscsi firmware | |
| Defaults: 1 | |
| Valid Values: 1,2,4,8 | |
| Note: Broadcom did find a single digit improvement in IOPS numbers on 1G chips | |
| But Broadcom has decided to disable interrupt coalescing for | |
| 5706/5708/5709 as our IOPS numbers are more than double the competition. | |
| However we believe this parameter makes more sense to 57710 (10G) | |
| ----------------------------------- | |
| bnx2i_nopout_when_commands_active : | |
| ----------------------------------- | |
| Description: "iSCSI NOOP even when connection is not idle", this parameter | |
| enables offload initiator to send iSCSI NOP-OUT on wire even when | |
| the link is not truely idle. This was introduced to avoid unnecessary | |
| session recoveries induced by some older targets when iSCSI NOP-OUT & | |
| iSCSI CMD pdus are intermixed. Newer iSCSI target systems are immune | |
| to this condition and this parameter is turned ON for quite some time. | |
| Defaults: 1 | |
| Values: Binary parameter, 0/1 | |
| The parameters can also be set in modprobe.conf. See the man page | |
| for more information. | |
| Unloading and Removing Driver | |
| ============================= | |
| To unload the driver, disconnect all active iSCSI sessions to targets and run | |
| the following command - | |
| rmmod bnx2i | |
| NOTE: refer to open-iscsi CLI tool, 'iscsiadm' for session teardown instructions | |
| If the driver was installed using rpm, do the following to remove it: | |
| rpm -e bnx2i | |
| If the driver was installed using make install from the tar file, the driver | |
| bnx2i.ko has to be manually deleted from the system. Refer to the section | |
| "Installing Source RPM Package" for the location of the installed driver. | |
| Driver Messages | |
| =============== | |
| The following are the most common sample messages that may be logged in the file | |
| /var/log/messages. Use dmesg -n <level> to control the level at which messages | |
| will appear on the console. Most systems are set to level 6 by default. To see | |
| all messages, set the level higher. | |
| BNX2I Driver signon: | |
| ------------------- | |
| Broadcom NetXtreme II iSCSI Driver bnx2i v1.0.30 (Sep 29, 2007) | |
| Network port to iSCSI transport name binding: | |
| --------------------------------------------- | |
| bnx2i: netif=eth2, iscsi=bcm570x-050000 | |
| bnx2i: netif=eth1, iscsi=bcm570x-030c00 | |
| Driver completes handshake with iSCSI Offload Enabled CNIC device: | |
| ------------------------------------------------------------------ | |
| bnx2i [05:00.00]: ISCSI_INIT passed | |
| NOTE: this message is displayed only when user attempts to make an | |
| iSCSI connection | |
| Driver detects iSCSI Offload is not enabled on the CNIC device: | |
| --------------------------------------------------------------- | |
| bnx2i: iSCSI not supported, dev=eth3 | |
| bnx2i: bnx2i: LOM is not enabled to offload iSCSI connections, dev=eth0 | |
| Driver unable to allocate TCP port for iSCSI connection: | |
| -------------------------------------------------------- | |
| bnx2i: unable to allocate tcp ports, make sure 'bnx2id' is running | |
| Exceeds maximum allowed iSCSI connection offload limit: | |
| ------------------------------------------------------- | |
| bnx2i: unable to allocate iSCSI context resources | |
| Network route to target node and transport name binding are 2 different devices: | |
| -------------------------------------------------------------------------------- | |
| bnx2i: conn bind, ep=0x... ($ROUTE_HBA) does not belong to hba $USER_CHOSEN_HBA | |
| where ROUTE_HBA --> net device on which connection was offloaded | |
| based on route information | |
| USER_CHOSEN_HBA --> HBA to which target node is bound (using | |
| iscsi transport name) | |
| Target cannot be reached on any of CNIC devices: | |
| ------------------------------------------------ | |
| bnx2i: check route, can't connect using cnic | |
| Network route is assigned to network interfce which is down: | |
| ------------------------------------------------------------ | |
| bnx2i: check route, hba not found | |
| Attempting to offload iSCSI connection onto Jumbo frame enabled device: | |
| ----------------------------------------------------------------------- | |
| bnx2i: eth# network i/f mtu is set to #mtu | |
| bnx2i: iSCSI HBA can support mtu of 1500 | |
| NOTE: user has to change 'mtu' to < 1500 using 'ifconfig' and restart the | |
| the interface in order to offload iSCSI connections | |
| SCSI-ML initiated host reset (session recovery): | |
| ------------------------------------------------ | |
| bnx2i: attempting to reset host, #3 | |
| CNIC detects iSCSI protocol violation - FATAL errors: | |
| ----------------------------------------------------- | |
| bnx2i: iscsi_error - wrong StatSN rcvd | |
| bnx2i: iscsi_error - hdr digest err | |
| bnx2i: iscsi_error - data digest err | |
| bnx2i: iscsi_error - wrong opcode rcvd | |
| bnx2i: iscsi_error - AHS len > 0 rcvd | |
| bnx2i: iscsi_error - invalid ITT rcvd | |
| bnx2i: iscsi_error - wrong StatSN rcvd | |
| bnx2i: iscsi_error - wrong DataSN rcvd | |
| bnx2i: iscsi_error - pend R2T violation | |
| bnx2i: iscsi_error - ERL0, UO | |
| bnx2i: iscsi_error - ERL0, U1 | |
| bnx2i: iscsi_error - ERL0, U2 | |
| bnx2i: iscsi_error - ERL0, U3 | |
| bnx2i: iscsi_error - ERL0, U4 | |
| bnx2i: iscsi_error - ERL0, U5 | |
| bnx2i: iscsi_error - ERL0, U | |
| bnx2i: iscsi_error - invalid resi len | |
| bnx2i: iscsi_error - MRDSL violation | |
| bnx2i: iscsi_error - F-bit not set | |
| bnx2i: iscsi_error - invalid TTT | |
| bnx2i: iscsi_error - invalid DataSN | |
| bnx2i: iscsi_error - burst len violation | |
| bnx2i: iscsi_error - buf offset violation | |
| bnx2i: iscsi_error - invalid LUN field | |
| bnx2i: iscsi_error - invalid R2TSN field | |
| bnx2i: iscsi_error - invalid cmd len1 | |
| bnx2i: iscsi_error - invalid cmd len2 | |
| bnx2i: iscsi_error - pend r2t exceeds MaxOutstandingR2T value | |
| bnx2i: iscsi_error - TTT is rsvd | |
| bnx2i: iscsi_error - MBL violation | |
| bnx2i: iscsi_error - data seg len != 0 | |
| bnx2i: iscsi_error - reject pdu len error | |
| bnx2i: iscsi_error - async pdu len error | |
| bnx2i: iscsi_error - nopin pdu len error | |
| bnx2i: iscsi_error - pend r2t in cleanup | |
| bnx2i: iscsi_error - IP fragments rcvd | |
| bnx2i: iscsi_error - IP options error | |
| bnx2i: iscsi_error - urgent flag error | |
| CNIC detects iSCSI protocol violation - non-FATAL, warning: | |
| ----------------------------------------------------------- | |
| bnx2i: iscsi_warning - invalid TTT | |
| bnx2i: iscsi_warning - invalid DataSN | |
| bnx2i: iscsi_warning - invalid LUN field | |
| NOTE: driver by default is configured to consider certain violation to be | |
| treated as warning and not as errors | |
| Driver puts a session through recovery: | |
| --------------------------------------- | |
| conn_err - hostno 3 conn 03fbcd00, iscsi_cid 2 cid a1800 | |
| REJECT iSCSI PDU recieved from the target: | |
| ------------------------------------------ | |
| bnx2i - printing rejected PDU contents | |
| [0]: 1 ffffffa1 0 0 0 0 20 0 | |
| [8]: 0 7 0 0 0 0 0 0 | |
| [10]: 0 0 40 24 0 0 ffffff80 0 | |
| [18]: 0 0 3 ffffff88 0 0 3 4b | |
| [20]: 2a 0 0 2 ffffffc8 14 0 0 | |
| [28]: 40 0 0 0 0 0 0 0 | |
| Open-iSCSI daemon handing over session to driver: | |
| ------------------------------------------------- | |
| bnx2i: conn update - MBL 0x800 FBL 0x800MRDSL_I 0x800 MRDSL_T 0x2000 | |
| User Application - 'bnx2id': | |
| ============================== | |
| 'bnx2id' application should be installed under '/sbin' when bnx2i RPM package | |
| is installed. Refer to the section "Installing Source RPM Package" for | |
| information. Run 'bnx2id' daemon before attempting to create iSCSI connections | |
| Driver won't be able to establish connections to iSCSI target without daemon's | |
| assistance | |
| # bnx2id | |
| bnx2id daemon requires mknod and sh shell, which are pretty much standard on | |
| any regular server. For iSCSI boot using NX2 offload support, binaries for | |
| 'mknod' and 'sh' needs to be bundled into initrd image. | |
| Open-iSCSI User applications: | |
| ============================= | |
| Install and run open-iscsi programs, 'iscsid' & 'iscsiadm' from Broadcom | |
| distributed open-iscsi packages, refer to "Packaging" section for more | |
| details. All pre-installed open-iscsi packages needs to be removed before | |
| Broadcom iSCSI supported packages could be installed. | |
| Redhat EL 5.0, 5.1, 5.2 - | |
| open-iscsi-package-name = iscsi-initiator-utils-6.2.0.868-0.7d | |
| Redhat EL 5.3 | |
| open-iscsi-package-name = iscsi-initiator-utils-6.2.0.868-0.18b | |
| SLES10 SP1 - | |
| open-iscsi-package-name = open-iscsi-2.0.707-0.25b | |
| SLES10 SP2 - | |
| open-iscsi-package-name = open-iscsi-2.0.707-0.44e | |
| SLES11 SP0 - | |
| open-iscsi-package-name = open-iscsi-2.0-870.26.5b | |
| 1. Remove all existing open-iscsi packages: | |
| RHEL5 - | |
| # rpm -e iscsi-initiator-utils | |
| SLES10 SP1 - | |
| # rpm -e open-iscsi | |
| 2. Install the source RPM package: | |
| # rpm -ivh <open-iscsi-package-name>.src.rpm | |
| 3. CD to the RPM path and build the binary driver for your kernel: | |
| # cd /usr/src/{redhat,OpenLinux,turbo,packages,rpm ..} | |
| # rpmbuild -bb SPECS/<open-iscsi-package-name>.spec | |
| Note that the RPM path is different for different Linux distributions. | |
| 4. Install the newly built package : | |
| # rpm -ivh RPMS/<arch>/<open-iscsi-package-name>.<arch>.rpm | |
| where <arch> is the machine architecture such as i386: | |
| # rpm -ivh RPMS/i386/<open-iscsi-package-name>.i386.rpm | |
| 5. Start the daemon | |
| # iscsid | |
| Making connections to iSCSI Targets: | |
| ==================================== | |
| Please refer to open-iscsi documentation for comprehensive list of 'iscsiadm' | |
| commands. Here is a sample list of commands to discover targets and | |
| create iscsi connections to target. | |
| *** Note - With iscsid v2.0-754 (open-iscsi-2.0.707-0.25b which is bundled for use | |
| with SLES10 SP1), the transport name is specified as 'node.transport_name' instead | |
| of 'iface.transport_name'. Also, the target portal group tag (TPGT) is not used. | |
| With the open-iscsi initiators bundled for use with Redhat and SLES10 SP2, the | |
| target portal group tag (TPGT) is needed if the user is creating a static entry *** | |
| Add static entry: | |
| ----------------- | |
| # iscsiadm -m node -p <ipaddr[:port],TPGT> -T <iqn.targetname> -o new | |
| iSCSI target discovery using 'SendTargets': | |
| ------------------------------------------- | |
| # iscsiadm -m discovery --type sendtargets -p <ipaddr[:port]> | |
| Bind iSCSI target to Broadcom NX2 transport (see next section for obtaining the transport name) : | |
| ------------------------------------------- | |
| # iscsiadm -m node -p <ipaddr[:port]> -T <iqn.targetname> --op=update | |
| --name=iface.transport_name --value=${XPORT_NAME} | |
| Login to target using 'iscsiadm' command: | |
| ----------------------------------------- | |
| # iscsiadm -mode node -p <ipaddr[:port]> -T <iqn.targetname> --login | |
| List all drives active in the system: | |
| ------------------------------------- | |
| # fdisk -l | |
| Bind iSCSI target to Broadcom NX2 iSCSI tranport name: | |
| ====================================================== | |
| *** Note - With iscsid v2.0-754 (open-iscsi-2.0.707-0.25b which is bundled for use | |
| with SLES10 SP1), the transport option is specified as 'node.transport_name' instead | |
| of 'iface.transport_name'. *** | |
| By default open-iscsi daemon connects to discovered targets using software | |
| initiatior (transport name = 'tcp'). Users who wish to offload iSCSI connection | |
| onto CNIC device should explicitly change transport binding of the iSCSI node. | |
| This can be easily done using 'iscsiadm' cli utlity as follows, | |
| # iscsiadm --mode node --targetname iqn.2004-06.com.broadcom:tg1 \ | |
| --portal 192.168.1.100 --op=update \ | |
| --name=iface.transport_name --value=${XPORT_NAME} | |
| where XPORT_NAME=bcm570x-xxyyzz | |
| xx - pci bus number of the NX2 device | |
| yy - pci device number of the NX2 device | |
| zz - pci function number of the NX2 device | |
| Network interface to iscsi transport name binding can be obtained by executing, | |
| # dmesg | grep "bnx2i: netif" | |
| Sample output in a system with 2 NetXtreme II devices - | |
| bnx2i: netif=eth1, iscsi=bcm570x-050000 | |
| bnx2i: netif=eth0, iscsi=bcm570x-030000 | |
| If user wishes to switch back to use software initiator for whatever reason, | |
| following command will do the trick, | |
| # iscsiadm --mode node --targetname iqn.2004-06.com.broadcom:tg1 \ | |
| --portal 192.168.1.100 --op=update \ | |
| --name=iface.transport_name --value=tcp | |
| Maximize iSCSI Offload Connections: | |
| =================================== | |
| With default driver parameter set which includes 128 outsanding commands, bnx2i | |
| can offload a maximum of 28 iSCSI connections. This is no hard limit, just a simple | |
| on chip resource allocation math. bnx2i will be able to offload > 28 connections | |
| by reducing the shared queue size which in turn limits the maximum outstanding | |
| tasks on a connection. Refer to section "BNX2I Module Parameters" for further | |
| details on sq_size, rq_size and cq_size. | |
| Driver logs the following message to syslog when maximum allowed connection | |
| offload limit is reached - "bnx2i: unable to allocate iSCSI context resources" | |
| Caveats: | |
| ======== | |
| 1. iSCSI support on CNIC devices: | |
| --------------------------------- | |
| Not all Broadcom NetExteme II device support iSCSI offload, please contact | |
| your server manufacturer on instructions to enable iSCSI offload | |
| 2) iSCSI Session won't recover after hot remove and hotplug: | |
| ------------------------------------------------------------ | |
| Cause is successive device registration with iSCSI transport layer would | |
| result in different 64-bithandles. Also when the device is removed, route | |
| could resolve to a different CNIC port in which case iSCSI session/connection | |
| are bound to HBA 1 (which just got hot removed) and TCP connection is | |
| established on HBA 2 and this is not an acceptable configuration. Broadcom | |
| advises administrator to logoff all iSCSI session before removing the HBA from | |
| the server | |
| 3. iSCSI Session recovery due to network device oparation: | |
| ---------------------------------------------------------- | |
| Following popular network device operations will result in iSCSI connection | |
| teardown, in most cases connection will recover but sometime it may not | |
| depending on the network configuration, | |
| a) Ethernet interface reset | |
| # service network restart | |
| # ifdown eth#; ifup eth# | |
| b) Change IP address | |
| # ifconfig eth0 192.168.1.20 | |
| c) MTU change | |
| # ifconfig eth0 mtu 1000 up | |
| d) ethtool selftest | |
| # ethtool -t eth0 | |
| 4. Jumbo frames not supported: | |
| ------------------------------ | |
| iSCSI driver/firmware will not offload iSCSI connection on a jumbo frame enabled | |
| CNIC device. Refer to section "Driver Messages", sub-section "Attempting to | |
| offload iSCSI connection onto Jumbo frame enabled device" for more details. | |
| 5. MPIO using open-iscsi needs pro-active iSCSI nopout's enabled: | |
| ----------------------------------------------------------------- | |
| For MPIO to work properly iSCSI nopout should be enabled on each iSCSI session. | |
| Please refer to open-iscsi documentation on how to set 'noop_out_interval' and | |
| 'noop_out_timeout' values. Default values could be differ between releases and | |
| is advisible to set them to 10 sec and 15 secs respectively | |
| 6. iSCSI boot using iSCSI Offload and multiple CNIC devices issue: | |
| ------------------------------------------------------------------ | |
| If there are multiple CNIC devices in the system and system is booted via' | |
| Broadcom's iSCSI boot solution, use has to make sure iscsi node under | |
| /etc/iscsi/nodes for the boot target is bound to NIC which is used for booting. | |
| If there is a mis-match in this configuration system will hang when boot | |
| session gets into recovery mode. | |
| e.g. | |
| NIC1 : net device = eth0, transport name = bcm570x-030200 | |
| NIC2 : net device = eth1, transport name = bcm570x-050000 | |
| boot target : name = iqn.2007-10.com.broadcom:target1, portal = 192.168.0.10:3260 | |
| If administrator chooses to boot using NIC1, parameter 'node.transport_name =' | |
| in file /etc/iscsi/nodes/iqn.2007-10.com.broadcom:target1/192.168.0.10:3260 | |
| should be set to 'bcm570x-030200'. Refer to section "Bind iSCSI target to | |
| Broadcom NX2 iSCSI tranport name" for more information on iscsi transport names |