| Installation Notes |
| Version 1.2.10 |
| 05/03/2011 |
| Broadcom's Linux iSCSI Offload Driver |
| |
| Broadcom Corporation |
| 5300 California Avenue, |
| Irvine, CA 92617 |
| |
| Copyright (c) 2006-2011 Broadcom Corporation |
| All rights reserved |
| |
| |
| Table of Contents |
| ================= |
| |
| Introduction |
| Limitations |
| Distros Supported |
| Load and Run Necessary iSCSI Software Components |
| BNX2I Driver Dependencies |
| BNX2I Driver Parameters |
| Unloading and Removing Driver |
| Driver Messages |
| User Application - 'brcm_iscsiuio' |
| Open-iSCSI User applications |
| Bind iSCSI target to Broadcom NX2 iSCSI tranport name |
| Making connections to iSCSI Targets |
| Maximize Offload iSCSI Connections |
| Caveats |
| |
| |
| Introduction |
| ============ |
| |
| This file describes the bnx2i Linux drivers for the Broadcom's |
| NetXtreme II BCM5706/BCM5708/5709/57710/57711/57711E/57712 |
| 10/100/1000/2500/10G Mbps PCI/PCI-X/PCIE CNIC Network Controller. |
| The bnx2i driver enables iSCSI offload on NetXtreme II family of |
| devices. |
| |
| |
| Distros Supported |
| ================= |
| Redhat Enterprise Linux 5.4, 5.5, 5.6 |
| Redhat Enterprise Linux 6.0, 6.1 |
| SUSE Linux Enterprise Server 11 SP1+ |
| |
| Limitations |
| =========== |
| |
| The current version of the driver has been tested on 2.6.x kernels starting |
| from 2.6.32 kernel. The driver may not compile on older kernels except the |
| distributions listed above. Broadcom QA validation is limited to i386 and |
| x86_64 architectures, Redhat EL5, EL6, and SUSE 11 SP1 distributions. |
| |
| |
| Load and Run Necessary iSCSI Software Components |
| ================================================ |
| Broadcom iSCSI Offload software suite consists of 3 kernel modules and |
| a user daemon (uIP). The user daemon is included in the iscsi-initiator-utils |
| package in the supported distributions. Refer to uIP README on how to update |
| the uIP daemon if necessary. |
| |
| Required software components can be loaded either manually or |
| through system services - |
| |
| 1. Unload existing driver if necessary: |
| Manual: |
| ------- |
| # rmmod bnx2i |
| |
| Please refer to the uIP README for manual termination of the uIP daemon. |
| |
| |
| 2. Load iscsi driver: |
| Manual: |
| ------- |
| # insmod bnx2i.ko |
| or |
| # modprobe bnx2i |
| |
| Please refer to the uIP README for manual initiation of the uIP daemon. |
| |
| |
| BNX2I Driver Dependencies |
| ========================= |
| |
| The driver uses library functions in the scsi_transport_iscsi, bnx2, bnx2x, |
| cnic and ipv6.ko. It is required to load these library modules either as |
| loadable module or as kernel built-in component before attempting to load |
| the driver or unresolved symbol errors will appear. Using modprobe will |
| resolve the dependencies automatically. |
| |
| |
| BNX2I Module Parameters |
| ====================== |
| |
| optional parameters "en_tcp_dack", "error_mask1", and "error_mask2" |
| can be supplied as a command line arguments to the insmod or modprobe |
| command for bnx2i. |
| |
| ---------------------------- |
| error_mask1 and error_mask2: |
| ---------------------------- |
| Description: "Config FW iSCSI Error Mask #", use to configure |
| certain iSCSI protocol violation to treated either as warning |
| or fatal error. All fatal iSCSI protocol voilations will result |
| in session recovery (ERL 0). These are bit masks |
| |
| Defaults: All violation will be treated as errors. |
| |
| CAUTION: Do not meddle with 'error_mask' if you are not sure about the |
| consequences. These values are to be discussed with Broadcom |
| development team on case by case basic. This is just a mechanism |
| to work around iSCSI implementation issues on the target side and |
| without proper knowledge of iSCSI protocol details, user are advised |
| not to experiment with these parameters |
| |
| |
| ------------ |
| en_tcp_dack: |
| ------------ |
| Description: "Enable TCP Delayed ACK", enables/disables TCP delayed ACK |
| feature on offloaded iSCSI connections |
| |
| Defaults: TCP delayed ACK is ENABLED |
| |
| e.g. |
| # insmod bnx2i.ko en_tcp_dack=0 |
| or |
| # modprobe bnx2i en_tcp_dack=0 |
| |
| |
| ------------- |
| time_stamps : |
| ------------- |
| Description: "Enable TCP TimeStamps", enables/disables TCP time stamp feature |
| on offloaded iSCSI connections |
| |
| Defaults: TCP time stamp option is DISABLED |
| |
| e.g. |
| # insmod bnx2i.ko time_stamps=1 |
| or |
| # modprobe bnx2i time_stamps=1 |
| |
| |
| ----------- |
| ooo_enable: |
| ----------- |
| Description: "Enable TCP Out-of-order Rx Feature", enables/disables TCP OOO |
| handling feature on offloaded iSCSI connections |
| |
| Defaults: TCP OOO rx feature is ENABLED |
| |
| e.g. |
| # insmod bnx2i.ko ooo_enable=1 |
| or |
| # modprobe bnx2i ooo_enable=1 |
| |
| |
| -------- |
| sq_size: |
| -------- |
| Description: "Configure SQ size", used to choose send queue size for offloaded |
| connections and SQ size determines maximum SCSI commands that can be |
| queued. SQ size also has a bearing on the number of connections that can |
| be offloaded, as QP size increases, number of connections supported will |
| decrease. With default values, 5706/5708 can offload 28 connections. |
| |
| Defaults: 128 |
| Range: 32 to 128 |
| Note: Broadcom validation is limited to power of 2, e.g. 32, 64, 128 |
| |
| |
| -------- |
| rq_size: |
| -------- |
| Description: "Configure RQ size", used to choose size of asynchronous buffer |
| queue size per offloaded connections and RQ size is not required be |
| greater than 16 as it is used to place iSCSI ASYNC/NOP/REJECT messages |
| and SCSI sense data. |
| |
| Defaults: 16 |
| Range: 16 to 32 |
| Note: Broadcom validation is limited to power of 2, e.g. 16, 32 |
| |
| |
| ---------------- |
| event_coal_min : |
| ---------------- |
| Description: "Event Coalescing Minimum Commands", performance tuning parameter |
| used to control the minimum rate of interrupt generation by the iscsi |
| firmware |
| |
| Defaults: 24 |
| Valid Values: 16-32 |
| Note: This parameter is meant for developers to tune the event coalescing for |
| performance adjustment and not intended for end users. |
| |
| |
| ---------------- |
| event_coal_div : |
| ---------------- |
| Description: "Event Coalescing Divide Factor", performance tuning parameter |
| used to moderate the rate of interrupt generation by the iscsi firmware |
| |
| Defaults: 1 |
| Valid Values: 1,2,4,8 |
| Note: Broadcom did find a single digit improvement in IOPS numbers on 1G chips |
| But Broadcom has decided to disable interrupt coalescing for |
| 5706/5708/5709 as our IOPS numbers are more than double the competition. |
| However we believe this parameter makes more sense to 5771x (10G) |
| |
| |
| ---------------------- |
| last_active_tcp_port : |
| ---------------------- |
| Description: "Last active TCP port", TCP port monitor parameter |
| used to indicate the last used TCP port by the iscsi firmware |
| |
| Defaults: N/A |
| Valid Values: N/A |
| |
| |
| The parameters can also be set in modprobe.conf. See the man page |
| for more information. |
| |
| |
| Unloading and Removing Driver |
| ============================= |
| |
| To unload the driver, disconnect all active iSCSI sessions to targets and run |
| the following command - |
| |
| rmmod bnx2i |
| |
| NOTE: refer to open-iscsi CLI tool, 'iscsiadm' for session teardown instructions |
| |
| If the driver was installed using rpm, do the following to remove it: |
| |
| rpm -e nextreme2 *** |
| |
| Note *** - this will remove bnx2, bnx2x and cnic modules as well |
| |
| If the driver was installed using make install from the tar file, the driver |
| bnx2i.ko has to be manually deleted from the system. Refer to the section |
| "Installing Source RPM Package" for the location of the installed driver. |
| |
| |
| Driver Messages |
| =============== |
| |
| The following are the most common sample messages that may be logged in the file |
| /var/log/messages. Use dmesg -n <level> to control the level at which messages |
| will appear on the console. Most systems are set to level 6 by default. To see |
| all messages, set the level higher. |
| |
| BNX2I Driver signon: |
| ------------------- |
| |
| Broadcom NetXtreme II iSCSI Driver bnx2i v2.1.2d (May 12, 2010) |
| |
| |
| Driver completes handshake with iSCSI Offload Enabled CNIC device: |
| ------------------------------------------------------------------ |
| |
| bnx2i [05:00.00]: ISCSI_INIT passed |
| |
| NOTE: this message is displayed only when user attempts to make an |
| iSCSI connection |
| |
| |
| Driver detects iSCSI Offload is not enabled on the CNIC device: |
| --------------------------------------------------------------- |
| |
| bnx2i: iSCSI not supported, dev=eth3 |
| bnx2i: LOM is not enabled to offload iSCSI connections, dev=eth0 |
| bnx2i: dev eth0 does not support iscsi |
| |
| |
| Exceeds maximum allowed iSCSI connection offload limit: |
| ------------------------------------------------------- |
| |
| bnx2i: alloc_ep: unable to allocate iscsi cid |
| bnx2i: unable to allocate iSCSI context resources |
| |
| |
| Network route to target node and transport name binding are 2 different devices: |
| -------------------------------------------------------------------------------- |
| |
| bnx2i: conn bind, ep=0x... ($ROUTE_HBA) does not belong to hba $USER_CHOSEN_HBA |
| where ROUTE_HBA --> net device on which connection was offloaded |
| based on route information |
| USER_CHOSEN_HBA --> HBA to which target node is bound (using |
| iscsi transport name) |
| |
| |
| Target cannot be reached on any of CNIC devices: |
| ------------------------------------------------ |
| |
| bnx2i: check route, can't connect using cnic |
| |
| |
| Network route is assigned to network interfce which is down: |
| ------------------------------------------------------------ |
| |
| bnx2i: check route, hba not found |
| |
| |
| SCSI-ML initiated host reset (session recovery): |
| ------------------------------------------------ |
| |
| bnx2i: attempting to reset host, #3 |
| |
| |
| CNIC detects iSCSI protocol violation - FATAL errors: |
| ----------------------------------------------------- |
| |
| bnx2i: iscsi_error - wrong StatSN rcvd |
| bnx2i: iscsi_error - hdr digest err |
| bnx2i: iscsi_error - data digest err |
| bnx2i: iscsi_error - wrong opcode rcvd |
| bnx2i: iscsi_error - AHS len > 0 rcvd |
| bnx2i: iscsi_error - invalid ITT rcvd |
| bnx2i: iscsi_error - wrong StatSN rcvd |
| bnx2i: iscsi_error - wrong DataSN rcvd |
| bnx2i: iscsi_error - pend R2T violation |
| bnx2i: iscsi_error - ERL0, UO |
| bnx2i: iscsi_error - ERL0, U1 |
| bnx2i: iscsi_error - ERL0, U2 |
| bnx2i: iscsi_error - ERL0, U3 |
| bnx2i: iscsi_error - ERL0, U4 |
| bnx2i: iscsi_error - ERL0, U5 |
| bnx2i: iscsi_error - ERL0, U |
| bnx2i: iscsi_error - invalid resi len |
| bnx2i: iscsi_error - MRDSL violation |
| bnx2i: iscsi_error - F-bit not set |
| bnx2i: iscsi_error - invalid TTT |
| bnx2i: iscsi_error - invalid DataSN |
| bnx2i: iscsi_error - burst len violation |
| bnx2i: iscsi_error - buf offset violation |
| bnx2i: iscsi_error - invalid LUN field |
| bnx2i: iscsi_error - invalid R2TSN field |
| bnx2i: iscsi_error - invalid cmd len1 |
| bnx2i: iscsi_error - invalid cmd len2 |
| bnx2i: iscsi_error - pend r2t exceeds MaxOutstandingR2T value |
| bnx2i: iscsi_error - TTT is rsvd |
| bnx2i: iscsi_error - MBL violation |
| bnx2i: iscsi_error - data seg len != 0 |
| bnx2i: iscsi_error - reject pdu len error |
| bnx2i: iscsi_error - async pdu len error |
| bnx2i: iscsi_error - nopin pdu len error |
| bnx2i: iscsi_error - pend r2t in cleanup |
| bnx2i: iscsi_error - IP fragments rcvd |
| bnx2i: iscsi_error - IP options error |
| bnx2i: iscsi_error - urgent flag error |
| |
| |
| CNIC detects iSCSI protocol violation - non-FATAL, warning: |
| ----------------------------------------------------------- |
| |
| bnx2i: iscsi_warning - invalid TTT |
| bnx2i: iscsi_warning - invalid DataSN |
| bnx2i: iscsi_warning - invalid LUN field |
| |
| NOTE: driver by default is configured to consider certain violation to be |
| treated as warning and not as errors |
| |
| |
| Driver puts a session through recovery: |
| --------------------------------------- |
| |
| conn_err - hostno 3 conn 03fbcd00, iscsi_cid 2 cid a1800 |
| |
| |
| REJECT iSCSI PDU recieved from the target: |
| ------------------------------------------ |
| |
| bnx2i - printing rejected PDU contents |
| [0]: 1 ffffffa1 0 0 0 0 20 0 |
| [8]: 0 7 0 0 0 0 0 0 |
| [10]: 0 0 40 24 0 0 ffffff80 0 |
| [18]: 0 0 3 ffffff88 0 0 3 4b |
| [20]: 2a 0 0 2 ffffffc8 14 0 0 |
| [28]: 40 0 0 0 0 0 0 0 |
| |
| |
| Open-iSCSI daemon handing over session to driver: |
| ------------------------------------------------- |
| bnx2i: conn update - MBL 0x800 FBL 0x800MRDSL_I 0x800 MRDSL_T 0x2000 |
| |
| |
| User Application - 'brcm_iscsiuio': |
| ================================= |
| |
| 'brcm_iscsiuio' application needs to be installed because iSCSI offload |
| connections can be made. Please refer to the 'brcm_iscsiuio' README for |
| details. Run 'brcm_iscsiuio' daemon before attempting to create iSCSI |
| connections |
| Driver won't be able to establish connections to iSCSI target without daemon's |
| assistance |
| |
| # brcm_iscsiuio |
| |
| |
| Open-iSCSI User applications: |
| ============================= |
| |
| Install and run open-iscsi programs, 'iscsid' & 'iscsiadm' from the |
| open-iscsi package, refer to "Packaging" section for more |
| details. |
| |
| 1. Start the daemon |
| # iscsid |
| |
| |
| Making connections to iSCSI Targets: |
| ==================================== |
| |
| Please refer to open-iscsi documentation for comprehensive list of 'iscsiadm' |
| commands. Here is a sample list of commands to discover targets and |
| create iscsi connections to target. |
| |
| |
| Add static entry: |
| ----------------- |
| |
| # iscsiadm -m node -p <ipaddr[:port],TPGT> -T <iqn.targetname> -o new |
| |
| |
| iSCSI target discovery using 'SendTargets': |
| ------------------------------------------- |
| |
| # iscsiadm -m discovery --type sendtargets -p <ipaddr[:port]> |
| |
| |
| Bind iSCSI target to Broadcom NX2 transport (bnx2i) : |
| ------------------------------------------------------------------------------------------------- |
| |
| # iscsiadm -m node -p <ipaddr[:port]> -T <iqn.targetname> --op=update |
| --name=iface_file_name |
| |
| |
| Login to target using 'iscsiadm' command: |
| ----------------------------------------- |
| |
| # iscsiadm -mode node -p <ipaddr[:port]> -T <iqn.targetname> --login |
| |
| |
| List all drives active in the system: |
| ------------------------------------- |
| # fdisk -l |
| |
| |
| Bind iSCSI target to Broadcom NX2 iSCSI tranport name: |
| ====================================================== |
| |
| By default open-iscsi daemon connects to discovered targets using software |
| initiatior (transport name = 'tcp'). Users who wish to offload iSCSI connection |
| onto CNIC device should explicitly change transport binding of the iSCSI node. |
| This can be easily done using 'iscsiadm' cli utlity as follows, |
| |
| # iscsiadm --mode node --targetname iqn.2004-06.com.broadcom:tg1 \ |
| --portal 192.168.1.100 --op=update \ |
| --name=iface_file_name |
| |
| where the iface file includes the following information: |
| |
| iface.net_ifacename = ethX |
| iface.iscsi_ifacename = <name of this iface file> |
| iface.hwaddress = XX:XX:XX:XX:XX:XX <make sure this is lower case> |
| iface.ipaddress = XX.XX.XX.XX |
| iface.transport_name = bnx2i |
| |
| Note that the iface.hwaddress must be in the lower case format. |
| |
| If user wishes to switch back to use software initiator for whatever reason, |
| following command will do the trick, |
| # iscsiadm --mode node --targetname iqn.2004-06.com.broadcom:tg1 \ |
| --portal 192.168.1.100 --op=update \ |
| --name=iface_file_name |
| |
| where the iface file specifies TCP transport |
| |
| iface.net_ifacename = ethX |
| iface.iscsi_ifacename = <name of this iface file> |
| iface.transport_name = tcp |
| |
| Please refer to the open-iscsi README for further details. |
| |
| |
| Maximize iSCSI Offload Connections: |
| =================================== |
| With default driver parameter set which includes 128 outsanding commands, bnx2i |
| can offload a following number of connections, |
| 5706/5708 - 28 |
| 5709 - 43 |
| 5771x - 128 |
| This is no hard limit, just a simple on chip resource allocation math. |
| bnx2i will be able to offload > 28 connections on 1G devices by reducing |
| the shared queue size which in turn limits the maximum outstanding tasks |
| on a connection. Refer to section "BNX2I Module Parameters" for further |
| details on sq_size and rq_size. Driver logs the following message to syslog |
| when maximum allowed connection offload limit is reached - |
| "bnx2i: unable to allocate iSCSI context resources" |
| |
| |
| Caveats: |
| ======== |
| |
| 1. iSCSI support on CNIC devices: |
| --------------------------------- |
| |
| Not all Broadcom NetExteme II device support iSCSI offload, please contact |
| your server manufacturer on instructions to enable iSCSI offload |
| |
| |
| 2) iSCSI Session won't recover after hot remove and hotplug: |
| ------------------------------------------------------------ |
| |
| Cause is successive device registration with iSCSI transport layer would |
| result in different 64-bithandles. Also when the device is removed, route |
| could resolve to a different CNIC port in which case iSCSI session/connection |
| are bound to HBA 1 (which just got hot removed) and TCP connection is |
| established on HBA 2 and this is not an acceptable configuration. Broadcom |
| advises administrator to logoff all iSCSI session before removing the HBA from |
| the server |
| |
| |
| 3. iSCSI Session recovery due to network device operation: |
| ---------------------------------------------------------- |
| |
| Following popular network device operations will result in iSCSI connection |
| teardown, in most cases connection will recover but sometime it may not |
| depending on the network configuration, |
| a) Ethernet interface reset |
| # service network restart |
| # ifdown eth#; ifup eth# |
| |
| b) Change IP address |
| # ifconfig eth0 192.168.1.20 |
| |
| c) MTU change |
| # ifconfig eth0 mtu 1000 up |
| |
| d) ethtool selftest |
| # ethtool -t eth0 |
| |
| |
| 4. MPIO using open-iscsi needs pro-active iSCSI nopout's enabled: |
| ----------------------------------------------------------------- |
| For MPIO to work properly iSCSI nopout should be enabled on each iSCSI session. |
| Please refer to open-iscsi documentation on how to set 'noop_out_interval' and |
| 'noop_out_timeout' values. Default values could be differ between releases and |
| is advisible to set them to 10 sec and 15 secs respectively |
| |
| |
| 5. iSCSI boot using iSCSI Offload with VLAN issue: |
| -------------------------------------------------- |
| For VLAN configuration, user has to the hardware net device identifier instead |
| of the VLAN net device identifier. For example: |
| For VLAN ethX.y net device, use ethX in the net device identification. |
| |
| Please refer to the uIP README for more details on VLAN operation. |
| |
| |
| 6. iSCSI Sessions can no longer be established after some strenuous reset test: |
| ------------------------------------------------------------------------------- |
| Please always use the latest open-iscsi util for the supported OS. |
