标签：cluster   hpc   mellanox   infiniband   

1      下载驱动

地址：http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers

根据操作系统版本进行驱动选择，建议使用ISO格式驱动包。

备注：RHEL5及以前版本选择1.5.3系列驱动，RHEL6及以后版本选择2.0及以上系列驱动。

2      驱动安装

2.1  将下载好的驱动传到服务器上，挂载到/public/ofed目录。

[root@node33 sourcecode]#mount -o loop MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64.iso /public/ofed/

[root@node33 sourcecode]# cd

[root@node33 ~]# df -h

Filesystem           Size  Used Avail Use% Mounted on

/dev/sda3            117G  9.8G  101G  9% /

/dev/sda1            494M   17M  452M  4% /boot

tmpfs                5.9G     0  5.9G  0% /dev/shm

/tftpboot/rhel.iso   3.9G  3.9G     0 100% /tftpboot/iso

/public/sourcecode/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64.iso

                      267M  267M    0 100% /public/ofed

[root@node33 ~]#

2.2  执行安装命令，开始软件包安装。

[root@node33 ~]# /public/ofed/mlnxofedinstall -y

 Usage:/public/ofed/mlnxofedinstall [OPTIONS]

 Options

          -c|--config <packages config_file> Example of the configurationfile

                                             can be found under docs

          -n|--net <network config_file> Example of the networkconfiguration file

                                          canbe found under docs

          -k|--kernel-version <kernel version> Use provided kernel versioninstead of ‘uname -r‘

          -p|--print-available       Printavailable packages for current platform

                                      Andcreate corresponding ofed.conf file

          --without-32bit            Skip32-bit libraries installation

          --without-depcheck         SkipDistro‘s libraries check

           --without-fw-update        Skip firmware update

          --fw-update-only           Updatefirmware. Skip driver installation

          --force-fw-update          Forcefirmware update

          --force                    Forceinstallation

           --all|--hpc|--basic|--msm  Install all, hpc, basic or Mellanox Subnetmanager packages

                                     correspondingly

          --vma|--vma-vpi            Installpackages required by VMA to support VPI

          --vma-eth                  Install packages required by VMA towork over Ethernet

          -v|-vv|-vvv                Setverbosity level

          --umad-dev-rw              Grantnon root users read/write permission for umad devices instead of default

           --hugepages-overcommit     Setting 80% of MAX_MEMORY as overcommitfor huge page allocation

          --pfc <0|bitmask>         Priority based Flow Control policy on TX and RX [7:0].

                                      Perpriority bit mask (uint). Default 0.

          -q                         Setquiet - no messages will be printed

[root@node33 ~]# echo y |/public/ofed/mlnxofedinstall --basic --msm --umad-dev-rw --hugepages-overcommit

This program will install the MLNX_OFED_LINUX packageon your machine.

Note that all other Mellanox, OEM, OFED, orDistribution IB packages will be removed.

Do you want to continue?[y/N]:

Starting MLNX_OFED_LINUX-1.5.3-4.0.42 installation...

Installing mlnx-ofa_kernel RPM

Preparing...               ##################################################

mlnx-ofa_kernel            ##################################################

Installing kmod-mlnx-ofa_kernel RPM

Preparing...                ##################################################

kmod-mlnx-ofa_kernel       ##################################################

Installing kmod-mlnx-ofa_kernel-xen RPM

Preparing...               ##################################################

kmod-mlnx-ofa_kernel-xen   ##################################################

Installing kernel-mft RPM

Preparing...               ##################################################

kernel-mft                 ##################################################

Installing user level RPMs:

Preparing...               ##################################################

mlnxofed-docs              ##################################################

Preparing...               ##################################################

ofed-scripts               ##################################################

Preparing...               ##################################################

libibverbs                 ##################################################

Preparing...               ##################################################

libibverbs                 ##################################################

Preparing...               ##################################################

libibverbs-utils           ##################################################

Preparing...               ##################################################

libmthca                   ##################################################

Preparing...                ##################################################

libmthca                   ##################################################

Preparing...               ##################################################

libmverbs                  ##################################################

Preparing...               ##################################################

libmverbs                  ##################################################

Preparing...               ##################################################

libmlx4                    ##################################################

Preparing...               ##################################################

libmlx4                    ##################################################

Preparing...               ##################################################

libcxgb3                   ##################################################

Preparing...               ##################################################

libcxgb3                    ##################################################

Preparing...               ##################################################

libnes                     ##################################################

Preparing...                ##################################################

libnes                     ##################################################

Preparing...               ##################################################

libipathverbs              ##################################################

Preparing...               ##################################################

libipathverbs              ##################################################

Preparing...               ##################################################

librdmacm                   ##################################################

Preparing...               ##################################################

librdmacm                  ##################################################

Preparing...                ##################################################

librdmacm-utils            ##################################################

Preparing...               ##################################################

mstflint                    ##################################################

Preparing...               ##################################################

libibumad                  ##################################################

Preparing...               ##################################################

libibumad                  ##################################################

Preparing...               ##################################################

libibmad                   ##################################################

Preparing...               ##################################################

libibmad                   ##################################################

Preparing...               ##################################################

mft                         ##################################################

Preparing...               ##################################################

opensm-libs                ##################################################

Preparing...                ##################################################

opensm-libs                ##################################################

Preparing...               ##################################################

infiniband-diags           ##################################################

Preparing...               ##################################################

opensm                     ##################################################

Preparing...               ##################################################

ibutils                    ##################################################

Device (06:00.0):

         06:00.0InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR/ 10GigE] (rev b0)

         LinkWidth: 8x

         PCILink Speed: 2.5Gb/s

Installation finished successfully.

Programming HCA firmware for /dev/mst/mt26428_pci_cr0device

Running: mlxburn -d /dev/mst/mt26428_pci_cr0 -fw/public/ofed/firmware/fw-25408/2_9_1000/fw-ConnectX2-rel.mlx -dev_type25408  -no

-I- Querying device ...

-I- Using auto detected configuration file:/public/ofed/firmware/fw-25408/2_9_1000/MHQH19B-XTR_A1-A3.ini (PSID =MT_0D90110009)

-I- Generating image ...

   Current FW version on flash: 2.7.626

   New FW version:              2.9.1000

Burning FW image without signatures  - OK 

Restoring signature                  - OK 

-I- Image burn completed successfully.

Configuring /etc/security/limits.conf.

Please reboot your system for the changes to takeeffect.

[root@node33 ~]#

备注：安装可选all、hpc、basic、msm四种方式。建议使用basic标准模式。管理节点需要安装msm和basic两种模式！！！安装过程中会强制刷新HCA卡固件，非独立HCA卡请严格注意固件版本！！！

2.3  配置IB网卡IP地址

[root@node33 ~]# cat <<EOF >> /etc/sysconfig/network-scripts/ifcfg-ib0

>DEVICE=ib0

>BOOTPROTO=none

>ONBOOT=yes

>NETMASK=255.255.255.0

>IPADDR=12.12.12.3

> EOF

[root@node33 ~]#

[root@node33 ~]# cat/etc/sysconfig/network-scripts/ifcfg-ib0

DEVICE=ib0

BOOTPROTO=none

ONBOOT=yes

NETMASK=255.255.255.0

IPADDR=12.12.12.33

[root@node33 ~]#

2.4  启动IB服务

[root@node33 ~]# chkconfig--list | grep open

openibd            0:off 1:off 2:on 3:on 4:on 5:on 6:off

opensmd        0:off 1:off 2:off 3:on 4:on 5:on 6:off

[root@node33 ~]# /etc/init.d/openibdrestart

Unloading HCA driver:                                      [  OK  ]

Loading HCA driver and Access Layer:                       [  OK  ]

Setting up InfiniBand network interfaces:

Bringing up interface ib0:                                 [  OK  ]

Setting up service network . . .                           [  done  ]

[root@node33 ~]# /etc/init.d/opensmdrestart

Stopping IB Subnet Manager.                                [FAILED]

Starting IB Subnet Manager.                                [  OK  ]

[root@node33 ~]# ibstat

CA ‘mlx4_0‘

         CAtype: MT26428

         Numberof ports: 1

         Firmwareversion: 2.9.1000

         Hardwareversion: b0

         NodeGUID: 0x0002c903000cc00e

         Systemimage GUID: 0x0002c903000cc011

         Port 1:

                   State:Active

                   Physicalstate: LinkUp

                   Rate:40

                   Baselid: 1

                   LMC:0

                   SMlid: 1

                   Capabilitymask: 0x0251086a

                   PortGUID: 0x0002c903000cc00f

                   Linklayer: InfiniBand

[root@node33 ~]#

备注：管理节点需要先启动openibd，后启动opensmd。计算节点只需要启动openibd。配置完成后注意通过ibstat检查速率和链路状态。

3      卸载IB驱动

[root@node33 ~]#echo y | /public/ofed.uninstall.sh

This program will uninstall allMLNX_OFED_LINUX-1.5.3-4.0.42 packages on your machine.

Do you want to continue?[y/N]:y

rpm -e --allmatches --nodeps kmod-mlnx-ofa_kernel-xen-1.5.3-OFED.1.5.3.4.0.42.g3cb72fe.rhel5u8libnes-1.1.1mlnx1-1 libcxgb3-1.3.1-1 libmverbs-0.1.0-3.15.gd28970elibibmad-1.3.8.MLNX_20120424-0.1 libmthca-1.0.6mlnx1-0.1.gbe5eef3 libibumad-1.3.7.MLNX_20130110_ff06102-0.1libibverbs-1.1.5mlnx2-1 libmlx4-1.0.2mlnx6-1 librdmacm-1.0.15-1kernel-mft-2.7.1-2.6.18_308.el5 libmverbs-0.1.0-3.15.gd28970elibipathverbs-1.2mlnx1-1 libibmad-1.3.8.MLNX_20120424-0.1mlnx-ofa_kernel-1.5.3-OFED.1.5.3.4.0.42.g3cb72fe.rhel5u8libibverbs-utils-1.1.5mlnx2-1 libcxgb3-1.3.1-1 mstflint-1.4mlnx4-1.21.gd948dddlibmlx4-1.0.2mlnx6-1 librdmacm-1.0.15-1 libmthca-1.0.6mlnx1-0.1.gbe5eef3libibumad-1.3.7.MLNX_20130110_ff06102-0.1 libibverbs-1.1.5mlnx2-1 librdmacm-utils-1.0.15-1mlnxofed-docs-1.5.3-4.0.42 libipathverbs-1.2mlnx1-1kmod-mlnx-ofa_kernel-1.5.3-OFED.1.5.3.4.0.42.g3cb72fe.rhel5u8libnes-1.1.1mlnx1-1 kernel-mft-2.7.1-2.6.18_308.el5ofed-scripts-1.5.3-OFED.1.5.3.4.0.42 mft-2.7.1a-1

Uninstall finished successfully

[root@node33 ~]#rm –rf/etc/infiniband

[root@node33 ~]#

4      排错

4.1  查看IB工作状态

[root@node33 ~]# ibstat

CA ‘mlx4_0‘

         CAtype: MT26428

         Numberof ports: 1

         Firmwareversion: 2.9.1000

         Hardwareversion: b0

         NodeGUID: 0x0002c903000cc00e

         Systemimage GUID: 0x0002c903000cc011

         Port 1:

                   State:Active

                   Physicalstate: LinkUp

                   Rate:40

                   Baselid: 1

                   LMC:0

                   SMlid: 1

                   Capabilitymask: 0x0251086a

                   PortGUID: 0x0002c903000cc00f

                   Linklayer: InfiniBand

[root@node33 ~]#

4.2  查看hosts信息

[root@node33 ~]# ibhosts

Ca    :0x0002c903000cc00a ports 1 "node34 HCA-1"

Ca    :0x0002c903000cc00e ports 1 "node33 HCA-1"

[root@node33 ~]#

4.3  查看switch信息

[root@node33 ~]# ibswitches

Switch      :0x0002c9020042bcc0 ports 36 "MF0;switch-1140a2:IS5030/U1" enhancedport 0 lid 4 lmc 0

[root@node33 ~]#

4.4  查看拓扑信息

[root@node33 ~]#ibnetdiscover

#

# Topology file: generated on Sun Mar  8 19:53:35 2015

#

# Initiated from node 0002c903000cc00e port0002c903000cc00f

vendid=0x2c9

devid=0xbd36

sysimgguid=0x2c9020042bcc3

switchguid=0x2c9020042bcc0(2c9020042bcc0)

Switch      36"S-0002c9020042bcc0"                #"MF0;switch-1140a2:IS5030/U1" enhanced port 0 lid 4 lmc 0

[30]  "H-0002c903000cc00e"[1](2c903000cc00f)          # "node33 HCA-1" lid 14xQDR

[31]  "H-0002c903000cc00a"[1](2c903000cc00b)                   # "node34HCA-1" lid 7 4xQDR

vendid=0x2c9

devid=0x673c

sysimgguid=0x2c903000cc00d

caguid=0x2c903000cc00a

Ca    1"H-0002c903000cc00a"                 #"node34 HCA-1"

[1](2c903000cc00b)        "S-0002c9020042bcc0"[31]              # lid 7 lmc 0"MF0;switch-1140a2:IS5030/U1" lid 4 4xQDR

vendid=0x2c9

devid=0x673c

sysimgguid=0x2c903000cc011

caguid=0x2c903000cc00e

Ca    1"H-0002c903000cc00e"                 #"node33 HCA-1"

[1](2c903000cc00f)         "S-0002c9020042bcc0"[30]              # lid 1 lmc 0"MF0;switch-1140a2:IS5030/U1" lid 4 4xQDR

[root@node33 ~]#

4.5  查看报错统计信息

[root@node33 ~]# ibdiagnet -Pall=1

Loading IBDIAGNET from:/opt/ibutils/lib64/ibdiagnet1.5.7

-W- Topology file is not specified.

    Reportsregarding cluster links will use direct routes.

Loading IBDM from: /opt/ibutils/lib64/ibdm1.5.7

-I- Using port 1 as the local port.

-I- Discovering ... 3 nodes (1 Switches & 2 CA-s)discovered.

-I---------------------------------------------------

-I- Bad Guids/LIDs Info

-I---------------------------------------------------

-I- No bad Guids were found

-I---------------------------------------------------

-I- Links With Logical State = INIT

-I---------------------------------------------------

-I- No bad Links (with logical state = INIT) werefound

-I---------------------------------------------------

-I- General Device Info

-I---------------------------------------------------

-I---------------------------------------------------

-I- PM Counters Info

-I---------------------------------------------------

-I- No illegal PM counters values were found

-I---------------------------------------------------

-I- Fabric Partitions Report (see ibdiagnet.pkey fora full hosts list)

-I---------------------------------------------------

-I-   PKey:0x7fff Hosts:2 full:2 limited:0

-I---------------------------------------------------

-I- IPoIB Subnets Check

-I---------------------------------------------------

-I- Subnet: IPv4 PKey:0x7fff QKey:0x00000b1bMTU:2048Byte rate:10Gbps SL:0x00

-W- Suboptimal rate for group. Lowest memberrate:40Gbps > group-rate:10Gbps

-I---------------------------------------------------

-I- Bad Links Info

-I- No bad link were found

-I---------------------------------------------------

----------------------------------------------------------------

-I- Stages Status Report:

   STAGE                                    ErrorsWarnings

   Bad GUIDs/LIDs Check                    0      0    

   Link State Active Check                 0      0    

   General Devices Info Report             0      0    

   Performance Counters Report             0      0    

   Partitions Check                        0      0    

   IPoIB Subnets Check                     0      1    

Please see /tmp/ibdiagnet.log for complete log

----------------------------------------------------------------

-I- Done. Run time was 1 seconds.

[root@node33 ~]#

4.6  查看全局详细报错信息

[root@node33 ~]# ibqueryerrors

Errors for 0x2c9020042bcc0"MF0;switch-1140a2:IS5030/U1"

   GUID0x2c9020042bcc0 port ALL: [PortRcvSwitchRelayErrors == 64] [PortXmitDiscards ==29] [PortXmitWait == 240663]

   GUID0x2c9020042bcc0 port 0: [PortXmitWait == 1232]

   GUID0x2c9020042bcc0 port 1: [PortRcvSwitchRelayErrors == 2] [PortXmitDiscards == 3]

   GUID0x2c9020042bcc0 port 2: [PortRcvSwitchRelayErrors == 3] [PortXmitDiscards == 3]

   GUID0x2c9020042bcc0 port 3: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 3]

   GUID0x2c9020042bcc0 port 4: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 1]

   GUID0x2c9020042bcc0 port 5: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 2]

   GUID0x2c9020042bcc0 port 6: [PortRcvSwitchRelayErrors == 2] [PortXmitDiscards == 3]

   GUID0x2c9020042bcc0 port 7: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 2]

   GUID0x2c9020042bcc0 port 8: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 2]

   GUID0x2c9020042bcc0 port 9: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 2]

   GUID0x2c9020042bcc0 port 10: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards ==2]

   GUID0x2c9020042bcc0 port 11: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards ==2]

   GUID0x2c9020042bcc0 port 12: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards ==2]

   GUID0x2c9020042bcc0 port 13: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards ==1]

   GUID0x2c9020042bcc0 port 14: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards ==1]

   GUID0x2c9020042bcc0 port 30: [PortXmitWait == 4294967295]

   GUID0x2c9020042bcc0 port 31: [PortRcvSwitchRelayErrors == 46] [PortXmitWait == 295]

   GUID0x2c9020042bcc0 port 34: [PortXmitWait == 892]

   GUID0x2c9020042bcc0 port 36: [PortXmitWait == 238245]

## Summary: 17 nodes checked, 1 bad nodes found

##          53ports checked, 19 ports have errors beyond threshold

## Thresholds:

## Suppressed:

[root@node33 ~]#

Infiniband驱动安装-RHEL5.8

标签：cluster   hpc   mellanox   infiniband   

原文地址：http://blog.csdn.net/xztjhs/article/details/44141389