Mellanox Technologies MT51336-EN-EVB Manual de usuario

www.mellanox.com

SwitchX® Evaluation Board Hardware User

Manual

PN: MT51336-EN-EVB, MT51336-VPI-EVB

Rev 1.5

Document Number: 3493

Mellanox Technologies

350 Oakmead Parkway Suite 100

Sunnyvale, CA 94085

U.S.A.

www.mellanox.com

Tel: (408) 970-3400

Fax: (408) 970-3403

Mellanox Technologies, Ltd.

Beit Mellanox

PO Box 586 Yokneam 20692

Israel

www.mellanox.com

Tel: +972 (0)74 723 7200

Fax: +972 (0)4 959 3245

Mellanox®, Mellanox logo, BridgeX®, ConnectX®, CORE-Direct®, InfiniBridge®, InfiniHost®, InfiniScale®,

MLNX-OS®, PhyX®, SwitchX®, UFM®, Virtual Protocol Interconnect® and Voltaire® are registered trademarks of

Mellanox Technologies, Ltd.

Connect-IB™, FabricIT™, Mellanox Open Ethernet™, MetroX™, MetroDX™, ScalableHPC™, Unbreakable-Link™ are

trademarks of Mellanox Technologies, Ltd.

All other trademarks are property of their respective owners.

NOTE:

THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED

DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY

KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE

THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT

HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE

PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND

DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST

QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED

WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT

ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES

FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND

(INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;

LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY

THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE

OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED

DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

SwitchX Switch EVB Hardware User Manual Rev 1.5

Mellanox Technologies 3

Table of Contents

Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Chapter 1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

1.1 Management CPU. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

1.2 PCIE/1GbE Extender Kit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

1.2.1 PCIE Extender Kit. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

1.2.2 1GbE Extender Kit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

1.3 Compliance Ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

1.4 Reference Traces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

1.5 Serial Number and Product Version Information. . . . . . . . . . . . . . . . . . . 11

Chapter 2 40 Gb/s Ethernet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Chapter 3 FDR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Chapter 4 Basic Operation and Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

4.1 Switch Platform Hardware Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

4.1.1 Status LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

4.1.2 Reset Button. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

4.2 QSFP Cable Power Budget Classification. . . . . . . . . . . . . . . . . . . . . . . . 22

4.3 SFP+ Cable Power Budget Classification. . . . . . . . . . . . . . . . . . . . . . . . 22

4.4 Interfaces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

4.4.1 Port Connector Interfaces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

4.4.2 Management and Firmware Updating Interfaces . . . . . . . . . . . . . . . . . . . 23

4.5 Switch Platform Installation and Operation . . . . . . . . . . . . . . . . . . . . . . . 25

4.5.1 Installation Safety Warnings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

4.5.2 Mechanical Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4.6 Package Contents and Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

4.6.1 Installing the Switch in the Rack. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

4.6.2 Grounding the Switch. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4.6.3 Power Connections and Initial Power On. . . . . . . . . . . . . . . . . . . . . . . . . 33

4.6.4 Extracting and Inserting the Power Supply Unit. . . . . . . . . . . . . . . . . . . . 33

4.6.5 Cable Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

4.6.6 Extracting and Inserting the Fan Unit . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

Chapter 5 Installing the PCIE/1GbE Server Side board . . . . . . . . . . . . . . . . . . 37

5.1 PCIE/1GbE Extender Kit parts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

5.2 Removing the Top Cover. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

5.3 Removing the Management CPU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

5.4 Installing the PCIE/1GbE Extender . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

5.5 Installing the Server Side Board . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

5.6 Board Connectors Numbering and Locations . . . . . . . . . . . . . . . . . . . . . 45

5.7 Switch Shut Down Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

5.8 Disassembly of the Switch from the Rack. . . . . . . . . . . . . . . . . . . . . . . . 48

5.9 Disposal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

Rev 1.5

Mellanox Technologies

5.10 Resetting the Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

5.11 Configuring the Switch for the First Time . . . . . . . . . . . . . . . . . . . . . . . . 50

Chapter 6 Connecting to the Switch Platform. . . . . . . . . . . . . . . . . . . . . . . . . . 52

6.1 I2C Connector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

6.2 How to Get Mellanox Firmware Tools (MFT). . . . . . . . . . . . . . . . . . . . . . 52

6.3 Updating Firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

Chapter 7 Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

SwitchX Switch EVB Hardware User Manual Rev 1.5

Mellanox Technologies 5

List of Figures

Figure 1: Connector and Power Side Panels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9

Figure 2: Pull Out Tab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12

Figure 3: Connector and Power Side Panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15

Figure 4: Power Side Panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15

Figure 5: Power, Fan, and System LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17

Figure 6: System Health LED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17

Figure 7: Power Side Panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18

Figure 8: PSU Status LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19

Figure 9: Fan Status LED Connector Side . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19

Figure 10: Fan Status LED Power side . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20

Figure 11: Reset Button . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21

Figure 12: Port Numbering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22

Figure 13: Top and Bottom Ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22

Figure 14: Management Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23

Figure 15: Rack Rail Kit Parts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .29

Figure 16: Mounting Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .30

Figure 17: Screwing on the Rail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31

Figure 18: Inserting the Caged Nuts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31

Figure 19: Slide the Rail into the Rail Slide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .32

Figure 20: Power Supply Unit Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .33

Figure 21: PSU Pulled Out . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34

Figure 22: Top and Bottom Ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .35

Figure 23: Air Flow Labels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .35

Figure 24: Fan Module Latches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .36

Figure 25: Server Side Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .38

Figure 26: PCIE/1GbE Extender . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .39

Figure 27: Screw Locations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .40

Figure 28: Slide the Top . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .40

Figure 29: Management Board Installed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .41

Figure 30: Unscrew the Management Board . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .41

Figure 31: Remove the Management Board . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .42

Figure 32: Holes for the Extender Board Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .42

Figure 33: Install the Extender Board . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .43

Figure 34: Connecting the EVB to the Server Side Board . . . . . . . . . . . . . . . . . . . . . . . . . .44

Figure 35: EVB Board Connections View 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .45

Rev 1.5

Mellanox Technologies

Figure 36: Reset Button . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .49

Figure 37: Management Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .50

Figure 38: Transfer Power Cord . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .59

Figure 39: Install the Switch Slide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .60

Figure 40: Transfer Power Cord Finished . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .60

Figure 41: Connector Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .61

SwitchX Switch EVB Hardware User Manual Rev 1.5

Mellanox Technologies 7

List of Tables

Table 1: Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8

Table 2: System Status LED Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17

Table 3: Connector Physical and Logical Link Indications . . . . . . . . . . . . . . . . . . . . . . . .18

Table 4: PSU Status LED Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19

Table 5: Fan Status LED Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20

Table 6: Bad Port LED Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21

Table 7: Port Connector Numbering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .46

Table 8: Serial Terminal Program Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .50

Table 9: Specifications for MT51336 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .56

Table 10: Replacement Parts Ordering Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .58

Table 11: Connector Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .61

Table 12: RJ-45 Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .63

Rev 1.5

Mellanox Technologies

Revision History

Table 1 - Revision History

Revision Description

1.5 Fixed Figure 35 and Table 7

Added Power up sequence to Section 5.5

1.4 Added Appendix B Thermal Definitions

1.3 Decision to include I2C power side interface

1.2 Changes recommended by the field application engineers

1.1 Added Internal I2C section

1.0 Initial Release

SwitchX Switch EVB Hardware User Manual Rev 1.5

Mellanox Technologies 9

1 Overview

This EVB is intended to be used as a platform to evaluate the SwitchX® silicon. SW engineers

can use it as a means for SW development using the SwitxchX SDK. SW developers can imple-

ment initial driver development/integration into non PPC CPU environments.

Signal integrity engineers can use it to evaluate SwitchX SI performance and compliance to spec.

This EVB can be used to:

• evaluate and qualify

• Ethernet SerDes

•IBSerDes

•FCSerDes

• Initial Ethernet and IB switch testing for L2/L2+

This EVB can be used both while rack mounted (standard 19” rack) and in a tabletop setup (for

Signal Integrity evaluation).

Figure 1: Connector and Power Side Panels

Full Feature List

• Reference traces SMA and mini-SMP

• Compliance ports SMA and mini-SMP

• 2 ports connect to mini SMP connectors for SI testing

• 24 QSFP ports

• 1 PSU

• 8 SFP+ ports

• 24 FDR (56Gb/s) ports in a 1U switch

• IBTA 1.3 and 1.21 complaint

• SDR/DDR/QDR/FDR10/FDR link speed

• FDR10 supports 20% more bandwidth over regular QDR using the same cables/connec-

tors

• All ports can be set to gateway ports connecting to 40GigE (over QSFP) or 1/10GigE

(over QSA)

• 4.032 Tb/s aggregate data switching capacity with ultra low latency

MT51336-EN-EVB

CONSOLE

MGT

RST

UID

PS1

5678910 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 35 36

33 34

124

Mellanox !!

OK !

I2C

Overview

Rev 1.5

Mellanox Technologies

1.1 Management CPU

The EVB comes with a Power PC 460E CPU for management. It also comes with a PCIE/1GbE

Extender that allows the user bypass the management CPU and to connect directly to the chip.

1.2 PCIE/1GbE Extender Kit

The Power PC management module can be removed and in its place a PCIE/1GbE extender can

be installed. When the Management module is removed the EVB becomes externally managed.

The PCIE/1GbE extender kit contains the system side board and the server side board.

The board has four connectors:

The server side board in the kit, gets inserted in a server. Connect the card into a 8X or 16X PCI

slot. This card has 3 ports:

When the PCIE/1GbE Extender is used the Power PC is removed from the sys-

tem and none of the management interfaces on the connector side will work.

When the Power PC is removed the switch/ EVB becomes unmanaged and

does NOT need Initial configuration.

Interface Description

CX4 connector for clocking and reset

2 QSFP connectors one is blue, one is green

• The green one goes from the green labeled connector on the EVBduaghter board to the

green label on the server side card included in the kit.

• The blue one goes from the blue labeled connector on the EVBduaghter board to the blue

label on the server side card included in the kit.

Server Side board a card that gets inserted into the server X8 PCI slot.

Use only one option the PCIE or the Ethernet, do not use both

RJ 45 connector for management through an ethernet port

Interface Description

CX4 connector for clocking and reset

2 QSFP connectors one is blue, one is green

• The green one goes from the green labeled connector on the EVBduaghter board to the

green label on the server side card included in the kit.

• The blue one goes from the blue labeled connector on the EVBduaghter board to the blue

label on the server side card included in the kit.

Otros manuales para MT51336-EN-EVB

Este manual sirve para los siguientes modelos

Tabla de contenidos

Otros manuales de Placa madre de Mellanox Technologies

Mellanox Technologies

Mellanox Technologies ConnectX-4 MCX4MHEVB-ECAA Manual de usuario

Mellanox Technologies

Mellanox Technologies MT51336-EN-EVB Manual de usuario

Manuales populares de Placa madre de otras marcas

DFI

DFI SD170 Manual de usuario

ASROCK

ASROCK ROMED8-2T Manual de usuario

Asus

Asus P2L97A Manual de usuario

Winmate

Winmate IC70 Manual de usuario

Asus

Asus MES-B Manual de usuario

Grantech

Grantech SYS8F377VGGA-3OMN Manual de usuario

Telit Wireless Solutions

Telit Wireless Solutions SL869-3DR Manual de usuario

Gigabyte

Gigabyte GA-9IVDT Manual de usuario

Texas Instruments

Texas Instruments ADS8372EVM Manual de usuario

Commell

Commell MS-C73 Manual de usuario

IBT Technologies

IBT Technologies MB860 Manual de usuario

Nvidia

Nvidia TEGRA DG-04927-001_V01 Manual de usuario

Biostar

Biostar M7VIT Bravo Instrucciones de funcionamiento e instalación

Intel

Intel D33217CK Manual de usuario

Asus

Asus F1A55-M LX PLUS R2.0 Manual de usuario

EPC

EPC EPC9147C Manual de usuario

Asus

Asus EX-B760M-V5 D4 Manual de usuario

EPOX

EPOX MU-8KHA0010 Manual de usuario