Page 1

Sun SPARC Enterprise Mx000 Servers.....An In-depth Look (Part 1) Presenter’s Name Presenter’s Title Sun Microsystems, Inc.

5.1


Sun SPARC Enterprise Midrange and High-end Servers

Page 2


The SPARC Enterprise Product Line M8000/M9000: High-End Systems

Midrange and High-End Servers • • • •

• Faster processors • New, high-speed, low latency interconnect • Industry standard PCIe I/O • High degree of modularity • Balanced SMP design

Highest availability Highest absolute performance Sophisticated control of resources Highest scalability

M4000/M5000: Midrange Systems • Price/performance targeted • Competitive RAS and S/W • Dense rack mount package

Page 3


Term Chart Term CPU memory board unit (CMU) IO board unit (IOU) eXtended System Board (XSB)

Logical System Board (LSB)

Definition Unit equipped with a CPU module, memory, and basic IO units Unit equipped with multiple PCI cards and disk drives (M8000/M9000 only) Either the entire CMU (Uni-XSB) or a quarter of the a CMU (Quad-XSB). Uni-XSB sometimes referred to as just PSB. Quad-XSB sometimes referred to as XSB. A number assigned to an XSB to identify it in a domain

XBU (M9000 Only)

Cross Bar Unit. Connects CMUs & IOUs

CLKU (M9000 Only)

Clock Unit that provides Clock for ASICs.

BPA (M8000 Only)

Backplane with integrated XB and CLK

XSCFU

System Control Facility. Two types:XSCFU_B (base) & XSCFU_C (Exp)

Page 4


Side note: Getting Our Terms Straight • Uni-XSB: When the system board is not logically divided, it consists of a single XSB • Quad-XSB: When the system board is logically divided, it consists of four XSBs

Page 5


M4000/M5000: Mid-Range Systems Overview • Application acceleration

> Uses dual core SPARC64 VI processor > Upgradable to SPARC64 VII (provides mixed speed

support) > New higher performance interconnect > New industry standard PCI express I/O

• Less down time > Greatly improved RAS functionality

• Economical, modular datacenter design > 6U and 10U, rack-optimized enclosures > Extended domaining and dynamic

reconfiguration support > Up to 4 fully usable domains

• Solaris 10 operating system Page 6


Mid-range Product Data Sheet Processor

Memory

Internal Disks Removable Media On-board IO PCI Slots External IO Expansion Enclosure Redundant Parts Hot-swappable parts Power Options Domains RAS Management Power BTU/hr * = 4 GB DIMMS support is post-RR

M4000 SPARC64 VI 2.15GHz, 5MB L2$ Max 4 (8 core)

M5000 SPARC64 VI 2.15GHz, 5MB L2$ Max 8 (16 core)

Max 128GB * 32 DIMM slots (ECC, Extended-ECC, Mirror)

Max 256GB * 64 DIMM slots (ECC, Extended-ECC, Mirror)

Max 2 (2.5” SAS)

Max 4 (2.5” SAS) DVD, DAT

2 GbE ports 4 PCIe, 1 PCI-X

4 GbE ports 8 PCIe, 2 PCI-X Yes

6U

10U Disk, PSU, Fan Disk, PSU, Fan

1-phase 1+1 cables 1-phase 2+2 cables Max 2 Max 4 XSCF (service processor) 2350W 4590W 8,018 16,036 Page 7


M4000/M5000 Competitive Positioning Comparison of Midrange Systems Sun Fire

SPARC Enterprise

V490

M4000

V890 E2900 E4900

M5000

IBM

HP

Fujitsu

eServer p5 550 eServer p5 550Q

Integrity rx4640-8 HP 9000 rp4410-4

PW450, PW650

eServer P5 560Q

HP 9000 rp4440-8 HP 9000 rp7420-16 Integrity rx7620-16 Integrity rx7640

PW650, pq420 PW850, pq420+ PW900, pq440

Page 8


Mid-range Product Comparison Processor

Memory Internal Disks Removable Media On-board IO Peak Bandwidth I/O Bandwidth PCI Slots External IO Expansion Enclosure Redundant Parts Hot-swappable parts Power Options Domains RAS Management * = 4 GB DIMMS support is post-RR

M4000 SPARC64 VI 2.15GHz, 5MB L2$ Max 4 (8 core)

V490 UltraSPARC IV+ 1.8 GHz, 2MB L2$, 32 MB L3$ Max 4 (8 core)

Max 128GB * Max 2 (2.5� SAS) DVD, DAT 2 GbE ports 32 GBps 8 GBps 4 PCIe, 1 PCI-X Yes 6U Disk, PSU, Fan Disk, PSU, Fan 1-phase 1+1 cables Max 2 XSCF (service processor)

Max 64GB Max 2 DVD 2 GbE ports 4.8 GBps 1.2 GBps 6 PCI No 5U Disk, PSU, Fan Disk, PSU, Fan 1-phase 1+1 cables 1 domain RSC Page 9


Mid-range Product Comparison Processor

Memory Internal Disks Removable Media On-board IO Peak Bandwidth I/O Bandwidth PCI Slots External IO Expansion Enclosure Redundant Parts Hot-swappable parts Power Options Domains RAS Management * = 4 GB DIMMS support is post-RR

M5000 SPARC64 VI 2.15GHz, 5MB L2$ Max 8 (16 core)

V890 UltraSPARC IV+ 1.8 GHz, 2MB L2$, 32 MB L3$ Max 8 (16 core)

Max 256GB * Max 4 (2.5� SAS) DVD, DAT 4 GbE ports 64 GBps 16 GBps 8 PCIe, 2 PCI-X Yes 10U Disk, PSU, Fan Disk, PSU, Fan 1-phase 2+2 cables Max 4 XSCF (service processor)

Max 128GB Max 12 DVD, DDS-4 1 GbE, 1 10/100E ports 9.6 GBps 1.2 GBps 9 PCI No 17U Disk, PSU, Fan Disk, PSU, Fan 1-phase 2+1 cables 1 Domain RSC Page 10


Methodology Theoretical Peak Data B/W The data B/W values come from the SC(System Controller) to MAC(Memory Access Controller) bus-width multiplied by the frequency, then multiplied by the number of SB(System Boards). ●

M4000: 32GB/s = (16B(in)+16B(out)) x 0.506GHz x 2SC x 1SB

M5000: 64GB/s = (16B(in)+16B(out)) x 0.506GHz x 2SC x 2SB

Page 11


Methodology Theoretical Peak I/O B/W The I/O B/W values come from the SC(System Controller) to Oberon(PCI Host Bridge) bus-width multiplied by the frequency. ●

M4000: 8GB/s = (4B(in)+4B(out)) x 0.506GHz x 2SC x 1IOB

M5000: 16GB/s = (4B(in)+4B(out)) x 0.506GHz x 2SC x 2IOB

Page 12


Streams Benchmark In GB/s

M4000 M5000

IBM_System p5-550Q IBM_System p5-560Q

STREAMS (Triad) 12.7 25.2

STREAMS (Copy) 12.5 24.8

STREAMS (Triad)

STREAMS (Copy)

17.3

15.7

38.9

32.8

Page 13


Mid-range System Cabinets Front

Fan

Back

DAT

DVD Disk

M4000 System

PCI

PSU CPU

Memory

XSCF

Front

Back

CPU Fan

M5000 System

Disk DAT PSU DVD

PCI Memory

XSCF Page 14


M4000 Product Concept 4x MEM Modules

2x HDD, DVD, DAT

2 x PSU 2 x 172mm Fans

Front Iso View

2 x CPU Modules

XSCFU 1 x IO Tray (1 PCI-X, 4 PCI-Express)

Rear Iso View

Page 15


M5000 Product Concept 4 x CPU Modules 8 x MEM Modules

4 x 172mm Fans

XSCFU 4 x HDD, DVD, DAT 4 x PSU

Front Iso View

2 x IO Trays (1 PCI-X, 4 PCI-Express per Tray)

Rear Iso View

Page 16


Mid-Range Systems Components CPU Module Connectors

Memory Board Connectors

M4000 Motherboard Page 17


Mid-Range Systems Components • 2 CPUs per module • 2 (M4000) or 4 (M5000) modules per system

CPU Module Page 18


Mid-Range Systems Components • 8 DIMM slots • 1, 2 and 4*GB DIMMs • ECC • Extended-ECC • Mirrored • 4 (M4000) or 8 (M5000) modules per system • Must be added in pairs

Memory Board * = 4 GB DIMMS on FF is post-RR

Page 19


Mid-Range Systems Components • DVD, DAT • 2 SAS disks • 4 x PCIe • 1 x PCI-X • 2 GbE ports • 1 (M4000) and 2 (M5000) per system

I/O Tray Page 20


System Views: M4000

Page 21


System Views: M5000

Page 22


PCI vs PCIexpress I/O Technology PCI (shared bus)

PCI EXPRESS

(switched point to point)

Page 23


PCIe Bandwidth Technology BUS SPEED BUS WIDTH BANDWIDTH

Comments

PCI

Shared bus 33MHz 33MHz 66MHz 66MHz

32 bit 64 bit 32 bit 64 bit

133 MB/s 266 MB/s 266 MB/s 532 MB/s

PCI-X

Shared bus 66MHz 100MHz 133MHz 266MHz

64 bit 64 bit 64 bit 64 bit

512 MB/s 800MB/s 1GB/s 2GB/s

PCI Express

Switched point to point X1 X2 X4 X8 X12 X16 X32

4Gbps (500 MB/s) 8Gbps (1GBps) 16Gbps (2GBps) 32Gbps (4GBps) 48Gbps (6GBps) 64Gbps (8GBps) 128Gbps (16GBps) Page 24


PCICS: PCI Cassette for M4000/M5000 IO Cage In PCICS, PCIexpress Short dedicated cassette provided Moveable carrier plate (plastic) Carrier main body (metal)

Carrier slot keyholes

Operating handle

Card retainer (5 total, 3 types)

Page 25


Midrange M4000 Block Diagram Memory Board 8 DIMMs

MAC1

Memory Board 8 DIMMs

MAC0

CPU Module CPU0

Internal Storage DVD

PCIexpress Slot

CPU3

MAC2

Memory Board

8 DIMMs

MAC3

8 DIMMs

System Control Facility

IOU

4

SATA x4

PCIexpress Slot

CPU2

Memory Board

SC1

PCIe to PCI-X Bridge

2 x GbE

PCIexpress Slot

CPU1

SC0

PCI-X Slot

PCIexpress Slot

CPU Module

8 8 8 8

PCIe Switch

8

PCIe PHY (serdes)

PCIe Bridge

PCIe PHY (serdes) Page 26


SC ASIC • The SC (System Controller) ASIC is responsible for setup and enforcement of domaining. • The multiple paths are for performance, not redundancy • The physical address space on each SB or CMU is divided equally across the SCs • Each SC stores L2$ tags for the processors on the same SB or CMU.

Page 27


Midrange M5000 Block Diagram Memory Board 8 DIMMs

Memory Board 8 DIMMs

MAC1

MAC0

SC0

CPU Module CPU0

CPU1

MAC4

Memory Board

CPU2

CPU3

Memory Board MAC2

MAC5

Memory Board

CPU4

CPU5

CPU Module

MAC3

8 DIMMs

MAC7

8 DIMMs

SC1

IOU #1 and Disks 2,3

8 DIMMs

Memory Board

8 DIMMs

IOU #0 and Disks 0,1 and DVD/DAT

SC2

8 DIMMs

CPU Module

SC3

CPU6

CPU7

CPU Module

MAC6

8 DIMMs

Memory Board

Memory Board Page 28


Midrange M4000 Mirrored Memory Memory Board

Memory Board

4 DIMMs

4 DIMMs

Mirror

Mirror

4 DIMMs

MAC1

MAC0

CPU Module

CPU0

CPU1

CPU Module

CPU2

CPU3

4 DIMMs

Memory Board 4 DIMMs

4 DIMMs

Mirror

Mirror

4 DIMMs

= Group A

Memory Board

MAC2

MAC3

4 DIMMs

= Group B

Page 29


Midrange M5000 Mirrored Memory Memory Board

Memory Board

4 DIMMs

4 DIMMs

Mirror

Mirror

MAC1

4 DIMMs

MAC0

CPU Module

CPU0

CPU1

CPU2

CPU3

4 DIMMs

Memory Board

Memory Board

4 DIMMs

4 DIMMs

Mirror

Mirror

MAC5

4 DIMMs

Memory Board

Memory Board

4 DIMMs

4 DIMMs

Mirror

Mirror MAC2

4 DIMMs

= Group A

4 DIMMs

CPU Module

MAC4

CPU Module

CPU4

CPU5

MAC3

4 DIMMs

= Group B

CPU Module

CPU6

CPU7

Memory Board

Memory Board 4 DIMMs

4 DIMMs

Mirror

Mirror

4 DIMMs

MAC6

MAC7

4 DIMMs

Page 30


Memory Mirroring Systems with no memory mirroring Data read from memory

Memory mirroring on Mx000 Data write to memory

Data read from memory

On write, same data is written to both memory 0 & 1 â—? On reads, reads from memory 0 & 1 â—?

Memory

Memory

Memory

Controller

Controller

Controller

Error

Error

Memory

Memory 0 Memory 1

Memory 0 Memory 1

Even with uncorrectable error in memory 0, correct data in memory 1 is used and business continuity is secured. Page 31


Memory Patrolling Detection of memory corruption now earlier Memory Patrol (other vendors) OS CPU

Burden on CPU ● Swift patrolling not enabled

Memory

Memory patrolling may affect application performance, so the frequency of patrolling is limited

OS

Software Memory Patrolling

Memory Controller

Mx000 Servers Memory Patrol

CPU

HW (MAC) incorporates memory patrolling function

No burden on CPU ● Swift patrolling enabled ●

Memory Controller

Memory

Less influence on applications, timely patrolling enabled Page 32


Mid-Range M5000 I/O Block Diagram PCI-X Slot

IOU #0

PCIe to PCI-X Bridge

2 x GbE

4 PCIexpress Slot PCIexpress Slot PCIexpress Slot PCIexpress Slot

8 8 8

PCIe Switch

8

M5000 Main Board

IOU #1

PCIe to PCI-X Bridge

2 x GbE

PCIe Bridge

PCIe PHY (serdes)

8

PCI-X Slot

PCIe PHY (serdes)

4 PCIexpress Slot PCIexpress Slot PCIexpress Slot PCIexpress Slot

8 8 8 8

PCIe Switch

8

PCIe PHY (serdes)

PCIe Bridge

PCIe PHY (serdes) Page 33


M5000 Block Diagram

Page 34


Extended System Board on M4000 CPU

MEM Board

Quad-XSB 00-0

DVD/DAT IO Disks #0 Channel & #1

PCI-X Slot

PCIe Slot 0 PCIe Slot 1

CPU Module

CPU

CPU

MEM Board

Quad-XSB 00-1

MEM Board

Quad-XSB 00-2

MEM Board

Quad-XSB 00-3

IO Channel

PCIe Slot 2 PCIe Slot 3

CPU Module

CPU

M4000 MBU

M4000 IOU Page 35


Extended System Board on M5000 M5000 IOU #0 CPU

MEM Board

Uni-XSB 00-0

DVD/DAT IO Disk 0,1 Ctlr

PCI-X Slot

PCIe Slot 1

CPU Module CPU

CPU CPU Module

SB0 SB1

MEM Board

MEM Board

CPU

MEM Board

Uni-XSB 01-0

Disk 2,3

IO Ctlr

CPU Module

CPU CPU Module CPU

M5000 MBU

PCIe Slot 2

IO Ctlr

PCIe Slot 3

MEM Board

CPU

CPU

PCIe Slot 0

MEM Board

IO Ctlr

PCI-X Slot

PCIe Slot 0 PCIe Slot 1 PCIe Slot 2 PCIe Slot 3

MEM Board MEM Board

M5000 IOU #1 Page 36


Extended System Board on M5000 M5000 IOU #0 MEM Board

Quad-XSB 00-0

DVD/DAT IO Disk 0,1 Ctlr

MEM Board

Quad-XSB 00-1

IO Ctlr

MEM Board

Quad-XSB 00-2

CPU

MEM Board

Quad-XSB 00-3

CPU

MEM Board

Quad-XSB 01-0

MEM Board

Quad-XSB 01-1

MEM Board

Quad-XSB 01-2

MEM Board

Quad-XSB 01-3

CPU

PCI-X Slot

PCIe Slot 1

CPU Module CPU

CPU CPU Module

SB0 SB1

Disk 2,3

IO Ctlr

CPU Module CPU

CPU CPU Module CPU

M5000 MBU

IO Ctlr

PCIe Slot 0

PCIe Slot 2 PCIe Slot 3

PCI-X Slot

PCIe Slot 0 PCIe Slot 1 PCIe Slot 2 PCIe Slot 3

M5000 IOU #1 Page 37


Extended System Board on M5000 M5000 IOU #0 MEM Board

Quad-XSB 00-0

DVD/DAT IO Disk 0,1 Ctlr

MEM Board

Quad-XSB 00-1

IO Ctlr

MEM Board

Quad-XSB 00-2

CPU

MEM Board

Quad-XSB 00-3

CPU

MEM Board

CPU

PCI-X Slot

PCIe Slot 1

CPU Module CPU

CPU CPU Module

SB0 SB1

Uni-XSB 01-0

Disk 2,3

IO Ctlr

CPU Module CPU

CPU CPU Module CPU

M5000 MBU

MEM Board

IO Ctlr

PCIe Slot 0

PCIe Slot 2 PCIe Slot 3

PCI-X Slot

PCIe Slot 0 PCIe Slot 1 PCIe Slot 2 PCIe Slot 3

MEM Board MEM Board

M5000 IOU #1 Page 38


M8000/M9000: Sun's NextGeneration High-end Servers

Page 39


M8000/M9000: High-End Systems Overview • Application acceleration > Uses dual core SPARC64 VI processor > Upgradable to SPARC64 VII (provides mixed speed

support) > New high performance interconnect (up to 737 GB/sec) > New industry standard PCIe I/O

• State-of-the-Art RAS capabilities • Unmatched configuration flexibility > Modular building block architecture

(CMU, IOU, expansion box, XSCF) > Initial SMP architecture up to 128 cores > Expanded dynamic system domains > Up to 24 fully usable domains

• Fine granularity of unit of domain > 1 CPU chip, 4 DIMMs, 2 PCIe slots at the smallest Page 40


High-end Product Data Sheet M8000 Processor

Memory

Internal Disks Removable Media CMU (System Board) and IOU PCI Slots (on IOU) External IO Expansion Redundant Parts Hot-Swappable Parts

SPARC64 VI 2.28- 2.4GHz, 6MB L2$ Max 16 (32 core)

SPARC64 VI 2.28- 2.4GHz, 6MB L2$ Max 32 (64 core)

SPARC64 VI 2.28- 2.4GHz, 6MB L2$ Max 64 (128 core)

Max 512GB 128 DIMM slots (ECC, Extended-ECC, Mirror)

Max 1TB 256 DIMM slots (ECC, Extended-ECC, Mirror)

Max 2TB 512 DIMM slots (ECC, Extended-ECC, Mirror)

Max 16 (2.5” SAS)

Max 32 (2.5” SAS) DVD, DAT Max 8 each Max 64 (PCIe) Yes

Max 64 (2.5” SAS)

Max 4 each Max 32 (PCIe) CMU, IOU, XSCF,Disk, PSU, Fan, CLKU, XBU CMU, IOU, XSCF, Disk, PSU, Fan

Power Options Max Power BTU/hr Domains RAS Management

M9000

10,500 [W] 11,000 [VA] 35,834 Max 16

Max 16 each Max 128 (PCIe)

CMU, IOU, XSCF, CLKU, Disk, PSU, Fan, XBU CMU, IOU, XSCF, XBU, CLKU, Disk, PSU, Fan 1-phase/3-phase/dual-grid 21,300 [W] 42,600 [W] 22,400 [VA] 44,800 [VA] 72,693 145,385 Max 24 XSCF (service processor) Page 41


M4000 - M9000 Competitive Positioning Comparison of Midrange and High-end Systems Sun Fire

SPARC Enterprise

V490

M4000

V890 E2900

IBM

HP

Fujitsu

eServer p5 550 eServer p5 550Q

Integrity rx4640-8 HP 9000 rp4410-4

PW450, PW650

eServer P5 560Q

HP 9000 rp4440-8 HP 9000 rp7420-16 Integrity rx7620-16 Integrity rx7640

PW650, pq420

M5000

E4900

PW850, pq420+ PW900, pq440

E6900

M8000

eServer P5 570

Integrity rx8620-32 Integrity rx8640 HP 9000 rp8420-32 Integrity Superdome 16 HP 9000 Superdome 16

PW900, PW1500, pq480, pq440+

E20K

M9000 (32 CPUs)

eServer p5 590

Integrity Superdome 32 HP 9000 Superdome 32

PW2500/64, pq480+

E25K

M9000 (64 CPUs)

eServer p5 595

Integrity Superdome 64 HP 9000 Superdome 64

PW2500/128 Page 42


High-end Product Comparison M8000

E6900

SPARC64 VI 2.28 – 2.4GHz, 6MB L2$ Max 16 (32 core)

UltraSPARC IV+

1.8 GHz, 2MB L2$, 32 MB L3$

System Bandwidth

Max 512GB 184 GBps

Max 384 GB 9.6 GBps

Memory Bandwidth

170 GBps

38 GBps

I/O Bandwidth

61 GBps

8 GBps

Max 16 (2.5” SAS) DVD, DAT Max 4 each Max 32 (PCIe) Yes

None None 6 CMU, 4 I/O Boats Max 32 PCI-X No

CMU, IOU, XSCF, Clock board, Disk, PSU, Fan, XB

SysBd, PSU, Fan, FireplaneSwitch, SC

CMU, IOU, XSCF, Disk, PSU, Fan 1-phase/3-phase/dual-grid Max 16 XSCF (service processor)

PSU, Fan, SC, SysBd, IO-Bd 1-phase/dual-grid Max 4 domains SC (System Controller)

Processor

Memory

Internal Disks Removable Media CMU (System Board) and IOU PCI Slots (on IOU) External IO Expansion Redundant Parts Hot-Swappable Parts Power Options Domains RAS Management

Max 24 (48 core)

Page 43


High-end Product Comparison Processor

Memory System Bandwidth Memory Bandwidth I/O Bandwidth Internal Disks Removable Media CMU (System Board) and IOU PCI Slots (on IOU) External IO Expansion Redundant Parts Hot-Swappable Parts Power Options Domains RAS Management

M9000

E25K

SPARC64 VI 2.28 – 2.4GHz, 6MB L2$ Max 64 (128 core)

UltraSPARC IV+

1.8 GHz, 2MB L2$, 32 MB L3$

Max 2TB 737 GBps 683 GBps 244 GBps Max 64 (2.5” SAS) DVD, DAT Max 16 each Max 128 (PCIe) Yes

Max 1TB 43 GBps 173 GBps 40 GBps None None Max 18 each 72 PCI-X No

CMU, IOU, XSCF, Clock board, Disk, PSU, Fan, XB

CMU, IOU, SC, PSU, Fan, Backplane

CMU, IOU, XSCF, Disk, PSU, Fan 1-phase/3-phase/dual-grid

CMU, IOU, SC, PSU, Fan 1-phase/dual-grid

Max 24

Max 18 domains

XSCF (service processor)

SC (System Controller)

Max 72 (144 core)

Page 44


Methodology Theoretical Peak Data B/W The data B/W values come from the SC(System Controller) to MAC(Memory Access Controller) bus-width multiplied by the frequency, then multiplied by the number of SB(System Boards). ●

M4000: 32GB/s = (16B(in)+16B(out)) x 0.506GHz x 2SC x 1SB

M5000: 64GB/s = (16B(in)+16B(out)) x 0.506GHz x 2SC x 2SB

M8000: 184GB/s = (16B(in)+8B(out)) x 0.480GHz x 4SC x 4SB

M9000 (32 CPUs): 368GB/s = (16B(in)+8B(out)) x 0.480GHz x 4SC x 8SB M9000 (64 CPUs): 737GB/s = (16B(in)+8B(out)) x 0.480GHz x 4SC x 16SB

Page 45


Methodology Theoretical Peak I/O B/W The I/O B/W values come from the SC(System Controller) to Oberon(PCI Host Bridge) bus-width multiplied by the frequency. ●

M4000: 8GB/s = (4B(in)+4B(out)) x 0.506GHz x 2SC x 1IOB

M5000: 16GB/s = (4B(in)+4B(out)) x 0.506GHz x 2SC x 2IOB

M8000: 61GB/s = (4B(in)+4B(out)) x 0.480GHz x 2SC x 2Oberon x 4IOU

M9000 (32 CPUs): 122GB/s = (4B(in)+4B(out)) x 0.480GHz x 2SC x 2Oberon x 8IOU M9000 (64 CPUs): 244GB/s = (4B(in)+4B(out)) x 0.480GHz x 2SC x 2Oberon x 16IOU

Page 46


Streams Benchmark In GB/s

M4000 M5000 M8000 M9000 (32 CPUs) M9000 (64 CPUs)

IBM_System p5-550Q IBM_System p5-560Q IBM_System p5-570 IBM_System p5-595

STREAMS (Triad) 12.7 25.2 69.6

STREAMS (Copy) 12.5 24.8 60.3

134.3

114.9

219.3

212.9

STREAMS (Triad)

STREAMS (Copy)

17.3

15.7

38.9

32.8

44.2

39

173.5

157.5 Page 47


High-end System Cabinets M8000 System

M9000 System **Rack for I/O Expansion

1200

M9000 System 1800

Fan 1200

Expansion Cabinet

720 PSU AC input selector XB Clock Board XSCF

Fan CMU

1800

IOU

Base Cabinet ** Rack for I/O Expansion ** Rack for I/O Expansion is optional and applies to M8000, M9000

IOU DVD

Front

810

Rear Page 48


M8000 and M9000 Comparison COMMONALITIES (M8000 vs. M9000) DIFFERENCIES (M8000 vs. M9000)

• CPU module • CPU memory unit (CMU) • I/O unit (IOU) • FAN trays ( Fan_A (3)) • Power Supply Unit (PSU) • XSCF board • Operator Panel board • 2.5” SAS disk • DVD, DAT • PCI cassette • Memory

• Max. number of CMU and IOU > M8000: 4CMU/IOU -> Max. 16 domain > M9000: 8CMU/IOU in one cabinet -> Max 24domain

• XB/CLK > M8000: XB/CLK exists on BP_A, Two clock source, but one clock ASIC > M9000: XB is in XBU, clock is in CLKU (Two clock source, and two clock ASICs)

• DDC for XB/CLK > M8000: Exist as a separate unit, DDC_A > M9000: Integrated to XBU/CLKU

• Expansion cabinet for M9000 only • Internal rack space > M8000: 12U available > M9000: None

• Power subsystem/Cooling subsystem > Different number of PSU/power cables > Different kind/number of Fans Page 49


M8000/M9000 System's RAS Management • Redundant interconnect • Redundant Service Processor * • No Single Failure can Prohibit Reboot • Most FRU's are Hot-Swappable

* = failover support is post-RR

Page 50


M8000 System Rack space for Dual Power Grid Option 19” Rack Space PSU 1-4 CMU Boards Fan Trays

XSCFU_B#0 XSCFU_B#1

1-4 IO Units (IOU) (4 Disk Bays Each)

Fan Trays

Rear View

Front View Page 51


M9000 System Fans PSU’s Crossbar Boards

Clock Boards XSCF Boards

DAT

IO Boards (4 Disk Bays Each)

DVD

Front View Page 52


M9000 System Fans

CMU Boards

IO Boards (4 Disk Bays Each)

Back View Page 53


M9000 System Fans PSU’s

Crossbar Boards

Crossbar Boards

Clock Boards XSCF Boards

Clock Boards XSCF Boards IO Boards DAT

IO Boards (4 Disk Bays Each)

DVD

DAT DVD

Front View

Page 54


M9000 System (no cables)

Crossbar Boards

Clock Boards XSCF Boards

Crossbar Boards

Clock Boards XSCF Boards

Page 55


M9000 System Fans

Fans

CMU Boards

CMU Boards

IO Boards IO Boards

Back View

Page 56


M9000+External Power Cabinets External Power Cabinet

External Power Cabinet

Page 57


Backplane BP_A (for M8000) Active Backplane

Page 58


Backplane BP_B (for M9000) Passive Backplane

Page 59


M8000/M9000 High-End Building Blocks CPU/Memory Unit (CMU) • Max 4 CPU sockets / 8 cores • 2x 16 DIMM slots (1 GB, 2 GB, 4 GB) • 16 GB, 32 GB, 64 GB memory expansions • Memory mirror option only in Uni-XSB mode

2x 16 DIMM

4 CPU Sockets

Red. DC-DC Converter

Page 60


Side note: Getting Our Terms Straight • Uni-XSB: When the system board is not logically divided, it consists of a single XSB • Quad-XSB: When the system board is logically divided, it consists of four XSBs

Page 61


M8000/M9000 High-End Building Blocks I/O Unit (IOU) • 1x IOU per CMU or 1x IOU per multiple CMU • 2x 4 PCIe slots • Optional base IO card (2x SAS, 2x GbE) using PCIe slot • PCIe slot assignment to 4x XSB • Link up to 4 full I/O expansion boats (link cards use PCIe slots) 2x2 Disk Bays

2x4 PCIe Slots

Page 62


PCICS: PCI Cassette for M8000/M9000 IOU In PCICS, PCIexpress Short dedicated cassette provided Moveable carrier plate (plastic) Carrier main body (metal)

Circular area for rotating the retainer Operating handle

Card retainer (5 total, 3 types)

Page 63


M8000/M9000 System Board Block Diagram All Buses are 4B Uni-Directional Paths with Multiplexed Address and Data Plus Control and ECC in Each Direction to/from XB way0

4B per path @1.07 GHz 2 paths each direction ~5 to 6 GB/sec/way peak (~30-35% data 65-70% broadcast address and control)

4 DIMMs

SC0 CPU0

CPU1

to/from XB way2 CPU2

to/from XB way3 4B per path @1.07GHz 1 path each direction

4 DIMMs

4 DIMMs

to/from XB way1

4B per path @ 0.53GHz 1 path each direction

MAC0

CPU3

SC1

MAC1

4 DIMMs

4 DIMMs

SC2

MAC2

SC3

4 DIMMs

4 DIMMs MAC3

4 DIMMs Page 64


M8000/M9000 System Board Block Diagram All Buses are 4B Uni-Directional Paths with Multiplexed Address and Data Plus Control and ECC in Each Direction 8.7 GB/S to/from XB way0

to/from XB way1

~40 GB/sec ~10 GB/sec peak

4 DIMMs

SC0

MAC0

CPU0

5~6 GB/sec peak

~20-24 GB/sec peak

4 DIMMs

4 DIMMs

CPU1

SC1

MAC1

35 GB/sec peak

to/from XB way2 CPU2

SC2

4 DIMMs

4 DIMMs MAC2

4 DIMMs

4 DIMMs

to/from XB way3 CPU3

SC3

MAC3

4 DIMMs Page 65


M8000/M9000 Mirrored Memory All memory is mirrored by groups. SC0

4 DIMMs

= Group B MAC0

CPU0

CPU1

SC1

= Group A

MAC1

4 DIMMs

Mirror

Mirror

4 DIMMs

4 DIMMs 4 DIMMs

CPU2

CPU3

SC2

SC3

MAC2

MAC3

4 DIMMs

Mirror

Mirror

4 DIMMs

4 DIMMs

Page 66


M8000/M9000 I/O Board Block Diagram Internal DVD Drive

External Drive

Internal DAT Drive

in cabinet

Base IO Board

PCIe to PCI-X bridge

SAS GBit Ethernet

PCIe bridge

to/from one way – 2 XB’s

PCIe PHY (serdes) PCIe PHY (serdes)

5-6 GB/sec peak

to/from one way – 2 XB’s

DAT

EXP

IOU

EXP

DVD

PCIe bridge

PCIe PHY (serdes) PCIe PHY (serdes)

8

PCIexpress Short Slot

8

PCIexpress Short Slot

8

PCIexpress Short Slot

8

PCIexpress Short Slot

8

PCIexpress Short Slot

8

PCIexpress Short Slot

8

PCIexpress Short Slot

8

PCIexpress Short Slot

Page 67


IOUA Internal DVD Drive

External Drive

Internal DAT Drive

@cabinet

Base IO Board

PCIe to PCI-X bridge

SAS GBit Ethernet

EXP

DVD DAT

> To IOU Internal HDD when intalled in PCI#0 or #4 > To DVD/DAT drive Via SAS exp when installed in PCI 0,2,4 or 6

Page 68


2-core Processor Memory Controller 2-core Processor 8 DDR2 DIMMs

Memory Controller 2-core Processor

8 DDR2 DIMMs

Memory Controller

PCIexpress slot PCIexpress slot PCIexpress slot PCIexpress slot PCIexpress slot PCIexpress slot PCIexpress slot PCIexpress slot

System Controller

8x8 Crossbar 8x8 Crossbar 8x8 Crossbar

~43 GBps

~48 GBps

I/O Board

8x8 Crossbar

System Controller

Memory Controller

XB Board

System Controller

System Controller

2-core Processor 8 DDR2 DIMMs

1.07 GHz

X8 X8 X8 X8 X8 X8 X8 X8

PHY (serdes)

Paths are 4 bytes each way and carry both address & data ~1/3 of traffic is data

PCIe x8 bridge

PHY (serdes) ~5.5 GBps

8 DDR2 DIMMs

0.53 Ghz ~40 GBps

~22 ~22 GBps GBps

System Board

24 Cables to other Xbars 0.5–1 GHz

~49 GBps

M8000/M9000 Boards

PHY (serdes) PHY (serdes)

PCIe x8 bridge Page 69


M9000 Block Diagram

Page 70


Quad-XSB Configuration System Board

I/O Board

2-core Processor 8 DDR2 DIMMs

Memory Controller

8 DDR2 DIMMs

Quad-XSB 00-0

PCIexpress slot

PCIe PHY (serdes)

Quad-XSB 00-1

PCIexpress slot

Memory Controller

PCIexpress slot

2-core Processor

PCIexpress slot

Memory Controller

PCIexpress slot

PCIe x8 bridge Quad-XSB 00-3

Memory Controller

PCIe PHY (serdes)

Quad-XSB 00-2

2-core Processor 8 DDR2 DIMMs

PCIexpress slot

PCIe x8 bridge 2-core Processor

8 DDR2 DIMMs

PCIe PHY (serdes)

PCIe PHY (serdes)

PCIexpress slot PCIexpress slot Page 71


M8000/M9000 Backplane Interconnect IOIO UUIO UIOU

M8000

184 GB/sec peak CC MM CMU UU

M9000 XB

XB

2 XB/way 4 ways

2 XB/way 4 ways

IO U IOIO U IOUU

C CM CMUM UU 368 GB/sec peak

XB 2 XB/way 4 ways

IO IOIO U IOUUU

C CM CMUM UU

Page 72


M8000/M9000 Backplane Interconnect IOIO UUIO UIOU

M8000

184 GB/sec peak CC MM CMU UU

M9000 XB

XB

2 XB/way 4 ways

2 XB/way 4 ways

737 GB/sec peak IOIO UUIO U IOU

CC MM CMU UU

IO U IOIO U IOUU

C CM CMUM UU 368 GB/sec peak

XB

XB

2 XB/way 4 ways

2 XB/way 4 ways

IO IOIO U IOUUU

C CM CMUM UU

M9000 Page 73


Operator Panel LED. Button, or Switch Name

Icon

Color

State/Setting Description

Power

Green

On Off

System power is on. System power is off.

Standby

Green

On Blinking

Indicates if the system can be powered on. XSCF on-Indicates that the system can be powered on. System initialization/power on processes in progress

Off

XSCF off-Indicates that the system cannot be powered on. (circuit breakers are off)

XSCF Check

Amber On Blinking

Off

POWER Button (recessed)

Less than 4 seconds 4seconds or longer (Service)

Indicates the operator panel is the maintenance target device. Indicates these conditions; ● Error detected. System startup disabled ● Power failure occurred Indicates the operating status of the system. Indicates one of three conditions; ● Normal operating state ● Circuit breakers are switched off ● Power failure occurred ● ●

Regardless of the MODE switch state, powers on all domains Skips processing for waiting for facility and power-on warm up.

Performs one of these functions; ● If power-on is being performed, the power-on processing is cancelled and the system is powered off. ● If the system is being powered off, the operator of the power switch is ignored. The power-off process continues until the system is powered off. Page 74


Operator Panel LED. Button, or Switch Name

MODE switch (1)

Icon

Color

State/Setting Description

Locked

● ● ● ●

MODE switch (2)

Service

● ● ● ●

Normal operator made. (recommended setting for day-to-day operators ) System can be powered on. System cannot be powered off. Key can be removed. Service should be provided. (switch must be set to service mode) Key cannot be removed. System can be powered on and off with the POWE R button. Prevents unintentional power-on.

Page 75


M9000 Cabling • • • • •

XB Data XB CLK CLK Data CLK CLK XSCF Data Total

: 48 cables : 8 cables : 2 cables : 4 cables : 2 cables : 64 cables

Page 76


M9000 Cabling

Page 77


M9000 Cabling: XB M9000 Base Cabinet

M9000 Ex. Cabinet

XBU #0 to #7

XBU #8 to #15

1.0m CLK Cable

Colored Tie Wraps Yellow Pink Brown Green Orange Blue

1.3m Data Cable

SPOF Page 78


M9000 Cabling: XB

M9000 Base Cabinet

M9000 Ex. Cabinet Page 79


M9000 Cabling: CLK M9000 Base Cabinet CLKU#0 out

out

in

M9000 Ex. Cabinet CLKU#2

CLKU#1 out

out

out

in

out

in

CLKU#3 out

out

in

CLK Cable Data Cable

SPOF Page 80


M9000 Cabling: XSCF M9000 Base Cabinet XSCFU_B#0

M9000 Ex. Cabinet XSCFU_C#0

XSCFU_B#1

XSCFU_C#1

Data Cable

Page 81


Airflow

Front

Back

M8000

M9000 Page 82


Dual Power Feed, 3-Phase Power for M8000

1 2 3

Descriptions

Remarks

Single phase DPF unit M8000 base cabinet + power cabinet for 3Phase (delta) DPF M8000 base cabinet + power cabinet for 3Phase (star) DPF

7U rack unit for DC1 base cabinet, field installable

Single phase input & Dual Power Feed

DPF Option 7U

Single Phase Input (standard) 3 Cables

M8000 Base cabinet + Power cabinet for 3-Phase(star)/ Dual Power Phase, 2 lines, Star(415V) is mainly used in EU region.

3-phase input (Delta) & Dual Power Feed

Base Cabinet

12U

M8000 Base cabinet + Power cabinet for 3-Phase(delta)/ Dual Power Feed Power, 2 lines, Delta (208V) is mainly used in Japan and North America.

Power Cabinet Single Phase Input (DPF Option)

Base Cabinet

3-phase input (Star) & Dual Power Feed Power Cabinet

12U 3 Phase (Delta) DPF

3 Phase Input (DPF)

Base Cabinet

12U 3 Phase (Star) DPF

3 Phase Input (DPF) Page 83


Dual Power Feed, 3-Phase Power for M9000, Single Cabinet 1 2 3

Descriptions

Remarks

power cabinet for single phase DPF M9000 base cabinet + power cabinet for 3Phase (delta) DPF M9000 base cabinet + power cabinet for 3Phase (star) DPF

Power cabinet, field installable

Single phase input & Dual Power Feed Power Cabinet

Base Cabinet

DPF Option

Single Phase Input (DPF Option)

M9000 base cabinet for 3-phase(delta)/ Dual Power Feed, 2 lines, Delta(208V) is mainly used in Japan and North America. M9000 base cabinet for 3-phase(star)/ dual Power Feed, 2 lines, Star(415V) is mainly used in EU region.

3-phase input & Dual Power Feed Power Cabinet

Base Cabinet

3 Phase (Delta or Star) DPF

Single Phase Input (Standard) 5 Cables

3 Phase Input (DPF) Page 84


Dual Power Feed, 3-Phase Power for M9000, Dual Cabinets 1 2 3

Descriptions

Remarks

Dual power cabinet for single phase DPF M9000 base + expansion cabinet, 3-Phase (delta), DPF M9000 base + expansion cabinet, 3-Phase (star), DPF

2x Power cabinet, field installable

Single phase input, Dual Power Feed Power Cabinet

Base Cabinet

Extension Cabinet

DPF Option

Single Phase Input (DPF Option)

Single Phase Input (Standard) 5 Cables

Single Phase Input (Standard)

M9000 Base cabinet + Power cabinet for 3-Phase(delta)/ Dual Power Feed Power, 4 lines, Delta(208V) is mainly used in Japan and North America. M9000 Base cabinet + Power cabinet for 3-Phase(star)/ Dual Power Phase, 4 lines, Star(415V) is mainly used in EU region.

3-phase input, Dual Power Feed Power Cabinet

Power Cabinet

DPF Option

3 Phase (Delta or Star) DPF

Single Phase 3 Phase Input Input (DPF) (DPF Option)

Base Cabinet

Extension Cabinet

Power Cabinet

3 Phase (Delta or Star) DPF

3 Phase Input (DPF) Page 85


Service Area: M8000

900 (35.4)

1260 (49.6)

800 (31.5)

M8000

Power Cabinet

800 (31.5)

M8000

Front

Front

750 (29.5)

1054 (41.5)

1260 (49.6)

900 (35.4)

Sun Rack

Front

850 (33.5)

Units in mm () in inches Page 86


Service Area: M9000 and M9000

900 (35.4)

M9000

M9000

Expansion Cabinet

M9000

Expansion Cabinet

Power Cabinet

M9000

Power Cabinet

1260 (49.6)

Power Cabinet

900 (35.4)

Front

Front

Front

Front

850 (33.5)

1154 (45.4)

1674 (65.9)

2282 Units in mm (90.0) () in inches Page 87


External IO Expansion Box

Page 88


External IO Expansion Unit • Common unit between mid-range and high-end servers • Rack mountable (4U) • All cabling rear accessible • Redundant and hot-swappable PSU’s and fans • Serial link connection between host using link cards > Copper cable (3m) and optical

fibre cable (up to 25m)

• Two IO boats with hot-pluggable PCI slots > 2 Boats with PCI-X slots > 2 Boats with PCIe slots > One boat with PCIe slots and one boat

with PCI-X slots

Page 89


I/O Box: FRUs and PCI Data Path NEC PCI-X to PCIe Bridge UPD720400

32 Lane PCIe Switch PLX PEX8532

Switch

Bridge

Bridge

Switch

Bridge

Switch

Switch 8 Lane PCIe

8 Lane PCIe Uplink

Link Slot Slot Slot Slot Slot Slot 0 1 2 3 4 5 6 6 PCI-X Slots

Link Slot Slot Slot Slot Slot Slot 0 1 2 3 4 5 6 6 PCIe Slots Page 90


What do you mean “bottleneck”? • 8 lane(8x) PCIe is approximately 4 times faster than what we support with 133Mhz 64 bit PCI-X cards (32 Gb -vs- 8 Gb) • Advantage to the PCIe architecture is that it is point to point, so issues around bus length are greatly reduced • Current cards are 2x and 4x Adapter Quad GbE 10 GbE JumboFrame 10 Gb iSCSI w/TOE 4/2/1 Gb FC DualPort 8/4/2 Gb FC DualPort IB 4x DualPort IB 8x (DDR) 40 GbE

Expected (Gb/s) 2.8 9 9 1 1.8 16 32 35

Theoretical (Gb/s) 4 20 20 1.6 3.2 20 40 80 Page 91


I/O Boat

Page 92


PCI Cassette for Boat in I/O Box In PCICS, PCIexpress Short dedicated cassette provided Moveable carrier plate (plastic) Carrier main body (metal) Carrier slot keyholes

Operating handle

Circular area for rotating the retainer

Card retainer (5 total, 3 types)

Page 93


Inserting and Removing a Carrier from the I/O Boat (1)Pulling the carrier handle raises PCI card out of PCI slot connector

(2)Pushing the carrier handle lowers PCI card into PCI slot connector

(3)Lever locks the handle in the extended position. Push the lever to the left to unlock the carrier handle.

Page 94


I/O Box: Link Card • Add-in card form factor • 8 lane PCIe path • Supports cable with sideband serial link • Does store and forward of I2C IO Box management traffic over sideband serial link • Bridge controller provides reliable transport protocol over sideband serial link • Supports an optional (non-standard) interrupt pin for host SP notifications

Page 95


Host to I/O Expansion Unit Link • Host to I/O Expansion Unit connectivity is obtained through either > (1) low height copper link

card kit > (2) full height fibre link card kit

• Host can have management control via side-band signals

Page 96


I/O Box: Interconnect Cable/Fibre • Copper cable - follows PCIexpress standard with sideband signals > Maximum length 5 meters

• Optical fibre - follows PCIexpress standard with sideband signals > Maximum length 25 meters

• Removal of cable will result in domain panic

Page 97


Copper Link Kit • Each half height PCIe based copper link card has 1 interconnect (i.e. cable plug) which carries data over a 5 meter distance > 8 lane PCIe (data) –

Bidirectional

> serial (mgmt)

Page 98


Fibre Link Kit • Each full height PCIe based fibre link card has: > 1 Transmit (TX) Port > 1 Receive (RX) Port

• Transceivers are/use: > 850nm wavelength > 50/125 MM Fibre > IEC 60825 & CDRH Class 1M laser eye safe

Page 99


Front View - Power Supplies (A195) • Two 600W 120-240V Hot Swap PS's > Redundant power & cooling (can run w/ 1 PS)

• Power supply and chassis status LED's Locate Fault Power Overtemp

Ready-to-remove Fault /Locate OK AC Power

PSU 0

PSU 1

600 Watts 120/240 VAC

600 Watts 120/240 VAC

Page 100


Rear View – I/O Expansion Boats • Two independent I/O “boats” Chassis Locator

> Mix/match either PCIe or PCI-X boat > Dual hosting is not supported (discussed later) Chassis Fault Chassis Power Chassis Overtemp

Boat Ok Boat Fault/Locate Ready to Remove

PCIe link To Host

Page 101


PCI cards for M8000 - M9000 • PCI Cards > Base I/O card can be populated into #0, #2, #4, #6, depending on hardware

implementation. > Link card can be populated into #1, #3, #5, #7, to avoid conflicting Base I/O card. IOU PCI-e Slot # #0

Base I/O Card Link Card Installable? Installable?

#2

Yes

#6 #7

IO boat x1

Yes

IO boat x1 Disk#2, #3, DVD(*)/DAT(*), 2xGB ethernet

Yes

#5

I n I / O Ca rd

DVD(*)/DAT(*), 2xGB ethernet

Yes

#3 #4

I n I OU Disk#0, #1, DVD(*)/DAT(*), 2xGB ethernet

Yes

#1

Av a ila ble res ources ( w he n ins ta lled B a s e I / O or L ink ca rd)

Yes

IO boat x1 DVD(*)/DAT(*), 2xGB ethernet

Yes Yes

IO boat x1 Page 102


IOUA and available resources in IOU • When populated into #0, #4: Disks, and DVD(*)/DAT(*) are available • When populated into #2, #6: DVD(*)/DAT(*) are available

(*) One DVD and one DAT (option) are installed per base cabinet and expansion cabinet(M9000). And, DVD or DAT drive are exclusively connected to one Base I/O card. Page 103


IO Box Options  Maximum Connections

M4000

M5000

M8000

M9000

M9000+Exp

Max IOU

1

2

4

8

16

Max I/O Boxes

2

4

8

16

32

Max I/O Boats

4

8

16

32

64

 IO Box Product List Descriptions 1

base unit, w. 2x PSUs, w 1 PCIe Boat

2

base unit, w. 2x PSUs, w 1 PCI-X Boat

3

PCI-X Boat

4

PCIe Boat

5

1x Copper Link card Kit (2 boards + metal cable) (FF only)

6

1x Optical Link card Kit (2 boards + 10M optical cable) (FF or DC)

7

1x Optical Link card Kit (2 boards + 25M optical cable) (FF or DC)

IO Box Configuration Rules ➢Mixing PCIe IO-Boat and PCI-X IO-Boat in an IO-Box is supported. ➢A Link option must be ordered per IO-Boat.

Page 104


Number of IO Expansion Units for M4000/M5000 IO-Box

• Concept

IOU

> Up to 2x IO Boxes per IOU

for M4000 & M5000

• Max. IO Boxes

PCI-X PCIe PCIe

Link Card Link Card

PCIe PCIe

Link Card Link Card

Link cards can be populated into any PCIex slots

Max I/O Boxes Max PCI Slots 2

25

1 IOU x 25 slots

M5000

4

50

2 IOU x 25 slots

Notes-1: As for 12 slots per IO-Box, either PCIe or PCI-X can be selected in 6 slots.

PCIe PCIe PCIe PCIe PCIe PCIe

PCIe PCIe PCIe PCIe PCIe PCIe

Remarks

M4000

PCIe PCIe PCIe PCIe PCIe PCIe

IO-Box

The number of PCI slots per 18 slots IOU : 25 slots IOU: 1 (PCI-X) IOU-Box: 2x 12 (see note-1)

PCIe PCIe PCIe PCIe PCIe PCIe Page 105


IO Box Configuration for M4000/M5000 M4000: Maximum of 2x IO Boxes (M4000: 6U, IO Box: 4U)

M5000: Maximum of 4x IO Boxes (M5000: 10U, IO Box: 4U)

19” Rack

19” Rack

IO-Box IO-Box

Link

IO-Box

IO-Box

IO-Box

IO-Box

M4000

M5000

IOU

IOU

IOU

#0

#0

#1

x2

Link

x2

x2

Notes: The physical mounting location of IO-Box is not fixed. Page 106


Number of IO Expansion Units for M8000/M9000 IO-Box

• Concept

IOU

> Up to 2x IO Boxes

per IOU

#0 #1

PCIe PCIe

Link Card

#2 #3

PCIe PCIe

Link Card

#4 #5

PCIe PCIe

Link Card

#6 #7

PCIe PCIe

Link Card

PCIe PCIe PCIe PCIe PCIe PCIe

PCIe PCIe PCIe PCIe PCIe PCIe

Max I/O Boxes

Max PCI Slots

Remarks

M8000

8

112

4 IOU x 28 slots

M9000

16

224

8 IOU x 28 slots

M9000+ Expansion

16

288

Limited to testing

Notes-1: As for 12 slots per IO-Box, either PCIe or PCI-X can be selected in 6 slots.

PCIe PCIe PCIe PCIe PCIe PCIe

IO-Box

PCIe PCIe PCIe PCIe PCIe PCIe Page 107


IO Box Configuration for M8000 & M9000 M8000: Maximum of 8x IO Boxes (IO Box: 4U) 19” Rack

M9000: Maximum of 16x IO Boxes (IO Box: 4U)

Base Cabinet

19” Rack

19” Rack

Base Cabinet

IO-Box

IO-Box

IO-Box

IO-Box

IO-Box

IO-Box

IO-Box

IO-Box

IO-Box

IO-Box

IO-Box

IO-Box

IOU IOU IOU IOU

IO-Box

IO-Box

IO-Box

#6

#4

#2

#0

IO-Box

IOU IOU IOU IOU

IO-Box

IO-Box

IOU IOU IOU IOU

IO-Box

#3

IO-Box

IO-Box

#7

IO-Box

IO-Box

#2

#1

#0

IO-Box Link

x2

x2

x2

x2

Rear View

Link

#5 x2

#3 x2

#1 x2

x2

Rear View

Notes: The physical mounting location of IO Box is not fixed. Page 108


IO Box Configuration for M9000 M9000: Maximum of 16x IO Boxes Configuration (IO-Box: 4U) 19” Rack

19” Rack

IO-Box

IO-Box

IO-Box

IO-Box

IO-Box

IO-Box

IO-Box

IO-Box

IOU

IOU

IOU

IOU

IOU

IOU

IOU

IOU

IO-Box

IO-Box

#14

#12

#10

#8

#6

#4

#2

#0

IO-Box

IO-Box

IO-Box

IO-Box

IO-Box

IO-Box

Base Cabinet

Expansion Cabinet

IOU

IOU

IOU

IOU

IOU

IOU

IOU

IOU

#15

#13

#11

#9

#7

#5

#3

#1

Link

Notes: The physical mounting location of IO Box is not fixed.

x1

x1

x1

x1

Link

x1

x1

x1

x1

Rear View Page 109


I/O Box Software OPL SCF Software

• IO Manager Software on SCF > Largely platform-independent, could be used on > > > >

OPL, Supernova or otherplatforms. Includes all the functionality required for IO Box management. Discovers IO Boxes and FRUs (part numbers, serial numbers, etc) when PCIe slots are powered on. Can collect environmental, voltage, status information from IO Box FRUs. Receives interrupts when the IO Box detects errors

Page 110


I/O Box Software OPL SCF Software

IO Box BUI

IO Box CLI Commands

Error Reporting

SunMC Agent

Module Wrapper IO Manager (libcharon) libsmbus Hardware Page 111


I/O Box CLI • The only IO Box CLI command on SCF is ioxadm Usage: ioxadm [-h] or: ioxadm [-fpv] list [TARGET] or: ioxadm [-fpv] env [-elt] [TARGET]... or: ioxadm [-fpv] reset TARGET or: ioxadm [-fpv] resetpci TARGET or: ioxadm [-fpv] poweroff TARGET or: ioxadm [-fpv] poweron TARGET or: ioxadm [-fpv] setled [-n|-f|-S|-F] TARGET LED or: ioxadm [-fpv] locator [-n|-f] TARGET

• TARGET is either a host pci slot location or a box/fru > PCI Slot: IOU#0-PCI#3 > FRU: iox@<serial id>[/fru] > iox@3B5A – The IO Box with serial id ending in 3B5A > iox@3B5A/ps1 – PSU #1 in IO Box > iox@3B5A/iob0/link – Link card in IO Boat #0 in IO Box Page 112


I/O Card Support: PCIe Part Number

Description

Bus

SEMY7BS1Z / SEMX7BS1Z SENY8LK2Z / SENX8LK2Z SENY8LK2 SENY8LK3Z SG-PCIE2FC-QF4 / SG-XPCIE2FC-QF4 SG-PCIE2FC-EM4 / SG-XPCIE2FC-EM4 SG-PCIE1FC-QF4 / SG-XPCIE1FC-QF4 SG-PCIE1FC-EM4 / SG-XPCIE1FC-EM4 7280A-2 / X7280A-2 7281A-2 / X7281A-2 6000A / X6000A SG-PCIE2SCSIU320Z / SG-XPCIE2SCSIU320Z SG-PCIE8SAS-E-Z / SG-XPCIE8SAS-E-Z 4447A-Z / X4447A-Z 1027A-Z / X1027A-Z 5558A / X5558A-Z 5560A-Z / X5560A-Z 3777 / X3777

Sun PCIe SE Base IO card Fiber interface card for External IO Expansion Unit Sun Optical Link card kit, 10m for Fiber Interface card Sun Optical Link card kit, 25m for Fiber Interface card Sun Sun PCIe 4GB FC Dual Port HBA Sun PCIe 4GB FC Dual Port HBA Sun PCIe 4GB FC Single Port HBA Sun PCIe 4GB FC Single Port HBA Sun PCIe Dual GigE UTP Sun PCIe Dual GigE MMF Sun PCIe Crypto Sun PCIe Dual Ultra320 SCSI HBA Sun PCIe SAS 8p HBA Sun PCIe Quad GigE Sun PCIe Dual 10 GigE Fiber Sun 10Gb SR Transceiver for 10GbE card Sun 10Gb LR Transceiver for 10GbE card Sun XVR-200 Graphics Card

PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe

Page 113


I/O Card Support: PCI-X Part Number

Description

Bus

4445A / X4445A SG-PCI2SCSI-LM320 / SG-XPCI2SCSI-LM320 SG-PCI2FC-QF4 / SG-XPCI2FC-QF SG-PCI2FC-EM4-Z / SG-XPCI2FC-EM4-Z SG-PCI1FC-QF4 / SG-XPCI1FC-QF4 SG-PCI1FC-EM4-Z / SG-XPCI1FC-EM4-Z SG-PCI8SAS-E-Z / SG-XPCI8SAS-E-Z 2156A / x2156A

Sun PCIx Quad GigaSwift Sun PCIx Dual Ultra320 SCSI HBA Sun PCIx 4GB FC Dual Port HBA Sun PCIx 4GB FC Dual Port HBA Sun PCIx 4GB FC Single Port HBA Sun PCIx 4GB FC Single Port HBA Sun PCIx SAS 8p HBA Sun Serial Asynchronous Interface (SAI/P) card

PCI-X PCI-X PCI-X PCI-X PCI-X PCI-X PCI-X PCI-X

Page 114


Storage Support • Disk Arrays and Storage Devices > Sun StorageTek SE D240, FLX280 (D280), 2510, 2530, 2540,

3120, 3320, 3510FC, 3511FC, 5210, 5310, 5320, 6120, 6140, 6540 (FLX380), 6920, 9910, 9960, 9970, 9980, 9985, 9990

> EMC Symmetrix DMX-3 product family

• Internal Boot Devices > Sun 73GB 10K RPM 2.5” SAS HDD

• External Boot Devices > Sun StorageTek 2510, 2530, 2540, 3120, 3510FC, 9980, 9985

Page 115


Tape Support â&#x20AC;˘ Tapes and Libraries > Libraries: > Sun StorageTek SL8500 Modular Library System, SL500, C4 Tape Library, L1400M libraries, L180/700e, LTO Rackmount, DAT72 Rackmount, SDLT Rackmount, LTO Desktop, DAT72 Desktop (both SCSI and USB),SDLT 600 Desktop, SDLT 320 Desktop, Sun StorEdge C2 autoloader > Legacy StorageTek L20/40/80

> Tape Drives: > HP LTO Gen 2,3,4 FC/SCSI , IBM LTO 2,3,4 FC/SCSI, 9840 B/C FC, T10000 2/4Gb FC, SDLT 600 FC/SCSI, DLTS4 FC/SCSI, DAT72, SDLT320 SCSI

Page 116


Presenterâ&#x20AC;&#x2122;s Name first.last @sun.com

Sun Servers  

Enterprise Severs

Read more
Read more
Similar to
Popular now
Just for you