CentOS 5 Hardware :: Infiniband (mlx4) Works On 16-core System But (now) Fails On 32-core System

We have a small cluster of 20 HP systems, all running CentOS 5.3 in an NFS-root environment. Half are quad-socket, quad-core Xeon E7340 @ 2.40GHz (total 16 cores), the other half are 8-socket, quad-core Opteron 8354 (total 32 cores). All systems have a Mellanox Infiniband adapter ("Mellanox Technologies MT25418 [ConnectX VPI PCIe 2.0 2.5GT/s - IB DDR / 10GigE] (rev a0)")

With kernel 2.6.18-128.1.6.el5, infiniband works fine on all systems.

With the update to kernel 2.6.18-164.11.1.el5 (and both types of node running the same NFS-root image), the 16-core Xeons still work fine. Infiniband no longer works on the 32-core Opterons. Specifically, either the ib0 interface fails to appear, or it does appear but when configured with an IP address, doesn't actually work. In either case, loading the IB kernel modules takes a long time, but I haven't instrumented the load script yet to see which module, if any, is at fault. More errors listed below.

However, if I tweak the BIOS of the 32-core systems to reduce the per-socket core count to 2 (so effectively 8-socket, dual-core, down to a total of 16 available cores), Infiniband starts working again. Putting it back to 32-cores makes it fail. Booting the older kernel makes it work again. In summary: old kernel, IB works on all systems. Newer kernel, IB only works on 16-core systems.

Updating the IB firmware from 2.5.0 to 2.7.0 (latest available) doesn't help. I also did a full 'yum update' to make sure that libmlx4, openibd all other associated packages were up-to-date. Doesn't help either.

Some errors that appear on 32-core nodes:

ib_query_port failed (-16) for mlx4_0
ib_query_port failed (-16) for mlx4_0
mlx4_core 0000:04:00.0: SW2HW_MPT failed (-16)
mlx4_core 0000:04:00.0: SW2HW_MPT failed (-16)

[Code]....