Networking :: Multicast UDP Packets Lost In Kernel

We have a high speed udp multicast connection. There are about 37 multicast groups and average incoming rate via the connection is about 20000 pkt/sec and peak 40000+ pkt/sec. Each packet is about 100-300 bytes. And there are 6 busiest multicast groups generating 80% of the traffic. No significant outgoing traffic via that interface. The problem is that, if our heavy-weight application create 37 sockets for all the groups, the packet loss become quite frequent. but if we create the 6 sockets for the busiest groups, the packet loss drops to 1/10 of the original level. We can confirm that the lost packets DO arrive in our box because a light-weight recorder on another box on the same ethernet segment can see the lost packets when listening to the same groups. If we start another heavy-weight application for the rest groups (31 groups ) on the same box, the packet loss just come back to the original level.

The packet loss happens not only to the heavy-weight application but also the light-weight recorder running on the same system. that is, the loss is system wide. The socket kernel buffer are all 4MB max and did not observe system buffer overflow. It seems the number of sockets listening to the same multicast group have the most negative impact on packet loss. And the higher traffic have a far less impact on packet loss than the number of sockets. It also seems that there is some limit in the kernel that when socket number are reduced under, the packet loss can significantly ease. How can I further diagnose the problem? Our system is RHEL 5.3 32bit i386, Xeon 2.9GHZ 16 Processors, 32GB memory. Two broadcom giga netowrk card and 6 Intel GIGA NIC, and only one interface card has so much traffic.