Fedora Security :: Setup IPSEC Encryption Between Two Boxes?

I have been trying to setup IPSEC encryption between two linux boxes. I have a server application which runs on Linux Box A and a client application which runs on Linux Box B. The client sends the data to server. I have captured wireshark logs at both server and client end. In the wireshark logs I can see that the Box B send ESP packets to the Box A.

But the server Application running at Box A is is not able to get any packets. If I turn the policy off at Box B, Box B sends normal UDP data packets to Box A, but still the Server Application running at box A doesn't get any packets.( Expected behavior since policy at Box A enforces that all packets coming from Box B should be encrypted.)

If I turn the policy off at Box A and Box B both, the server application receives the unencrypted data which is also expected behavior. But when the policy is turned on at both the boxes the encrypted packets reach the Box A but are not delivered to the server application. If anyone has faced such issue please help me to debug this issue. I have attached the ifconfig and policy settings at Box A and Box B for your reference.