Thanks to you both Lee and Jesse.

Yes I've followed the buffer setting as per the link you sent Lee. I've slowly increased the values though as I've run into this issue.

The asterisk server is located on the same router as the Master RTCM (latency <2ms).

Just today another ham mentioned to me that having the RTCM set on auto negotiation can cause ethernet issues... so I've changed that to negotiate only at 10Mbps 1/2 duplex on the Master site.