Have you quantified your network parameters and set your buffers using the tuning procedure?
Where is your asterisk/allstar server, in relation to your MASTER site? Per the votersystem.pdf:
There must be a VOTER (RTCM) board on the same LAN (very low latency) as the Asterisk server
implementing the “master site”, which acts as the Master Timing source. This allows chan_voter to
have a consistent, reliable, accurate timing source with which the timing information from all other
inbound packets are compared and appropriately processed, and from which to generate accurate
timing information for time-consistent transmission purposes.