I'm running a Pi 3 too.

Using http://vklink.com.au/ as the image for local use in VK.

I logged in shortly after I had a QSO on the repeater and the load average was 0.00, 0.01, 0.00. I also noticed while having another conversation on the repeater this morning, the problem started after 5-10 minutes or so, up to almost 7 syllables. Then the other user went clear for 5 mins, then came back and we continued the conversation with no delay for a further 10 minutes. 
I've disabled the voter record function in the channel driver for now, just in case that is the issue.

I guess I could try another RTCM/Voter unit sitting at the Master site with no radio attached at all purely for timing.  

Sam - what did you change your buflen too?
My current config - 

buflen = 140 in voter.conf
Tx Buffer Length in RTCM's = 1120

Ping to site 2 (slowest site) - Packets tx: 100, rx: 100, oos: 0, Avg.: 33.720 ms
 Worst: 53 ms, Best: 32 ms, 100.0% Packets successfully received (0.0% loss)

Ping to site 1 (master site) - Packets tx: 100, rx: 100, oos: 0, Avg.: 3.450 ms
Worst: 4 ms, Best: 3 ms, 100.0% Packets successfully received (0.0% loss)

Regards
Hayden VK7HH