[App_rpt-users] hub crash help

[Peter]

Hi...

I'm one the minions that help run M0HOY's Allstar hubs in the UK. Node 
41522 & 41223, There is an ongoing problem where the hub will crash 
randomly when a someone comes on with a flaky internet connection, the 
crash is associated with segfault in app_rpt.so and and usually one of 
the following in the logs:


Undecodable frame received from 'xxx.xxx.xxx.xxx'
PBX may not have been terminated properly on 
'IAX2/xxx.xxx.xxx.xxx:4572-5435'
lots of.... Max retries exceeded to host xxx.xxx.xxx.xxx


Sometimes the crash is a 'general protection ip' in app_rpt.so but 
usually nothing in the logs just before to give any clues.

Average uptime is 7 - 14 days, a flaky incoming connection will kill it 
within minutes dropping all the connections.

Both of the hubs are hosted in a DC on VPS running on Debian 7.4, 3.2.81 
X64, built with source from the github repo.

There tends to be on average 60 - 80 ASL connections spread across the 
two hubs to share the load.

Recently we've had a few users on the hub form the US and Australia 
repeatedly crashing the hubs sometimes more than once a day.


Some of the ideas we've done/are looking at are:

More hubs to spread the load and minimise the disruption
A script to keep a check on the running process and restart asterisk QUICKLY
A scripts to alert us of potential problems
On some occasions have had to use iptables to block the host causing the 
problem were we've not been able to make contact.


I wonder if anyone has any suggestions?

TIA

Peter
G7RPG