I have long complained about the uptime of App_Rpt. I've identified two repeatable problems.

First one is if a node is connected more than about 650 hours or so there is a buffer overflow. I notice this most prominently with my RTCMs that are all on the same server connected to a hub on the same server. Allmon will show an negative number of connected hours followed shortly by a crash. Apparently this not just an RTCM problem, it just shows up there more readily because the nodes on the same server and aren't disconnecting due to network issues.

The second problem can be recreated in two ways. Set up a bunch of nodes to repeatedly connect and disconnect to a hub with some simple bash scripting. After a random time there will be a crash. The second way is to crank in simulated packet loss with the IAX2 test command. It takes more time and patients for it to crash under this test. I believe both these tests demonstrate the same problem. There is something flaky in the area of IAX links within App_Rpt or Asterisk

The Ham VoIP guys say they have really improved the uptime of their distro. I think that's great but I find it terribly upsetting that the DIAL folks don't get these fixes.

On Wed, Nov 15, 2017 at 8:36 AM, Peter <g7rpg@hotmail.com> wrote:

Hi...

I'm one the minions that help run M0HOY's Allstar hubs in the UK. Node
41522 & 41223, There is an ongoing problem where the hub will crash
randomly when a someone comes on with a flaky internet connection, the
crash is associated with segfault in app_rpt.so and and usually one of
the following in the logs:

Undecodable frame received from 'xxx.xxx.xxx.xxx'
PBX may not have been terminated properly on
'IAX2/xxx.xxx.xxx.xxx:4572-5435'
lots of.... Max retries exceeded to host xxx.xxx.xxx.xxx

Sometimes the crash is a 'general protection ip' in app_rpt.so but
usually nothing in the logs just before to give any clues.

Average uptime is 7 - 14 days, a flaky incoming connection will kill it
within minutes dropping all the connections.

Both of the hubs are hosted in a DC on VPS running on Debian 7.4, 3.2.81
X64, built with source from the github repo.

There tends to be on average 60 - 80 ASL connections spread across the
two hubs to share the load.

Recently we've had a few users on the hub form the US and Australia
repeatedly crashing the hubs sometimes more than once a day.

Some of the ideas we've done/are looking at are:

More hubs to spread the load and minimise the disruption
A script to keep a check on the running process and restart asterisk QUICKLY
A scripts to alert us of potential problems
On some occasions have had to use iptables to block the host causing the
problem were we've not been able to make contact.

I wonder if anyone has any suggestions?

TIA

Peter
G7RPG

_______________________________________________
App_rpt-users mailing list
App_rpt-users@lists.allstarlink.org
http://lists.allstarlink.org/cgi-bin/mailman/listinfo/app_rpt-users

To unsubscribe from this list please visit http://lists.allstarlink.org/cgi-bin/mailman/listinfo/app_rpt-users and scroll down to the bottom of the page. Enter your email address and press the "Unsubscribe or edit options button"
You do not need a password to unsubscribe, you can do it via email confirmation. If you have trouble unsubscribing, please send a message to the list detailing the problem.

Tim