AllStarLink Maintenance followup
Our maintenance was completed successfully. AllStar Link is now running a geographically redundant registration cluster! This removes any single failure point from core AllStar services; a major advance in network reliability. Should any node in this cluster go offline, it will be removed and registration will fail over to another server. Most users have registered on the new system without issue. Some older installs may need to restart asterisk or their server to pickup the new IP address of the servers. After some late night debugging, the phone portal is working too. Should you still have issues, please drop a reply on list or to helpdesk@allstarlink.org and the admin team will look into it asap. We'd also like to announce a system services front end: https://grafana.allstarlink.org/ This is a public view of the core systems we run and their real time status. We'll post a more detailed update later today, however now we all need to get some sleep. 73's The AllStarLink Operations Team
Bryan, I know it has been a long night for you all. But, my logs are filling with registration errors--see below. These errors are going to make it impossible to troubleshoot registration related problems. Any ideas on solving this?? Thanks, 73, David KB4FXC Asterisk CLI output: ---------------------------------------------------------------------------------------------- [Aug 12 05:28:15] NOTICE[20463]: dnsmgr.c:178 dnsmgr_refresh: host 'register.allstarlink.org' changed from 44.72.21.13 to 44.103.0.49 [Aug 12 05:28:15] NOTICE[20463]: dnsmgr.c:178 dnsmgr_refresh: host 'register.allstarlink.org' changed from 44.98.254.145 to 44.72.21.13 [Aug 12 05:28:15] NOTICE[20463]: dnsmgr.c:178 dnsmgr_refresh: host 'register.allstarlink.org' changed from 44.72.21.13 to 44.103.0.49 [Aug 12 05:28:15] NOTICE[20463]: dnsmgr.c:178 dnsmgr_refresh: host 'register.allstarlink.org' changed from 44.72.21.13 to 44.103.0.49 [Aug 12 05:28:15] NOTICE[20463]: dnsmgr.c:178 dnsmgr_refresh: host 'register.allstarlink.org' changed from 44.98.254.145 to 44.72.21.13 [Aug 12 05:28:19] WARNING[20472]: chan_iax2.c:7395 iax2_ack_registry: Received unsolicited registry ack from '44.98.254.145' [Aug 12 05:28:19] WARNING[20472]: chan_iax2.c:9820 socket_process: Registration failure [Aug 12 05:28:19] WARNING[20471]: chan_iax2.c:7395 iax2_ack_registry: Received unsolicited registry ack from '44.72.21.13' [Aug 12 05:28:19] WARNING[20471]: chan_iax2.c:9820 socket_process: Registration failure [Aug 12 05:29:13] NOTICE[20472]: chan_iax2.c:9828 socket_process: Registration of '2290' rejected: '<unknown>' from: '44.103.0.49' [Aug 12 05:29:51] NOTICE[20470]: dnsmgr.c:178 dnsmgr_refresh: host 'register.allstarlink.org' changed from 44.103.0.49 to 44.98.254.145 -- Registered IAX2 to '44.98.254.145', who sees us as 12.17.28.14:4569 with no messages waiting [Aug 12 05:30:27] NOTICE[20470]: dnsmgr.c:178 dnsmgr_refresh: host 'register.allstarlink.org' changed from 44.103.0.49 to 44.72.21.13 -- Registered IAX2 to '44.72.21.13', who sees us as 12.17.28.14:4569 with no messages waiting [Aug 12 05:31:10] NOTICE[20471]: dnsmgr.c:178 dnsmgr_refresh: host 'register.allstarlink.org' changed from 44.72.21.13 to 44.98.254.145 [Aug 12 05:31:10] NOTICE[20472]: chan_iax2.c:9828 socket_process: Registration of '2289' rejected: '<unknown>' from: '44.98.254.145' [Aug 12 05:31:33] NOTICE[20472]: chan_iax2.c:9828 socket_process: Registration of '2290' rejected: '<unknown>' from: '44.72.21.13' [Aug 12 05:31:34] NOTICE[20471]: dnsmgr.c:178 dnsmgr_refresh: host 'register.allstarlink.org' changed from 44.103.0.49 to 44.72.21.13 [Aug 12 05:31:34] NOTICE[20472]: chan_iax2.c:9828 socket_process: Registration of '2286' rejected: '<unknown>' from: '44.72.21.13' Allstar-inttek*CLI> iax2 show registry Host dnsmgr Username Perceived Refresh State 44.72.21.13:4569 Y 2290 12.17.28.14:4569 60 Rejected 44.98.254.145:4569 Y 2289 <Unregistered> 60 Rejected 44.98.254.145:4569 Y 2288 12.17.28.14:4569 60 Registered 44.98.254.145:4569 Y 2287 12.17.28.14:4569 60 Registered 44.72.21.13:4569 Y 2286 <Unregistered> 60 Rejected 44.72.21.13:4569 Y 2285 12.17.28.14:4569 60 Registered On Sun, 12 Aug 2018, Bryan Fields wrote:
Our maintenance was completed successfully.
AllStar Link is now running a geographically redundant registration cluster!
This removes any single failure point from core AllStar services; a major advance in network reliability. Should any node in this cluster go offline, it will be removed and registration will fail over to another server.
Most users have registered on the new system without issue. Some older installs may need to restart asterisk or their server to pickup the new IP address of the servers. After some late night debugging, the phone portal is working too.
Should you still have issues, please drop a reply on list or to helpdesk@allstarlink.org and the admin team will look into it asap.
We'd also like to announce a system services front end: https://grafana.allstarlink.org/ This is a public view of the core systems we run and their real time status.
We'll post a more detailed update later today, however now we all need to get some sleep.
73's
The AllStarLink Operations Team _______________________________________________ App_rpt-users mailing list App_rpt-users@lists.allstarlink.org http://lists.allstarlink.org/cgi-bin/mailman/listinfo/app_rpt-users
To unsubscribe from this list please visit http://lists.allstarlink.org/cgi-bin/mailman/listinfo/app_rpt-users and scroll down to the bottom of the page. Enter your email address and press the "Unsubscribe or edit options button" You do not need a password to unsubscribe, you can do it via email confirmation. If you have trouble unsubscribing, please send a message to the list detailing the problem.
On 8/12/18 5:36 AM, David McGough wrote:
Bryan,
I know it has been a long night for you all. But, my logs are filling with registration errors--see below. These errors are going to make it impossible to troubleshoot registration related problems.
I've been busy cleaning up for the evening here. The team has been hammering this system with over 15k node registrations for the last couple months. Our use of IAX2 on the registration server is quite limited, it's there to get the IAX port (as it appears), and the port from authenticated hosts. This gets put in a database with a time stamp. The system considers someone registered if they have a timestamp in the last 600 seconds, it's not based on IAX status. So long as you hit one registration server and register to it in a 10 minute interval, you're in the nodes file. With dnsmgr running, IAX2 will tear down reestablish a connection every time DNS changes. This has the effect of spreading load out on the servers, and giving us redundancy. Should a server go down it will stop updating the timestamp on DNS entries in the database and that IP will time out of the DNS. The connections between nodes are built using IAX2 between nodes, so long as they have a copy of the nodes file they will connect up. These connections don't hit the registration server, and only depend on it insofar as getting the updated nodes list. You'll note the nodes list now is updated much faster, and in near real time from the database. This is a huge improvement in the 5+ min delay on the prior nodes list. For checking if your registered from the perspective of the servers use https://allstar.aargdmr.net/nodelist/ or the main allstarlink.org portal and check you last registration time and if you're green. You may also pull the extnodes file directly from http://nodes.regsvcs.allstarlink.org/cgi-bin/nodes.pl if you're on a registered node. The "errors" you are seeing in the logs are just the notice of the IP address resolving for register.allstarlink.org. Some of the rejections we due to the loading, when 2500 nodes hit the server all at once, it can take a bit of time to get registered. We're in steady state now and registration is stable. Example:
Itchy*CLI> iax2 show registry Host dnsmgr Username Perceived Refresh State 44.72.21.13:4569 Y 44233 44.98.249.6:4569 60 Registered 44.72.21.13:4569 Y 42032 44.98.249.6:4569 60 Registered 44.98.254.145:4569 Y 41618 44.98.249.6:4569 60 Registered 44.98.254.145:4569 Y 40821 44.98.249.6:4569 60 Registered
The time/cpu it takes to build a new connection is the same as refreshing an existing connection. Most people will not notice anything, and hopefully if we have a server fail, they will continue not to notice anything. I'm off to bed now. -- Bryan Fields 727-409-1194 - Voice http://bryanfields.net
participants (3)
-
ARS W5OMR -
Bryan Fields -
David McGough