Hi Loren If ezstream completely stops, for any reason, asterisk will restart it on the very next call. So killing ezstream fixes the problem without impacting repeater operation. It is not necessary to kill asterisk for the “lame problem”. I f you are having to kill asterisk (for some reason) then something different is going on. If you find one of the CPU’s is in a 100% wait state, you can run this little script to see what process it is waiting on for x in `seq 1 1 10`; do ps -eo state,pid,cmd | grep "^D"; echo "----"; sleep 5; done Regards Ken Also see http://bencane.com/2012/08/06/troubleshooting-high-io-wait-in-linux/ From: Loren Tedford [mailto:lorentedford@gmail.com] Sent: Wednesday, May 04, 2016 11:25 PM To: Ken <ke2n@cs.com> Cc: app_rpt mailing list <app_rpt-users@ohnosec.org> Subject: Re: [App_rpt-users] ezstream 100% cpu I restarted asterisk with this command here.. So far we are backup and running.. root@server:/etc/asterisk# cat restart.sh #!/bin/bash kill -9 $(pgrep ezstream)& kill -9 pgrep ezstream service asterisk stop sleep 1s killall asterisk kill -9 $(pgrep ezstream)& service asterisk stop sleep 1s service asterisk start root@server:/etc/asterisk# Below is my server at idle I do have other projects going on with the server but I don't believe any of them would have affected ezstream. http://kc9zhv.com/wp-content/uploads/2016/05/2016-05-04-1.png <http://kc9zhv.com/wp-content/uploads/2016/05/2016-05-04-1.png> Loren Tedford (KC9ZHV) Email: <mailto:lorentedford@gmail.com> lorentedford@gmail.com <http://www.lorentedford.com/> http://www.lorentedford.com http://www.kc9zhv.com http://forum.kc9zhv.com http://hub.kc9zhv.com http://www.newwavesucks.com http://forum.newwavesucks.com On Wed, May 4, 2016 at 10:07 PM, Loren Tedford <lorentedford@gmail.com <mailto:lorentedford@gmail.com> > wrote: I don't think i am running out of disk space but looking now to check logs It did run the system log at least thats what i seem to see.. May 4 21:20:01 server CRON[22247]: (root) CMD (/usr/local/sbin/./check_stream) May 4 21:30:01 server CRON[25488]: (root) CMD (/usr/local/sbin/./check_stream) May 4 21:40:01 server CRON[29164]: (root) CMD (/usr/local/sbin/./check_stream) May 4 21:50:01 server CRON[9769]: (root) CMD (/usr/local/sbin/./check_stream) May 4 22:00:01 server CRON[12927]: (root) CMD (/usr/local/sbin/./check_stream) Cron job seems to be running correctly and things seem to be pointed at the script not sure what i am missing.. root@server:/etc/asterisk# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 26 Stepping: 5 CPU MHz: 2800.000 BogoMIPS: 5600.43 Virtualisation: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 8192K NUMA node0 CPU(s): 0-7 root@server:/etc/asterisk# df -h Filesystem Size Used Avail Use% Mounted on udev 7.9G 4.0K 7.9G 1% /dev tmpfs 1.6G 1.2M 1.6G 1% /run /dev/md2 77G 11G 63G 15% / none 4.0K 0 4.0K 0% /sys/fs/cgroup none 5.0M 0 5.0M 0% /run/lock none 7.9G 4.1M 7.9G 1% /run/shm none 100M 16K 100M 1% /run/user /dev/md3 1.8T 208G 1.5T 13% /home Loren Tedford (KC9ZHV) Email: <mailto:lorentedford@gmail.com> lorentedford@gmail.com <http://www.lorentedford.com/> http://www.lorentedford.com http://www.kc9zhv.com http://forum.kc9zhv.com http://hub.kc9zhv.com http://www.newwavesucks.com http://forum.newwavesucks.com On Wed, May 4, 2016 at 10:00 PM, Ken <ke2n@cs.com <mailto:ke2n@cs.com> > wrote: When the script runs you will find an indication in /var/log/cron Lines like this: May 2 20:50:01 localhost crond[30559]: (root) CMD (/usr/local/sbin/./check_stream) One thing that can cause a Linux system to grind to a near-halt is if you have run out of disk space …. But I see you are running VBoxHeadless – I have no familiarity but a quick Google of that finds something related https://forums.virtualbox.org/viewtopic.php?f=8 <https://forums.virtualbox.org/viewtopic.php?f=8&t=68525> &t=68525 Ken From: Loren Tedford [mailto:lorentedford@gmail.com <mailto:lorentedford@gmail.com> ] Sent: Wednesday, May 04, 2016 10:42 PM To: Brent Weatherall <va3bfw@gmail.com <mailto:va3bfw@gmail.com> > Cc: Ken <ke2n@cs.com <mailto:ke2n@cs.com> >; app_rpt mailing list <app_rpt-users@ohnosec.org <mailto:app_rpt-users@ohnosec.org> > Subject: Re: [App_rpt-users] ezstream 100% cpu Well my server is at it again 100% cpu usage for no reason kinda odd.. I did try Ken's script however I am not sure that its working the way it should... I did place the info in crontab -e I wonder if we are missing some sort of dependency that isn't getting installed.. Maybe their is an easier way to broadcast to Broadcastify.. Here is an image of what it looks like this evening.. http://kc9zhv.com/wp-content/uploads/2016/05/2016-05-04.png <http://kc9zhv.com/wp-content/uploads/2016/05/2016-05-04.png> Loren Tedford (KC9ZHV) Email: <mailto:lorentedford@gmail.com> lorentedford@gmail.com <http://www.lorentedford.com/> http://www.lorentedford.com http://www.kc9zhv.com http://forum.kc9zhv.com http://hub.kc9zhv.com http://www.newwavesucks.com http://forum.newwavesucks.com On Mon, May 2, 2016 at 4:46 PM, Brent Weatherall <va3bfw@gmail.com <mailto:va3bfw@gmail.com> > wrote: Ken, definitely confirmed on my last CPU spiking that lame's process was no longer running. I'll give your monitoring/restart script a try. Thanks for the info! On Mon, May 2, 2016 at 11:26 AM Ken <ke2n@cs.com <mailto:ke2n@cs.com> > wrote: Of course the 100% CPU is probably due to it waiting for some resource, rather than actually being loaded to 100%. Press “1” in top and look for the “%wa” I found (like some others) that the problem is actually the “lame” program which vanishes for some reason. I run a script every 10 minutes that checks the pid for lame and restarts ezstream – but only if needed. I think that is better than killing ezstream when it is, in fact, running fine. This is the script (it is not my invention) #!/bin/bash lamenumber=$(/sbin/pidof lame) if [ "$lamenumber" = "" ] then eznumber=$(/sbin/pidof ezstream) echo "$eznumber" kill -9 "$eznumber" echo "restarting" date else eznumber="" fi Regards Ken From: Brent Weatherall [mailto:va3bfw@gmail.com <mailto:va3bfw@gmail.com> ] Sent: Monday, May 02, 2016 11:09 AM To: Loren Tedford <lorentedford@gmail.com <mailto:lorentedford@gmail.com> > Cc: app_rpt mailing list <app_rpt-users@ohnosec.org <mailto:app_rpt-users@ohnosec.org> > Subject: Re: [App_rpt-users] ezstream 100% cpu A quick look at that cron schedule looks like it would run every minute of the 0 hour and 12 hour. Did you just want it to run only once at 0 and 12? If that is the case you'd want: 0 */12 * * *. I'll let you know how it works out for stability. Thanks again. On Mon, May 2, 2016 at 11:03 AM Loren Tedford <lorentedford@gmail.com <mailto:lorentedford@gmail.com> > wrote: The biggest issue with ezstream is that for some reason you get multiple instances of the program running.. Not sure exactly what would be the best solution to actually fix this but the cron script is a big bandage with duck tape.. I really wish I knew how to write code.. Loren Tedford (KC9ZHV) Email: lorentedford@gmail.com <mailto:lorentedford@gmail.com> Phone: 618-553-0806 <tel:618-553-0806> Fax: 16185512755 <tel:16185512755> http://www.lorentedford.com http://kc9zhv.com Sent from Droid Turbo from Verizon wireless network On May 2, 2016 10:00 AM, "Brent Weatherall" <va3bfw@gmail.com <mailto:va3bfw@gmail.com> > wrote: Thanks for the cron script to try Loren. Hopefully the daily restart of ezstream will cure any issues. On Mon, Apr 25, 2016 at 3:57 PM Loren Tedford <lorentedford@gmail.com <mailto:lorentedford@gmail.com> > wrote: Here is what i did to get ezstream working I will be updating this forum as well as to my fix for ezstream locking open randomly basicly it kinda goes like this.. http://forum.kc9zhv.com/index.php/topic,23.0.html crontab -e * */12 * * * sh /etc/asterisk/stopez.sh script that i use to reset ezstream.. root@server:/etc/asterisk# cat stopez.sh #!/bin/bash kill -9 $(pgrep ezstream)& sleep 2s kill -9 pgrep ezstream sleep 2s /usr/sbin/asterisk -rx "module reload" now to restart asterisk i run this customized script. root@server:/etc/asterisk# cat restart.sh #!/bin/bash kill -9 $(pgrep ezstream)& kill -9 pgrep ezstream service asterisk stop sleep 1s killall asterisk kill -9 $(pgrep ezstream)& service asterisk stop sleep 1s service asterisk start So far I have had no issues with ezstream since i did this.. Loren Tedford (KC9ZHV) Email: <mailto:lorentedford@gmail.com> lorentedford@gmail.com <http://www.lorentedford.com/> http://www.lorentedford.com http://www.kc9zhv.com http://forum.kc9zhv.com http://hub.kc9zhv.com http://www.newwavesucks.com http://forum.newwavesucks.com On Mon, Apr 25, 2016 at 12:10 PM, Brent Weatherall <va3bfw@gmail.com <mailto:va3bfw@gmail.com> > wrote: Hello, I've recently setup an allstar hub node and decided to stream it via broadcastify. Setup has gone well - I'm running debian, with the allstar node running without issue, as well as streaming now being delivered via Broadcastify. I've ran in to an issue where the ezstream process will occasionally pin at 100%. I can kill it fine, and asterisk gracefully resumes by creating a new working output stream. Has anyone else encountered this cpu pinned usage? Will I have to set up something to monitor the process to kill it, so it restarts? Thanks in advance for any suggestions VA3BFW - Brent _______________________________________________ App_rpt-users mailing list App_rpt-users@ohnosec.org <mailto:App_rpt-users@ohnosec.org> http://ohnosec.org/cgi-bin/mailman/listinfo/app_rpt-users To unsubscribe from this list please visit http://ohnosec.org/cgi-bin/mailman/listinfo/app_rpt-users and scroll down to the bottom of the page. Enter your email address and press the "Unsubscribe or edit options button" You do not need a password to unsubscribe, you can do it via email confirmation. If you have trouble unsubscribing, please send a message to the list detailing the problem.