[App_rpt-users] ezstream 100% cpu

Thu May 5 14:49:35 UTC 2016

Hi Loren

If ezstream completely stops, for any reason, asterisk will restart it on the very next call. 

So killing ezstream fixes the problem without impacting repeater operation.

It is not necessary to kill asterisk for the “lame problem”.  I

f you are having to kill asterisk (for some reason) then something different is going on. 

If you find one of the CPU’s is in a 100% wait state, you can run this little script to see what process it is waiting on

for x in `seq 1 1 10`; do ps -eo state,pid,cmd | grep "^D"; echo "----"; sleep 5; done

Regards

Ken

Also see

http://bencane.com/2012/08/06/troubleshooting-high-io-wait-in-linux/ 

From: Loren Tedford [mailto:lorentedford at gmail.com] 
Sent: Wednesday, May 04, 2016 11:25 PM
To: Ken <ke2n at cs.com>
Cc: app_rpt mailing list <app_rpt-users at ohnosec.org>
Subject: Re: [App_rpt-users] ezstream 100% cpu

I restarted asterisk with this command here.. So far we are backup and running..

root at server:/etc/asterisk# cat restart.sh

#!/bin/bash

kill -9 $(pgrep ezstream)&

kill -9 pgrep ezstream

service asterisk stop

sleep 1s

killall asterisk

kill -9 $(pgrep ezstream)&

service asterisk stop

sleep 1s

service asterisk start

root at server:/etc/asterisk#

Below is my server at idle I do have other projects going on with the server but I don't believe any of them would have affected ezstream.

http://kc9zhv.com/wp-content/uploads/2016/05/2016-05-04-1.png

  <http://kc9zhv.com/wp-content/uploads/2016/05/2016-05-04-1.png> 

Loren Tedford (KC9ZHV) 
Email:  <mailto:lorentedford at gmail.com> lorentedford at gmail.com

 <http://www.lorentedford.com/> http://www.lorentedford.com

http://www.kc9zhv.com

http://forum.kc9zhv.com

http://hub.kc9zhv.com

http://www.newwavesucks.com

http://forum.newwavesucks.com

On Wed, May 4, 2016 at 10:07 PM, Loren Tedford <lorentedford at gmail.com <mailto:lorentedford at gmail.com> > wrote:

I don't think i am running out of disk space but looking now to check logs

It did run the system log at least thats what i seem to see..

May  4 21:20:01 server CRON[22247]: (root) CMD (/usr/local/sbin/./check_stream)

May  4 21:30:01 server CRON[25488]: (root) CMD (/usr/local/sbin/./check_stream)

May  4 21:40:01 server CRON[29164]: (root) CMD (/usr/local/sbin/./check_stream)

May  4 21:50:01 server CRON[9769]: (root) CMD (/usr/local/sbin/./check_stream)

May  4 22:00:01 server CRON[12927]: (root) CMD (/usr/local/sbin/./check_stream)

Cron job seems to be running correctly and things seem to be pointed at the script not sure what i am missing..

root at server:/etc/asterisk# lscpu

Architecture:          x86_64

CPU op-mode(s):        32-bit, 64-bit

Byte Order:            Little Endian

CPU(s):                8

On-line CPU(s) list:   0-7

Thread(s) per core:    2

Core(s) per socket:    4

Socket(s):             1

NUMA node(s):          1

Vendor ID:             GenuineIntel

CPU family:            6

Model:                 26

Stepping:              5

CPU MHz:               2800.000

BogoMIPS:              5600.43

Virtualisation:        VT-x

L1d cache:             32K

L1i cache:             32K

L2 cache:              256K

L3 cache:              8192K

NUMA node0 CPU(s):     0-7

root at server:/etc/asterisk# df -h

Filesystem      Size  Used Avail Use% Mounted on

udev            7.9G  4.0K  7.9G   1% /dev

tmpfs           1.6G  1.2M  1.6G   1% /run

/dev/md2         77G   11G   63G  15% /

none            4.0K     0  4.0K   0% /sys/fs/cgroup

none            5.0M     0  5.0M   0% /run/lock

none            7.9G  4.1M  7.9G   1% /run/shm

none            100M   16K  100M   1% /run/user

/dev/md3        1.8T  208G  1.5T  13% /home

Loren Tedford (KC9ZHV) 
Email:  <mailto:lorentedford at gmail.com> lorentedford at gmail.com

 <http://www.lorentedford.com/> http://www.lorentedford.com

http://www.kc9zhv.com

http://forum.kc9zhv.com

http://hub.kc9zhv.com

http://www.newwavesucks.com

http://forum.newwavesucks.com

On Wed, May 4, 2016 at 10:00 PM, Ken <ke2n at cs.com <mailto:ke2n at cs.com> > wrote:

When the script runs you will find an indication in  /var/log/cron

Lines like this:

May  2 20:50:01 localhost crond[30559]: (root) CMD (/usr/local/sbin/./check_stream)

One thing that can cause a Linux system to grind to a near-halt is if you have run out of disk space ….

But I see you are running VBoxHeadless – I have no familiarity but a quick Google of that finds something related

https://forums.virtualbox.org/viewtopic.php?f=8 <https://forums.virtualbox.org/viewtopic.php?f=8&t=68525> &t=68525

Ken

From: Loren Tedford [mailto:lorentedford at gmail.com <mailto:lorentedford at gmail.com> ] 
Sent: Wednesday, May 04, 2016 10:42 PM
To: Brent Weatherall <va3bfw at gmail.com <mailto:va3bfw at gmail.com> >
Cc: Ken <ke2n at cs.com <mailto:ke2n at cs.com> >; app_rpt mailing list <app_rpt-users at ohnosec.org <mailto:app_rpt-users at ohnosec.org> >

Subject: Re: [App_rpt-users] ezstream 100% cpu

Well my server is at it again 100% cpu usage for no reason kinda odd..

I did try Ken's script however I am not sure that its working the way it should...

I did place the info in crontab -e I wonder if we are missing some sort of dependency that isn't getting installed.. Maybe their is an easier way to broadcast to Broadcastify..

Here is an image of what it looks like this evening.. 

http://kc9zhv.com/wp-content/uploads/2016/05/2016-05-04.png

  <http://kc9zhv.com/wp-content/uploads/2016/05/2016-05-04.png> 

Loren Tedford (KC9ZHV) 
Email:  <mailto:lorentedford at gmail.com> lorentedford at gmail.com

 <http://www.lorentedford.com/> http://www.lorentedford.com

http://www.kc9zhv.com

http://forum.kc9zhv.com

http://hub.kc9zhv.com

http://www.newwavesucks.com

http://forum.newwavesucks.com

On Mon, May 2, 2016 at 4:46 PM, Brent Weatherall <va3bfw at gmail.com <mailto:va3bfw at gmail.com> > wrote:

Ken, definitely confirmed on my last CPU spiking that lame's process was no longer running. I'll give your monitoring/restart script a try. Thanks for the info!

On Mon, May 2, 2016 at 11:26 AM Ken <ke2n at cs.com <mailto:ke2n at cs.com> > wrote:

Of course the 100% CPU is probably due to it waiting for some resource, rather than actually being loaded to 100%. Press “1” in top and look for the “%wa”

I found (like some others) that the problem is actually the “lame” program which vanishes for some reason.  I run a script every 10 minutes that checks the pid for lame and restarts ezstream – but only if needed.  I think that is better than killing ezstream when it is, in fact, running fine.

This is the script (it is not my invention)

#!/bin/bash 

lamenumber=$(/sbin/pidof lame)

if [ "$lamenumber" = "" ]

then

        eznumber=$(/sbin/pidof ezstream)

        echo "$eznumber"

        kill -9 "$eznumber"     

        echo "restarting"

        date 

else

        eznumber=""

fi

Regards

Ken

From: Brent Weatherall [mailto:va3bfw at gmail.com <mailto:va3bfw at gmail.com> ] 
Sent: Monday, May 02, 2016 11:09 AM
To: Loren Tedford <lorentedford at gmail.com <mailto:lorentedford at gmail.com> >
Cc: app_rpt mailing list <app_rpt-users at ohnosec.org <mailto:app_rpt-users at ohnosec.org> >
Subject: Re: [App_rpt-users] ezstream 100% cpu

A quick look at that cron schedule looks like it would run every minute of the 0 hour and 12 hour. Did you just want it to run only once at 0 and 12? If that is the case you'd want:

0 */12 * * *.

I'll let you know how it works out for stability. Thanks again.

On Mon, May 2, 2016 at 11:03 AM Loren Tedford <lorentedford at gmail.com <mailto:lorentedford at gmail.com> > wrote:

The biggest issue with ezstream is that for some reason you get multiple instances of the program running.. Not sure exactly what would be the best solution to actually fix this but the cron script is a big bandage with duck tape.. I really wish I knew how to write code..

Loren Tedford (KC9ZHV) 
Email: lorentedford at gmail.com <mailto:lorentedford at gmail.com> 

Phone: 618-553-0806 <tel:618-553-0806> 
Fax: 16185512755 <tel:16185512755> 
http://www.lorentedford.com
http://kc9zhv.com

Sent from Droid Turbo from Verizon wireless network

On May 2, 2016 10:00 AM, "Brent Weatherall" <va3bfw at gmail.com <mailto:va3bfw at gmail.com> > wrote:

Thanks for the cron script to try Loren. Hopefully the daily restart of ezstream will cure any issues.

On Mon, Apr 25, 2016 at 3:57 PM Loren Tedford <lorentedford at gmail.com <mailto:lorentedford at gmail.com> > wrote:

Here is what i did to get ezstream working I will be updating this forum as well as to my fix for ezstream locking open randomly basicly it kinda goes like this..

http://forum.kc9zhv.com/index.php/topic,23.0.html

crontab -e 

* */12 * * * sh /etc/asterisk/stopez.sh

script that i use to reset ezstream..

root at server:/etc/asterisk# cat stopez.sh

#!/bin/bash

kill -9 $(pgrep ezstream)&

sleep 2s

kill -9 pgrep ezstream

sleep 2s

/usr/sbin/asterisk -rx "module reload"

now to restart asterisk i run this customized script.

root at server:/etc/asterisk# cat restart.sh

#!/bin/bash

kill -9 $(pgrep ezstream)&

kill -9 pgrep ezstream

service asterisk stop

sleep 1s

killall asterisk

kill -9 $(pgrep ezstream)&

service asterisk stop

sleep 1s

service asterisk start

So far I have had no issues with ezstream since i did this..

Loren Tedford (KC9ZHV) 
Email:  <mailto:lorentedford at gmail.com> lorentedford at gmail.com

 <http://www.lorentedford.com/> http://www.lorentedford.com

http://www.kc9zhv.com

http://forum.kc9zhv.com

http://hub.kc9zhv.com

http://www.newwavesucks.com

http://forum.newwavesucks.com

On Mon, Apr 25, 2016 at 12:10 PM, Brent Weatherall <va3bfw at gmail.com <mailto:va3bfw at gmail.com> > wrote:

Hello,
I've recently setup an allstar hub node and decided to stream it via broadcastify.

Setup has gone well - I'm running debian, with the allstar node running without issue, as well as streaming now being delivered via Broadcastify.

I've ran in to an issue where the ezstream process will occasionally pin at 100%. I can kill it fine, and asterisk gracefully resumes by creating a new working output stream.

Has anyone else encountered this cpu pinned usage? Will I have to set up something to monitor the process to kill it, so it restarts?

Thanks in advance for any suggestions

VA3BFW - Brent

_______________________________________________
App_rpt-users mailing list
App_rpt-users at ohnosec.org <mailto:App_rpt-users at ohnosec.org> 
http://ohnosec.org/cgi-bin/mailman/listinfo/app_rpt-users

To unsubscribe from this list please visit http://ohnosec.org/cgi-bin/mailman/listinfo/app_rpt-users and scroll down to the bottom of the page. Enter your email address and press the "Unsubscribe or edit options button"
You do not need a password to unsubscribe, you can do it via email confirmation. If you have trouble unsubscribing, please send a message to the list detailing the problem. 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.keekles.org/pipermail/app_rpt-users/attachments/20160505/e7ca0de2/attachment.html>