Hi Shaun,<div><br></div><div>Looking at your log watch (attached file), I don't think disk space is the issue. You still have 28G of free space which is quite a lot. I would first start looking in you log files for clues. Take a look at /var/log/messages file. If you have a specified time when you noticed the node hang, look around for that time. </div>
<div><br></div><div>When linux hangs, it can be caused by a memory issue, hardware issue, or kernel issue. Hopefully the log file will give you some clues. </div><div><br></div><div>Here are some commands that you can used to look for large files and why your system is using so much disk space. </div>
<div><br></div><div>This will list the disk usage of a directory and list the top offenders:</div><div><br></div><div> <b>du -skh * | sort -k1 -n -r | head </b></div><div><b><br></b></div><div>I would check /var, first... for example:</div>
<div><br></div><div> <b> cd /var</b></div><div><b> du -skh * | sort -k1 -n -r | head</b><br></div><div><br></div><div>Here is the output example:</div><div><br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px">
<div><div><i>[root@AllStar var]# du -skh * | sort -k1 -n -r | head</i></div></div><div><div><i>220K<span class="Apple-tab-span" style="white-space:pre"> </span>spool</i></div></div><div><div><i>188K<span class="Apple-tab-span" style="white-space:pre"> </span>run</i></div>
</div><div><div><i>40M<span class="Apple-tab-span" style="white-space:pre"> </span>lib</i></div></div><div><div><i>40K<span class="Apple-tab-span" style="white-space:pre"> </span>lock</i></div></div><div><div><i>37M<span class="Apple-tab-span" style="white-space:pre"> </span>cache</i></div>
</div><div><div><i>28K<span class="Apple-tab-span" style="white-space:pre"> </span>db</i></div></div><div><div><i>24K<span class="Apple-tab-span" style="white-space:pre"> </span>empty</i></div></div><div><div><i>20K<span class="Apple-tab-span" style="white-space:pre"> </span>yp</i></div>
</div><div><div><i>12K<span class="Apple-tab-span" style="white-space:pre"> </span>account</i></div></div><div><div><i>8.0K<span class="Apple-tab-span" style="white-space:pre"> </span>tmp</i></div></div></blockquote><div>
<br></div><div>I normally start at the root level and work down...</div><div><br></div><div> cd /</div><div> du -skh * | sort -k1 -n -r | head</div><div><br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px">
<div><div><i>[root@AllStar /]# du -skh * | sort -k1 -n -r | head</i></div></div><div><div><i>146M<span class="Apple-tab-span" style="white-space:pre"> </span>lib</i></div></div><div><div><i>124M<span class="Apple-tab-span" style="white-space:pre"> </span>root</i></div>
</div><div><div><i>124K<span class="Apple-tab-span" style="white-space:pre"> </span>dev</i></div></div><div><div><i>101M<span class="Apple-tab-span" style="white-space:pre"> </span>etc</i></div></div><div><div><i>88K<span class="Apple-tab-span" style="white-space:pre"> </span>tmp</i></div>
</div><div><div><i>87M<span class="Apple-tab-span" style="white-space:pre"> </span>opt</i></div></div><div><div><i>78M<span class="Apple-tab-span" style="white-space:pre"> </span>var</i></div></div><div><div><i>34M<span class="Apple-tab-span" style="white-space:pre"> </span>sbin</i></div>
</div><div><div><i>16K<span class="Apple-tab-span" style="white-space:pre"> </span>lost+found</i></div></div><div><div><i>12M<span class="Apple-tab-span" style="white-space:pre"> </span>boot</i></div></div></blockquote><div>
<br></div><div><br></div><div>This will find large files > 10M in size and help you find the large files.</div><div><br></div><div> <b> find / -size +10M -exec ls -latr {} \;</b></div><div><br></div><div>As far as files to delete, I would start in <b style="font-style:italic">/var/log</b>, there are log files that could be large. If log rotation is working, it would be self limiting, the fact that you are up in to 100G+ disk space, I am not sure what is eating the disk usage. I find the /var is the usual problem area for disk issues. Let me know what you find out and I will be glad to let you know what files you can remove and what files you should not delete.</div>
<div><br></div><div>Chris,</div><div><br></div><div>w0anm (allstar node 29061, irlp node 3106)</div><div><br></div><div><br><div class="gmail_quote">On Fri, Dec 21, 2012 at 7:10 PM, Shaun Rudling <span dir="ltr"><<a href="mailto:shaun@pmbnet.co.za" target="_blank">shaun@pmbnet.co.za</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
<br>
My AllStar node just hangs every so often. I have to physically drive up<br>
to the remote high-site and pull the power to get it working again. I<br>
don't know much about fault finding in Linux so could someone give me a<br>
pointer.<br>
<br>
I attached the last bit of text from this email "You have new mail in<br>
/var/spool/mail/root" .<br>
<br>
I suspect that the hard drive is running out of space but why? And what<br>
files can I delete to give me more HDD space?<br>
<br>
Shaun<br>
zr5s (zs5pmb)<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>_______________________________________________<br>
App_rpt-users mailing list<br>
<a href="mailto:App_rpt-users@ohnosec.org">App_rpt-users@ohnosec.org</a><br>
<a href="http://ohnosec.org/cgi-bin/mailman/listinfo/app_rpt-users" target="_blank">http://ohnosec.org/cgi-bin/mailman/listinfo/app_rpt-users</a><br>
<br></blockquote></div><br></div>