Issue: When the /var partition reaches 75%, 85%, or 95% of capacity, an event is logged and an alert is sent.
Fix: Rotate logs
If the /var partition returns to a normal usage level, review the list of recently written logs to determine if a specific log is rotating frequently. Rotation can resolve the full-partition issue by compressing or removing large logs and old logs, thereby automatically reducing partition usage.
Check the percentage of free isilon nodesOpen an SSH connection to the node that reported the error and log in using the "root" account.
Run the following command:
df -i |grep var |grep -v crash
Output similar to the following appears:
Filesystem 1K-blocks Used Avail Capacity iused ifree %iused Mounted on
/dev/mirror/var0 1013068 49160 882864 5% 1650 139276 100% /var
If the %iused value is 90% or higher, reduce the number of files in the /var partition using one of the methods described below:
Remove files that do not belong in the /var partition.
On the node that generated the alert, run the following command to list files in the /var partition that are greater than 5 MB:
find -x /var -type f -size +10000 -exec ls -lh {} \; | awk '{ print $9 ": " $5 }'
In the output, look for files that do not typically belong in the /var partition. For example, a OneFS installer file, log gathers, or a user-created file.
Remove the files or move them to the /ifs directory. If you are unsure what to remove, contact Isilon Technical Support for assistance.
Determine if a process is holding a large file open
You can use the fstat command to list the open files on a node or in a directory, or to list the files that were opened by a particular process. A list of the open files can help you monitor the processes that are writing large files. See How to use the fstat command to list the open files on a node, 16648 .
If neither of the above tasks resolves the issue, continue with the following solution:
Limit the rollover file size and compress the file
Open an SSH connection on any node in the cluster and log in using the "root" account.
Run the following commands to create a backup of the /etc/newsyslog.conf file:
cp /etc/newsyslog.conf /ifs/newsyslog.conf
cp /etc/newsyslog.conf /etc/newsyslog.bak
Open the /ifs/newsyslog.conf file in a text editor.
Locate the following line:
/var/log/wtmp 644 3 * @01T05 B
Change the line to:
/var/log/wtmp 644 3 10000 @01T05 ZB
These changes instruct the system to roll over the /var/log/wtmp file when it reaches 10 MB and to compress the file with gzip.
Save and close the /ifs/newsyslog.conf file.
Run the following command to copy the updated file to all nodes on the cluster:
isi_for_array 'cp /ifs/newsyslog.conf /etc/newsyslog.conf'
If other logs are rotating frequently, or if the preceding solutions do not resolve the issue, run the isi_gather_info command to gather logs, and then contact Isilon Technical Support for assistance.
Ref: EMC KB Article 000471789
Filesystem 1K-blocks Used Avail Capacity iused ifree %iused Mounted on
/dev/mirror/var0 1013068 49160 882864 5% 1650 139276 100% /var
If the %iused value is 90% or higher, reduce the number of files in the /var partition using one of the methods described below:
Remove files that do not belong in the /var partition.
On the node that generated the alert, run the following command to list files in the /var partition that are greater than 5 MB:
find -x /var -type f -size +10000 -exec ls -lh {} \; | awk '{ print $9 ": " $5 }'
In the output, look for files that do not typically belong in the /var partition. For example, a OneFS installer file, log gathers, or a user-created file.
Remove the files or move them to the /ifs directory. If you are unsure what to remove, contact Isilon Technical Support for assistance.
Determine if a process is holding a large file open
You can use the fstat command to list the open files on a node or in a directory, or to list the files that were opened by a particular process. A list of the open files can help you monitor the processes that are writing large files. See How to use the fstat command to list the open files on a node, 16648 .
If neither of the above tasks resolves the issue, continue with the following solution:
Limit the rollover file size and compress the file
Open an SSH connection on any node in the cluster and log in using the "root" account.
Run the following commands to create a backup of the /etc/newsyslog.conf file:
cp /etc/newsyslog.conf /ifs/newsyslog.conf
cp /etc/newsyslog.conf /etc/newsyslog.bak
Open the /ifs/newsyslog.conf file in a text editor.
Locate the following line:
/var/log/wtmp 644 3 * @01T05 B
Change the line to:
/var/log/wtmp 644 3 10000 @01T05 ZB
These changes instruct the system to roll over the /var/log/wtmp file when it reaches 10 MB and to compress the file with gzip.
Save and close the /ifs/newsyslog.conf file.
Run the following command to copy the updated file to all nodes on the cluster:
isi_for_array 'cp /ifs/newsyslog.conf /etc/newsyslog.conf'
If other logs are rotating frequently, or if the preceding solutions do not resolve the issue, run the isi_gather_info command to gather logs, and then contact Isilon Technical Support for assistance.
Ref: EMC KB Article 000471789
