InsightIQ Error: The InsightIQ user on the cluster does not have permission to view quota reports.


Error: The InsightIQ user on the cluster does not have permission to view quota reports.

Fix
  1. Verify from InsightIQ server configuration to confirm the user being used for cluster connection.  Go to InsightIQ server WebUI page, and click on Settings -> Configure (click on the link associated with the cluster in question).  Check what username is being used for monitoring.  By default we use user - insightiq, If it is Different user follow the below Steps for that particular User ID instead of UserID: InsightIQ (If it a domain user, use the user id as <User_ID@Domain>).

  1. Login to cli of the cluster node in question as root user.

  1. Verify the current set of privileges associated with insightiq user:
//isi auth mapping token <User_ID@Domain>
# isi auth mapping token insightiq
                   User
                       Name : insightiq
                        UID : 15
                        SID : S-1-22-1-15
                    On Disk : 15
                    ZID: 1
                   Zone: System
             Privileges: ISI_PRIV_LOGIN_PAPI
                         ISI_PRIV_EVENT
                         ISI_PRIV_JOB_ENGINE
                         ISI_PRIV_STATISTICS

  1. Check if any role is assigned to user account InsightIQ, If it is there; like I Bolded below; skip step 5 and go to Step 6. If no role assigned then go to Step 5
IsilonCluster# isi auth roles list --v | grep -B4 -i insightiq
      Read Only: True
--------------------------------------------------------------------------------
       Name: InsightIQRole
Description: -
    Members: insightiq

  1. Create a new Role and add user to that Role
IsilonCluster# isi auth roles create InsightIQRole --description "Allows access to read and write SmartQuotas data and settings, as well as read filesystem namespace, snapshot, and smartpools information."
IsilonCluster# isi auth roles modify InsightIQRole --add-user insightiq

  1. Run the following set of commands to give insightiq user additional privileges:

IsilonCluster# isi auth roles modify InsightIQRole --add-priv ISI_PRIV_QUOTA
IsilonCluster# isi auth roles modify InsightIQRole --add-priv ISI_PRIV_NS_IFS_ACCESS
IsilonCluster# isi auth roles modify InsightIQRole --add-priv ISI_PRIV_NS_TRAVERSE
IsilonCluster# isi auth roles modify InsightIQRole --add-priv ISI_PRIV_SNAPSHOT
IsilonCluster# isi auth roles modify InsightIQRole --add-priv ISI_PRIV_SMARTPOOLS

  1. Verify again the privileges associated with insightiq user to make sure you see the following:

IsilonCluster# isi auth mapping token insightiq
                   User
                       Name : insightiq
                        UID : 15
                        SID : S-1-22-1-15
                    On Disk : 15
                    ZID: 1
                   Zone: System
             Privileges: ISI_PRIV_LOGIN_PAPI
                         ISI_PRIV_EVENT
                         ISI_PRIV_JOB_ENGINE
                         ISI_PRIV_QUOTA
                         ISI_PRIV_SMARTPOOLS
                         ISI_PRIV_SNAPSHOT
                         ISI_PRIV_STATISTICS
                         ISI_PRIV_NS_TRAVERSE
                         ISI_PRIV_NS_IFS_ACCESS
Logon to the InsightIQ server CLI (through Putty), and restart IIQ service with the following command:
InsightIQ# sudo /sbin/service insightiq stop
Stopping insightiq:                                        [  OK  ]
InsightIQ# sudo /sbin/service insightiq start
Starting insightiq:                                        [  OK  ]

  1. Verify if the Quota Reporting page can be displayed correctly from the InsightIQ WebUI page.

Ref:

ViPR SRM 4.3 - Troubleshooting : SocketWriter::sendBuffer(): Can't write to Backend Host


Issue:
  1. ViPR SRM Missing Alert Definition default templates under > SRM Admin UI > Config > Alerts > Manage Alert Definitions.

Errors in Logs:
Server: Primary Backend - Event-Processing-Manager - Alert-Consolidation
Install Path:  Event-Processing/Event-Processing-Manager/Alert-Consolidation
Viewing File 'logs/processing-0-0.log'
WARNING  -- SocketWriter::sendBuffer(): Can't write to <Primary Backend>/xx.xxx.xxx.xxx:2013. Retrying 10 times…
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at com.watch4net.events.processing.writers.generic.nio.ChannelWriter.completeConnection(ChannelWriter.java:102)
WARNING  -- [2019-11-21 17:14:31 EST] -- StreamFlusher::run(): Unable to flush the buffer. Some events might be lost...
java.io.IOException: Can't write to <Primary Backend>/xx.xxx.xxx.xxx:2013
    at com.watch4net.events.processing.writers.generic.SocketWriter.sendBuffer(SocketWriter.java:311)

Server: Primary Backend - Alerting-Backend - Default
Install Path:  Backends/Alerting-Backend/Default
Viewing File 'logs/alerting-0-0.log'
WARNING  -- SocketSource$DataReaderWorker::run(): An incoming event could not be processed: com.watch4net.events.common.serialization.SerializationException: Invalid data type
SEVERE   -- SocketSource$DataReaderWorker::run(): Unable to read from socket... Aborting
java.net.SocketException: Socket closed

Cause: dropping of data due to contention (or) Connection Brocken b/w Alert Consolidation and Alerting Blackened.

Fix/Resolution/Work Around:

  1. Increase the Available memory for the service 'Event-Processing-Manager - Alert-Consolidation'
             >> Go to Physical Overview in SRM > Select "Primary BackEnd' > Click on 'Services' Tab > Click on 'Event-Processing-Manager - Alert-Consolidation' > Click on Gear box 'Configure Service' > Increase the Available memory for the service (Custom 2GB) '3000' MB > Save it


  1. Increase the Available memory for the service 'Event-Processing-Manager - Alert-Consolidation'
             >> Go to Physical Overview in SRM > Select "Primary BackEnd' > Click on 'Services' Tab > Click on 'Alerting-Backend - Default' > Click on Gear box 'Configure Service' > Increase the Available memory for the service (Custom 2GB) '3000' MB > Save it
  1. Restart the Service 'Alerting-Backend - Default'
  2. Restart the Service 'Event-Processing-Manager - Alert-Consolidation'
  3. Give couple of Mins, and check whether you can see the Alert Definition templates under > SRM Admin UI > Config > Alerts > Manage Alert Definitions.

FYI:

Isilon Cluster AD Integration & Root Privileges to AD Group

Error1:
zsh: permission denied: isi_for_array
Fix:
>> normal you see this issue, if you dont have permission to run the command
or you are not running commands with root user
or you forgot to put sudo in fornt of the command when you are using non root user account
>> you can fix this issue by modify the sudo file to give access to all users to all commands
//Open the sudo file to add access to the Groups or Users
isi_visudo -f /etc/mcp/override/sudoers
>>click "insert" button to edit the file, add the below any entry based on your requirement
#Below Command to give access to all users
ALL ALL=(ALL) NOPASSWD: ALL
#Below Command to give access to Domain User to all CMDs
<DOMAIN>\\<USERID> ALL=(ALL) NOPASSWD: ALL
#Below Command to give access to Islon Cluster Local User to all CMDs
<USERID> ALL=(ALL) NOPASSWD: ALL
#Below Command to give access to Islon Cluster Local Group users to all CMDs
%<LocalGroupName> ALL=(ALL) NOPASSWD: ALL
>>Press Esc and enter ":wq" to write the changes to the file
Now run your command with "sudo" in front of the command, like below
Example:
IsilonCluster1-2% isi_for_array -s 'isi_hw_status | grep -i serno'
zsh: permission denied: isi_for_array
IsilonCluster1-2% sudo isi_for_array -s 'isi_hw_status | grep -i serno'
IsilonCluster1-1:   SerNo: XXXXXXXXXXX
IsilonCluster1-2:   SerNo: XXXXXXXXXXX

Error2: 
Could not chdir to home directory /ifs/home/DOMAIN/USERID: No such file or directory
Fix:
Normally you see this error, if the user that logged in doest have a Home Directory, either you create manually.
if you are a domain user, make you "Create home directories on first login" option is checked under the domain in Active Directory
GUi > Access > Authrentication Providers > Active Directory

How to setup Full Previllages to AD Groups on Isilon Cluster ?
1. Login to the GUi > Access > Authrntication Providers > Active Directory > + Join a Domain > Fill the details > Join.
Once it is joined succussfully, and status is showing "Online", goto next step
Note: for Isilon OneFS v8.1.2.0 and above make sure "Create home directories on first login" option is check.
if it is not checked, Users after loggin into putty, maybe be able to use Tab Functionality

2. Login to Isilon Cluster CLI as root through SSH tools like Putty. and run the below commands

//Create a Role First like "StorageAdmins"
isi auth roles create --name StorageAdmins --description "Full Admin Privileges to the Cluster"

//Add AD Group to the Role
isi auth roles modify StorageAdmins --add-group=ADGroup1@Domain

//you can add more groups to the same role like below
isi auth roles modify StorageAdmins --add-group="ADGroup2@Domain"

//Add Previllages to the Role that was created earlier "StorageAdmins" (Below is list of full Previllages)
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_ANTIVIRUS
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_AUDIT
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_AUTH
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_CERTIFICATE
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_CLOUDPOOLS
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_CLUSTER
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_DEVICES
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_ESRS_DOWNLOAD
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_EVENT
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_FILE_FILTER
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_FTP
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_HARDENING
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_HDFS
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_HTTP
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_IFS_BACKUP
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_IFS_RESTORE
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_IFS_WORM_DELETE
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_JOB_ENGINE
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_LICENSE
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_LOGIN_CONSOLE
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_LOGIN_PAPI
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_LOGIN_SSH
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_MONITORING
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_NDMP
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_NETWORK
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_NFS
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_NS_IFS_ACCESS
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_NS_TRAVERSE
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_NTP
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_QUOTA
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_REMOTE_SUPPORT
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_ROLE
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_SMARTPOOLS
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_SMB
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_SNAPSHOT
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_SNMP
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_STATISTICS
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_SWIFT
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_SYNCIQ
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_SYS_SHUTDOWN
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_SYS_SUPPORT
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_SYS_TIME
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_SYS_UPGRADE
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_VCENTER
isi auth roles modify StorageAdmins --add-priv ISI_PRIV_WORM

//Open the sudo file to add access to the Groups or Users
isi_visudo -f /etc/mcp/override/sudoers
>>click "insert" button to edit the file, add the below entry
ALL ALL=(ALL) NOPASSWD: ALL
>>Press Esc and enter ":wq" to write the changes to the file

3. Verify the access by relogin to the putty
login as: <DOMAIN>\<USERID>
Using keyboard-interactive authentication.
Password: **********

How to expand disk space in VIPR SRM Vapp?

Issue: Low File System Space on root; Due to increased Data Center growth on ViPR SRM

Procedure for expand disk space in VIPR SRM Vapp:
1. Add new disk on the VM (Dont Expand the Existing disk)
2. SSH into the SRM Vapp
3. run command >> expand_disk.pl
4. verify the expanded capacity: CMD:~# df -l 
5. restart the service: CMD:~# ./manage-modules.sh service restart all

Reference:
494944 : ViPR SRM: How to expand  disk space in VIPR SRM? https://support.emc.com/kb/494944
459138 : ViPR SRM: Reclaiming disk space after incorrectly expanding the vApp virtual machine disk provisioning https://support.emc.com/kb/459138

Sample Output:



In Unity Array, How to Transfer Files from the Peer SP


In Unity Array, In general only the primary SP has direct access to the network, if you want to download files from the peer SP there is no direct network access.You can access the peer SP, for file transfer, via an ssh tunnel via Putty.
The advantage of doing this, you dont need to copy files from peer SP to Primary SP.

On a Unity Array, each SP has an internal network interface named: eth_int
For SPA the address assigned to eth_int is: 128.221.255.100
For SPB the address assigned to eth_int is: 128.221.255.101

you can verify this by logging putty session into Unity with service account, and ping SPA & SPB like below



Create Session with Unity Mgmt IP <10.1.2.3> and using port 22



Go to: "Connection > SSH > Tunnels" Click "Local ports accept connections from other hosts" and create a tunnel for SPA (128.221.255.100) on tcp port 22 then Click Add Button


This is the result of clicking Add Button, Now Click on Open Button


Open up putty Session and login to the Primary SP, leave this session open.


Open Up a WinSCP session and login to SPB using 'localhost" as the address


Then transfer Your Files


Ref: https://support.emc.com/kb/499330

You can also transfer the file to primary SP, and download using FTP tools like WinSCP
Syntax:
scp your-file peer:/home/service

Example:
service@(none) spa:/home/service/> scp test_file.tar peer:/home/service

Ref: https://community.emc.com/docs/DOC-66234