ViPR SRM 4.3 - Troubleshooting : SocketWriter::sendBuffer(): Can't write to Backend Host


Issue:
  1. ViPR SRM Missing Alert Definition default templates under > SRM Admin UI > Config > Alerts > Manage Alert Definitions.

Errors in Logs:
Server: Primary Backend - Event-Processing-Manager - Alert-Consolidation
Install Path:  Event-Processing/Event-Processing-Manager/Alert-Consolidation
Viewing File 'logs/processing-0-0.log'
WARNING  -- SocketWriter::sendBuffer(): Can't write to <Primary Backend>/xx.xxx.xxx.xxx:2013. Retrying 10 times…
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at com.watch4net.events.processing.writers.generic.nio.ChannelWriter.completeConnection(ChannelWriter.java:102)
WARNING  -- [2019-11-21 17:14:31 EST] -- StreamFlusher::run(): Unable to flush the buffer. Some events might be lost...
java.io.IOException: Can't write to <Primary Backend>/xx.xxx.xxx.xxx:2013
    at com.watch4net.events.processing.writers.generic.SocketWriter.sendBuffer(SocketWriter.java:311)

Server: Primary Backend - Alerting-Backend - Default
Install Path:  Backends/Alerting-Backend/Default
Viewing File 'logs/alerting-0-0.log'
WARNING  -- SocketSource$DataReaderWorker::run(): An incoming event could not be processed: com.watch4net.events.common.serialization.SerializationException: Invalid data type
SEVERE   -- SocketSource$DataReaderWorker::run(): Unable to read from socket... Aborting
java.net.SocketException: Socket closed

Cause: dropping of data due to contention (or) Connection Brocken b/w Alert Consolidation and Alerting Blackened.

Fix/Resolution/Work Around:

  1. Increase the Available memory for the service 'Event-Processing-Manager - Alert-Consolidation'
             >> Go to Physical Overview in SRM > Select "Primary BackEnd' > Click on 'Services' Tab > Click on 'Event-Processing-Manager - Alert-Consolidation' > Click on Gear box 'Configure Service' > Increase the Available memory for the service (Custom 2GB) '3000' MB > Save it


  1. Increase the Available memory for the service 'Event-Processing-Manager - Alert-Consolidation'
             >> Go to Physical Overview in SRM > Select "Primary BackEnd' > Click on 'Services' Tab > Click on 'Alerting-Backend - Default' > Click on Gear box 'Configure Service' > Increase the Available memory for the service (Custom 2GB) '3000' MB > Save it
  1. Restart the Service 'Alerting-Backend - Default'
  2. Restart the Service 'Event-Processing-Manager - Alert-Consolidation'
  3. Give couple of Mins, and check whether you can see the Alert Definition templates under > SRM Admin UI > Config > Alerts > Manage Alert Definitions.

FYI: