CRS-4535: Cannot communicate with Cluster Ready Services

This error is caused by crsd not running, while both cssd and evmd processes are running. Also all the database instances are running.

Environment is 12.1.0.2 GI and 11.2.0.4 databases.

$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services  
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
$ps -eaf|grep crsd 

No process found

Try to start up CRS

# crsctl start crs
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.

CRS alert.log

2017-01-22 00:41:14.042 [OHASD(3987)]CRS-2757: Command 'Start' timed out waiting for response from the resource 'ora.storage'. Details at (:CRSPE00163:) {0:17:9} in /u01/app/grid/diag/crs/racnode1/crs/trace/ohasd.trc.
2017-01-22 00:41:14.053 [OHASD(3987)]CRS-2878: Failed to restart resource 'ora.crsd'
Non critical error DIA-48113 caught while writing to trace file "/u01/app/grid/diag/crs/racnode1/crs/trace/osysmond.trc"
Error message:
Writing to the above trace file is disabled for now on...

osysmond.trc

2017-01-21 23:08:41.450704 : CRFMOND:2125666048: crfmond_csvdump: Write failed with rc -2
2017-01-21 23:08:41.450910 : CRFMOND:2125666048: Terminating csv dump thread as it has hit a fatal error!
2017-01-21 23:08:46.450992 : CRFMOND:2125666048: *** Thread csv dump stopped ***

CRSD trace file ( crsd.trc )

ORA-09817: Write to audit file failed.
Linux-x86_64 Error: 28: No space left on device
Additional information: 12
. For details refer to "(:CLSN00109:)" in "/u01/app/grid/diag/crs/racnode1/crs/trace/crsd_oraagent_grid.trc".

Check GI_HOME, the file system is 100% full.

$ df -h $ORACLE_HOME
Filesystem Size Used Avail Use% Mounted on
/dev/sda5   59G  59G  0.0G 100%       /u01

Cleaned the log files , and start the crsd process

# crsctl start crs
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.

# crsctl start res ora.crsd -init
CRS-2672: Attempting to start 'ora.crsd' on 'racnode1'
CRS-2676: Start of 'ora.crsd' on 'node1' succeeded

Check crs status

# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online  
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
Advertisements
This entry was posted in GI and RAC and tagged , , , , , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s