In 11gR2 when you set up your cluster and RAC database with different OS users, you may get ORA-12537 when connecting to the database through SCAN listener. Listener and SCAN listeners are started in grid infrastructure home, which is owned by a different OS user from RDBMS home owner. In such configuration two things can cause ORA-12537:
1) Grid owner cannot see RDBMS home binary
As grid owner, do “ls -l <RDBMS Home>/bin/oracle”. If getting “permission denied” then this is access problem.
2) The permission “-rwxr-x–x” is wrong as it’s missing suid bit, oracle binary should have permission of 6751
/u01/app/oracle/product/11.2.0/dbhome_1/bin >ls -l oracle
-rwsr-s–x 1 oracle asmdba 228934980 Jul 25 14:35 oracle
Fixing these two issues can solve ORA-12537 error.