Wednesday 7 February 2018

R12 : All Concurrent Request are Running Normal Forever not Completing


I face one internal issue that all standard concurrent requests in EBS R12.1.3  are not completing and its continuously in status "Running Normal" forever. This issue was reported on of their recently cloned TEST Environment.
Environment details:
EBS : 12.1.3
DB   : 11.1.0.7
OS   : IBM-AIX -7.1 (64-bit)
If we check the log file for any standard concurrent request there is no error reported. Below is the concurrent manager logfile for Active users:



 The concurrent request will stuck with no further progress.
As a workaround tried restarting database, concurrent manager, restarted all Application services and re linking libraries. But none of them worked.
Cause:
To troubleshoot further tried executing Standard Concurrent request "Active Users" from CLI.

$ $APPLORC userid=apps/ap**** mode=character report=$FND_TOP/reports/US/FNDSCURS.rdf batch=yes destype=file desname=FNDSCURS.txt desformat=$FND_TOP/reports/HPL.prt 

REP-3000: Internal error starting Oracle Toolkit.
REP-3000: Internal error starting Oracle Toolkit.

Report Builder: Release 10.1.2.3.0 - Production on Wed Feb 7 12:22:28 2018

Copyright (c) 1982, 2005, Oracle. All rights reserved.

REP-0069: Internal error
REP-57054: In-process job terminated:Terminated with error:
REP-3000: Internal error starting Oracle Toolkit.

Note :-
==========
Also, could you check the user permissions and dba/os group for the $APPLCSF/APLLOUT and APPLLOG directories.

Solution:

The problem is with VNCSERVER files.

Usually Rep - 3000 errors occur when the DISPLAY is not set corrrectly.

Please verify if this is the correct value set for DISPLAY = erptestappldb 0.0
++++++++++++++++++++++++++++

Perform the below action plan

1. Login in to putty as applmgr user in the AIX machine which is running the EBS application. On this machine a vncserver software has to be installed.

2. Start a new vnc server process as applmgr user with a new port eg. vncserver :3 or ebis:0.0

3. Start the vncviewer (from your client desktop) and login as applmgr user on the new port

4. Check echo $DISPLAY to confirm the display value

5. Type xclock and check whether clock is displayed

6. Once all the above actions are successful, open the applications context file on all nodes and change the s_display variable to the new value displayed on vncviewer

7. Save the changes

8. Shutdown the application services and run autoconfig on all nodes

9. Edit the . .profile file to change the DISPLAY to new value and save it

10. Restart the application services and retest the issue by running the report and confirm if the REP - 3000 error occurs still.