REP-3000: Internal error starting Oracle Toolkit.

Issue:

Recently we had issue with Concurrent requests which having print format of PDF or XML, and excel etc. 

User reported that all reports are failing with the below error and when we did the sanity from our side we too got the same issue while opening the view log and view output of a concurrent request and error is:

"REP-3000: Internal error starting Oracle Toolkit"

 Solution:

1. Log on as root on the UNIX box.

2. Start The VNC server:
Example:
$ vncserver :0
If it's not installed on the box, have your Unix Admin's to install the VNC on the server.
3. Set the DISPLAY environment variable for Oracle Reports by performing the
following steps:
a. $ set DISPLAY=hostname:0; export DISPLAY
b. $ xhost +
c. Set the DISPLAY parameter in adcmctl.sh, adrepctl.sh,adfrmctl.sh
and gsmstart.sh to hostname:0
d. Restart concurrent manager and report server

Example to check vnc server status.

How to Verify:

1. Login with system administrator responsibility
2. Run the Active Users Concurrent Request
4. It should be completed without any issue.

Oracle Application Performance Issue.

Please follow below steps:
1.Where more specific are you encounter performance issues. While login, accessing forms, jsp pages, etc?
2. Check  what performance issue means what is the expected time and what is normal time on this situation?
3. Check context file for investigation
4. How many concurrent users do you have at peak?

If uncertain, you could get a round about number using:

REM
REM SQL to count number of Apps users
REM Run as APPS user
REM
select 'Number of user sessions : ' || count( distinct session_id) How_many_user_sessions
from icx_sessions icx
where disabled_flag != 'Y'
and PSEUDO_FLAG = 'N'
and (last_connect + decode(FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT'), NULL,limit_time, 0,limit_time,FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT')/60)/24) > sysdate
and counter < limit_connects;
REM
REM END OF SQL
REM

5. How many web servers do you have? How many cpu's on each web server?

6. Check Below Script result.
 
monitor_jdbc_conn.sql - Script to monitor JDBC connections in Apps eBusiness Suite ( Doc ID 557194.1 ).

7. Check txk report with the following:

$ADPERLPRG $FND_TOP/patch/115/bin/TXKScript.pl -script=$FND_TOP/patch/115/bin/txkInventory.pl -txktop=$APPLTMP -outfile=$APPLTMP/Report_Inventory.html

8. How many users are connecting at peak?

9. What values you have defined for xms and xmx for jvm startup?

10. To check SGA size?

Example:

1. There is a delay while login to the oracle application, there is a delay in accessing forms and jsp pages.

2. As a DBA i am observing the performance issue while accessing the server using putty, For e.g. when i go to $AD_TOP directory and execute "du -sk *" normally it takes few seconds to output the result but currently it is taking so much time to output the result.

3. Check context file for investigation

4. Number of user sessions : 121

5. We don't have a webserver outside the Oracle EBS
# lsdev -Cc processor
proc0 Available 00-00 Processor
proc8 Available 00-08 Processor
proc16 Available 00-16 Processor

# bindprocessor -q
The available processors are: 0 1 2 3 4 5 6 7 8 9 10 11

# pmcycles -m
CPU 0 runs at 4116 MHz
CPU 1 runs at 4116 MHz
CPU 2 runs at 4116 MHz
CPU 3 runs at 4116 MHz
CPU 4 runs at 4116 MHz
CPU 5 runs at 4116 MHz
CPU 6 runs at 4116 MHz
CPU 7 runs at 4116 MHz
CPU 8 runs at 4116 MHz
CPU 9 runs at 4116 MHz
CPU 10 runs at 4116 MHz
CPU 11 runs at 4116 MHz

# prtconf -s
Processor Clock Speed: 4116 MHz

6. Check Below Script result.

monitor_jdbc_conn.sql

7. Check txk report.

$ TOP/patch/115/bin/txkInventory.pl -txktop=$APPLTMP -outfile=$APPLTMP/Report_Inventory.html <
Enter Apps password ?
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** STDOUT = /prodappl/inst/apps/PROD_erpprodappl/logs/appl/rgf/TXK/txkInventory_Wed_Jun_29_16_25_45_2016_stdout.log

9. values you have defined for xms=256 and xmx=128 for jvm startup.

10. SGA Size 1G.

Solution:
Please review document : JVM: Guidelines to setup the Java Virtual Machine in Apps Ebusiness Suite 11i and R12 ( Doc ID 362851.1 ) and according with this please perform the following:

1. Please increase Xmx with 2048 and Xms with 2048
2. Please make sure that you are using no more than 1 oacore per 100 user, in case you are using more than 100 user you will need to add more oacores, for this please follow the bellow note:
Managing Configuration of Oracle HTTP Server and Web Application Services in Oracle E-Business Suite Release 12.2 ( Doc ID 1905593.1 )
3. Please make sure that you add the following parameter to the DBC file:
JDBC\:oracle.jdbc.maxCachedBufferSize=262144 

Unable To Avoid Potential Divide By Zero Situation

Issue:
Most of time oracle developer facing this type of issue while they have been doing report development activity.
Unable To Avoid Potential Divide By Zero Situation

Solution:
Please speak with your in-house developer about this error and try to fix it within the rtf template.

The following syntax will avoid a potential divide by zero situation:

<?if: VALUE2=0?>0<?end if?><?if: VALUE2!=0?><?VALUE1 div VALUE2?><?end if?>

This will not fail if VALUE2 does have the value zero (0).

Reference:

Unable To Avoid Potential Divide By Zero Situation Using If-then-else Construct (Doc ID 429808.1)

Unable to Find the Concurrent Request Output Post Processor (OPP) Service

Description:

AQs tables are used to look for "subscriptions" by FNDSMs. That is, when ICM calls for FNDSM to start, they "subscribe" to this queue to identify its status. The time taking for the process cleanup prior to the ICM starting up the regular CMs is correlated to the number of processes that were not stopped cleanly. In case of un-clean shutdown, the process to restart will be longer as manager spends extra cycles to perform housekeeping tasks.

Solution:

A permanent solution is to schedule a cron job or something which queries records in APPLSYS.FND_CP_GSM_OPP_AQTBL to monitor it and use DBMS_AQADM.PURGE_QUEUE_TABLE to purge the table as needed.

The issue has been resolved after we implemented the solution from DOC- How To Purge FND_AQ Tables ( Doc ID 1156523.1 )  

Also check
Increase the value of following profiles:
Concurrent:OPP Process Timeout
Concurrent:OPP Response Timeout

Concurrent manager become up after 3-4 hours after start Oracle Application.

Description:
These two tables FND_ENV_CONTEXT FND_CONCURRENT_REQUESTS grown to 4 GB size . To speed up.

Solution:
1. First Purge and then De-fragmentation to improve the performance of the DML fired on FND_CONCURRENT tables.

Refer Note 1057802.1 for more details.

1.1. Truncation of FND_ENV_CONTEXT

1.1.1. Take a backup of FND_ENV_CONTEXT

1.1.2. Shutdown concurrent manager

1.1.3. Truncate FND_ENV_CONTEXT

1.2. Defragment the tables below to reclaim unused space / improve performance

FND_CONCURRENT_REQUESTS
FND_CONCURRENT_PROCESSES
FND_CRM_HISTORY
FND_TEMP_FILES
FND_LOG_TRANSACTION_CONTEXT

Refer Note 1057802.1 for more details (how to defragment)

2. Start the concurrent manager

Monitor AQ$ tables as well.

The AQ$_FND_CP_GSM_OPP_AQTBL_H is a history table.Ensure you purge at regular interval.

select count(*) from AQ$_FND_CP_GSM_OPP_AQTBL_H;

If the table is grown, then one can purge using DBMS_AQADM.PURGE_QUEUE_TABLE. Refer to: ( Doc ID 1156523.1 )

Tuning Output Post Processor (OPP) to Improve Performance ( Doc ID 1399454.1 ) 

Guest user/pwd does not exist or match: GUEST/ORACLE1 In Oracle apps R12

Please perform the below steps.

1. Shutdown the EBS services.

2. The only supported way to change the Guest user password is to update the context variable s_guest_pass and run AutoConfig, which runs the AdminAppServer utility internally.

Check your context file for variable s_guest_pass, ensure it has the value ORACLE1 set, if its not set then you need to set it manually.

Note : Restriction on the GUEST User Password : The GUEST User password cannot include the special character "#".

3. Run autoconfig on DB Node and then application node.

4. Execute the below sql again:

select fnd_web_sec.validate_password('GUEST','ORACLE') from dual;

Check whether query output is showing Y.

If no, Please check whether the below error is seen on autoconfig log:

Unable to update GUEST_USER_PWD in database to GUEST/ORACLE - Password was not changed, this point to the DB parameter JAVA_JIT_ENABLED which is set as TRUE.

5. On 11g DB you need to have the below settings for EBS specifically, run the below sql

alter system set JAVA_JIT_ENABLED= FALSE scope = both;

6. Follow the steps 2 and 3 again.

7. Run the below command:

perl $FND_TOP/patch/115/bin/ojspCompile.pl --compile --flush -p 2

7. Check whether the compile completed successfully.

8. Restart the application services.

9. Retest the issue.