Thursday 1 October 2020

Warning: VKTM detected a time drift in 11gR2

ALERT LOG:

Warning: VKTM detected a time drift.
Time drifts can result in an unexpected behavior such as time-outs. Please check trace file for more details.

 

Cause:


This problem is due to the Bug 11837095 -"TIME DRIFT DETECTED" APPEARS INTERMITTENTLY IN ALERT LOG"

 

Solution:


Time drift" error is a " message which can be ignored, to remove the "Warning: VKTM detected set event 10795 or Apply patch 11837095

 

SQL> alter system set event="10795 trace name context forever, level 2" scope=spfile;

 

Then bounce the DB instances to implement the event/change.

Oracle EBS-Blank Page Appslogin/AppsLocal Login Page in R12

There were quite a few times that I have encountered this blank front-end issue. The root cause was not the same every time it was investigated. There are many reasons for this issue to occur which I have compiled to make it easier for Apps DBA’s. Listed below are some of them.

Here is the error page while logging into Oracle E-Business Suite front end.

These are some of the main reasons for the APPS blank login page in R12:

·    JSP pages not compiled properly after the clone

·    Data block corruption on Cloned environment with bulk synchronization issues

·    Database Listeners are down

·    Archive log space is full

·    DB or APPS file system is Full including /tmp location

·    Java Cache issues between the multiple middle-tiers

 

Here are different scenarios and work around/Fixes:

Scenario 1:  After the clone if any of the users experience user login issues other than super user with the below error message then please follow the steps in scenario 1.

The login in page may show Http-500 Internal server error (or) Http-404

1.         http 500 internal server error:

              java.lang.NoClassDefFoundError
              at oracle.apps.fnd.sso.AppsLoginRedirect.doGet(AppsLoginRedirect.java:623)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:743)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
              at com.evermind[Oracle Containers for J2EE 10g (10.1.3.0.0)].
              server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:64)
              at oracle.apps.jtf.base.session.ReleaseResFilter.doFilter(ReleaseResFilter.java:26)

Solution: (from Application Tier)


$cd $ADMIN_SCRIPTS_HOME

$adopmnctl.sh stopall — to shutdown opmn managed services viz., opmn, forms, oacore & oc4j 

$ cd $COMMON_TOP/_pages

$ mv _pages _pages_old

$mkdir _pages

$ cd $FND_TOP/patch/115/bin or export PATH=$FND_TOP/patch/115/bin:$PATH

Compile all jsps.

$perl ojspCompile.pl –compile –flush -p 2 (or) ojspCompile.pl --compile --fast --quiet

(-p represents parallel threads, if you have enough horse power on your server, you can increase it to 8 or 10)

$cd $ADMIN_SCRIPTS_HOME

adopmnctl.sh startall

OR:

ORA-01578: ORACLE data block corrupted (file # 23, block # 221085)
ORA-01110: data file 23: ‘/RDBMS ORACLE_HOME/app/a_txn_data02.dbf’
ORA-26040: Data block was loaded using the NOLOGGING option

Solution: Login as apps user

$ sqlplus apps/apps_pwd


set serveroutput on size 100000;

 declare

 begin

 WF_LOCAL_SYNCH.BULKSYNCHRONIZATION(

 P_ORIG_SYSTEM => 'ALL',

 P_PARALLEL_PROCESSES => null,

 P_LOGGING => null,

 P_RAISEERRORS => TRUE);

 exception

 when others then

 dbms_output.put_line('sqlerrm = ' || sqlerrm);

 end;

 /

Scenario 2: Database Listener is down

Solution:  Start the database listener.

Scenario 3: Database Archive Locaion is 100% full or any mount point or tmp location is full(including concurrent log and output storage mount point)

Solution: Either temporarily move the files/archivelogs to another location or delete older files/archives (provided you had taken a cold backup sometime).

Scenario 4: IAS Cache initialization failed (Doc ID 1484840.1)

E-Business Suite Applications Technology Stack,  HTTP / IAS Server related issues

Users report cache initialization errors in OA Framework related pages.

The oacorestd.err file shows: 

Exception in static block of jtf.cache.CacheManager. Stack trace is: oracle.apps.jtf.base.resources.FrameworkException: IAS Cache initialization failed. 

The Distributed Caching System failed to initialize on port: 12360. The list of hosts in the distributed caching system is: 10.24.248.41 . 

The port 12360 should be free on each host running the JVMs. 

Solution:

To resolve the issue test the following steps in a development instance and then migrate accordingly:

I. For Autoconfig enabled environment:
   1. Bring down opmn services on all boxes.
   2. Verify no process is listening for s_java_object_cache_port value:

  netstat -an | grep [port number] | grep LISTEN

    Identify the OS process associated with a port:

  lsof -i tcp:[port number]

   3. Change s_java_object_cache_port on all the nodes in $CONTEXT_FILE.

   4. Run autoconfig.

II. For Non-Autoconfig enabled environments:

  s_java_object_cache_port value is referrenced in two places

   1. $INST_TOP/ora/10.1.3/javacache/admin/javacache.xml

   2.  JTF_DIST_CACHE_PORT profile option.

   Change the port value to a new number in the $CONTEXT_FILE and bounce opmnctl services.

   3. Verify no process is listening for s_java_object_cache_port value : 

  netstat -an | grep [port number] | grep LISTEN

   Identify the OS process associated with a port

  lsof -i tcp:[port number]

If the newly assigned port still does not listen on the server you can use workaround given bellow. This workaround will resolve the “IAS Cache initialization failed” even if the port is not listening.

 Workaround: (Note: The below workaround is intended for testing purposes only to help identify RCA)

a. Set LONG_RUNNING_JVM=false in $INST_TOP/ora/10.1.3/j2ee/oacore/config/oc4j.properties.

b. Bounce opmn services.
Note: This happens when Distributed Java Caching is enabled in a multi middle tier environment

Note: In 11i, if the database listener is down or if the archive location is full, may get the internal server page while trying to access the login page. In R12 may get a blank page and apache error log will show the “oc4j_socket_recvfull timed out” and "mod_oc4j: request to OC4J [mid-tier servername:ajp port] failed: Connect failed" error messages.

Scenario 5: mod_oc4j: Failed to find a failover oc4j process for session request for destination

Error Noticed in the log file 

$LOG_HOME/ora/10.1.3/Apache

Solution: 

1. Noticed the When we have started the Application Services the load average was 1.

2. When executed ps -ef  |grep applmgr  noticed many processes.

3. Killed the process via 

ps -ef |grep applmgr|grep -v grep |awk '{print $2}'|xargs kill -9

Issue got resolved.

a. Started the services.

b. Via forms fixed restart and all the concurrent managers are up.

XML Publisher Reports End With Warnings - oracle.xdo.parser.v2.XPathException: Cannot convert to number

XML Publisher Reports End With Warnings -(Doc ID 1505402.1)

APPLIES TO:

BI Publisher (formerly XML Publisher) - Version 10.1.3.2.0 to 11.1.1.7.0 [Release 10.1 to 11.1]
Information in this document applies to any platform.

SYMPTOMS

On Oracle Applications 12.1.3 version, Reports for Applications Technology

When attempting to run some XML reports the following error occurs:

java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor46.invoke

Caused by: oracle.xdo.parser.v2.XPathException: Cannot convert to number.

The issue can be reproduced at will with the following steps:
1. Run XML reports.

The issue has the following business impact:
Due to this issue, users cannot run XML reports

CAUSE

The cause was from the template design or the data.

SOLUTION

The above error suggests there is a problem with either the template design or the data.
Somewhere there is text data that the code in the layout expects to be numerical.
Correct the template or fix the program that generates the XML data to make sure the data is always numeric.

 

Autoconfig Error: adgentns.pl exiting with status 2 after Clone/Refresh

Error: adgentns.pl exiting with status 2

[CVM Error Report]

The following report lists errors encountered during CVM Phase

/UATappl/apps/apps_st/appl/ad/12.0.0/bin/adgentns.pl  2

No of scripts failed in CVM phase: 1

AutoConfig is exiting with status 1 

Solution:

Open a new terminal and source the APPS-Tier Environment.

$sqlplus apps/apps_pwd

SQL> exec fnd_conc_clone.setup_clean

SQL>commit;

Open a new terminal and source the DB-Tier Environment.

Execute Autoconfig on the DB Tier.

Switch back to the APPS environment or start a new terminal and source the APPS-Tier environment.

Execute Autoconfig at the Apps Tier.

Check the Autoconfig log for any errors and ensure that the services start correctly.

R12 Apache Issue: error while loading shared libraries: libdb.so.2: cannot open shared object file: No such file or directory

Problem

Web server were not up while start the Application server

Installation Log File

You are running adapcctl.sh version 120.7.12010000.2

Starting OPMN managed Oracle HTTP Server (OHS) instance ...

opmnctl: opmn is already running.

opmnctl: starting opmn managed processes...

================================================================================

opmn id=erpuatappl.com.in:6215

    0 of 1 processes started.

ias-instance id=UAT_erpuatappl.erpuatappl.com.in

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

--------------------------------------------------------------------------------

ias-component/process-type/process-set:

    HTTP_Server/HTTP_Server/HTTP_Server/

Error

--> Process (index=1,uid=606170722,pid=28344)

    failed to start a managed process after the maximum retry limit

    Log:

    /UATappl/inst/apps/UAT_erpuatappl/logs/ora/10.1.3/opmn/HTTP_Server~1.log

adapcctl.sh: exiting with status 0

adapcctl.sh: check the logfile /UATappl/inst/apps/UAT_erpuatappl/logs/appl/admin/log/adapcctl.txt for more information ...  

Apache Log File

09/30/20-13:46:10 Start process

--------

/UATappl/inst/apps/UAT_erpuatappl/ora/10.1.3/Apache/Apache/bin/apachectl startssl: execing httpd

/UATappl/apps/tech_st/10.1.3/Apache/Apache/bin/httpd: error while loading shared libraries: libdb.so.2: cannot open shared object file: No such file or directory

Status of the Services

[applmgr@erpuatappl scripts]$ ./adopmnctl.sh status apps/apps

09/30/20-13:46:16 :: adopmnctl.sh version 120.6.12010000.5

09/30/20-13:46:16 :: adopmnctl.sh: Checking the status of Oracle Process Manager (OPMN)

Processes in Instance: UAT_erpuatappl.erpuatappl.com.in

---------------------------------+--------------------+---------+---------

ias-component                    | process-type       |     pid | status  

---------------------------------+--------------------+---------+---------

OC4JGroup:default_group          | OC4J:oafm          |    4503 | Alive   

OC4JGroup:default_group          | OC4J:forms         |    4434 | Alive   

OC4JGroup:default_group          | OC4J:oacore        |    4287 | Alive   

HTTP_Server                      | HTTP_Server        |     N/A | Down    

09/30/20-13:46:17 :: adopmnctl.sh: exiting with status 0

Cause

http web server unable to start due to missing library.

Solution

1. Shutdown all application services.

2. Create link for library file as below

$ ln -s /usr/lib/libgdbm.so.2.0.0 /usr/lib/libdb.so.2

OR

OS Library Patch for Oracle HTTP Server (on Oracle Linux 5 and RHEL 5 only)
Download and apply the patch 6078836 from My Oracle Support to fix an issue with the Oracle HTTP Server (missing libdb.so.2) bundled with the E-Business Suite technology stack.

Link to Berkeley DB library for HTTP server (on Oracle Linux 7 and RHEL 7 only)

$ cd /usr/lib

$ ln -s libdb-4.7.so libdb-4.3.so

Reference : Attempting to Start Apache (adapcctl.sh) Throws an 'Error While Loading Shared Libraries: libdb.so.2' on Linux (Doc ID 879522.1)

Monday 7 September 2020

Gather Schema Statistics "program error with Locks and Duplicate columns"

Oracle E-Business Suite Release 12 uses cost–based optimization in order to choose the most efficient execution plan for SQL statements.Gather Schema Statistics is  the favourite one among Apps DBA's and developers. Gather Schema Statistics program generates statistics that quantify the data distribution and storage characteristics of tables, columns, indexes, and partitions.

On Oracle Applications 12.1.3,
Gather schema statistics (GSS) is completing with error in our instances. 

When attempting to run Gather Schema Statistics request, the following error occurs:

Error

*******

**Starts**13-APR-2019 23:59:45

**Ends**14-APR-2019 05:30:40

ORA-0000: normal, successful completion

+---------------------------------------------------------------------------+

Start of log messages from FND_FILE

+---------------------------------------------------------------------------+

In GATHER_SCHEMA_STATS , schema_name= ALL percent= 10 degree = 24 internal_flag= NOBACKUP

stats on table AQ$_WF_CONTROL_P is locked

stats on table FND_CP_GSM_IPC_AQTBL is locked

stats on table FND_SOA_JMS_IN is locked

stats on table FND_SOA_JMS_OUT is locked

Error #1: ERROR: While GATHER_TABLE_STATS:

object_name=GL.JE_BE_LINE_TYPE_MAP***ORA-20001: invalid column name or duplicate columns/column groups/expressions in method_opt***

Error #2: ERROR: While GATHER_TABLE_STATS:

object_name=GL.JE_BE_LOGS***ORA-20001: invalid column name or duplicate columns/column groups/expressions in method_opt***

Error #3: ERROR: While GATHER_TABLE_STATS:

object_name=GL.JE_BE_VAT_REP_RULES***ORA-20001: invalid column name or duplicate columns/column groups/expressions in method_opt***

+---------------------------------------------------------------------------+

End of log messages from FND_FILE

+---------------------------------------------------------------------------+

Successfully resubmitted concurrent program FNDGSCST with request ID  4981147 to start at 21-APR-2019 00:00:00 (ROUTINE=AFPSRS)

Solution

***********

1. Please follow Doc ID 375351.1 to unlock the tables:

                AQ$_WF_CONTROL_P
                FND_CP_GSM_IPC_AQTBL
                FND_SOA_JMS_IN
                FND_SOA_JMS_OUT

Do the following select from dba_tab_stats to show all tables with locked statistics on them:

    SQL> select owner, table_name, stattype_locked from dba_tab_statistics 
           where stattype_locked is not null 
           and owner not in ('SYS');          

From DB node run below to unlock tables

        SQL>exec dbms_stats.unlock_table_stats('APPLSYS','AQ$_WF_CONTROL_P');
        SQL>exec dbms_stats.unlock_table_stats('APPLSYS','FND_CP_GSM_IPC_AQTBL');
        SQL>exec dbms_stats.unlock_table_stats('APPLSYS','FND_SOA_JMS_IN');
        SQL>exec dbms_stats.unlock_table_stats('APPLSYS','FND_SOA_JMS_OUT');

2. Please perform the following action plan from Doc ID 781813.1 :

   Find out all duplicates and/or obsolete rows in FND_HISTOGRAM_COLS and delete one of them.

   Remember to take backup of the FND_HISTOGRAM_COLS table before deleting any data.

   SQL> create table FND_HISTOGRAM_COLS_BKP as select * from FND_HISTOGRAM_COLS;

 -- Identify duplicate rows

  SQL>select table_name, column_name, count(*) from FND_HISTOGRAM_COLS

             group by table_name, column_name 

             having count(*) > 1;

 -- Use above results on the following SQL to delete duplicates

 SQL>delete from FND_HISTOGRAM_COLS

           where table_name = '&TABLE_NAME'

           and column_name = '&COLUMN_NAME'

           and rownum=1;

 -- Use following SQL to delete obsoleted rows

  SQL>delete from FND_HISTOGRAM_COLS

            where (table_name, column_name) in

                  (

            select hc.table_name, hc.column_name

             from FND_HISTOGRAM_COLS hc , dba_tab_columns tc

            where hc.table_name ='&TABLE_NAME'

            and hc.table_name= tc.table_name (+)

            and hc.column_name = tc.column_name (+)

            and tc.column_name is null

                  );


   SQL>commit;

3. Run "Gather Schema Statistics" program again.

    Fix worked successfully for our env.

[Solved]How to open o123.out file in excel format

An *.out file is a compiled executable file created by various source code compilers in Unix-like operating systems, such as Linux and AIX. *.out files are most commonly seen as a.out files. The name stands for "assembler output". *.out files may also be used to collect data that has been dumped from a program to standard output.

Our goal is to help you understand what a file with a *.out suffix is and how to open it.

We can follow below steps to open *.out files:

Step 1: First of all change the o123.out file extension to o123.xml extension.

Step 2: Now change the First line in o123.xml extension file.

Example: <?xml version="1.0" encoding="UTF-8"?>    replace from

                  <?xml version="1.0" encoding="WINDOWS-1252"?>

Step 3: Load this o123.xml file in RTF Template if getting any Out of Memory error in RTF Template then doing the required changes the Java Heap size in RTF Template( Now change –Xmx256M to -Xmx1500M).

Go To=> Add-Ins=>Tools=>Options

Click on OK.

Step 4: Again load this o123.xml file in RTF Template.

Step 5: View output in as you want.

Friday 4 September 2020

How to Create a Custom Concurrent Manager

Adding Concurrent Manager is a crucial decision. Lots of things needs to be examined before adding new Concurrent Manager. One of the major area is resources.If you have less number of concurrent managers then overall concurrent processing gets delayed with a lot many requests having the status pending.On the other hand if you have too many Concurrent Managers then there will be a heavy load on your Operating System.So before adding any new Concurrent Managers a proper analysis needs to be done.

General Steps for creating new custom concurrent manager in Oracle App R12

1. Navigate to Concurrent / Manager / Define.

2. Manager Field: Custom Manager.

3. Short Name: CUSTOM.

4. Type: Concurrent Manager.

5. Program Library: FNDLIBR.

6. Enter desired cache.

7. Work Shifts: Standard.

8. Enter number of Processes.

9. Provide Specialization Rules(you can include or exclude program,id,user,types or combination).

10. Save.

11. Navigate to Concurrent / Manager / Administer.

12. Activate the Custom Manager.

Why need to create a custom concurrent manager?

It is avoid to long running concurrent request during business hours.this improves overall performance of the system.create a Specialized Manager to process the specific Long Running requests. assign the Specialized Manager with a work shift with Off peak time.


Assume, the Users submit the Concurrent Requests during peak time, then it will get assigned to the Specialized Manager.But Specialized Manager will not process it during peak time.

You can define as any concurrent managers as you want. For defining Concurrent Manager we need to login to the e-business suite as a 'SYSADMIN' and need to choose the System Administrator responsibility.

Follow below steps for create custom concurrent manager 

1. Navigate to Concurrent==>Manager==>Define

2. Manager Field: XXX Heavy Report Manager (Custom Manager).

3. Short Name: CUSTOM.

4. Type: Concurrent Manager.

5. Program Library: FNDLIBR.

6. Enter desired cache.

Go To Work Shifts: Choose Standard.

7. Enter number of Processes: 5

8. Sleep Seconds: 30

Define Request Type

9. Navigate to Concurrent==>Program==>Type

10. Provide Specialization Rules:

11. Save

12. Exclude the Request Type from Standard Manager’s Specialization Rules.

13. Navigate to Concurrent==>Manager==>Administer.

14. Activate the Custom Manager.