RT_SCTEMP Error : Not able to run Datastage Jobs

RT_SCTEMP/jobName.fifo Problem and Solution:DK®

Problem in Details :DK®

DataStage jobs fail with error message:
Message: Error setting up internal communications (fifo RT_SCTEMP/jobName.fifo)

Solution of the problem :DK®

The following error occurs when DataStage is unable to create, delete, read, or write a temporary fifo file for a job to the RT_SCTEMP directory within the project owning the job.
Error setting up internal communications (fifo RT_SCTEMP/jobName.fifo)

The above error has two primary causes:

1. Unable to process file due to locks.
2. Unable to process file due to file permissions or other file system problems.
3. Virus Scan or backup program interferes with writing fifo files to temp or scratch directories as discussed in technote:
   https://www-304.ibm.com/support/docview.wss?uid=swg21445893&wv=1

If the failure is due to locks, then the above error should have additional text at the end or in subsequent message stating the lock status, for example:
Error setting up internal communications 
(fifo RT_SCTEMP/MyTestJob.fifo LOCKED STATUS () -1); file is locked .

In that situation, refer to the following technote for instructions on how to clear locks for a job:
https://www-304.ibm.com/support/docview.wss?uid=swg21438482

If the issue is not due to locks, then review the following checklist to resolve other common causes for this error:

1. Check the file limits at job runtime, especially if all jobs run under a common userid such as DSADM. You can check the limits used at DataStage job run-time even if you cannot run jobs, by running the command through the DataStage Administrator client. Login to DataStage Administrator Client, select the failing project, click the COMMAND button, and then enter command:
   sh ulimit -a
   
   If the number is under 2048, consider increasing it. On busy systems it may need to be higher. In this situation, you can add command to set the limit to $DSHOME/dsenv script such as:
   ulimit -n 10240
   
   After making the change you will need to stop and restart DataStage and then perform above test again to ensure new limit is in effect.
   
2. Check available space on volume containing the RT_SCTEMP directory. If the project containing failing jobs is named "MyProject", then the path to RT_SCTEMP will be similar to:
   /opt/IBM/InformationServer/Server/Projects/MyProject/RT_SCTEMP
   
3. Check permissions for the RT_SCTEMP directory and the files inside it. Ensure that the userid running jobs (which should be listed on event messages in job log) has read and write permission to the directory and the files within it either via ownerid, group membership, or public permissions.
   
   A quick test to confirm if permissions are the problem is to set directory permissions temporarily to 777 so that all users can write to it.
   
   
4. Confirm that DSADM, or whichever userid runs the failing jobs, can create a file in this directory using the following steps:
   -- Login to server operating system as the userid who runs the failing DataStage jobs.
   -- Change directory to InformationServer/Server/Projects/projectname/RT_SCTEMP directory.
   -- Enter command: touch test.fifo
   -- If the above command fails, then the userid is unable to create a file at that location and that issue must be resolved before DataStage jobs can run correctly.

Ultimately, if this issue is not due to locks, then the DataStage error occurs due to an inability to correctly create, read, write, or delete the temporary fifo files. If the above tests to not isolate the cause of file system i/o problem, then it may be necessary to contact Information Server support for assistance in performing a system trace (truss or strace) of the dsapi process launching the failing jobs to track down the actual OS operations which are failing.