Troubleshooting Automation

Isolate the behavior of each component in the system. In particular, determine if the following components can be ruled out as causing of the problem:

•

The script or program—Running the custom logic, REST calls, and file handling.

•

The AI nodes/automation workers—Calling the command line and invoking the script or program.

•

The network—Providing reliable and timely TCP/IP packet transfers.

•

The client—Completing the process/step and notifying the server.

•

The server—Responding to client notifications and dispatching to AI nodes/automation workers.

The script provides many options for troubleshooting. For example, increase logging to rule out unexpected behavior.

•

Printing to stderr in the script writes a line to the Error Handling automatedinformatics.log file. This file is a great source of information.

•

The records in the log file allow for emulating the command-line call for unit testing the script, and calling it manually on the command-line prompt (see Review Automated Informatics Log File).

•

Validate AI Node/Automation Workerhelps verify the automation program on the AI node/automation worker. If the script and the AI node/automation worker are functioning, review the log file entries for any warning (WARN) or error (ERR) lines near the time-stamp of the process completion event sent from the client.

•

If the issue is related to the client or server software, contact the Illumina Support team, providing:

–

The automatedinformatics.log file

–

The server log

–

The results of the isolation tests (in the previous section).

Use the following steps to test and verify the setup.

Test and Verify Setup on a Windows AI Node/Automation Worker

1.

Create a process/step that generates a result file.

2.

Configure an automation on the process/step. Associate it with the channel on which the AI node/automation worker is configured to communicate.

3.

Add the following command line string.

cmd /c "C:\ai\ai.bat {outputFile0}"

4.

On the AI node/automation worker machine, create an ai folder in C:\ so the system has an C:\ai path. Create a new file named ai.bat.

5.

Edit the ai.bat file and add the following line:

echo Data for Output File LIMS ID %1 > %1.txt

6.

Run the process/step created on an existing attached result file.

•

The step passes the LIMS ID of its output file placeholder to the script.

•

The script creates a file in the working directory.

•

When the script exits, this file transfers to the LIMS and associated with the step. The file contains a single line of text that includes the LIMS ID of the output file for easy verification.

Test and Verify Setup on a Linux AI Node/Automation Worker

1.

Create a process/step that generates a result file.

2.

Configure an automation on the process/step. Associate it with the channel on which the AI node/automation worker is configured to communicate.

3.

Add the following command line string:

bash -c "echo Automation Test > {outputFile0}.txt"

4.

Run the process/step created on an existing attached result file.

•

The step passes the LIMS ID of its output file placeholder to the script.

•

The script creates a file in the working directory with a file name that contains this LIMS ID.

•

When the script exits, this file transfers to the LIMS and associated with the step. The file contains the text "Automation Test". When open, the file opens in the default program associated with *.txt files.

With automation, the command-line information is important. The actual values sent on the command line are recorded in the AI log file as the AI node/automation worker receives them. Copy the parameters/tokens from the log and use them on the command line to troubleshoot. If scripting in Groovy, the cli class handles command-line tokens well. See the example *.groovy files used with automation and the utility class section in Work with EPP/Automation and Files.

AI nodes/automation workers are installed using the Automated Informatics (Automation Worker for LIMS v5 and later) software package.

In the installation directory:

1.

Find the /log directory, which contains an automatedinformatics.log file.

2.

Use this file to locate log lines near the time of the process/step completion event.

a.

Locate the log line containing the parameter/token command string, and manually run and test the script.

b.

Locate the working directory to review temporary files created.

Step 1: Locate log lines near time of the process completion event

This step can indicate if the cause of the error lies with a network or other issue external to the computer running the AI node / automation worker.

•

If no records are found, use the Validate AI Node/Automation Worker procedure to confirm that logging is functional.

Step 2: Locate log line containing parameter/token command string to manually run/test script

Running the script manually forms a unit test. The script is run on the command line, without being invoked by automation.

To locate the line containing the command-line string, search for the Command string, or externalprogram.runExternalProgram.

An example is shown in the following abridged log file section. Copy the line to a text file and modify it for script testing.

2014-02-28 21:24:48,688 INFO ... definitions.behaviour.automatedinformatics.plugins.externalprogram.runExternalProham as ...

2014-02-28 21:24:49,323 INFO ... (ExternalProgramBehaviour.java:133)... Command string: bash -c "~/scripts/HelloWorld.sh http://###.###.###.###/api/v2/processes/A30-MXX-110228-24-2320 > 92-2869.txt"

Step 3: Locate working directory to review temporary files created

Used when temporary files are left by the script, this automation script removes the temporary working directory, unless there was an error. In case of an error, the directory provides clues to the root cause of the error.

To locate the working directory, search for "Working directory:" in the log file.

An example is shown in the following abridged log file section. Use the recorded directory to list and review temporary files.

2014-02-28 21:24:49,324 INFO ... Working directory: /home/gls/GenoLogicsAutomatedInformatics/temp/runExternalProgram-28022014-432350866632555775.A30-MXX-110228-24-2320

2014-02-28 21:24:49,324 INFO ... Retrieved files. Executing command.