Known Issues on the HTC

This page documents some common and known issues encountered on the HTC system. While this page can be beneficial in troubleshooting, it does not contain a comprehensive list of errors.

Visit our Get Help page to find more resources for troubleshooting.

Table of Contents

[General] When submitting a job, it doesn't run / goes on hold and shows the error "Job credentials are not available".

Cause:

This is a complicated bug that can strike randomly. We’re working on a fix.

Solution:

To work around this issue, run the following command on the access point before resubmitting the job.

echo | condor_store_cred add-oauth -s scitokens -i -

[General] My job exits with an "Illegal instructions" error.

Cause:

The instructions for communicating with the processor does not match between software and hardware.

Solution:

In the submit file, try adding one of the following to your requirements line:

requirements = (has_avx || has_avx2)
requirements = has_avx2
requirements = (Microarch > x86-64-v3)

Then resubmit and check if this resolves the issue.

[General] I used generative AI to create my submit file and the job is stuck on "Idle".

Cause:

Generative AI (such as Copilot, ChatGPT, etc.) may be inaccurate. In our experience, generative AI almost always gets something wrong when it comes to an HTCondor submit file. This is because HTCondor has significantly changed since its creation over 40 years ago, and is highly configurable. This means that while there is a lot of information online in forums and site-specific documentation, the information is not always relevant to our local HTC system. As such, submit files created by generative AI are usually not applicable to our HTC system.

Solution:

The CHTC website provides documentation - including example submit files - for how to submit jobs on the HTC system.

[Container] When building an Apptainer, "apt" commands in the %post block fail to run.

Example error message:

Couldn't create temporary file /tmp/apt.conf.9vQdLs for passing config to apt-key

Cause:

The container needs global read/write permissions in order to update or install packages using the apt command.

Solution:

Add chmod 777 /tmp to the front of your %post block. See the example below:

Bootstrap: docker
From: ubuntu:22.04

%post
    chmod 777 /tmp
    apt-get update -y

We also recommend using the -y option to prevent installation from hanging due to interactive prompts.

[Container] When attempting to run a Docker container, it fails with the error message "[FATAL tini (7)] exec ./myExecutable.sh failed: Exec format error".

Cause:

The Docker container is likely built on an Apple computer using an ARM processor, which is incompatible with Linux machines.

Solution:

To resolve this, when building your Docker container, use the command:

docker build --platform linux/amd64 .

[Container] My interactive Apptainer job is failing with the error message, "Can't open master pty Bad file descriptor".

Cause:

Interactive Apptainer jobs are incompatible with CentOS7 machines.

Solution:

To your submit file, add the following line and resubmit:

requirements = (OpSysMajorVer > 7)

[GPU] My GPU job has been in the queue for a long period of time and is not starting.

Cause:

To use our shared use GPU machines, you must opt into the GPU Lab.

Solution:

To your submit file, add the following line and resubmit:

+WantGPULab = true


[General, Python] My job has an error that mentions `/home/netid/` or the root directory `/`.

Cause:

Sometimes programs assume that they can write into the /home or root / directory. However, these directories are not writeable on the Execution Points.

Solution:

To your executable (.sh script), add the following line and resubmit:

export HOME=$PWD

This will change the HOME environment variable to the working directory on the Execution Point.

For any items referencing the root directory /, try the solution suggested above. If that does not fix the issue, you will need to find where your code or program is referencing the root directory and change it to the current working directory. Please email us or drop by office hours if you need assistance with this.


Can’t find your issue?

Visit our Get Help page.

HTC guides