User Guidebook – CMS Caltech Tier2

Credentials

If you are not familiar on how to create an ssh key, from a remote client (your laptop) run the following command

ssh-keygen -t rsa -b 4096 -C "your_email@example.com"

and place the content of the public key in the ~/.ssh/authorize_keys on any machine (As it is shared file system, same files are available on all machines)

Nodes and Access

Caltech Tier2 and iBanks GPU Cluster use share CEPH Shared Filesystem and everyone has it’s own home directory on CEPH (default home directory for all users).

All servers have a public (regular network) and a private (10G) IP. An SSH key is the only authentication (Administrators use 2FA). Please let the admins know (hep-wheel AT caltech.edu) in case of issues.

login-1.hep.caltech.edu and login-2.hep.caltech.edu – can be used to access Caltech Tier2 and GPU Clusters. Be aware that login nodes do not have any GPUs attached. (There is an option to use GPU HTcondor Scheduling, but it is WIP.)

Data Storage on GPU Nodes

The home directory should be used for software and although there is room, please prevent putting too much data within your home directory.

The /wntmp/ volume is mounted on all nodes, not all on SSD. This is the preferred temporary location for HTCondor jobs.

The /data/ volume is mounted on some GPU nodes, not all on SSD. This is the preferred temporary location for data needed for intensive I/O.

The /imdata/ volume on GPU Nodes is a ramdisk of 40G with very high throughput, but utilizing the RAM of the machine. Please use this in case of need of very high i/o, but clean the space tightly, as this will use the node memory. There is a 2-day-since-last-access retention policy on it.

The /mnt/hadoop/ path is the read-only access to the full Caltech Tier2 Hadoop Storage.

The /storage/group/gpu/shared path is CEPH volume.

The /storage/user/${USER} path is your home directory.

Grid proxy

Before working on the Tier2, make sure you have a valid grid proxy, also known as the VOMS proxy or X509 proxy. The first time you log in, you have to copy the following files from lxplus or via the browser as described here: https://twiki.cern.ch/twiki/bin/view/CMSPublic/SWGuideRunningGridPrerequisites#Grid_certificate

$ ls -al ~/.globus
-rw-r--r--. 1 user user 3766 Sep 24 2018 mycert.p12
-r--------. 1 user user 3307 Sep 24 2018 usercert.pem
-r--------. 1 user user 1945 Sep 24 2018 userkey.pem

You can chech that the proxy is valid using the following command

$ voms-proxy-info
...
path      : /tmp/x509up_p3526184.filek30oxu.1 <- this can be different, depending on how you have logged in to Tier2 or configured the X509_USER_PROXY variable
timeleft  : 16:28:33

Accessing bulk CMS storage

Once you have a valid proxy, use the gfal-ls, gfal-rm, gfal-copy tools to access the large-scale Hadoop/HDFS storage. This should be used to store multi-terabyte datasets for longer-term and for interfacing with the CMS grid.

$ gfal-ls -l gsiftp://transfer.ultralight.org//store/user/
drwxr-xr-x   1 0     11606      4096 Jun 20  2010 user1
drwxrwxr-x   1 0     12014      4096 Jun 13  2011 user2
drwxr-xr-x   1 0     11774      4096 May  4  2012 user3

Accessing T2 fast shared storage

Smaller files that need to be accessed often can be stored at /storage/user/$USER/, this is a shared filesystem that is read/write accessible from all interactive and worker nodes. Generally, it can be treated as a fast local disk.

Setup CMSSW

Works the same as on lxplus:

$ source /cvmfs/cms.cern.ch/cmsset_default.sh 
$ cmsrel CMSSW_7_6_6
$ cd CMSSW_7_6_6/src/
$ cmsenv

Singularity

If your CMSSW release needs SL6, you should run it inside singularity. singularity shell -B /cvmfs /cvmfs/singularity.opensciencegrid.org/bbockelm/cms:rhel6export SCRAM_ARCH=slc6_amd64_gcc700 source /cvmfs/cms.cern.ch/cmsset_default.sh

then cmsrel as usual to get the release.

Submitting HTCondor jobs

1. HTCondor script jobs (batch):

In order to submit a condor job, you need a JDL (job description language) file and an executable shell script. JDL Example:

Universe = vanilla
Executable = example_job.sh
Arguments = 10

#The logs directory must exist before submitting job.
Log = logs/example_job.$(Cluster).log
Output = logs/example_job.out.$(Cluster).$(Process)
Error = logs/example_job.err.$(Cluster).$(Process)

Requirements=(TARGET.OpSysAndVer=="CentOS7" && regexp("blade-.*", TARGET.Machine))

#This is necessary to choose either rhel8(slc8)/rhel7 (slc7)/rhel6 (slc6) as needed
+RunAsOwner = True
+InteractiveUser = true
+SingularityImage = "REPLACE_ME"
# REPLACE_ME with a path to your own singularity image or to cms based images.
# Latest CMS Singularity images are available here:
# /cvmfs/singularity.opensciencegrid.org/cmssw/cms:rhel6 - latest RHEL 6 image
# /cvmfs/singularity.opensciencegrid.org/cmssw/cms:rhel7 - latest RHEL 7 image
# /cvmfs/singularity.opensciencegrid.org/cmssw/cms:rhel8 - latest RHEL 8 image.

+SingularityBindCVMFS = True
run_as_owner = True

#Provide information on proxy in order to access storage
x509userproxy = $ENV(X509_USER_PROXY)

#Don't request more than needed, otherwise, your job will wait longer in queue
RequestDisk = 4
RequestMemory = 2000
RequestCpus = 1

#transfer this file back to the login node
#use this for small files, like plots or txt files to an existing output directory on login-1
#Big outputs should be transferred within the job to /mnt/hadoop using `gfal-copy`
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
transfer_output_files = output_small.txt
transfer_input_files = my_calibrations.txt
transfer_output_remaps = "output_small.txt=outputs/output_small.txt.$(Cluster).$(Process)"

#This number can be used to queue more than one job
Queue 1

and example_job.sh script file:

#!/bin/sh

#Print out all bash commands
set -x

#Abort bash script on any error
set -e

#Print some basic debugging info
echo "whoami="`whoami`
echo "pwd="`pwd`
echo "hostname="`hostname`
echo "date="`date`
env

#print out proxy
voms-proxy-info -all

#Inside singularity, the scratch directory is here
#This is also where the job starts out
echo "TMP:" `df -h $TMP`
echo "looking inside scratch directory BEFORE job"
ls -al $TMP

#This will fail, cannot write to home directory (/data/...) from jobs
#echo "test" > /data/$USER/testfile.txt

#This will fail, cannot write to storage (/mnt/hadoop/...) from jobs
#echo "test" > /mnt/hadoop/store/user/$USER/testfile.txt

#Run cmsenv in an existing CMSSW directory on login-1
cd /storage/user/user/CMSSW_10_2_0/src
source /cvmfs/cms.cern.ch/cmsset_default.sh
eval `scramv1 runtime -sh`

#go back to scratch directory on worker node
cd $TMP

#your transfer_input_files are located in the working directory
cat my_calibrations.txt

#Run some ROOT code, produce output
echo "my job output datacard or plot" > $TMP/output_small.txt
echo "this is a placeholder for a big output file" > $TMP/output_big.dat

#Return to non-CMSSW environment, which is required for gfal-copy
eval `scram unsetenv -sh`

#Output can be copied using `gfal-copy` to /mnt/hadoop (large files only) or handled by condor using `transfer_output_files = output.txt` (for small files only)
#do NOT copy more than one file per job to hadoop - instead create compressed archives in case you need to produce multiple outputs per job
gfal-copy -f --checksum-mode=both file://$TMP/output_big.dat gsiftp://transfer.ultralight.org//store/user/user/output_big.dat

echo "looking inside scratch directory AFTER job"
ls -al $TMP

2. HTCondor interactive jobs:

For interactive jobs (either singularity or bare metal job), you need to use condor_submit -i <yourjdlfile> . Make sure you commented out ‘Executable’ parameter inside the JDL.

3. HTCondor GPU jobs:

To request GPU assigned to your job, you need to set 2 parameters inside your JDL:

+InteractiveGPUUser = true
Request_gpus = 1

This will ensure that matching looks and matches your jobs to machines which have GPUs. Once your job starts to run, it will have environment variable exported with specific GPU assigned to job, like:

CUDA_VISIBLE_DEVICES=0
_CONDOR_AssignedGPUs=CUDA0

Please ensure that these variables are used in your scripts.

Condor remarks:

Submit the job as follows:

$ condor_submit example_job.jdl
Submitting job(s).
1 job(s) submitted to cluster 4.

$ condor_q
-- Schedd: login-1.tier2 : <10.3.10.4:9618?... @ 03/27/18 09:13:17
OWNER BATCH_NAME SUBMITTED DONE RUN IDLE TOTAL JOB_IDS
user CMD: /storage/user/user/sleep.sh 3/27 09:12 _ _ 1 1 4.0

$ condor_history
-- Schedd: login-1.tier2 : <10.3.10.4:9618?... @ 03/27/18 09:16:13
ID OWNER SUBMITTED RUN_TIME ST COMPLETED CMD
4.0 user 3/27 09:14 0+00:00:04 C 3/27 09:14 /storage/user/user/sleep.sh 1

To check to status of the job:

$ condor_q -bet <JOB-ID>

Things to remember when submitting local condor jobs:

The software environment is provided by Singularity, you need to specify it in the job description using +SingularityImage = "/cvmfs/singularity.opensciencegrid.org/bbockelm/cms:rhel7" (to mimic lxplus7)
You can ask to run on specific machines using the following job description expression: Requirements=(TARGET.OpSysAndVer=="CentOS7" && regexp("blade-.*", TARGET.Machine))
In order to submit jobs successfully, the following needs to be defined in the job description: x509userproxy = $ENV(X509_USER_PROXY), and the corresponding environment variable must be defined.
It’s good practice to keep your jobs between 10 minutes and 1-2h in length

More generic information on condor batch jobs can be found on the lxbatch documentation: http://batchdocs.web.cern.ch/batchdocs/local/quick.html

More condor examples are here: http://research.cs.wisc.edu/htcondor/manual/v8.7/2_5Submitting_Job.html

Debugging jobs while running

Any running job allows user to ssh to it and use any debugging if needed. e.g.

$ condor_ssh_to_job 594650.0
Welcome to slot2@blade-1.tier2!
Your condor job is running with pid(s) 24522.
$ pwd
/wntmp/condor/execute/dir_24509

Make sure to log out of the job once you are done debugging.

Access on the Tier2

Login to lxplus.cern.ch or login-1; create your own proxy with voms-proxy-init -voms cms. For any command below, you always need to have a valid proxy. Additionally, keep in mind that the gfal-tools are not compatible with the CMSSW / cmsenv environment, therefore it needs to be unset: eval `scram unsetenv -sh` before using gfal-tools.

To list a specific directory (e.g. bottom list all /store/user/ directory and all users):

gfal-ls -l gsiftp://transfer.ultralight.org//store/user/

To remove a specific file from T2_US_Caltech:

gfal-rm gsiftp://transfer.ultralight.org//store/user/jbalcas/my_file/from_my_analysis/file_name.root

You can also use rm -r to remove everything recursively (Be careful and double check full path what you delete)

gfal-rm -r gsiftp://transfer.ultralight.org//store/user/jbalcas/my_file/

To copy a file from T2_US_Caltech to your working directory:

gfal-copy gsiftp://transfer.ultralight.org//store/user/jbalcas/a.test a.test

To copy file from your working directory to T2_US_Caltech storage:

gfal-copy a.test gsiftp://transfer.ultralight.org//store/user/jbalcas/a-new.test