Data access using DQ2

Page under construction

Until further notice it is recommended to run dq2 commands at CERN on machines with SL3

>   ssh lxslc3

Selected references

DQ2 clients -- documentation: installation, end user tools, command line utilities, FAQ
DQ2 clients -- URL for documentation, current releases
dq2 commands -- full description of commands (old page)
Athena using DQ2 -- Analysis of CSC samples using DQ2 (experts)
DDMEndUserTutorial -- very useful tutorial with exercises and solutions, demonstrates the use of the commands listed below

The following are summaries.

Reporting problems

Set up of environment

Environment for LCG user interface at CERN/lxplus, and VOMS-enabled grid proxy, and DQ2 user tools

 >   source /afs/cern.ch/project/gd/LCG-share/current/etc/profile.d/grid_env.sh  [.csh]
			   >   voms-proxy-init --voms=atlas 
			   >   source /afs/cern.ch/atlas/offline/external/GRID/ddm/DQ2Clients/setup.sh [.csh]

If dq2 commands do not work from a CERN computer and you have access to BNL machines, try there; get a fresh shell first, e.g. logout and login again.

On acas nodes:
	 >   source /afs/usatlas.bnl.gov/lcg/current/etc/profile.d/grid_env.sh [.csh]
   On gridui01:
	 >   source /etc/glite/glite.sh [.csh]
   then independent of node:
	 >   grid-proxy-init    
	 >   source /afs/usatlas.bnl.gov/Grid/Don-Quijote/dq2_user_client/setup.sh.BNL [.csh.BNL]

Summary of commands

dq2_xyz commands are executed as soon as possible and are therefore "stressing" the storage system. It is preferable to use the DDM system, but as long as this is not fully functional, use dq2_, but use it with care.

To get the full description of the command, click on the command. The complete list of commands is part of the TWiki page "UsingDQ2". The "\" in some of the commands indicates "concatenation" of datafiles/sets and is compulsory (true? TBD).

dq2_ls [-g] dataset ["condition1" "condition2"]: -g list the file content of the dataset, use wildcard character "*"
-f list the file content of the dataset
-fp list the addr in this site and in DDM
example for condition: "Total>10" means, list only datasets with more than 10 files
dq2_get [-rva] dataset file1 file2: -r copy via Grid, without -r the copy uses only the local storage
-a scan all remote sites to find replicas -v verbose
dq2_sample dataset_name number_of_files: copy a portion of an existing dataset and registers it to DQ2;
the sample dataset name is generated automatically
dq2_put [-d directory] dataset file1 file2: copy dataset or files to directory
-d target directory
dq2_cleanup [-l] [-t timeout] dataset file1 file2: delete a dataset from a site's catalog and storage
-l list the files to be deleted, do not delete them -t timeout in seconds per file (on LCG only)
dq2_register [-q] dataset site: register dataset or files, upload local files to the Gris storage; see the details of this command (location of script, requirements on Java, ...)
-q quiet, suppress logging output
dq2_poolFCjobO [-p] [-j] dataset: Create PoolFileCatalog.xml and/or files_jobO.py
-p Pool file catalog
-j Athena job options fragment

Data access using DQ2

Selected references

Reporting problems

Set up of environment

Summary of commands

See also