Data Management
From EUAGwiki
Contents |
[edit] Lectured by
Giuseppe LA ROCCA
Italian National Institute of Nuclear Physics
Italy
mailto:giuseppe.larocca@ct.infn.it
[edit] About this page
The aim of this tutorial is to introduce file management in gLite.
This is achieved by running two sets of commands:
- "lfc-" commands interact with the LFC File Catalog server that maps logical filenames to "site URLs"
- "lcg-" commands include ones used to copy files to and from an SE, and to replicate files.
The lcg-commands performs also interaction with both the SEs and the catalogue server.
[edit] Environment variables
Several environment variables need to be set before you start to ensure that the correct catalog service is used. The default settings for these variables in your account should be correct, however this needs to be checked. The variables which need to be checked are:
$ echo $LCG_GFAL_INFOSYS bdii.grid.sinica.edu.tw:2170 $ echo $LCG_CATALOG_TYPE lfc $ echo $LFC_HOST lfc.grid.sinica.edu.tw
If one or more of them has different or empty value, please set it (them) in this way:
$ export LCG_GFAL_INFOSYS=bdii.grid.sinica.edu.tw:2170 $ export LCG_CATALOG_TYPE=lfc $ export LFC_HOST=lfc.grid.sinica.edu.tw
Remember to initialize your proxy certificate as follow:
$ voms-proxy-init --voms euasia Cannot find file or dir: /home/larocca/.glite/vomses Enter GRID pass phrase: Your identity: /C=IT/O=INFN/OU=Personal Certificate/L=Catania/CN=Giuseppe La Rocca Creating temporary proxy ......................................................................................................... Done Contacting voms.grid.sinica.edu.tw:15015 [/C=TW/O=AS/OU=GRID/CN=voms.grid.sinica.edu.tw] "euasia" Done Creating proxy ..................................................................... Done Your proxy is valid until Fri Jul 17 20:30:19 2009 $ voms-proxy-info --all subject : /C=IT/O=INFN/OU=Personal Certificate/L=Catania/CN=Giuseppe La Rocca/CN=proxy issuer : /C=IT/O=INFN/OU=Personal Certificate/L=Catania/CN=Giuseppe La Rocca identity : /C=IT/O=INFN/OU=Personal Certificate/L=Catania/CN=Giuseppe La Rocca type : proxy strength : 1024 bits path : /tmp/x509up_u560 timeleft : 11:58:02 === VO euasia extension information === VO : euasia subject : /C=IT/O=INFN/OU=Personal Certificate/L=Catania/CN=Giuseppe La Rocca issuer : /C=TW/O=AS/OU=GRID/CN=voms.grid.sinica.edu.tw attribute : /euasia/Role=NULL/Capability=NULL timeleft : 11:54:54 uri : voms.grid.sinica.edu.tw:15015
Now, you are ready to start.
[edit] LFC and LCG commands
[edit] Listing file and directory
For each of the supported VO a separate "top level" directory exists under the "/grid" directory. You can see all the files that are stored for the euasia VO. First ensure you have a running VOMS proxy and then type:
$ lfc-ls -l /grid/euasia drwxr-xr-x 0 248 121 0 Jul 09 03:58 Bazli drwxr-xr-x 2 157 121 0 May 30 14:32 charon-admin drwxr-xr-x 1 157 121 0 May 30 14:52 euasia006 drwx------ 1 157 121 0 May 30 14:37 euasia008 drwxrwxr-x 4 132 121 0 Jul 01 10:42 farhan drwxrwxr-x 4 106 121 0 Jul 14 17:11 generated drwxrwxr-x 1 157 121 0 May 30 14:21 kmunicek [..]
You will see a listing of the contents of the /grid/euasia directory.
Rather than having to type an absolute path for every file and directory you use, it is instead possible to define a HOME directory from which you may use relative file/directory paths. Set this now by setting the environment variable LFC_HOME as follows:
E.g.: $ export LFC_HOME=/grid/euasia/generated $ lfc-ls 2009-04-30 2009-06-29 2009-06-30 2009-07-14
[edit] Create directory
Before creating and uploading any of your own files create a personal directory for storage by typing (of course, here and in the next examples, you have to replace userdirectory with your account name; so, don't just cut and paste !):
$ lfc-mkdir /grid/euasia/tutorials/
To check that you have created your directory type:
$ lfc-ls -l /grid/euasia
and you should see your directory (plus possibly those of other attendees)
$ lfc-ls /grid/euasia Bazli charon-admin euasia006 euasia008 farhan generated kmunicek text_file.txt tutorials
[edit] Renaming an entry
Suppose you have just created an entry, for example having uploaded a file with a valid LFN (as shown here), you can change, for some reason known to you, the logical file name.
$ lfc-rename /grid/euasia/tutorials/myfile /grid/euasia/tutorials/myfile_renamed $ lfc-ls /grid/euasia/tutorials myfile_renamed
[edit] Setting Access Control List for directories
LFC has a very powerful instrument given by access control list, which allows to a file/directory owner to grain finely access rights for that entry for any other users. For example, let's create a new directory and see which are the default access rights with lfc-getacl
$ lfc-mkdir /grid/euasia/tutorials/larocca $ lfc-getacl /grid/euasia/tutorials/larocca # file: /grid/euasia/tutorials/larocca # owner: /C=IT/O=INFN/OU=Personal Certificate/L=Catania/CN=Giuseppe La Rocca # group: euasia user::rwx group::r-x #effective:r-x other::r-x
Note that ownership is expressed through DN of certificate, in order to grant uniqueness and avoid conflicts, while group is expressed through VO membership. Then it is shown the present ACL for the entry : user and group have full privileges while other can just read (execution rights make no sense here). Finally it is shown the default, which applies for each new entry created within this directory. To have an example of this, firstly we will create a file in the directory, and we will see applied default ACL
$ echo "bla bla bla" > message.txt $ export LFC_HOME=/grid/euasia/tutorials/larocca $ lcg-cr -d dpm.biruni.upm.my --vo euasia -l lfn:$LFC_HOME/first file:$PWD/message.txt guid:22dc22fd-1229-48da-a795-3b0f4287cb9d $ lfc-ls -l $LFC_HOME/first -rw-r--r-- 1 232 121 12 Jul 17 08:57 /grid/euasia/tutorials/larocca/first
now lets's change default ACL, with read/write permission for user and group, and no privileges for others. The syntax we apply here is modify (-m) default (d:) for user (u::), and the same of course for group and others. The conventions for rights is the usual UNIX-like (7 all, 6 rw...)
$ lfc-setacl -m d:u::6,d:g::6,d:o:0 $LFC_HOME/
this applies to all the entries you put later. In this way you can so modify according to your will the default privileges for directories.
$ lcg-cr -d dpm.biruni.upm.my --vo euasia -l lfn:$LFC_HOME/second file:$PWD/message.txt guid:7fccc641-fc6d-4a9c-b7c8-81dab8cb54a3 $ lfc-ls -l $LFC_HOME/ -rw-r--r-- 1 232 121 12 Jul 17 08:57 first -rw-rw---- 1 232 121 12 Jul 17 09:00 second
[edit] Upload file into SE
The next step is to upload a file into the directory you just created. First create locally a simple text file:
$ echo "Put something here" > text_file.txt
The command used for this is lcg-cr (LCG copy and register). Type the following to store this file on the dpm01.grid.sinica.edu.tw Storage Element : (use [lcg-infosites] in order to find which the available SE are)
$ lcg-infosites --vo euasia se Avail Space(Kb) Used Space(Kb) Type SEs ---------------------------------------------------------- 132348283 32067548 n.a se01.knowledgegrid.net.my 1027170119 55083888 n.a se02.knowledgegrid.net.my 933501500 50478916 n.a dpm.biruni.upm.my 2740000 56320000 n.a dpm01.grid.sinica.edu.tw 1 1 n.a rhino.lsr.nectec.or.th
The output should be something like this:
$ lcg-cr --vo euasia -l lfn:$LFC_HOME/text_file.txt -d dpm.biruni.upm.my file:$PWD/text_file.txt guid:33ffa464-1dc8-43be-b068-ded652586992
Of course, the guid you will have is different, since it is an unique identifier for each file (except for the case when you are replicating a file, as we will see). Check that the file is there by listing the contents of your directory.
Before continuing it is worth noting the difference between the command used to store the file and the creation of the directory in previous. The directory created is just a virtual directory and only exists within the catalog of lfn's. On the other hand the file physically exists on an SE but has an additional "virtual" filename in the catalog. This is connected to why the commands just handling the lfn namespace tend to start with "lfc" whilst the commands manipulating the file directly tend to start with "lcg".
[edit] Get the file SURL
For several purpose, FTS for instance, is useful to know the file SURL (they can be many if the file has replicas somewhere). The appropriate command is lcg-lr (list-replicas) [lfn | guid]
$ lcg-lr --vo euasia lfn:$LFC_HOME/text_file.txt srm://dpm.biruni.upm.my/dpm/biruni.upm.my/home/euasia/generated/2009-07-17/file0ed4e479-d920-4496-ba34-e60996d20735
[edit] Replicate file between SE
gLite supports file replication. A file can be stored on multiple SE's and then a running job can access the closest SE with the file on it, thus giving faster access times to the data. This also helps protect against failures/access difficulties with a particular SE. To find the list of SE'S available to you see the tutorial on lcg-infosites which can be found here. We will replicate the file just created to the SE dpm.biruni.upm.my with the command
$ lcg-rep --vo euasia -d dpm01.grid.sinica.edu.tw lfn:$LFC_HOME/text_file.txt
There is no output from this command on success, but you can check that the replica was created by listing all the replicas of your file, that is done by using the LCG list replicas command:
$ lcg-lr --vo euasia lfn:/grid/euasia/text_file.txt srm://dpm.biruni.upm.my/dpm/biruni.upm.my/home/euasia/generated/2009-07-17/file0ed4e479-d920-4496-ba34-e60996d20735 srm://dpm01.grid.sinica.edu.tw/dpm/grid.sinica.edu.tw/home/euasia/generated/2009-07-16/filed9e307d3-caf0-4734-9466-a63696ecd3bb
Note how the path to where each file is stored is different. This demonstrates how the use of a "lfn" avoids the need to understand the local filesystem where the replica is actually stored.
[edit] Download a file from SE to UI
Having already uploaded a file the next step is to show downloading a file. To download the file you already uploaded using the new lfn you have just created use the command:
$ lcg-cp --vo euasia lfn:$LFC_HOME/text_file.txt file://$HOME/text_file_copy.txt
You can check that the file you just downloaded is the same as the file you uploaded with cat command, you should find the same text you entered below.
[edit] Clean all!
You can delete a file from SE with lcg-del
$ lcg-del -a --vo euasia lfn:$LFC_HOME/text_file.txt
This will remove File Catalog entries as well
$ lfc-ls -l $LFC_HOME -rw-r--r-- 1 232 121 12 Jul 17 08:57 first -rw-rw---- 1 232 121 12 Jul 17 09:00 second
