Data Management Java API
From EUAGwiki
[edit] SEE-GRID File Management Java API
[edit] Lectured by
Giuseppe LA ROCCA
Italian National Institute of Nuclear Physics
Italy
mailto:giuseppe.larocca@ct.infn.it
[edit] Overview
SEE-GRID File Management Java API supports most of data and logical file management operations available in gLite environment through LFC and lcg-utils C APIs. It is developed and maintained at Belgrade University Computer Centre. This API was previously known as "LFC Java API" but the name was changed after its expansion with additional features. The API is compatible with LCG 2.7.x and gLite grid middleware.
The initial version was created within SEE-GRID project during the gridification of Volumetric Image Visualization Environment (VIVE) application. Its further development within SEE-GRID-2 will hopefully help gridification of other applications.
One of artifacts of VIVE development is Web-based LFC Data Management Front End which uses our SEE-GRID File Management Java API. The paper "Grid Data Management Web Portal" describes the implementation of this portal and its features.
Another usage, providing higher-level features suitable for usage from command line or Java code, is Managing Sets of Files and Replicas Within LFC Catalog.
This API is also used in this GILDA tutorial, which includes several exercises that cover various aspects of API usage.
[edit] Available features
* Uploading files from UI to grid
* Downloading files from grid to UI
* Reading file information
* permissions
* GUID
* comment
* Listing file replicas
* File management operations
* replicating files (copying physical files from one SE to another)
* deleting files and file replicas
* unregistering files and file replicas
* Reading directory information
* permissions
* comment
* Listing directory contents
* Reading user and group ids for files and directories
* Reading user and group names for files and directories
* Reading filemode information as described in LFC C/C++ API manual
* Listing file/directory aliases
* Alias management operations (create/rename/delete)
* Reading file/directory date information
* Additional file management operations (rename/move)
* Directory management (create/rename/move)
* File/directory comment modification
* Setting of operational parameters using environment variables and configuration file
* Listing of SE from BDII with the ability to ignore some SEs set in a properties file
* Directory management (delete)
[edit] Planned features
* Basic FTS support: initiating FTS transfers and reading their status * FPS using LFC and FTS * The InputStream implementation for files will be developed if not provided by future versions of gLite middleware. Developers can also use GFAL API for Java (javadoc).
[edit] Installation and Configuration
The following APIs should be already installed and configured by your system administrator. This is done by following the gLite installation guides 1, 2. These APIs come with gLite middleware:
* GFAL C/C++ API * lcg-utils C/C++ API * LFC C/C++ API
Download the SEE-GRID File Management API. Unpack the archive into a directory present within CLASSPATH environment variable, or add the target directory to CLASSPATH.
When using this API, the used virtual organisation, BDII, and configuration file can be set through the parameters of LFCDataStorage constructor. If default constructor is used or null values are set as parameters, the actual used values are the following:
* If configuration file pathname is null, "repmngr.properties" is used, pointing to a file
of the same name in current folder;
* The virtual organization name value is obtained from vo parameter. If null, the used virtual organization is obtained from the properties file in the current folder. If properties file or VO value in the file is missing, the value from environment variable LCG_GFAL_VO is used; * The parameter bdii defines the host name and port number of BDII being used to obtain the list of available storage elements. It can be also set in properties file. If not, the
value LCG_GFAL_INFOSYS of environment variable is used as default.
Please note that the usage of GFAL related operations, i.e. file content transfers between a storage element and local filesystem require LCG_GFAL_INFOSYS to be consistent with the value in properties, if any. If some of required parameters is not set, an exception will be thrown.
Properties file configuration:
* Set the value of property VO to your Virtual Organization name (E.g. VO = gilda); * Set the value of properties bdii.host.name and bdii.host.port to point to the BDII
(E.g. bdii.host.name=gilda-bdii.ct.infn.it, bdii.host.port=2170);
* Optionally, set the value of property SE.ignore.list - the list of SEs to be ignored when obtaining available SEs from the BDII. Use space to separate list items
(E.g.: SE.ignore.list = se1.ignoredhost.org se2.ignoredhost.org)
[edit] Documentation
http://grid02.rcub.bg.ac.yu/LFCJavaAPI/files/docs/javadoc/version1.3/index.html
[edit] Downloads
[edit] SEE-GRID File Management Java API
[edit] GILDA exercises
Source code, classes, properties file and examples (zip file, 50 KB) Media:Api-Java-LFC-1.3.tar.gz
[edit] Version 1.3 (released 11/07/2007)
Source code, classes, properties file and examples (zip file, 87 KB) http://grid02.rcub.bg.ac.yu/LFCJavaAPI/files/downloads/SEE-GRIDFileManagementAPI-v1.3.zip
[edit] Version 1.2 (released 10/30/2006)
Source code, classes, properties file and examples (zip file, 99 KB) http://grid02.rcub.bg.ac.yu/LFCJavaAPI/files/downloads/SEE-GRIDFileManagementAPIv1.2.zip
[edit] Version 1.1
Source code, classes, properties file and examples (zip file, 42 KB) http://grid02.rcub.bg.ac.yu/LFCJavaAPI/files/downloads/SEE-GRIDFileManagementAPIv1.1.zip
[edit] Version 1.0
Source code, classes, properties file and examples (zip file, 40 KB) http://grid02.rcub.bg.ac.yu/LFCJavaAPI/files/downloads/SEE-GRIDFileManagementAPIv1.0.zip
[edit] Compile examples
Uncompress the tar with the exercises as follow:
tar zxvf Api-Java-LFC-1.3.tar.gz
listing the directory you should see
$ ll Api-Java-LFC-1.3 -rw-r--r-- 1 larocca users 780 Oct 10 2008 aliases.java -rw-r--r-- 1 larocca users 2055 Nov 10 03:40 compile-exercises.sh -rw-r--r-- 1 larocca users 1196 Oct 10 2008 copyAndRegister.java -rw-r--r-- 1 larocca users 777 Oct 10 2008 download.java -rw-r--r-- 1 larocca users 786 Oct 10 2008 getParent.java -rw-r--r-- 1 larocca users 974 Oct 10 2008 GetReplicas.java drwxr-xr-x 2 larocca users 8192 Nov 7 2007 JNI -rw-r--r-- 1 larocca users 399 Oct 10 2008 lfc-api-java.cfg -rw-r--r-- 1 larocca users 1155 Oct 10 2008 LfcLsDate.java -rw-r--r-- 1 larocca users 813 Oct 10 2008 LfcLs.java -rw-r--r-- 1 larocca users 657 Oct 10 2008 listSE.java -rw-r--r-- 1 larocca users 675 Oct 10 2008 mkdir.java -rw-r--r-- 1 larocca users 765 Oct 10 2008 mkdirWithPerm.java -rw-r--r-- 1 larocca users 1180 Oct 10 2008 remove.java -rw-r--r-- 1 larocca users 894 Oct 10 2008 rename.java -rw-r--r-- 1 larocca users 724 Oct 10 2008 replicate.java -rw-r--r-- 1 larocca users 396 Nov 10 03:39 repmngr.properties -rw-r--r-- 1 larocca users 618 Oct 10 2008 rmdir.java -rw-r--r-- 1 larocca users 776 Oct 10 2008 setComm.java -rw-r--r-- 1 larocca users 1059 Oct 10 2008 symlink.java -rw-r--r-- 1 larocca users 897 Oct 10 2008 type.java -rw-r--r-- 1 larocca users 740 Oct 10 2008 userAndGroup.java drwxr-xr-x 3 larocca users 8192 Nov 7 2007 yu
To compile all the Java files run the compile-exercise.sh script as follow:
$ cd Api-Java-LFC-1.3 $ source compile-exercises.sh ++ export CLASSPATH=./:../../../../../:./bin:./repmngr.properties ++ CLASSPATH=./:../../../../../:./bin:./repmngr.properties ++ javac -classpath ./:../../../../../:./bin:./repmngr.properties -verbose -sourcepath aliases.java copyAndRegister.java download.java getParent.java GetReplicas.java LfcLsDate.java LfcLs.java listSE.java mkdir.java mkdirWithPerm.java remove.java rename.java replicate.java rmdir.java setComm.java symlink.java type.java userAndGroup.java -d /home/larocca/HEPIX/Api-Java-LFC-1.3 [parsing started copyAndRegister.java] [parsing completed 43ms] [parsing started download.java] [parsing completed 17ms] [parsing started getParent.java] [parsing completed 19ms] [parsing started GetReplicas.java] [parsing completed 3ms] [parsing started LfcLsDate.java] [parsing completed 3ms] [parsing started LfcLs.java] [parsing completed 2ms] [parsing started listSE.java] [parsing completed 2ms] [parsing started mkdir.java] [parsing completed 30ms] [parsing started mkdirWithPerm.java] [parsing completed 3ms] [parsing started remove.java] [parsing completed 4ms] [parsing started rename.java] [parsing completed 3ms] [parsing started replicate.java] [parsing completed 17ms] [parsing started rmdir.java] [parsing completed 2ms] [parsing started setComm.java] [parsing completed 3ms] [parsing started symlink.java] [parsing completed 3ms] [parsing started type.java] [parsing completed 2ms] [parsing started userAndGroup.java] [parsing completed 2ms] [search path for source files: []] [search path for class files: [/usr/java/jdk1.5.0_14/jre/lib/rt.jar, /usr/java/jdk1.5.0_14/jre/lib/jsse.jar, /usr/java/jdk1.5.0_14/jre/lib/jce.jar, /usr/java/jdk1.5.0_14/jre/lib/charsets.jar, /usr/java/jdk1.5.0_14/jre/lib/ext/localedata.jar, /usr/java/jdk1.5.0_14/jre/lib/ext/dnsns.jar, /usr/java/jdk1.5.0_14/jre/lib/ext/sunpkcs11.jar, /usr/java/jdk1.5.0_14/jre/lib/ext/sunjce_provider.jar, ./, ../../../../../]] [loading /usr/java/jdk1.5.0_14/jre/lib/rt.jar(java/lang/Object.class)] [loading /usr/java/jdk1.5.0_14/jre/lib/rt.jar(java/lang/String.class)] [loading /usr/java/jdk1.5.0_14/jre/lib/rt.jar(java/util/Date.class)] [checking copyAndRegister] [loading /usr/java/jdk1.5.0_14/jre/lib/rt.jar(java/lang/System.class)] [loading /usr/java/jdk1.5.0_14/jre/lib/rt.jar(java/io/PrintStream.class)] [loading /usr/java/jdk1.5.0_14/jre/lib/rt.jar(java/io/FilterOutputStream.class)] [loading /usr/java/jdk1.5.0_14/jre/lib/rt.jar(java/io/OutputStream.class)] [loading ./yu/ac/bg/rcub/grid/dataManagement/DataStorageInterface.class] [loading ./yu/ac/bg/rcub/grid/dataManagement/LFCDataStorage.class] [loading ./yu/ac/bg/rcub/grid/dataManagement/DirectoryItem.class] [loading ./yu/ac/bg/rcub/grid/dataManagement/LFCDirectoryItem.class] [loading ./yu/ac/bg/rcub/grid/dataManagement/Item.class] [loading /usr/java/jdk1.5.0_14/jre/lib/rt.jar(java/lang/StringBuilder.class)] [loading /usr/java/jdk1.5.0_14/jre/lib/rt.jar(java/lang/AbstractStringBuilder.class)] [loading /usr/java/jdk1.5.0_14/jre/lib/rt.jar(java/lang/CharSequence.class)] [loading /usr/java/jdk1.5.0_14/jre/lib/rt.jar(java/io/Serializable.class)] [loading /usr/java/jdk1.5.0_14/jre/lib/rt.jar(java/lang/Comparable.class)] [loading /usr/java/jdk1.5.0_14/jre/lib/rt.jar(java/lang/StringBuffer.class)] [wrote /home/larocca/HEPIX/Api-Java-LFC-1.3/copyAndRegister.class] [checking download] [loading ./yu/ac/bg/rcub/grid/dataManagement/FileItem.class] [loading ./yu/ac/bg/rcub/grid/dataManagement/LFCFileItem.class] [wrote /home/larocca/HEPIX/Api-Java-LFC-1.3/download.class] [checking getParent] [wrote /home/larocca/HEPIX/Api-Java-LFC-1.3/getParent.class] [checking GetReplicas] [wrote /home/larocca/HEPIX/Api-Java-LFC-1.3/GetReplicas.class] [checking LfcLsDate] [loading ./yu/ac/bg/rcub/grid/dataManagement/ItemIterator.class] [loading ./yu/ac/bg/rcub/grid/dataManagement/LFCFileMode.class] [wrote /home/larocca/HEPIX/Api-Java-LFC-1.3/LfcLsDate.class] [checking LfcLs] [wrote /home/larocca/HEPIX/Api-Java-LFC-1.3/LfcLs.class] [checking listSE] [loading ./yu/ac/bg/rcub/grid/dataManagement/SEList.class] [wrote /home/larocca/HEPIX/Api-Java-LFC-1.3/listSE.class] [checking mkdir] [wrote /home/larocca/HEPIX/Api-Java-LFC-1.3/mkdir.class] [checking mkdirWithPerm] [wrote /home/larocca/HEPIX/Api-Java-LFC-1.3/mkdirWithPerm.class] [checking remove] [wrote /home/larocca/HEPIX/Api-Java-LFC-1.3/remove.class] [checking rename] [wrote /home/larocca/HEPIX/Api-Java-LFC-1.3/rename.class] [checking replicate] [wrote /home/larocca/HEPIX/Api-Java-LFC-1.3/replicate.class] [checking rmdir] [wrote /home/larocca/HEPIX/Api-Java-LFC-1.3/rmdir.class] [checking setComm] [wrote /home/larocca/HEPIX/Api-Java-LFC-1.3/setComm.class] [checking symlink] [loading ./yu/ac/bg/rcub/grid/dataManagement/AliasItem.class] [wrote /home/larocca/HEPIX/Api-Java-LFC-1.3/symlink.class] [checking type] [wrote /home/larocca/HEPIX/Api-Java-LFC-1.3/type.class] [checking userAndGroup] [wrote /home/larocca/HEPIX/Api-Java-LFC-1.3/userAndGroup.class] [total 751ms] Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. ++ set +x
Before to start don't forget to create your proxy
$ voms-proxy-init --voms gilda Your identity: /C=IT/O=GILDA/OU=Personal Certificate/L=INFN Catania/CN=Giuseppe La Rocca/Email=giuseppe.larocca@ct.infn.it Enter GRID pass phrase: Creating temporary proxy ........................................ Done Contacting voms.ct.infn.it:15001 [/C=IT/O=INFN/OU=Host/L=Catania/CN=voms.ct.infn.it] "gilda" Done Creating proxy ......................................................... Done Your proxy is valid until Fri Dec 22 23:45:13 2006
[edit] Listing directory content
$ java LfcLs List files for a given directory. Usage: java LfcLs grid_directory $ java LfcLs /grid/gilda/ drwxrwxr-x 124 104 0 aula_grid drwxrwxr-x 124 104 0 aula_grid_11 drwxrwxr-x 152 104 0 balasko drwxrwxr-x 137 104 0 cdg drwxrwxr-x 129 104 0 clermont drwxrwxr-x 101 104 0 corsogrid drwxrwxr-x 112 117 0 emidio [..cut..]
[edit] Copy and Register a file
Usage: copyAndRegister \ sourceFilePath \ gridDestinationDir \ gridDestFileName \ SEName_withSfnOrSrmPrefix $ java copyAndRegister /home/larocca/HEPIX/Api-Java-LFC-1.3/compile-exercises.sh \ /grid/gilda/tutorials \ compile-exercises.sh \ aliserv6.ct.infn.it File /home/larocca/HEPIX/Api-Java-LFC-1.3/compile-exercises.sh copied and registed as: /grid/gilda/tutorials/compile-exercises.sh.
[edit] Remove a file
$ java remove Remove file from the grid Usage: remove FileToRemove E.g.: remove /grid/gilda/tutorials/myfile $ java remove /grid/gilda/tutorials/compile-exercises.sh File /grid/gilda/tutorials/compile-exercises.sh successfully removed from the Grid.
[edit] Get Replicas
$ java GetReplicas List all the replica(s) for a given file. Usage: java GetReplicas gridDestFileName E.g.: java GetReplicas /grid/gilda/tutorial/mylastfile.txt $ java GetReplicas /grid/gilda/tutorials/compile-exercises.sh SURL: srm://aliserv6.ct.infn.it/dpm/ct.infn.it/home/gilda/generated/2009-11-03/fileec0c6b10-e28f-4d88-b980-c068e79988cb
[edit] Download a file
$ java download Downloads file from grid. Usage: java download <lfc_file_to_download> <sourceSURL> <destinationFileName> $ java download /grid/gilda/tutorials/compile-exericse.sh \ srm://aliserv6.ct.infn.it/dpm/ct.infn.it/home/gilda/generated/2009-11-03/fileec0c6b10-e28f-4d88-b980-c068e79988cb \ $PWD/myfile unable to get file information, invalid file name or credentials Download successful: true $ ls -l myfile -rw-r--r-- 1 larocca users 2053 Nov 3 02:25 myfile
[edit] Get parent entry directory
$ java getParent Gets the parent for given LFC entry. Usage: java getParent <lfc_catalogue_entry> $ java getParent /grid/gilda/tutorials Parent: /grid/gilda
[edit] Retrieve the list of SE
$ java listSE se.hpc.iit.bme.hu aliserv6.ct.infn.it gilda-02.pd.infn.it sirius-se.ct.infn.it vega-se.ct.infn.it iceage-se-01.ct.infn.it dgt02.ui.savba.sk gilda-se.rediris.es fn2.hpcc.sztaki.hu Returned 9 entries
[edit] Create a directory in the LFC File Catalog
$ java mkdir Creates directory with default permissions (755). Usage: java mkdir <parentDir> <dirNameToCreate> $ java mkdir /grid/gilda/tutorials qwert success: true
[edit] Remove a directory in the LFC File Catalog
$ java rmdir Remove directory. Usage: java rmdir <dirNameToRemove> $ java rmdir /grid/gilda/tutorials/qwert success: true
[edit] Replicate entry
$ java replicate Replicate file to given SE. Usage: java replicate <gridDestFileName> <target_SE> $ java replicate /grid/gilda/tutorials/compile-examples.sh vega-se.ct.infn.it Replication successful: true
[edit] Set/delete comment
$ java setComm Set/delete item comment. To set: java setComm <itemPath> <newComment> To delete: java setComm <itemPath> $ java setComm /grid/gilda/tutorials/compile-examples.sh "My Comment" Comment for /grid/gilda/tutorials/compile-examples.sh: My Comment $ java setComm /grid/gilda/tutorials/compile-examples.sh Comment for /grid/gilda/tutorials/compile-examples.sh: null
[edit] Type
$ java type Tests the type of a given item. Usage: java type <lfnItemPathName> $ java type /grid/gilda/tutorials/compile-examples.sh /grid/gilda/tutorials/compile-examples.sh is a file $ java type /grid/gilda/tutorials/ /grid/gilda/tutorials/ is directory
[edit] User & Group
$ java userAndGroup Gets user & group of a given item. Usage: java userAndGroup <lfnItemPathName> $ java userAndGroup /grid/gilda/tutorials/compile-examples.sh User: /C=IT/O=GILDA/OU=Personal Certificate/L=INFN Catania/CN=Giuseppe La Rocca (104) Group: gilda (104)
[edit] Symbolic link
$ java symlink Creates alias of given item. Usage: java symlink <lfnItemPathName> <newAliasPathname> $ java symlink /grid/gilda/tutorials/compile-examples.sh /grid/gilda/tutorials/mylink /grid/gilda/tutorials/compile-examples.sh is file Alias created /grid/gilda/tutorials/mylink for target /grid/gilda/tutorials/compile-examples.sh: true
[edit] References
Retrieved from http://wiki.egee-see.org/index.php/SEE-GRID_File_Management_Java_API LFC Documentation https://twiki.cern.ch/twiki/bin/view/LCG/DataManagementDocumentation
[edit] Contact
Dragan Okiljevic (oki AT rcub.bg.ac.yu)
