The AMGA Metadata Service
From EUAGwiki
Contents |
[edit] Lectured by
Jingya You
Academia SINICA Grid Computing
Taiwan
mailto:jingya.you@twgrid.org
[edit] Objective
In this page, you will learn some AMGA commands during this hands-on session and also be familiar with the AMGA operation more.
[edit] Before Launching AMGA
- Set environment
[UPM04@ui ~]$ export PATH=/opt/glite/bin/mdclient:$PATH
[edit] Setup the .mdclient.config configuration file
- Before starting the AMGA client application, it is necessary to copy a configuration file into the home directory. There is a template file provided by the AMGA client installation that can be customised and used to access your own AMGA server.
- Copy template configuration file to the HOME directory
- Default configuration file is in the path, "/opt/glite/etc/"
[UPM04@ui ~]$ cp /opt/glite/etc/mdclient.config $HOME/.mdclient.config (The file can be hidden or not by adding/avoiding the '.' character at the beginning of the file name).
Once you have copied your configuration file (mdclient.config) it is necessary to open the file and change some values with a text editor. The following fields should be modified:
# Connection options Host = ui.euag.org Port = 8822 # User settings Login = UPM04 ( eg. if your account is UPM01, please use UPM01 ) PermissionMask = rwx GroupMask = r-x Home = / #Name= # Security options UseSSL = no # Values: require, try, no. If off, all options below are ignored AuthenticateWithCertificate = 0 # Use certificate to authenticate # Certificates used for authentication: ... either normal certs #CertFile=/home/koblitz/.globus/usercert.pem #KeyFile=/home/koblitz/.globus/userkey.pem # ... or a grid proxy certificate #UseGridProxy = 1 # Use password instead of certificate to authenticate Password = UPM_euag.09 ( eg. This is the password of your account.) #VerifyServerCert = 1 #IgnoreCertificateNameMismatch = 0 # If server certificates are verified, CA certificates need to be loaded: TrustedCertDir = /etc/grid-security/certificates
[edit] Connect to the AMGA server
It is possible to get help anytime on the client just using the 'help' command.
[UPM04@ui ~]$ mdclient -c ~/.mdclient.config Connecting to ui.euag.org:8822... ARDA Metadata Server 1.9.0 Query> help createdir >> createdir /parentdir/dir [option] >> Creates a new directory.. Query> help rmdir >> rmdir directory >> Removes a directory. Query>
Make sure you have logined with the right account
Query> whoami >> UPM04
[edit] General Operations on AMGA
AMGA provides a directory-like structure of the collections of entries
Query> ls -l >> drwxrwxr-x UPM04 /UPM04/ Query> createdir /UPM04/tutorial ( create the directory under "UPM04" directory ) Query> ls -l >> drwxrwxr-x root /ESS >> drwxrwxr-x UPM04 /UPM04/ Query> cd tutorial Query> pwd >> /UPM04/tutorial/ Query>
[edit] Let us start with an example
- create cities directory which has longitde and latitude as attributes.
- the data types of longitde and latitude are int.
Query> pwd /UPM04/ Query> createdir cities Query> addattr cities longitude int Query> addattr cities latitude int Query> listattr cities >> longitude >> int >> latitude >> int
- create weather directory having city, temp_hi, temp_lo, humidity, time as attributes.
- the data type of city attribute is varchar with 30 bytes.
- temp_hi means the high temperature and its data type is float.
- temp_lo means the low temperature and its data type is float.
- the data type of humidity is also float.
- time means time stamp of inserting entry.
Query> createdir weather Query> addattr weather city varchar(30) temp_hi float temp_lo float humidity float time timestamp Query> listattr weather >> city >> varchar(30) >> temp_hi >> float >> temp_lo >> float >> humidity >> int >> time >> timestamp
Valid datatypes are summarized in the following table where the corresponding AMGA DB backend datatype is also shown.
| AMGA | PostgreSQL | MySQL | Oracle | SQLite | Python |
| int | integer | int | number(38) | int | int |
| float | double precision | double precision | float | float | float |
| varchar(n) | character varying(n) | character varying(n) | varchar2(n) | varchar(n) | string |
| timestamp | timestamp w/o TZ | datetime | timestamp(6) | unsupported | time(unsupported) |
| text | text | text | long | text | string |
| numeric(p,s) | numeric(p.s) | numeric(p.s) | numeric(p.s) | numeric(p.s) | float |
- Fill the tables with entries
- ingest the city name as entry ID in cities directory
Query> addentry cities/KualaLumpur latitude 3 longitude 101 Query> addentry cities/Taipei latitude 25 longitude 121 Query> addentry cities/Tokyo latitude 35 longitude 139 Query> addentry cities/Hongkong latitude 22 longitude 110 Query> ls -l cities >> -rwxr-xr-x root KualaLumpur >> -rwxr-xr-x root Taipei >> -rwxr-xr-x root Tokyo >> -rwxr-xr-x root Hongkong Query> addentry weather/record_hk01 city 'Hongkong' temp_hi 30 temp_lo 23 humidity 65 time '2009-07-28 11:00:06' Query> addentry weather/record_hk02 city 'Hongkong' temp_hi 32 temp_lo 24 humidity 70 time '2009-07-29 11:03:17' Query> addentry weather/record_tk01 city 'Tokyo' temp_hi 24 temp_lo 20 humidity 61 time '2009-07-27 11:03:17' Query> addentry weather/record_tk02 city 'Tokyo' temp_hi 17 temp_lo 11 humidity 52 time '2009-07-29 09:03:17' Query> addentry weather/record_tp01 city 'Taipei' temp_hi 25 temp_lo 21 humidity 68 time '2009-07-28 11:03:17' Query> addentry weather/record_tp02 city 'Taipei' temp_hi 28 temp_lo 22 humidity 65 time '2009-07-29 12:03:17' Query> addentry weather/record_kl01 city 'KualaLumpur' temp_hi 32 temp_lo 28 humidity 66 time '2009-07-27 12:03:17' Query> addentry weather/record_kl02 city 'KualaLumpur' temp_hi 30 temp_lo 24 humidity 65 time '2009-07-27 12:03:17' Query> addentry weather/record_kl03 city 'KualaLumpur' temp_hi 33 temp_lo 26 humidity 67 time '2009-07-27 12:03:17' Query>
Now, we can get some information back
Which are the cities further north then 30 degrees north, and how high are they?
Query> selectattr cities:FILE longitude 'latitude > 30' >> Tokyo >> 139
Show all cities and their latitude
Query> selectattr cities:FILE latitude ' ' >> KualaLumpur >> 3 >> Taipei >> 25 >> Tokyo >> 35 >> Hongkong >> 22
Which city has low temperatures higher than 25 degrees?
Query> selectattr weather:city weather:temp_lo 'weather:temp_lo > 25' >> KualaLumpur >> 28 >> KualaLumpur >> 26
Which city has low temperatures smaller than 15 degrees?
Query> selectattr weather:city weather:temp_lo 'weather:temp_lo < 15' >> Tokyo >> 11
[edit] Exercises
Ex1. In the weather directory, we would like to retain the data about cities in ASIA. So, we have to remove the record_ma01entry. What could we do?
Hint: Manipulating Entry Command Answer: Query> rm /UPM04/weather/record_01
Ex2. Which city has high temperatures higher than 25 degrees and the city name starts with "K"?
Hint: use query condition with "like" Answer: Query> selectattr weather:city ' weather:temp_hi > 25 and like (weather:city, "K%") ' >> KualaLumpur
Ex3. Which city has the humidity smaller than 60% and shows its latitude, longitude, temp_lo, temp_hi?
Hint: one of the join condition is "cities:FILE=weather:city" Answer: Query> selectattr weather:city cities:longitude cities:latitude weather:temp_hi weather:temp_lo ' cities:FILE=weather:city and weather:humidity < 60 ' >> Tokyo >> 139 >> 35 >> 17 >> 11
[edit] Enable Authentication by Grid Proxy
- Server Configuration
- amgad.config is the configuration file of Server
- Default path of amgad.config is in /opt/glite/etc/
- The host certificate is required for enabling authentication by certificate or proxy.
[root@ui ~]# ls /opt/glite/etc/ amgad.config init.d mdclient.config mdclient.config.rpmsave
To enable authentication by certificates, you need to load your server certificates first. The configuration in amgad.config shall follows by the settings below.
[root@ui ~]# vim /opt/glite/etc/amgad.config
# Secure Connections UseSSL = 1 # Authentication options RequireAuthentication = 1 # If this is off, no authentication is done! AllowCertificateAuthentication = 1 AllowPasswordAuthentication = 1 # If you use SSL, you need to load server certificates: CertFile = /etc/grid-security/hostcert.pem KeyFile = /etc/grid-security/hostkey.pem # If Certificate based authentication allowed, you need to load server certs TrustedCertDir = /etc/grid-security/certificates AllowGridProxyLogin = 1 # Requires also AllowCertificateAuthentication
- AMGA Server settings
- Adding each DN of the users by the command user_subject_add username subject(DN)
[root@ui ~]# user_subject_add UPM04 '/C=IT/O=GILDA/OU=Personal Certificate/L=Kuala Lumpur/CN=KUALALUMPUR04'
- Client Configuraton
- mdclient.config or .mdclient.config is the configuration file of clients
- This config file is required when you connect to AMGA server. You can store your mdclient.config in your home directory (eg. /home/UPM04/). Or you can copy from a template from /opt/glite/etc/mdclient.config
- You are able to start a connection with the specific config file
[UPM04@ui ~]# cp /opt/glite/etc/mdclient.config mdclient.config [UPM04@ui ~]# ls mdclient.config [UPM04@ui ~]# vim mdclient.config
# Security options UseSSL = no # Values: require, try, no. If off, all options below are ignored AuthenticateWithCertificate = 1 # Use certificate to authenticate #Certificates used for authentication: ... either normal certs CertFile=/home/UPM04/.globus/usercert.pem KeyFile=/home/UPM04/.globus/userkey.pem # ... or a grid proxy certificate UseGridProxy = 1 # Use password instead of certificate to authenticate #Password = secret # If server certificates are verified, CA certificates need to be loaded: TrustedCertDir = /etc/grid-security/certificates
Initiate your proxy before connecting to AMGA server
[UPM04@ui ~]$ voms-proxy-init --voms gilda Cannot find file or dir: /home/UPM04/.glite/vomses Enter GRID pass phrase: Your identity: /C=IT/O=GILDA/OU=Personal Certificate/L=Kuala Lumpur/CN=KUALALUMPUR04 Creating temporary proxy ..................................................... Done Contacting voms.ct.infn.it:15001 [/C=IT/O=INFN/OU=Host/L=Catania/CN=voms.ct.infn.it] "gilda" Done Creating proxy................................................................................... Done Your proxy is valid until Fri Jul 31 16:55:04 2009
Try to initiate your connection with your mdclient.config
[UPM04@ui ~]# mdclient -c mdclient.config Connecting to t-ap27.grid.sinica.edu.tw:8822... ARDA Metadata Server 1.9.0 Query>
[edit] Reference
- Configure of the AMGA client : http://amga.web.cern.ch/amga/mdclient_config.html
- AMGA Commands : http://amga.web.cern.ch/amga/commands.html
- Metadata Queries : http://amga.web.cern.ch/amga/queries.html
- script for training course ingestData.sh
