In this page we present installation instructions for the gredia middleware platform. Specifically, we present instructions for the following components: RDLS, SFC, Gridtorrent.
RDLS
RDLS (Rich Data Location Services) is the layer that exposes the middleware's functionality through standard Web Services functionality. In the following, we give installation instructions of how to setup a gredia middleware server and how to install the client software and access the functionality of the installed services.
Server setup
RDLS is based on the web services layer of Globus Toolkit v.4. Globus can be installed as a whole, or only the web services layer. To install the whole toolkit, download the file
Globus Toolkit v.4.01 and follow the installation instructions from
here. If you wish to install only the web services layer, download it from
here and follow the installation instructions from
here. When the installation is completed,
download the gredia middleware services package and install it using the following command:
globus-deploy-gar gredia_MetaData.gar
The source code can be found
here
Client setup
The command line client software can be downloaded from
here and the source code from
here . The client software enables the user to perform rich queries and to store metadata descriptions on the RDLS. The client can be executed both from windows/linux machines with the only prerequisite is a java installation. After the zip is extracted, the directory config contains the file metadataclient.properties. The propery servername is the dns name of the server executing RDLS. The property proxyfile is the file name of a valid proxy certificate (that must reside in this directory). The executables have a .sh extension for linux and a .bat extension for windows. We now describe the client executables and their usage.
queryMedia : this executable is used to query the media overlay. It takes the following parameters: author date_created date_modified category type status privacy_status keywords(separated by " "). When searching for a range, the attribute value must be replaced by
from_value..to_value (with ".." between them). For the special ALL
value, just give "..".
queryBanking : The same for the banking overlay. the parameters are the following: customer_name vat_number.
storeMedia: This executable is used to upload a metadata description to the overlay. the parameters are : path_to_local_filename author date_created date_modified category type status privacy_status keywords(separated by " ")
storeBanking: Upload metadata description to the banking overlay. The parameters are: path_to_local_filename customer_name vat_number privacy_status.
SFC module and Kademlia overlay
The Python implementation of this server software responsible for constructing a Kademlia-based overlay supporting the storage and querying of metadata files and can be downloaded from
here. Just unzip it to a directory. Python2.5.x and the package
pysqlite-2.5.x are required. The indexing is based on Space Filling Curves (Z-curve or Hilbert curve). The initialization of the nodes that accommodate the enabled operations in the Metadata overlay is accomplished by the execution of peerPlatform.py. The description of the nodes and the indexing method is included in a INI file, which is given as parameter to the specific executable (-c option). Examples of INI files for the media scenario (
here), the banking scenario (
here) and keyword searches (
here) have been attached. For the media overlay for instance, type:
python peerPlatform.py -c MediaZ.ini
GridTorrent
GridTorrent constitutes the data transfer layer of the Gredia middleware. Based on peer-to-peer techniques, it allows clients to download files from multiple sources while uploading them to other users at the same time, rather than obtaining them from a central server. At the same time, it takes advantage of the striped version of GridFTP protocol to directly communicate with GridFTP servers. Moreover, instead of the centralized RLS Grid component, GridTorrent relies on a Kademlia-based overlay for efficient and scalable Replica Location.
GridTorrent Installation
GridTorrent is written in Java and doesn't need any installation. Just download the code from
here and unzip it to a directory
GridTorrent Operation Modes
GridTorrent can run as :
a. a standalone torrent application to upload/download a file
b. a server attached in a terminal which can accept remote requests for download/upload files
c. a daemon (linux service) that is detached from any terminals and has the same functionality as the above server
d. a shell that provides functionality of remote administrating GridTorrent servers/daemons
GridTorrent standalone application
To run the GridTorrent standalone application execute gtorrent.sh in Linux or gtorrent.exe in Windows.
With the -h option a help message is provided.
[azisi@gridvm5 GrediaGridTorrent]$ gtorrent.sh -h
usage: java -jar GridTorrent.jar [ -f filename | -i id ] -r rls
-b,--blocksize <arg> The block size that will be used when
transmitting a file
-e,--failurerate <arg> The failure rate that will be used when
sending a file
-f,--filename <arg> The filename to seed
-h Print help for this application
-i,--id <arg> The id/hash of the file to download
-o,--propertiesfile <arg> The GridTorrent.properties file
-p,--piecesize <arg> The piece size that will be used when
uploading a file
-r,--rls <arg> The rls endpoint (DRLS/
-t,--seedtime <arg> The seconds to seed after downloading the
file
-u,--uploadservers <arg> A comma separated list of servers to use for
uploading the file <host:port,host:port>
For easier visual integration the command line arguments can be included in a file and then the file can be given as a unique command line argument. For example we make the following sample with gtorrent file extension:
[azisi@gridvm5 GrediaGridTorrent]$ cat sample.gtorrent
-r
https://gridvm6.cslab.ece.ntua.gr:8443
-f
dummyfile
-u
gridvm1.cslab.ece.ntua.gr:48620,gridvm2.cslab.ece.ntua.gr:48620
If the user has selected the gtorrent.sh/gtorrent.exe to open the .gtorrent by default, then by double-clicking the .gtorrent file, GridTorrent will find the file named dummyfile in the data/ directory, connect to the DRLS to publish it and send remote download command to gridvm1/gridvm2 GridTorrent servers to start downloading the file.
GridTorrent server
The GridTorrent server can be invoked by executing gtserver.sh.
We can interact with the server only with remote commands that can be issued by the GridTorrent shell or the GridTorrent standalone application. According to the etc/log4j.properties diagnostic messages will be print in the terminal or the log file.
GridTorrent daemon
The GridTorrent daemon can be invoked by executing gtdaemon.sh start and can be stopped with gtdaemon.sh stop
We can interact with the daemon only with remote commands that can be issued by the GridTorrent shell or the GridTorrent standalone application. According to the etc/log4j.properties diagnostic messages will be print in the log file.
GridTorrent shell
Executing gtshell.sh the GridTorrent shell will be initiated and we will be able to connect to remote servers. The help command shows the implemented remote commands
[azisi@gridvm5 GrediaGridTorrent]$ gtshell.sh
Java GridTorrent Shell starting
# help
GridTorrent commands
start name rlsip:rlsport ip:port
Starts a new download on server listening on ip:port
if the LFN=name exists in the data directory then GT starts to share the file
else GT search in the RLS for the UID=name and tries to downloaded
stop uid ip:port
Stops an existing download on server listening on ip:port
delete lfn ip:port
Stop and deleting local files for existing download on server listening on ip:port
list ip:port
Listing existing downloads UIDs/LFNs on server listening on ip:port
stats uid ip:port
Show download statistics for the specific UID on server listening on ip:port
shutdown ip:port
Shutting down server listening on ip:port
sleep sec
The shell sleeps for sec seconds
exit
Exiting/Quiting GridTorrentClient
quit
Exiting/Quiting GridTorrentClient
GridTorrent and Globus Security Infrastructure (GSI)
GridTorrent uses GSI to secure the data and control channel when communicating with other peers.
These features can be enabled/disabled using the suitable options in etc/GridTorrent.properties
By default all security checks are disabled.
The available security checks are:
- authentication: Both client and server check the remote peer's certificate in order to authenticate. The certificate must be signed by a mutually trusted CA. (the CA files - cert/policy - must stored in /etc/grid-security/certificates/ or $HOME/.globus/certificates/)
- verification: Authentication is done as described above and also all messages have a cryptographic checksum in order to prevent data modification.
- encryption: Authentication is done as described above and also all messages are transmitted in encrypted form.
When GridTorrent starts, it expects to find a valid Globus proxy credential in the default location. This proxy credential is the output of a grid-proxy-init command or the fivo-authn login operation. If the proxy credential is not found or is not valid, GridTorrent tries to find the default Globus certificates usually situated in $HOME/.globus/user{cert,key}.pem In windows $HOME usually points to C:\Documents and Settings\username