Schlouder
IaaS cloud broker to execute bag-of-tasks on public or private cloud
 All Classes Files Functions Pages
Schlouder Documentation

Schlouder is a cloud broker written in Perl. It is part of a research project. If you have any questions, do not hesitate to contact us at schlouder-devs@lists.gforge.inria.fr.

While using a meta-scheduler is the normal procedure on grids, IaaS clouds usually require users to directly interact with the started VMs (e.g., through ssh). Hence, we aim to offer to cloud clients an interface able to select the resources onto which the jobs are to be run. This interface actually wraps a broker, which implements scheduling strategies allowing a user to better control the execution. We introduce Schlouder, whose scope is to act as a client-side cloud broker. It is a home-made software which implements the provisioning and scheduling strategies, and connects to third-party resource managers to handle VM creation and job-to-VM assignment.

Installing Schlouder

Prerequisites

The prerequisites are to have Perl 5 installed as Schlouder is written in this language, and to have a batch scheduler and a VM manager with a valid Perl lib. For our tests, we used Slurm as a batch scheduler and Eucalyptus v3 to manage the private cloud platform. We explain in the two following subsection how to have a Slurm working installation although you can install another job scheduler.

Installing MUNGE

We use MUNGE for the authentication part in Slurm. We use the MUNGE version from the repository. To install it, just type apt-get --yes install munge libmunge-dev. Then create a MUNGE key to be distributed on all the VM.

Installing Slurm

Download Slurm here. The script we use to install Slurm is located in the repository in conf/slurm/install-slurm.sh with a configuration file example and the init script we use. You must specify the controller address in the configuration file (ControlMachine and ControlAddr) and the VMs name (NodeName and PartitionName).

The Slurm perl library is located in the source folder of Slurm at this location: contribs/perlapi. You must type make contrib, then go to the contribs/perlapi/libslurm folder. Here, type make install and finally copy all files in the /opt/slurm/lib/perl/5.14.2 folder into a folder of your Perl array.

The problem now is that Slurm uses the VMs hostname to communicate with them. Thus we need a DNS server in order to match the hostname with the VMs IP address. We explain in the following subsection the configuration of Bind.

Installing the DNS server

First install the bind version from the repository on the controller: sudo apt-get install --yes bind9.

An example of customizable configuration file is provided in the conf/bind folder. Do not forget to generate a couple of private/public keys.

If you are still experiencing problem, ensure that the files and folders rights are correct for the user bind and that AppArmor is not blocking the access to a file. You can have a look at /var/log/syslog to ensure.

When each VM boots, we add two records in the DNS server: one for the standard zone and on for the reverse one. To do so, we call nsupdate:

$ cat<<EOF | /usr/bin/nsupdate -k $DNS_KEY -v
server $CTL_NAME.$DOMAIN
zone $DOMAIN
update delete $HOSTNAME.$DOMAIN A
update add $HOSTNAME.$DOMAIN 60 A $FULL_IP
send
EOF
$ cat<<EOF | /usr/bin/nsupdate -k $DNS_KEY -v
server $CTL_NAME.$DOMAIN
zone $NETWORK_REV.in-addr.arpa
update delete $IP.$NETWORK_REV.in-addr.arpa PTR
update add $IP.$NETWORK_REV.in-addr.arpa 60 PTR $HOSTNAME.$DOMAIN.
send
EOF

When all these steps are done, you should see in Slurm that the VMs are listed using the sinfo command.

Retrieving the latest stable version

Dependencies

We now describe the steps to install and configure Schlouder. It uses some well-known Perl libraries. You can find a list in the Makefile.PL file. Note that these libraries are automatically installed when you install Schlouder. You still first need to install some packages beforehand: File::Remove, LWP::UserAgent, YAML::Tiny.

apt-get install libfile-remove-perl libwww-perl libyaml-tiny-perl

Schlouder

The latest version of Schlouder can be retrieved from the git repository:

git clone git://scm.gforge.inria.fr/schlouder/schlouder.git
  1. Generate the Makefile
    cd schlouder
    perl Makefile.PL
  1. It is possible that you don't have all the required dependencies. In this case, there are two ways to do:

    • Say yes to auto-install during the make command. In this case, the packages will be downloaded from CPAN, then built on your computer
    • Most of CPAN packages are pre-packaged in your distribution, for instance if it is said that the package Text::ASCIITable is missing, just install libtext-asciitable-perl. Then rerun
      perl Makefile.PL
      Most of the dependencies must be resolved.

Note that if you want to use Eucalyptus, you may have trouble with the Net::Amazon::EC2 depending on your Eucalyptus version. If you cannot connect to your private cloud, please use the library version you can find here.

  1. Then build and install Schlouder (and potentially install dependencies)
    make
    make install
    You must run the make install command as root.

SimSchlouder

If you plan to provide the Provisioning Simulation Module (PSM) to the clients, you also need to install a simulator. We developped SimSchlouder but you can develop and use your own simulator. In order to install SimSchlouder, please refer to its own website. You will need to configure it properly in the configuration files of Schlouder (see next section).

Notes

If you want to use OpenStack, you should use our lib in the lib/Net/Openstack folder. The official version does not work with perl thread (see here).

Using Schlouder

Once Schlouder is successfully installed, we highlight what needs to be done in order to execute a job.

Configuration files

All configuration files must be by default in /etc/schlouder. However you can specify to the client and the server a different folder with the -c option.

Server configuration

You normally have your configuration files in the /etc/schlouder directory. On the server side, you must have the following tree:

    /etc/schlouder/
    |---config_server.cfg
    |---log/
    |---|---log4perl_server.conf
    |---cloud/
    |---|---_cloudname1_.cfg
    |---|---_cloudname1_.cfg

In config_server.cfg, the default values are commented. The format is Perl.

The main keys to fill are:

  • users: Array of users name. The user name is actually the name of the key (ssh) created in the cloud
  • cloud: List of clouds name. Schlouder will import conf/cloud/_cloudname_.cfg files.
  • strategies: Hash of provisioning and scheduling strategies. It must match a Schlouder module name in lib/Schlouder/{Provisioning,Scheduling}/
  • multicloud: If you have multiple cloud, we propose a set of strategy to automatically select which cloud execute the job. Values must match a Schlouder module name in lib/Schlouder/Multicloud
  • metastrategy: Configuration of the Schlouder::Metastrategies::Deadline metastrategy

There is a specific configuration file by cloud. An example of such configuration is available in the eucalyptus-icps.cfg file.

The main keys to fill are:

  • api: Type of the API compatible with you cloud (e.g. EC2, OpenStack...)
  • connection: Some important value to connect to the cloud (e.g. ip address, port...)
  • users: Hash to associate a username and an image id
  • boot: Name of the module which predict the boot time of the VM

Client configuration

You normally have your configuration files in the /etc/schlouder directory. On the client side, you must have the following tree:

    /etc/schlouder/
    |---config_client.cfg
    |---log/
    |---|---log4perl_client.conf

The config_client.cfg file is easy to understand. The comments in this file may be sufficient.

Running the server

Schlouder provide a single executable to run the server. Just type:

schlouder-server

Note that you may need to be root to execute it.

By default the logs are in /var/log/schlouder_server.log.

Custom image

The VM image need some small modification to interact with Schlouder. First of all, to communicate with the batch scheduler we need to install an agent in the VM which start at boot time. In our case we used Slurm. We provide an example of script we use to customize a VM image in the repository in the file script/install_vm.bash.

Executing my first script on the cloud through Schlouder

Schlouder provides a single executable to run the client. Its options are:

    schlouder-client [-c/--config config_folder] [-h/--help | -o/--output {txt,json} | -p/--psm] [-d/--deadline deadline_timestamp] bag-of-tasks file

    Options list :

    -h/--help       Print this help notice.
    -p/--psm        Use the Provisioning Simulation Module

    -o/--output {txt,json}
            Dump the information about your nodes and jobs in ascii tables (txt) or json 
            array (json).

    -c/--config <config_folder> 
            Use an alternate config folder. By default the software will check in 
            /etc/schlouder/.

    -d/--deadline <deadline_timestamp>
            Use a deadline constrained strategy and execute the workload before 
            deadline_timestamp.

    The bag-of-tasks file contains a list of batch script with arguments plus 
    possibly some dependency. Each batch script respect the following model.

    Batch:

      The batch file is simply a script which will be executed on the allocated node, in
      this script you have to pass some argument. You simply have to add some comments using
      this syntax :

        #SBATCH --option=value

      Options list :

        --job-name/-J Name of the job
        --workdir/-D Working directory in the node
        --time/-t Length of the job in seconds

    Dependencies:

      You can specify some dependencies to a job by adding to the beginning of the
      file the following comment:

        #DEPENDENCIES <job_name_dependence_1> <job_name_dependence_2> ... <job_name_dependence_n>

The mandatory workload file given to the schlouder-client executable is in a specific format. Each line contains a job. Find below an example of such a file:

    /path/to/job1.bash
    /path/to/job2.bash
    [...]
    /path/to/jobN.bash <arg1> <arg2> ... <argN>

Executing my first script through the PSM

The Provisioning Simulation Module (PSM) is a convenient way to test the outcome of using a provisioning strategy with a given workload. We can simulate the provisioning strategies using adapted simulation tools to predict their results. That is why we offer in Schlouder a Provisioning Simulation Module (PSM). This module call a simulator built upon the discrete event simulation toolkit SimGrid and can predict the number of BTUs (i.e. the cost) and the makespan for all the available strategies in the provisioning library. The PSM input is the same file describing the workload described in the previous section. To take into account the data transfer amount, you can specify the data input and output (in bytes). You must also specify the runtime prediction (in seconds). You just have to specify in the begining of the file the following lines:

    #PSM --data_in=XXXX
    #PSM --data_out=XXXX
    #PSM --runtime_prediction=XXXX

The simulator we rely on is called SimSchlouder and is described in its own website. Basically, the input is twofold: the target workload (i.e. duration and communications of its tasks) provided by the Schlouder engine, and a description of the platform. The description follows the SimGrid formalism and XML file format. Plug-and-play platform files, such as Amazon EC2, are already available. Its main output is the cost and makespan for each available provisioning strategy. However it can also provide very precise information about the simulation, which is useful to analyze and improve provisioning strategies offline.

You can call the PSM using the following command:

    schlouder-client -p|--psm <workload file>

The output will be a list of the strategies with the makespan and the cost like the following:

    PSM results are: 
    ==================================
    Strategy: AFAP
    Makespan: 3002.392708333152 seconds
    Total BTU: 1.0
    ==================================
    Strategy: ASAP
    [...]
    ==================================

Monitor your cloud

In order to have a good overview of the real characteristics of your cloud, we also provide a way to monitor the CPU power and the network bandwidth and latency of your VMs. The monitoring takes the form of a script executed on all your VMs.

Prerequisites

The monitoring script uses iperf to test the bandwidth, ping for the latency and linpack for the CPU power. The script copies the results in a file in /tmp on the controller via scp. Please ensure that the VM has an access to the controller or modify the script accordingly.

Monitoring 101

The script is located on the repository in the script/schlouder-job-monitoring.sh file. Its location is then specified in the configuration file of the server. In order to execute the monitoring jobs, use the following command:

    schlouder-client -m/--monitoring[=all|network-only|cpu-only] <workload file>

The monitoring option can take one of these three arguments:

  • all: Default value. The job will monitor both CPU and network.
  • network-only. The job will only monitor the network bandwidth and latency.
  • cpu-only. The job will only monitor the CPU.

Schlouder Output

For the moment, Schlouder provides two different outputs to understand the execution of an experiment: Text and JSON. We describe them in the following sections.

Schlouder Text Output

The text output is a convenient and easy to read way of showing the results of an execution for human being. It provides a list of the VMs started with the jobs executed on them along with different dates: Start/Stop dates of the VMs, beginning and end of the jobs' execution... This view does not provide a comprehensive overview of the system but provide enough information to visualize the state of the system at a given time. To display this output you need to call on the CLI:

    schlouder-client -o|--output txt

It will display on the standard output:

    .----------------------------------------------------------------------------.
    | ID         | Name       | Boot Date | BootTime  | Jobs End Date | Shutdown |
    +------------+------------+-----------+-----------+---------------+----------+
    | i-42F007D7 | icps-vm-14 | 11:56:47  | 63 (50)   | 12:03:08      | 12:55:48 |

    ...

    | i-4BDD0843 | icps-vm-39 | 11:56:47  | 305 (301) | 12:03:20      | 12:56:45 |
    '------------+------------+-----------+-----------+---------------+----------'
    .------------------------------------------------------------------------------------------------------------------------------.
    |                                         i-42F007D7 - icps-vm-14 (11:56:47) + 63 (50)                                         |
    +------+-------+-----+---------------------+------------+----------------------------+------------+----------------------------+
    | ID   | State | Pos | Name                | Submission | Duration (real / in batch) | Start Date | End Date (real / in batch) |
    +------+-------+-----+---------------------+------------+----------------------------+------------+----------------------------+
    | 9372 | F     |     | XXXXXX              | 11:56:37   | (198 / 195)                | 11:57:50   | (12:01:08 / 12:01:05)      |
    | 9414 | R     |     | YYYYYY              | 11:56:41   | (120 / 120)                | 12:01:08   | (12:03:08 / 12:03:08)      |
    | 9430 |       |  1  | ZZZZZZ              | 11:56:41   | (60 / 60  )                | 12:03:10   | (12:04:10 / 12:04:10)      |
    '------+-------+-----+---------------------+------------+----------------------------+------------+----------------------------'

    ...

    .------------------------------------------------------------------------------------------------------------------------------.
    |                                        i-4BDD0843 - icps-vm-39 (11:56:47) + 305 (301)                                        |
    +------+-------+-----+---------------------+------------+----------------------------+------------+----------------------------+
    | ID   | State | Pos | Name                | Submission | Duration (real / in batch) | Start Date | End Date (real / in batch) |
    +------+-------+-----+---------------------+------------+----------------------------+------------+----------------------------+
    | 9431 | F     |     | AAAAAA              | 11:56:43   | (88 / 58)                  | 12:01:52   | (12:03:20 / 12:02:50)      |
    '------+-------+-----+---------------------+------------+----------------------------+------------+----------------------------'

Schlouder JSON Output

The JSON output is the only way to get comprehensive data about the execution. It provides the list of the VMs and the jobs executing on it along with all the date and time gathered by Schlouder in the JSON format. To display this output you need to call on the CLI:

    schlouder-client -o|--output json

It will display on the standard output:

    {
       "info" : {
          "version" : "4ae8a592e79c4f820603e326f484084a76d08ec7",
          "start_date" : 1351853693,
          "description" : ""
       },
       "nodes" : [
          {
             "instance_id" : "i-42F007D7",
             "start_date" : 1351853807,
             "stop_date" : 1351857348,
             "boot_time" : 63,
             "boot_time_prediction" : 99,
             "host" : "ec2-vm-14",
             "cloud" : "amazon_ec2",
             "user" : "user1",
             "instance_type" : "m1.small",
             "jobs" : [
                {
                   "id" : 9372,
                   "walltime" : 198,
                   "walltime_prediction" : "195",
                   "command_line" : "/tmp/exec_script <options>",
                   "name" : "Job name",
                   "provisioning_strategy" : "Schlouder::Provisioning::Asap",
                   "start_date" : 1351853870,
                   "submission_date" : 1351853797
                },

                ...

             ]
          },

          ...
       ]