Biowulf/IME

 

The Biowulf/IME cluster is a Beowulf parallel processing system built at Instituto de Matemática e Estatística (IME-USP). Managed by Martha Torres Ph.D.

This cluster is funded by FAPESP in the Cooperation for Analysis of Gene Expression Project CAGE (process No.99/07390-0)

The Biowulf/IME provides a high performance computing platform for development of computational tools utilized in bioinformatic applications as the sequence data mining procedure, Gene expression Clustering and techniques for lattice dynamical system identification.

 

 

 

Hardware Configuration

Software Configuration

            Filesystem Overview

Benchmarking

Using the Biowulf/IME

Publications

Research

Biowulf/IME hardware configuration

 

 

The 16 PCs are connected by a Fast Ethernet switch (using hostnames node1 through node16).

 Node1 is the server and its name is tiramisu.ime.usp.br for external access. The cluster is connected to the IME network via 100 Mbit/sec

The installation of all nodes was done by the automatic installation tool FAI.

 

The Biowulf/IME currently consists of 16 AMD PCs. Each AMD PC is configured as follows:

top

Biowulf/IME Software configuration

 

 

top

Benchmarking

Sustainable Memory Bandwidth in High Performance Computers is a benchmark to test memory bandwidth of a computer system.

 

 All results are in MB/s --- 1 MB=10 6 B, *not* 220 B

 

 

 

Machine ID               

ncpus         

COPY

SCALE

ADD          

TRIAD

Biowulf/IME node                        

1   

    581.8182      

556.5217       

662.0690       

518.9189           

Asus_Athlon_1300_7ZX     

1 

    474.1

474.1         

600.0   

518.9

 

data1

data2 

 

Comparison with other beowulfs:

swarm

galaxy

 

 

Node 1
 
Node 1

 

Comparison with other beowulfs:

galaxy

    NetPipe maps the performance of a network. The following graphs illustrates latency and throughput for fast ethernet with a switched configuration.

 

Figure 1: Mbits/sec vs Packet size on Biowulf/IME for TCP Send/Recv tests

 

Figure 2: Mbits/sec vs Packet size on Biowulf/IME for MPI Send/Recv tests

 

Comparisons with other beowulfs:

IBMCluster

Brahma                       

Yggdrasil 

Valhal

 

 

Mpptest is a program that measures the performance of some of the basic MPI message passing routines.

Ping Pong Latency

Figure 3 Roundtrip results - processes on two nodes connected to the same switch

 

Comparison with other beowulf

hive

goptest is a program that measures the performance of MPI collective operations.

 

Barrier

Broadcast

top 

Using the Biowulf/IME

 

There is only one way to connect to the Biowulf/IME, that being the public domain secure shell, or ssh.  All other utilities--including rsh, telnet,

and rlogin--will not work because their corresponding daemons have been turned off due to security concerns.   Therefore, you must  connect from a machine that has ssh installed. 

The ssh utility works very much like rsh.  Most often, you will use it to log into another machine, for example:

                ssh tiramisu.ime.usp.br  

 

 Another tool in the ssh package is called scp, which works very much like rcp.   This utility is used to copy a file from the local machine to a remote machine, for example:

              scp file1 nome.domain:file2  

 though it can be used in the opposite direction as well, for example:

                scp nome.domain:file1 file2  

The Biowulf/IME is accessed from the IME network via ssh to tiramisu.ime.usp.br. Access to any of the 15 processing nodes is gained from node1 (tiramisu.ime.usp.br) using ssh. The node names have been assigned as node#, where # is a number ranging 1 to 16.

 

In order to log in all nodes without passwd. Please execute the following steps:

  1. execute "ssh-keygen -t rsa"

2. change to ~user/.ssh

3. copy "id_rsa.pub" file for "authorized_keys2" file

4. Log with ssh for all nodes. It is created the "known_hosts" file.

5. After the login first, the system will not ask you, your password.  

  • Compiling MPI Programs
    MPICH 1.2.1, a public-domain implementation of MPI from Argonne National Labs, is already installed.There is extensive documentation online, but the general use of MPICH is describe here, in particular how to build and run MPI programs with it. The MPICH API conforms to the MPI specification.
    To compile an MPI program you should use mpicc (C programs), mpif77 (F77 programs), mpif90 (F90 programs), or mpiCC(C++programs).For example, the following command should be used to compile a program named MPIprogram.c, placing the resulting
    executable in a file named MPIprogram
                   /usr/local/mpich/bin/mpicc -O2 MPIprogram.c -o MPIprogram  
     
    To run this executable across 4 nodes, log into node1 (tiramisu.ime.usp.br) and run:
     
                  /usr/local/mpich/bin/mpirun -np 4 -nolocal -machinefile machines.mpich MPIprogram  
     
    If the program needs arguments passed to it, try;
     
                   /usr/local/mpich/bin/mpirun -np 4 -nolocal -machinefile machines.mpich MPIprogram Arg1 Arg2 ... ArgN  
     
    You can specify which machines to run the computation on via a machine file.   A machine file is an ASCII file containing the names of   machines, one per line, to use as nodes in your computation. For example, suppose the file machines.mpich contains the following four   lines:
                   node2
                   node4
                   node6
    then your computation will be run on the three nodes node2, node4 and node6. 

 

User Manual
 

top

Filesystem Configuration

 

Path

Type

Size

Notes

/

local

500MB

Local node storage

/tmp

local

1 GB

Temporary node storage

/var

local

500MB

Local node storage

/usr

local

3 GB

Local node storage

/usr/local

NFS

3 GB

Aditional software

/swap

local

2 x 768MB

Local node storage

/home

NFS

14 GB

Your home directory - (Class Users)

/scratch1

local

5 GB

Local node storage
  • Filesystems available on processing nodes:

Path

Type

Size

Notes

/

local

500MB

Local node storage

/tmp

local

1 GB

Temporary node storage

/var

local

1 GB

Local node storage

/swap

local

2 x 768MB

Local node storage

/scratch1

local

5 GB

Local node storage

/scratch2

local

17 GB

Reserved to PVFS

 

home: Your home directory is in NFS (Network File System) mount. The same directory will be mounted on all of the nodes, and data written there will be visible everywhere. You can use this to import files to each node, and to store long term data on.

scratch: This directory exists on all the nodes, but each one is a seperate, locally mounted disk. You can use this for temporary scratch or mid-term storage on each node.

tmp: This is a locally mounted disk on each node. You can use this for scratch, but do not store anything that you care about here. It will get wiped from time to time.

Note that NFS mounts are signifigantly slower than local disks. The NFS mounts are there for your convinience, and centralization of data. The local disks are there for speed and locality.

Local Disk

Local Scratch space is the storage space unique to each individual node. Local Scratch can be accessed on each node through  /scratch1/. This space will be the fastest. Users would run their jobs from the local scratch space.

 PVFS

 In the future, this space will have a medium  size of about 255 GB, which is shared throughout all of the nodes.  

 NFS

This space is visible to all of the nodes of The Biowulf/IME  through an auto-mounting system when users   log in. This space has ample storage but not much speed.

top

Publications

Parallel Incremental Splitting of Intervals for Boolean Function Minimization, Martha Torres, Nina S. T. Hirata and Junior Barrera. Submitted to IPDPS 2002

top

Research

top

This page was written and is maintained by Martha Torres mxtd@vision.ime.usp.br
Last modified 4/12/2001