Useful commands for Dolphin PCIe

Dolphin SuperSockers & IP over PCI Express (IPoPCIe)

Simple SuperSockets Instructions

#Run with supersockets
dis_ssocks_run iperf -c <hostname>
#Remember to use dis_ssocks_run on both sides

#Run with disip / IPoPCIe
iperf -c 192.168.0.xxx

#Run with ethernet
iperf -c <hostname>

SuperSockets Configuration 

#All tunables are in the configuration file
cat /etc/dis/supersockets_profiles.conf

#Set a tunable by setting the corresponding environment variable
export SSOCKS_DMA_MODE=0
#or temporaily for a single command:
SSOCKS_DMA_MODE=0 iperf ....

#Note: The configuration variables are per-node!

SuperSockets Diagnostics

#Statistics
watch -n .5 "/opt/DIS/bin/dis_ssocks_stat -G"

#Live transfer rates
/opt/DIS/bin/dis_ssocks_rate -d 0.5 

#Histogram -H 0 for operations, -H b for bytes
watch -n -.5 "/opt/DIS/bin/dis_ssocks_stat -H o"

MPI Commands

OpenMPI (mpirun)

#Note: mpirun will start the mpi program on all nodes - i.e. you don't need to run it on multiple machines yourselfes

#The -hostfile <file> is a text file with hostnames/ip addresses and parameters. You can also use -H host1,host2

#To avoid openmpi using multiple networks use:
#Only ethernet (or supersockets)
#-mca btl_tcp_if_include 10.174.0.0/16
#Only disip / IPoPCIe-mca btl_tcp_if_include 192.168.0.0/16
#-mca btl_tcp_if_include 192.168.0.0/16

#Ethernet
mpirun -mca btl_tcp_if_include 10.174.0.0/16 -display-map -hostfile hostfile --mca orte_base_help_aggregate 0 -n 2 /opt/osu/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw

#DISIP
mpirun -mca btl_tcp_if_include 192.168.0.0/16 -display-map -hostfile disiphost --mca orte_base_help_aggregate 0 -n 2 /opt/osu/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw

#Supersockets
mpirun -x LD_PRELOAD=libksupersockets.so -mca btl_tcp_if_include 10.174.0.0/16 -display-map -hostfile hostfile --mca orte_base_help_aggregate 0 -n 2 /opt/osu/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw

#Getting fancy
mpirun -x SSOCKS_MIN_LONG_SIZE=4081 -x SSOCKS_DMA_MODE=0 -x LD_PRELOAD=libksupersockets.so -mca btl_tcp_if_include 10.174.0.0/16 -display-map -hostfile hostfile --mca orte_base_help_aggregate 0 -n 2 /opt/osu/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw

Sample "hostfile" (for mpg-2014-13 & mpg-2014-14 - edit for your config)

#mpg-2014-11.mlab.no slots=1
#mpg-2014-12.mlab.no slots=1
mpg-2014-13.mlab.no slots=1
mpg-2014-14.mlab.no slots=1
#mpg-2014-15.mlab.no slots=1
#mpg-2014-16.mlab.no slots=1
#mpg-2014-17.mlab.no slots=1
#mpg-2014-18.mlab.no slots=1