Cluster computing 2

Improved efficiency

Copying the python code to all the nodes is tedious and time consuming so here is a better way to do things. This could be done with a bash script but since python can execute bash commands we will stick with that.

nano execute.py

import os
import sys
from time import sleep

programme_name = sys.argv[1]
nodes = sys.argv[2]

for i in range(1, 4):
	cmd = 'scp ' + programme_name + ' andrew@node' + str(i) + ':/home/andrew'
	os.system(cmd)
sleep(1)
cmd = 'mpiexec -hostfile machinefile -n ' + nodes + ' python ' + programme_name
os.system(cmd)

To use this script to run send_recv_2.py use the following command.

python execute.py send_recv_2.py 4

The script uses the argv's passed from the command line. Argv[0] is send_recv_2.py, argv[1] is send_recv_2.py and argv[2] is 4. Note that the script being executed, send_recv_2.py in this case has to be configured for the correct number of notes.

andrew@master:~ $ python execute.py send_recv_2.py 4
send_recv_2.py                         100%  682   305.8KB/s   00:00    
send_recv_2.py                         100%  682   327.9KB/s   00:00    
send_recv_2.py                         100%  682   342.4KB/s   00:00    
Rank 1 on node node1 received 5.
Rank 2 on node node2 received 10.
Sent 5 to dest 1
Sent 10 to dest 2
Sent 15 to dest 3
Rank 3 on node node3 received 15.
andrew@master:~ $ 

Last modified: 22 April 2024