Write your parallel code as if the machine doesn't have a shared memory system (more distributed), and then if it has, spend time to optimize it for shared memory.
If there is a doubt in your mind that you need a sync statement -
use it!. Also, help yourself (and potentially others) out by commenting on how confident you are on this sync statement.
Optimize later is a good motto to keep in mind when parallelizing code.
They usually have very short walltimes (maybe something like 5 minutes or so) Good for rapidly debugging your code, but they might have very short walltimes - (like 5 minutes!).
Examples of such clusters here at Penn State: Tesla (has GPUs), and Hammer
More queue oriented. Has a scheduler, and a load-balancer
Gigabit has very low latency vs Infiniband. Moreover, Infiniband has more bandwidth.
When you are starting out it might get good get the cluster to send you an email when your program starts, ends, and/or aborts. It can be done per-core or per-node.
Too frequent updates on caches between different processors. This can severly hurt application performance.
You can intentionally group all your writes into a single job.
Can be spread across many processors. Each processor owns a chunk of the array. Other processors can ask for other chunks of it - costs communication time though. As slow as the network is, it is still faster than a hard-disk, if you really need a lot of memory using distributed arrays might be a good choice.
This type of array is shared between multiple cores at a workstation. So with a shared array, there is one copy of the array in the shared memory, and the multiple cores each have a part of it, but all of them can access the other parts that are on the other cores (this access is slower than just accessing the local part of the array).
Downside: You have to be careful if you have multiple processes that are reading and writing simultaneously to the shared array. You can use
sync or other types of
barriers to prevent running into unvanted troubles, but if you have a lot of synchronization statements then the parallization might not work to well for you.
Advice: If there is a doubt in your mind that you need a sync statement -
Also, help yourself (and others) out by commenting on how confident you are on this sync statement.
Very good for debugging parallel code, and stress test it. People who work in cloud computing really love this.
Guðmundur Kári Stefánsson| 500px | vimeo | facebook |
email: gws5257 [at] psu.edu
09 March 2015 Installing a CDK24 Telescope at Penn State
06 December 2014 A Day in Pittsburgh
12 November 2014 HET trip - Results
11 November 2014 HET trip - day 1
01 October 2014 Black Moshannon State Park Observing
29 September 2014 HPF MLI blanket fabrication
19 September 2014 MLI Blankets
11 August 2014 HPF subsystem assembly
21 July 2014 Astrofest 2014
13 June 2014 HPF - Keeping it cool