Up until now we looked at:
- parallel / concurrent
- fork-join / threads
- oop on shared memory
- locking / lock-free / transactional
- semaphores / memory
→ all of these assumed shared memory space
18.1 Distributed Memory
There are two alternatives:
- functional programming
- immutable state → no synchronization required
- message passing: isolated mutable state
- state is mutable → but not shared: each thread/task has it’s private state
- tasks cooperate via message passing
18.1.1 Isolated Mutable State
Each thread has it’s own isolated mutable state. They then exchange messages:
Example the basic counter
With isolated mutability: each thread increments their own counter locally
if we want to get a global count
reducefunction (map-reduce) → gets the count from each and sums them up
Example Bank Account
Example bank account in a distributed memory model
18.1 Types of Messages
Types of messages
18.2 Message Parsing Interface (MPI)
We hide the hardware/software details behind a standardised software library.
- MPI has C/C++, FORTRAN, etc… libraries
18.2.1 MPI Specifics
We arrange processes into communicators.
There is initially a COMM_WORLD communicator which is global.
We can then have communicators with only certain processes.
Example Different communicators
we can create copies of the communicators and / or create new ones.
Different communicators assign different ranks to processes.
- So one process has multiple ranks (one for every communicator).
Code Example Access rank in a communicator
18.2.2 Single Program Multiple Data (SPMD)
we write a single program that has branching that differentiates each processes job based on their ranks.
- they then work on different data
18.2.3 Communication
To communicate we use the Comm.send function. We send a certain memory region (offset = pointer, count (bytes to send, calculated with sizeof(type)).
Datatype: We can only send basic data types. To send a list, we need to “assemble” it ourselves on the other end.
Tag: integer marking a message with a certain type, to have different message types, handled differently.
Message Matching: how messages get to the right place
Receiving messages:
Note: there are special arguments to receive any message, any tag for catch-all.
Specifically Synchronous Message Passing (rarely used)
Asynchronous Send (usually used)
Note When we send very large messages it may be synchronous, when there is not enough memory.
→ 1 GB messages will be stuck, because we need to get that memory somewhere to buffer the messages.
18.2.4 Blocking / Nonblocking
We can have different types of calls, depending on if we want to re-use the memory allocated for keeping the memory.
Blocking
- when the MPI function returns, we are free to re-use the memory → already sent
Non-blocking
- we reserve the memory until completed and return immediately
- this frees us up to do other work.
→ but we may never modify that memory
We get a handle that we can use to test if message has been sent and buffer can be re-used.
Example Streaming
Using non-blocking sending, we can compute while sending has started → streaming.
18.2.5 Summary
18.3 Deadlocks in MPI
When both processes send first, if we have insufficient storage at the destination
→ blocks
thus this could get stuck.
We need to order our operations more carefully.
or use special functions.
Even more solutions:
We use non-blocking operations on the data.
Since they immediately return, we can’t block.
→ always solves the deadlock problem.
18.4 MPI Recap
