SoA Vectors on SPU

One approach is to use an AoSoA (read: Array of Struct of Array) approach which is a hybrid of AoS and SoA. The idea is to store N structs worth of data in a contiguous chunk in SoA form, then the next N structs worth in SoA form.

Your AoS form for 16 vectors (labelled 0,1,2...F), swizzled at granularity of 4 structs is:

000111222333444555666777888999AAABBBCCCDDDEEEFFF
XYZXYZXYZXYZXYZXYZXYZXYZXYZXYZXYZXYZXYZXYZXYZXYZ

for SoA, this is:

0123456789ABCDEF
XXXXXXXXXXXXXXXX

0123456789ABCDEF
YYYYYYYYYYYYYYYY

0123456789ABCDEF
ZZZZZZZZZZZZZZZZ

for AoSoA, this becomes:

01230123012345674567456789AB89AB89ABCDEFCDEFCDEF
XXXXYYYYZZZZXXXXYYYYZZZZXXXXYYYYZZZZXXXXYYYYZZZZ

The AoSoA approach has the following benefits of AoS:

• Only a single DMA transfer is required to transfer a chunk of structs to SPU local memory.
• structs still have a chance of all data fitting in a cacheline.
• Block prefetching is still very easy.

The AoSoA approach also has these benefits of SoA form:

• You can load data from SPU local memory directly into 128-bit vector registers without having to swizzle your data.
• You can still operate on 4 structs at once.
• You can fully utilize the SIMD'ness of your vector processor if there is no basic branching (ie. no unused lanes in your vector arithmetic)

The AoSoA approach still has some of these drawbacks of SoA form:

• object management has to be done at swizzling granularity.
• random access writes of a full struct now needs to touch scattered memory.
• (these can turn out to be non-issues depending on how you organize/manage your structs and their lifetime)

BTW, these AoSoA concepts apply very well to SSE/AVX/LRBni, as well as GPUs which can be likened to very wide SIMD processors eg. 32/48/64 wide depending on the vendor/architecture.

SPUs are actually an interesting special case when it comes to vectorizing code. Instructions are divided into "arithmetic" and "load/store" families, and the two families run on separate pipelines. The SPU can issue one of each type per cycle.

Math code is obviously heavily bound by math instructions - so usually mathy loops on SPU will have lots and lots of open cycles on the load/store pipe. Since shuffles happen on the load/store pipe, you often have enough free load/store instructions to swizzle xyzxyzxyzxyz form into xxxxyyyyzzzz form without any overhead at all.

This technique is in use at Naughty Dog at least - see their SPU assembly presentations (part 1 and part 2) for details.

Unfortunately the compiler is often not smart enough to do this automatically - if you decide to go this route you'll need to either write assembly yourself or unroll your loops using intrinsics and check the assembler to make sure it's what you want. So if you are looking to write general cross-platform code that happens to run well on SPU, you may want to go with SoA or AoSoA (as jpaver suggests.)

As with any optimisations, profile! Readability comes first, and should only be sacrificed when profiling identifies a particular bottleneck and you have exhausted all your options for tuning the high level algorithm (the fastest way to do the work is to not have to do the work!) You should always reprofile following any low level optimisation to confirm that you really have made things faster rather than the opposite, especially with pipelines as quirky as the Cell's.

What techniques you use then will depend on the particulars of the bottleneck. In general, when working with vector types, a vector component you ignore in a result represents work wasted. Switching SoA / AoS does not make sense unless it allows you to do more useful work by filling such unused components (e.g. one dot product on PS3's PPU vs four dot products in parallel in the same amount of time). To address your question, spending time shuffling components around just to perform one operation on a single vector sounds like a pessimisation to me!

The flip side on SPUs is that the bulk of the cost of small DMA transfers is in setup; anything less than 128 bytes will take the same number of cycles to transfer, and anything less than about a kilobyte only a few cycles more. So don't worry about DMAing more data than you strictly need; reducing the number of sequential DMA transfers triggered, and performing work while DMA transfers are happening - and therefore unfolding loop prologues and epilogues to form software pipelines - is key to good SPU performance, and it is easiest to deal with corner cases by fetching extra data / discarding partially computed results than jumping through hoops to try to arrange for the exact amount of data that is necessary to be read and processed.

No, that would not make much sense in general as most vector opcodes operate on a vector as a whole and not on seperate components. So you can already multiply a vector in 1 instruction, whereas with splitting up the seperate components you'd spend 4 instructions on it. So since you basically do a lot of operations in general on part of a struct you're better of packing them in an array, but you hardly ever do things only on one component of a vector, or wildly different on each component so breaking them out wouldn't work.
Of course, if you do find a situation where you have to do something to only the (say) x components of vectors it might work, however the penalty of swizzling everything back when you need the actual vector wouldn't be cheap so you could wonder if you shouldn't use vectors to begin with but just an array of floats that happen to allow for vector opcodes to do their specific calculations.