This comes up every time someone orders a Z4 and then adds memory later. HP's own service documentation for the Z4 family specifies the population order, and it is not simply "fill the slots nearest the processor".
The documented behaviour is that populating fewer channels reduces available memory bandwidth regardless of total capacity. A machine with more gigabytes across fewer channels can be slower on bandwidth-sensitive work than a smaller balanced configuration.
Check the service guide for your exact Z4 generation before ordering. The slot map is not consistent across generations and guessing from a previous machine is how people end up with an unbalanced configuration.