Size not a problem, nor is desert heat
A bigger structure of the same shape has less surface area per unit of volume, thus is easier to insulate. The small solar house is really more of a challenge, not easier.
As for the Arizona desert brought up in the first comment - that's where solar energy is most abundant, the easiest place to operate an all-solar house. The fact that it's hot outside just means you need lots of insulation.