If the tree grows up against the garden wall then either the tree will be crowded and stunted or the wall will be pushed out.