What all ages books do best is they don't discriminate. They widen the audience without necessarily succumbing to a lowest common denominator.

If I were to set the target audience to say a 4th grade reading level, I would probably already cut out about 1/3 of all the fourth graders around because they have tested at a higher reading level and will be bored by my restricted vocabulary and the sense of the text talking down to them. And needless to say I've cut out most everyone above that age range as well. Where if, instead, I make an all ages book where the concepts covered are not inappropriate for that age group, but I use a higher vocabulary level, the child will either be challenged and figure it out on their own through context, or ask for help. I love that Mouse Guard is a book parents and their kids read and enjoy together.