Unfortunately, all your projects will grow when the library is getting bigger - even if most contents of that library are not used. I tested it myself by creating an app A and a library L. If L is a library used in A, the classes.dex file (and therefore the A.apk file) is growing if I add more classes - even if they are not used.

To sum up: Right now I would create a basic library for certain things that are small and that may be used by many projects, and create a new library for every new component that is going to be larger and only is used by some projects. A good candidate for a new library would be a new UI component with multiple images defined in the resources. A good candidate for the base library are commonly-used methods and things like file caches, for example. Compiled code is compressed quite heavily for Dalvik, which you can see here. (The whole presentation is actually fun to watch :-)

Edit: If ProGuard is activated, it will also remove unused code for you. The default proguard.cfg is sufficient. It will not run on the (default) debug built, but when the final .apk is compiled. So it actually is possible!