There are already fine answers related to build systems and shared libs, but I'll tackle this from another angle I don't see people applying so often.

To me it's useful to separate stable code (as in unlikely to ever need further changes) from unstable code (code that naturally warrants further changes as the software expands). If you cannot do this and see the entirety of your codebase as potentially moving parts, even if the interfaces are completely stable, I think you are doing something wrong.

We already tend to do this naturally for open source third party libraries. We generally build them once, create a dylib ideally to avoid static linking times, and just look up symbols and call functions in them at runtime. We don't find a need to constantly rebuild them over and over since we have no interest in touching their source code, only using the available functionality -- basically build once and use forever. The code is 100% stable in that regard (there is no reason for it to ever change, at least not in our hands).

So for third parties I personally put them in a "third_parties" subdirectory and I only build that when I add a new third party library or replace an old version with a new version which is extremely rare, like once every few months minimum. I'm not rebuilding those over and over on a daily basis.

Same thing should be able to apply to your own codebase. I have a similar thing where I have a "libs" subdirectory with my own code and rarely ever build the resulting libraries since the code is very stable, reliable, efficient, tested thoroughly with unit tests, been through static analysis, and not something I need to change in the future. It's separated far away from the unstable code outside of the "libs" subdirectory which does need to change on a daily basis which does get rebuilt repeatedly.

So separating your codebase this way where stable parts you rarely, if ever, need to change from unstable parts which constantly warrant changes can be a useful way to organize your codebase -- not only in terms of optimizing build times but in order to better separate away stable packages from unstable packages with a goal to end up with as much stable code as possible that you can confidently say will not warrant any changes in any near future.

When you do that, those stable parts of your codebase can be built separately and away from the unstable parts using them. The unstable parts shouldn't take so long to build as your stable set of libraries expand and expand while the unstable parts shrink and shrink.

However, this tends to imply some code duplication. To be able to create a stable image library, it can't depend on, say, some unstable math library or else changes to the math library will warrant a rebuild if not further source changes to the image library. So I actually find it useful sometimes for, say, an image library to duplicate some math routines in order to become independent from any auxiliary math library. In that case, decoupling the code in this way through some modest duplication of logic can actually help make those packages a lot more stable, eliminating reasons for them to have to change and/or be rebuilt over and over. If the code is well-tested and works beautifully for years to come and might hardly ever have to be rebuilt, the modest duplication required for the library to achieve its independence and stability is hardly a problem.

Further it helps to achieve minimalism in your interfaces. If your image library aims to implement every single image operation ever imaginable to mankind, then its ambitions and monolithic nature will warrant neverending changes. It can never hope to become a stable library with such goals. If it only aims to provide basic image operations or even none at all with the ability to create image operations outside of the library using what the library provides, then it can potentially achieve a state of perfect stability (finding no reason to change in any near future).

So anyway, if you are going to start splitting up a codebase, I'd suggest to start off splitting the stable, well-tested parts where you're pretty sure you don't need to change them any further from the unstable parts where you at least anticipate some possibility of a need for future changes. Building the stable parts should invoke a separate build process (rarely applied) from the unstable parts (frequently rebuilt).