I have been trying to get rid of Google completely on my Android phone. As I have made some nice progress I'd like to share what I did and which problems I had to overcome:

OS I removed my stock version of Android and replaced it with Cyanogen Mod (http://www.cyanogenmod.org/) CM doesn't ship the Google Services, you have to install them manualy afterwards. I didn't do that, so most of Google was gone already. This produced a lot of problems, because Google Play isn't available plus the Maps Framework is used by many 3rd party apps.

Appstore The biggest showstopper is the missig appstore. I installed F-Droid (https://f-droid.org/) which is a nice appstore that provides open-source apps only. This provided me with some apps i use regualry (Twitter-client and some others) but is in no way a replacement for Google Play. Now I had to decide if I wanted to trust other 3rd party appstores or not. I decided to don't. Amazon has it's own "trustworthy" appstore, but apps are patched and crippled. didn't want to have that. Decided to download needed apps with another Android device. After that I grab them from /data/app on the other device with Root Explorer (https://play.google.com/store/apps/details?id=com.speedsoftware.rootexplorer&hl=de) and sync them to my real device with Owncloud. This ugly process basicly made me use less apps on my phone because it's such a hassle to get them on it. I got used to it after some time and reduced my app selection to a sane minimum.

Contacts, Mail and Calendar sync I setup an installation of Horde Groupware (http://www.horde.org) on my personal server. Horde features "Microsoft Active Sync" compatibility. On Android, I just had to add an Active Sync Account to sync all my stuff to my own server. Works perfectly.

Maps Framework Many apps that embed a Google Map somewhere use the "Google Maps Framework" which comes with Google Maps. A specific app I use often depends on the framework. The app won't start without the framework, even I don't need the embedded maps inside. Luckyly some guy from XDA wrote a script that simulates an installed Maps framework without it being there actually. (Couldn't find link, too long ago)

Maps Checked out many mapping applications based on OpenStreetMaps. MapFactor worked best for me (https://play.google.com/store/apps/details?id=com.mapfactor.navigator&hl=de). It's not as good as Google Maps but it works.

Browser Andorid browser that ships with Android has builtin Google-Search. I replaced it with Lightning (https://play.google.com/store/apps/details?id=acr.browser.barebones), a small and quick open souce browser which can be configured to use many different search engines. I set it to Startpage.

Music My PlayStore license for Poweramp didn't work on my devices without Google Licensing Services, provided by Play (a problem with many pay-apps, not all though). Luckily the dev offers a version, bound to an e-mail account. I re-bought the software.

That's basicly it. I'll sniff the traffic of my phone, if I find the time, to verfiy I really got rid of Google on my phone. (I bet i forgot something)