One of the great white hopes of the part of the Unix world that never liked X Windows was Sun's NeWS. Never mind all of its practical flaws, all sorts of people held NeWS up as the better way and the bright future that could have been if only things had been different (by which they mean if people had made the 'right' choice instead of settling for X Windows). One of the reasons people often give for liking NeWS is that it put much of the windowing toolkit into the server instead of forcing every client to implement it separately.

Unfortunately for all of these people, history has fairly conclusively shown that NeWS was a mistake. Specifically the core design of putting as much intelligence as possible into the server instead of the clients has turned out to be a terrible idea. There are at least two big reasons for this.

The first is parallelization. In the increasingly multi-core world you desperately want as much concurrent processing as possible and it's much easier to run several clients in parallel than it is to parallelize a single server. Even if you do get equal parallelization, separate clients are inherently more resilient because the operating system intrinsically imposes a strong separation of address space and so on, something that's very hard to get in server where everything is jumbled together.

(I believe that this is one reason that modern X font rendering has been moved from the server to the client. XFT font rendering is increasingly complex and CPU-consuming, so it's better to stick clients with that burden than dump all of it on the server.)

The second is that if you put the toolkit in the server you make evolving the toolkit and its API much more complicated and problematic. The drawback of having everyone use the server toolkit is that everyone has to use the same server toolkit. Well, not completely. You can introduce a mechanism to have multiple toolkit versions and APIs all in the same server and allow clients to select which one they want or need and so on and so forth. The mess of a situation with the current X server and its extensions make a very educational example of what happens if you go down this path; not very much of it is good.

(Some X extensions are in practice mandatory but still must be probed for and negotiated by the clients, while others are basically historical relics but they still can't be dropped because some client somewhere may ask for them.)

Toolkits in the client push the burden of dealing with the evolution of the toolkit into the clients. It is clients that carry around old or new versions of the toolkit, with various different APIs, and you naturally have old toolkit versions (and even old toolkits) go away entirely when they are no longer used by any active clients (or even any installed clients, when things get far enough).

(I'm ignoring potential security issues for complex reasons, but they may be a good third reason to be unhappy with server-side toolkits.)