Tk-Cocoa 2.0



I've recently committed several updates to to the Cocoa/Mac port of the Tk GUI toolkit that change the toolkit extensively enough that I'm calling it, only somewhat in jest, Tk-Cocoa 2.0.

Tk was initially ported to Cocoa in 2009 by Daniel Steffen, who had maintained Tk for several years. Apple sponsored the port of Tk to Cocoa after it deprecated its Carbon frameworks, which had previously been the foundation for Tk. The Cocoa port was a top-to-bottom rewrite, essentially cut from whole cloth rather than a simple update or port. Daniel Steffen had to create an entirely new code architecture for Tk to work with Cocoa, which has a significantly different, higher-level approach to GUI programming than Tk, based on a lower-level model derived from Unix's Xlib library (and which had strong parallels with Carbon). The resulting Cocoa-based Tk was a remarkable piece of design, and brought many advantages in speed and system integration.

After Tk-Cocoa was released in the summer of 2009, Daniel did a major round of updates and bug fixes to smooth out some rough edges in the initial release, and then left Tcl/Tk development to join Apple full-time. A bit later, after releasing some Tcl/Tk extensions that integrated with parts of Cocoa and contributing some patches to the Tk core, I took over as maintainer.

Over the next couple of years, as users gained more experience with Tk-Cocoa, some limitations of the system became apparent. Apart from the standard-issue bugs and glitches that are part of any software project, and which were relatively easy to fix, the differing way Tk and Cocoa handled user input and processing of events caused a variety of issues. Applications would occasionally freeze, and stop responding to user input, because they would become overloaded with events to process. There would be occasional issues with drawing and flickering. These issues were structural, not simply glitches that can be fixed with a few lines of code. The complexity involved would have required a major redesign of Cocoa and Tk's integration, something Daniel Steffen was precluded from doing as an Apple employee (Apple's developers are not allowed to participate in open source projects in most cases); in any case it was not even entirely clear if the integration could be designed differently.

Another issue also arose, not technical but one of policy: Tk-Cocoa, at a deep level, made use of private, undocumented Apple API's to improve drawing performance, using methods borrowed from Apple's open-source WebKit project. This practice, while always discouraged by Apple, took on greater importance when Apple introduced the Mac App Store, which stated that apps making use of private API's would be rejected. This was a serious issues for developers who, like me, wanted to distribute Tcl/Tk apps through the Mac App Store; the only workaround at the time was to link my applications to the somewhat buggy version of Tk-Cocoa that Apple shipped with OS X. (System libraries are allowed to use private API's.)

This was the situation for a couple of years: a lot of low-level frustration with the structural limitations of Tk-Cocoa, no clear path to fixing them, just incremental fixes of bugs, and a handful of Tcl/Tk apps in the Mac App Store linking to an aging version of Tk that did not include any improvements. In the summer of 2014, however, after working with Marc Culler on some patches he submitted to improve how images are rendered in Tk/Cocoa (some OS updates had broken Tk's previous mechanism), I decided to take another look at the deeper issues in Tk.

I found, to my surprise, that Tk would work without the private API calls--so I removed them altogether. I also saw, however, that Tk worked much better with them included. Their removal opened a Pandora's box of bugs and very serious drawing issues: vast amounts of flickering, ghostly images of buttons and scrollbars appearing at random whenever a window was resized or changed in any way. Except for Apple's policy about private API's, which presented a major obstacle to using Tcl/Tk in via its premier distribution channel, I would rather have left the private API's in. They made Tk/Cocoa run much better. But because of Apple's policies, they had to go--and their removal left a huge mess behind.

After doing some more work with Tk without the private API's, the issues started to sort themselves out into some predictable patterns, which made it possible to chart a plan to address them. These patterns included:

Flickering during resizing and moving of windows.

Weird flickering and ghostly images of Tk widgets that were based on native Cocoa widgets--specifically, buttons and menubuttons (NSButton) and scrollbars (NSScroller).

Widgets not based on Cocoa controls, such as listboxes, and widgets based on a different native API, HITheme, did not display the same flickering.

The common denominator between these bugs seemed to be that Tk-Cocoa had trouble managing a complex hierarchy of Cocoa views in a window. Tk-Cocoa's design involves using a single NSView in an NSWindow to render the layout of a Tk interface. It handles widgets drawn by Tk that are not directly derived from Cocoa widgets, such as listboxes, just fine. But NSButton and NSScroller widgets (themselves based on NSView) present added complexity that Tk seems unable to manage easily without the private API's.

The solution to these issues focused on reducing the complexity of what Tk-Cocoa had to draw:

Suppressing drawing during a window resize operation, which reduced flickering in that context down to almost nothing.

Replacing the NS-based widgets (buttons and scrollbars) with widgets based on the HITheme API. HITheme, while nominally part of Carbon, is still fully supported on the Mac because it is only a drawing API, useful in a variety of contexts, and still widely used by Apple in WebKit.

The window-resize code I developed on my own, with a lot of trial and error and lots of time on Google. The HITheme code, however, was heavily based on work that others had done in past years--a good thing, because that was a much more complicated task.

The button code, both for regular pushbuttons and popup menubuttons, was based on a patch developed a decade ago by Revar Desmera, a polyglot developer who has done a number of projects in Tcl/Tk. Revar's patch, which added support for the HITheme API to Tk buttons, was never adopted, but it was still sitting in Tk's code tracker and provided the perfect foundation. I had to tweak and update some things, but that work only took a week or two. I am grateful to Revar for his initial work.

The scrolling code was the hardest piece of this puzzle. There was no existing HITheme scrolling implementation for me to tweak, so I had to start from scratch; I worked on the scrolling code for about a month with only modest progress, borrowing some code from Tk's older Carbon implementation and its Unix implementation, which is the foundation for both the Mac and Windows ports of Tk. I got the scrolling code to a point where it would draw the scrollbar, and nothing more. Finally, after rummaging around in Tk's source tree, I found an old patch from Donal Fellows, one of Tcl/Tk's core developers, which fixed a bug in the Mac's other HITheme-derived widgets (scale and progress bar) that seemed suspiciously like the scrollbar issue: the widgets had to apply some kind of multiplier to set their values correctly, and thus redraw. (The math was, and is, over my head.) Once I got that placed into the scrollbar code, things progressed quickly. The final piece of the puzzle was to get the scrollbar to respond to being dragged by the mouse; my initial work here focused on bringing a lot of the old Carbon code back into Cocoa, and updating it somehow. The final solution was beautifully simple and elegant. Based on an observation from Jim Ingham, Tk's original maintainer on the Mac, that the HITheme API would allow Tk/Mac's scrollbar code to become more like its Unix version, that's what I decided to do: throw out all the old Carbon code, import mostly Unix code, and just let the HITheme scrollbar be redrawn by the native API's, not moved and driven by them. Bingo.

I've committed all these updates to Tk's main line of development (trunk) and its still-supported 8.5 version this week, and they are ready to be tested and used. Here is the difference these changes make in Tk-Cocoa:

Resizing windows now works almost completely without flicker. I accomplished this by disabling drawing during the resize event and deferring visual updates until the resize is done. The windows no longer have the "live-resize" visual effect that most Cocoa applications have; they simply snap to the new dimensions when the mouse is lifted. This is far better, however, than having flicker and ghostly images.

Flicker with the buttons and the scrollbars is gone. Visually the HITheme buttons and scrollbar are almost identical to the Cocoa-based ones; any difference is subtle and is not likely to be noticed by most users.

While this is subjective, to me Tk-Cocoa feels a bit more responsive, a bit snappier, with these changes. Even with the private API's to ensure smooth drawing without much flicker, Tk-Cocoa sometimes felt sluggish and unresponsive. I used to blame this on the event-loop issues, but now I suspect that feeling was more directly related to the overhead that Tk-Cocoa had in trying to manage multiple NSViews within a single application window. Tk-Cocoa's drawing is greatly simplified without that overhead; now all widgets in the window are Tk-owned, just regions on a screen to be drawn in a single NSView, without added complexity of Cocoa's responder chain and view hierarchy.

Let me be clear: Tk-Cocoa remains Cocoa-based. I have left most of Daniel Steffen's brilliant architecture unchanged: Tk's app structure is still based on the NSApp API, it still has the same complex event loop integration, and it still draws into an NSWindow containing an NSView. There still may be bugs owing to the complexity that cannot be changed. However, by making these changes to Tk-Cocoa's drawing code, I have removed some of the complexity that was there before, and brought Tk a bit closer to the level of abstraction it had under Carbon, and retains under both X11 (Unix) and Windows, where it is most performant. As a result, I feel this is a significant evolution of Tk-Cocoa, and an improvement, and I do not feel embarrassed in calling it Tk-Cocoa 2.0.

To all Tk/Mac developers and users out there: I hope you find these changes helpful. And to the developers named here and to those who have provided encouragement and feedback: thank you. While at times I have complained that I work alone, this is not true: I have relied on the contributions of many, both of code and feedback, and I am grateful.

[/software] permanent link