Some time ago I wrote a post on form generation. It mostly came from my experience with FunFormKit, early versions of FormEncode, and then what FormEncode is now (which is strictly a validation library).

Now there's some new form generation libraies in the works; ToscaWidgets, Django newforms, and I hear rumors of a few others. I feel bad poo-pooing other people's work, but I don't think this approach is a good approach.

A form generation library is a hard library to write. You have to balance complexity and features. There are a lot of potential features. How do you tweak the forms? How do you control layout? How do you handle repeating form elements? How do you handle elements that require Javascript? Server-side resources like CSS and Javascript? Server-side communication like auto-complete? A simple library cannot handle a lot of these. But very simple forms don't need form generation libraries. So a simple library only handles a narrow band of use cases. A complex library can potentially handle all these cases, but can be difficult to use for the most common cases.

I will assert that as the libraries are usually designed, there is no good balance. Not all problems have feasible solutions. Form generation libraries are one such problem, but they don't seem to be such a problem.

There's a cycle going on in the library development. A developer has certain requirements. Usually they also hate HTML, and may very well be working with an awkward templating language, and maybe they aren't very excited about the task at hand and building a library is a pleasant diversion. But anyway, they Do The Right Thing (or so they think) and try an existing library. It's complex, it doesn't match the way they think about the problem, the bugs in their code are hard to figure out, and adjusting the things they (or their boss or client) cares about is difficult.

And they think "I can do this so much more simply!" The developer is kind of right. They know exactly what they want to do (now), and they can improve their productivity some.

The problem is that no library ends there. It can't end there. Because someone uses the library, and the code must be maintained. Form generation is a hard thing to maintain, because it has to be actively maintained. New fields are added. New cases and exceptions come into being. Anything that touches UI can be redefined in weird and unexpected ways. Good UI is usually eclectic and nonsensical from the perspective of a programmer. UI should be consistent with the user's notion of the domain, not the program's internal modeling of the domain, and it is uncommon that the two really match up.

So you get new requirements over time. A straight list of checkboxes used to be reasonable, but then the number of options expanded too far and now you need two columns of checkboxes. So you add in a little bit of code, both in the library and the application, to handle that case. You want to have a text field that contains its own label when its empty. The validation becomes increasingly complex. Maybe you have a multi-screen form, or maybe a form with optional fields, or who knows. So you add code to the library and the application each time (if you control both of them). It gets more and more complex while at the same time the developer's familiarity with the library code becomes more and more remote. And that's not even to mention when a non-programmer wants to tweak the HTML, or there is developer turnover, or when you don't control the library at all...

It's a mess. And so someone comes along and says "I can do this so much more simply!" and it starts all over again.

The problem domain is wrong. I think there's a better approach, but I'm too lazy to finish it. So I'll lay down the design here, with little expectation that anything will happen because of it, but just maybe this will connect with someone. (Ideally someone who will write code ;)

First, validation: I like how FormEncode works, I think it's the right model. Validation completely separate from the rest of the system, and you can programmatically construct the validation so you can embed it in other systems if you want (and embedding it is probably a good idea).

Second: forms. First, the simplest model: write it in a normal template. But this is annoying people will cry. Why?

Filling in defaults is annoying. You have to do this little dance: is the value in the request (e.g., resubmission)? Then use that. Is it passed in through some defaults? Then use that. Otherwise? Maybe another default, or blank. Filling in errors is annoying. Similar dance to defaults. The <label> tag is somewhat annoying to generate, which is perhaps why many very high-traffic websites still don't properly do labels. I suppose the HTML is annoying in general, though it's only slightly more verbose than the programming code that might replace it. Javascript can be annoying, and many web developers still aren't comfortable with Javascript. Getting all the right Javascript and CSS files in place, and linking them up with the appropriate fields is awkward. Ajax callbacks are really annoying. Nested and repeating fields compound every other kind of annoying. It's hard to reuse any of this.

My response is that these issues can be resolved without form generation libraries. formencode.htmlfill addresses 1 and 2. It's not perfect, but it's usable. Things like WebHelpers can help with 4. Neither of these needs to be the implementation, and both could be improved. But while remaining in their scope they can only be improved a little, which is important. They will not become endless time sinks.

Unobtrusive Javascript solves 5 quite well. I think a tool for solving 6 -- the proper inclusion of dependent CSS and Javascript -- could be handy. I imagine <input type="date" js-require="DateInput"> calling some code that figures out where DateInput is (probably building on something like JSAN package metadata) and puts the appropriate code in the head of the page.

7 and 8 are addressed quite nicely by the Web Forms 2.0 WHAT-WG specification, and tools building on those models would be excellent.

And once you have all this, I think 9 is much more approachable. Because you don't need a framework, you just need stuff that generates blank HTML that also uses a few of these simple conventions.

Notably, these techniques embrace the abstractions HTML and Javascript already allow. Also, each tool can be useful on its own, and can be used, not used, or reimplemented on its own. The hard part is that any one tool seems terribly simple and lacking features compared to a form generation framework. But if slow and steady can win the race it might not matter; the other form generation frameworks will suffer the churn of the reimplementation cycle while these simpler and duller tools will better stand the test of time. And even if they don't, unlike a framework the healthy decoupled set of tools can suffer the failure of one piece (in design, maintenance, or implementation) without compromising the entire stack.

Now someone please go build the missing pieces, because I'm trying really hard not to start more projects.