On Reflection and DSLs

Smalltalk is a worldview, a way of thinking, a completely different programming paradigm. The concepts at the heart of Smalltalk are so profound, and so foundational, that all serious programmers should learn the language, even if they don’t ever expect to actually use it professionally. Learning C++, Java or C# is not any sort of substitute for learning the real thing. None of those languages satisfy Dr. Kay’s definition of OOP [2].

Smalltalk is considerably simpler than most other languages, both conceptually and syntactically. But Smalltalk is also considerably more powerful and expressive than most other languages, due in part to its greater simplicity, uniformity and symmetry. Smalltalk is comprehensively uniform and consistent, and comprehensively symmetrical — which is an important part of the reason why it can be both simple and powerful at the same time. The flexibility provided by Smalltalk’s high degree of symmetry and extreme late binding have proven to help programmer productivity and creativity more than any theoretical benefits that might be derived from the use of static type checking, symmetry-breaking primitive data types or symmetry-breaking syntactic sugar.

Smalltalk is also pervasively dynamic, pervasively reflective, and pervasively open. Its metalevel is completely symmetric with its base level: all the same things can be done at the metalevel as at the base level, using the same syntax.

Stated concisely, the symmetry, the uniformity, the consistency and the simplicity of Smalltalk lie in the following: all values are objects, all behavior is invoked by sending messages, and almost nothing is done by syntax when it can instead be done by sending messages to objects.

But most importantly, Smalltalk is fun. It’s fun because its programming tools make use of Smalltalk’s superior reflective capabilities to provide a development environment unlike any other.

Smalltalk is also fun because defining and using domain specific languages isn’t an afterthought, it’s the only way Smalltalk works at all. As a result, Smalltalk code lets both the reader and the writer focus on the problem domain, using language and notation that is natural to that domain.

The Smalltalk Computational Model

In Smalltalk, all values are objects. Almost all computation is done by sending messages to objects.

Messages and Methods

A message can be thought of as a request to invoke or perform a computation. When a message is sent to an object, the receiving object (usually referred to as the “receiver of the message,” or simply as “the receiver”) is responsible for deciding how to fulfill the request to invoke (perform) the computation named by the message — or in Smalltalk jargon, how to “respond to the message.” An object that receives a message responds to the message by (dynamically, at runtime) finding a method in its method namespace whose name matches that of the message, and then executing that method.

A method is an object that embodies a computational algorithm that, when executed, computes a value — possibly with side effects. To execute a method is to execute the algorithm embodied by the method. Conceptually, a method is analogous to a function (or subroutine), and a message is analogous to a function/subroutine call (a function or subroutine “call” represents a request for it to be executed). But the analogy between methods and functions (subroutines) is only that — a conceptual analogy — for reasons that are explained below.

The name of a message or method is known as its selector. The selector of a message or method identifies its intentional semantics (logical meaning) and is what distinguishes one message or method from another. In addition to its selector, a message may have zero or more arguments (parameters). And a method may accept zero or more arguments — although any particular method requires a particular number of arguments. Each argument in a message becomes an argument to whatever method is executed in response to the message.

A method namespace operates as a mapping (functional predicate) that, for each unique method selector (method name) in its domain, associates that method selector (method name) with a particular method. It is used to lookup a method by its selector (name).

Dynamic message dispatch is the term used to refer to the process of using a method namespace to (dynamically, at runtime) find the method associated with a particular method selector (name). It should be noted that Dr. Kay (and Smalltalk) use the term “method” to mean a function or subroutine that is invoked in response to the sending of a message, by means of dynamic message dispatch at run time (and not by static binding at compile time). This is one of the two differences between a method and a function/subroutine. It’s also the essential difference between Smalltalk methods and the functions/subroutines in other programming languages that, in disrepect to those who coined the term, are often referred to as “methods.”

Because of dynamic message dispatch, the semantics of sending a message to an object depend absolutely on the object that receives the message. The same message may be interpreted quite differently by different objects, due to the fact that each object may use a different method namespace in order to lookup a method based on the selector of a message, and so may associate a different method with the same message selector. Consequently, a message can be thought of as an abstract function call which will result in the execution of whatever concrete function (subroutine) is chosen by the object that receives the message. The idea is that a message should name the logical function to be computed, but the object receiving the message should decide how best to physically compute the desired result. The universal, pervasive application of this distinction between logical function (semantics) and physical function (implementation strategy) is the central point of object oriented programming, as defined and envisioned by Dr. Kay.

Classes

A class defines the structure of its instances because it specifies the set of instance variables they will have. A class defines the behavior of its instances because it specifies and implements the method namespace used by its instances to lookup methods in response to messages.

The methods that a class specifies for use by its instances are known as instance methods. The instance methods specified by a class define which methods will be executed in response to messages sent to its instances — and so define the meaning of those messages when sent to any of its instances. By defining the mapping of messages to methods, a class also defines the operational meaning of the instance variables of its instances.

Note, however, that there are OOP languages that don’t use classes. The best known such language is Self, which uses objects and methods, but does not use classes. The fact that an OO language can be implemented without classes is rather inconvenient for those who think classes are central to the definition of OO. The truth is that classes and methods are both simply convenient implementation techniques for that which truly is central to OO: message sending. There is an analogy here with Lisp, where s-expressions are simply a convenient implementation technique for that which is truly central to Lisp: the lambda calculus.

A class is also an object. Like any other object, a class is itself an instance of a class — which is called the metaclass of the class. A class is the sole instance of its metaclass. The metaclass of a class named Foo is always referred to as “Foo class.” All metaclasses are instances of the class Metaclass. The class Metaclass is an instance of the metaclass (canonically referred to as) “Metaclass class,” which is then (recursively) an instance of Metaclass.

Since a class is an object, and is also an instance of a class (the metaclass of the class), each class can have its own methods which can be executed by sending messages to the class. In fact, instances of a class are created by sending it messages.

The methods that are used by a class to respond to messages sent to the class itself (not its instances) are known as class methods. Note that the class methods of a class are one and the same as the instance methods defined by its metaclass — and that the “class instance variables” of a class are one and the same as the instance variables defined by its metaclass.

Variables, Types and Classes

Smalltalk variables have no type constraint syntactically associated with them. Any variable can have any value assigned to it — in other words, the value of a variable can be an object that is an instance of any class (all values are objects).

It should be noted that a Smalltalk class is not a type, although it implements one (or more). A Smalltalk type is defined as the power set of messages to which an object can meaningfully respond — a definition that differs non-trivially from that of the term type as commonly used in most other programming languages. Smalltalk essentially uses what has come to be called duck typing. The author likes to call it “optimistic typing.”

Different classes can all implement the same Smalltalk (“duck”) type — which is a key ingredient to the extreme polymorphic power of Smalltalk. Unlike a class, a Smalltalk type is not an object — although you can define a class whose instances model a type, and can modify Smalltalk at the metalevel so that classes can inform you of their type (or types) using one or more instances of the type-modelling class you’ve defined.

It is not correct to say that Smalltalk is untyped, or that it is weakly typed. Smalltalk is strongly typed by means of dynamic typing, instead of by means of static typing. Strong typing requires that type errors be detected and prevented — it does not require that type errors must be detected and prevented statically by a compiler, using syntactic analysis of source code.

The class (and hence the type) of a Smalltalk object is encapsulated as part of the value (internal structure) of an object. The type (or types) of an object is an intrinsic part of the value of an object, and is exclusively determined by the class of the object, and not at all by what variable happens to reference it. To make either the class or the type of an object in any way dependent on the variable that references it is a violation of the encapsulation principle, with consequences that are every bit as serious as any other failure to observe good information hiding protocol. In spite of that hard fact, it is a common feature of “OO” languages to actually allow the exact same “message” to have different semantics (behavior), when sent to the exact same object, depending on the static type constraint of the variable which references the object to which the message is sent. That interpretation of “OO” and “message” statically binds the meaning of a “message” to the variable used to reference an object, and prevents the object itself from having full control over the meaning of the messages it is sent. An object that does not fully control the semantics of the messages it is sent does not fully encapsulate its behavior — hence the violation of encapsulation.

The class (and hence the type) of a value is an essential property of both the input and output data used and produced by a computational process. In other words, the class (and hence the type or types) of objects are just as much a part of the values to be computed dynamically at run time as is any other data involved in a computation. Attempting to statically restrict the set of values that may be involved in a computation, based on syntactic constraints enforced at compile time, not only artificially and unneccesarily reduces the set of valid computations that can be performed, but makes the compiler responsible for a task that should rightfully be done by the programmer.

If the programmer can be trusted to use and produce the right values in the code he writes, then he can be trusted to use and produce values of the right types — since the type of a value is no different than any other aspect of a value. Conversely, if the compiler can rightfully be assigned the task of determining which types are or are not valid, why isn’t it also assigned the task of determining the validity of all other aspects of the values to be assigned to variables? Of course, even if the compiler were assigned all such responsibility, the additional constraints that would need to be provided in the code, so that the compiler could enforce those constraints on all aspects of all values, would still have to written by a programmer. There’s a reason that imperative code is not written that way: there usually isn’t enough information when imperative code is compiled to make such decisions — especially not optimal ones! And that’s one reason why mainstream (imperative) languages with static typing tacitly admit the error in their design by permitting programmers to “cast” values from one type to another at run time, thus circumventing the static type system.

Without static type checking, some typing errors will only be detected at runtime, and not at compile time. But catching all type errors at compile time has a cost (relative to relying solely on dynamic typing), and the cost is not worth the benefit. From an engineering perspective, dynamic typing is good enough. The fact that some academics think otherwise is nothing new.

Strong typing requires that type errors be detected and prevented — it does not require that type errors must be detected and prevented statically by a compiler, using syntactic analysis of source code.

There are two self-consistent programming paradigms: 1) imperative programming, where program semantics emerge operationally as a result of behavior that is explicitly specified using imperative commands; and 2) declarative programming, where program behavior emerges operationally as a result of semantics that is explicitly specified using the rules of logic. Static type checking is a poor attempt to mix the two paradigms. Instead of trying to compute correct behavior from explicitly specified semantics, it merely tries to prove whether explicitly specified behavior conforms to constraints on the usage of types and function calls — something that it can only do by preventing correct behavior from being specified, simply because it can’t prove that the usage constraints won’t be violated dynamically. The “impedence mismatch” caused by static type checking, which is a natural consequence of mixing the two basic programming paradigms, imposes costs that are greater than the benefits.

In a system where behavior is explicitly specified using imperative statements, guaranteeing the semantics of a message (function call) is something that no system of static type constraint checking can provide, period. It’s the implementation of the methods (functions) that operationally distinguish the semantics of “fire an employee” from that of “fire a gun.” A static type checking system can guarantee that the function “fire” is applicable to an Employee or to a Gun, and can prevent the assignment of a Gun object to a variable whose static type constraint says “Employee.” But it cannot guarantee what the semantics of the function (or method) that responds to the message “fire” might be — which is why it’s essentially useless, from an engineering perspective.

When strong typing is provided by dynamic typing, the presence of static typing causes more damage than benefit. In practice dynamic typing is a better engineering solution. That’s a conclusion that many Smalltalk programmers don’t believe initially, but come to appreciate with ever greater certainty the longer they use the language. That’s why, in spite of how easy it is to define objects that model or represent types, and how easy it would be to modify the Smalltalk compiler to add static type checking, and despite the fact that serious efforts to do one and/or the other of those things have been expended more than once, neither static type checking, nor reifying types as objects, have ever really caught on in the Smalltalk programming community [3].

Defining Classes

A Smalltalk class is a living object — it is not a static declaration in a text file. A Smalltalk class is not an object created only at “runtime” as an afterthought based on a declarative definition — it is an object that functions as its own specification, even while the program is being written! The object that is the class comes into existence as a result of sending messages as the programmer writes the code (usually, the development tools send the necessary messages, so the programmer does not have to do it directly). In fact, the class object must exist first, before any instance variables can be defined or methods can be “declared.” And that’s why there is no distinction between “compile time” and “run time.”

Because Smalltalk classes function as their own specifications (i.e., models), Smalltalk is intrinsically based on model-driven engineering from design to execution, and from top to bottom. Smalltalk programs model themselves intrinsically, pervasively, comprehensively, completely and down to the core. Introspection/reflection arises as a natural property of the Smalltalk computational model — it does not need to be added as an “extra cost” afterthought. Deep introspection is a natural result of unifying “compile time” with “run time,” specifications (models) with their instances, values with objects, and (last but not least) the base level and the meta-level.

Smalltalk programs are built dynamically and incrementally, by sending messages to live objects, which result in the creation of live objects which can themselves be sent messages. When you add, rename or remove the instance variables specified by a class, the effect is immediate: all the existing instances of the class are atomically modified right then and there. There is no waiting until the “program” is recompiled. Such is the value of using live objects as the governing specifications of your system. Smalltalk “programs” are not declaratively specified in flat, static text files!

Introspection/reflection arises as a natural property of the Smalltalk computational model — it does not need to be added as an “extra cost” afterthought.

The following example shows a message that defines (or more likely, redefines) a class named Point as a subclass of the class Object, and also specifies the class’s instance variables and category:

Object subclass: #Point

instanceVariableNames: 'x y'

classVariableNames: ''

poolDictionaries: ''

category: 'Graphics-Primitives'

The category of a class has no affect on the behavior of the class or its instances, but does affect how the development environment organizes the class among all the other classes in the class library (in other words, it’s an annotation for the benefit of programmers). In the example above, the class is assigned to the category ‘Graphics-Primitives’.

Based on the message sent to Object in the example above, instances of Point will have two instance variables, named x and y. Also, the class Point will not have any class variables. It also won’t have any shared pool dictionaries (i.e., it doesn’t import any namespaces).

Here’s a slightly more elaborate example showing a message that defines a subclass of Point named ThreeDPoint , which specifies that ThreeDPoint will have the instance variable z (in addition to x and y, which are inherited from its superclass, Point), and has a class variable named Zero and imports a shared pool dictionary (namespace) named TrigConstants:

Point subclass: #ThreeDPoint

instanceVariableNames: 'z'

classVariableNames: 'Zero'

poolDictionaries: 'TrigConstants'

category: 'Graphics-Primitives'

To add an accessor method and a mutator method for ThreeDPont’s instance variable z, we can send the following messages:

ThreeDPoint

compile:

'z

"Answer the value of my instance variable z." ^z'

classified: #accessing. ThreeDPoint

compile:

'z: newValue

"Set the value of my instance variable z to be that of the argument newValue." z := newValue'

classified: #accessing.

The argument following the keyword #classified: specifies the category of the method being defined. As was the case with the category of a class, the “category” of a method (which is more usually referred in Smalltalk jargon as a method protocol) is simply an annotation to be used by the development environment for organizing methods into groups, for the convenience of programmers.

Messages such as those in the examples above may be sent at any time. Programmers don’t usually send such messages directly, however. The development tools do it for them. But when needed, programmers can and do “do it themselves.”

Final Thoughts

Notation vs. Turing Equivalence: All programming languages are Turing Equivalent. So why does it matter which programming language one uses? It matters because Turing Equivalence only defines what can be computed, not how quickly or easily the program that performs the computation can be designed, coded, implemented, tested and deployed to production. If you don’t think notation matters in that regard, then consider the difference between doing arithmetic using Roman Numerals versus using Arabic Numerals. Ever try doing long division using Roman Numerals?

There’s a reason most programs aren’t written in assembly language anymore. Differences in the expressive power of the programming notation used do matter.

Smalltalk as a Domain-Specific Meta-Language: Because almost all computation in Smalltalk is invoked and implemented by sending messages, and because the syntax of sending messages is the same regardless of when, where, why or by whom a message is first defined, Smalltalk syntax unifies the language, the class library and the applications written by programmers.

You can add your own control structures — and they will not only work just like any other control structures, they will look like them, too. You can add your own number types — and they will look and act just like those in the base language. You can add your own collection types — and those, too, will look and act just like those in the standard class library.

Smalltalk essentially eliminates any distinction between “built-in” and “programmer defined,” and does so more thouroughly than just about any other language — Lisp and Forth would be the only well-known exceptions. Lisp syntax is uniformly based on prefix symmetry. Forth syntax is uniformly based on postfix symmetry. Smalltalk syntax is uniformly based on infix symmetry.

You can add your own control structures — and they will not only work just like any other control structures, they will look like them, too.

Writing a program in Smalltalk essentially defines a domain-specific language — which can then be recursively used by programmers to extend Smalltalk further, with ever richer domain-specific languages. Ultimately, the entire application becomes a single message send which executes a method whose code is written in a high-level domain specific language, and so on down the chain of abstractions until the primitive methods are reached.

It’s all just messages, “all the way down.”

Messages are Primary: In Smalltalk, messages are “the message” (with apologies to Marshall McLuhan). Messages and their semantics are primary: classes and methods only exist to satisfy the expected semantics of the messages that are sent, and not the other way around. That’s why extreme programming and test-driven development both originated in Smalltalk. Both assume that implementation artifacts exist to satisfy the users of those artifacts, that classes and methods exist to satisfy the requirement specification(s) embodied by the application/business logic that uses them. That’s why those methodologies have the programmer first write test code, and then implement classes and methods so that the test code satisfies the requirements implied by the messages sent by that code.

In extreme programming, in test-driven development, and in Smalltalk, it’s the semantics of messages that matter, and that are primary. Classes and methods are written to conform to the expected semantics of the messages they will be sent — to appropriately respond to the messages sent by applications or business logic. If the code fails, it could just as easily be the fault of the implementation of the objects that are responding (or failing to respond) to the messages, as the fault of the programmer who sent the messages. The compiler is not smart enough to know which is the case — so why expect it to assume that responsibility?

From the Smalltalk perspective, it makes just as much sense for the development environment to prevent the programmer from sending whatever messages he desires as it would for the existing class library (of any programming language) to prevent business users from writing use cases as they wish. Imagine a “use case compiler” rejecting a statement in a use case because there’s no such class or method in the existing class library. That would make no sense at all. To someone used to programming in Smalltalk, neither does static typing, and for the same reason: the programmer is the master, and the class library is his servant. The responsibility of the class library is to serve the needs of the programmer; satisfying the constraints of the class library is not the proper function of a programmer.

Of course, both use cases and methods can incorrectly specify system behavior. But static type checking cannot prevent that from happening, because it’s not at all equivalent to declaratively specifying the desired semantics, so that the correct behavior can be computed by the system. The behavior of a program must be tested for correctness and acceptability in any case — and the faster the programmer can get the code ready for such testing, the better. Dynamic typing gets that done more effectively.

Without static typing on which to rely, Smalltalk programmers put much more focus on the semantics of messages. The Smalltalk programming culture puts great weight and importance on having well-defined, reliable message semantics. Informally using the semantics of messages as (near) invariants not only provides many of the same benefits as would static typing, it also provides some of the benefits that would ensue from declarative programming. Why? Because a body of code that sends messages (whose semantics are reliably defined) functions, in a sense, as a declarative specification of the desired semantics of a computation — a property of code on which test-driven development relies. Test cases validate that the messages have the expected semantics — which static typing cannot do.

With reliable message semantics (validated and enforced by test cases), methods and classes become nothing more than implementation details — as they should be.