Q: How do I embed a newline in a regexp I'm typing into the minibuffer?

"images/foo_bar.gif"

"images/foo_bar.gif"

I finally upgraded to Emacs 22 a few weeks ago, and now I'm wishing I'd braved it sooner. Technically it's not released yet; I'm working from a build of a cvs snapshot from a month or so ago. But the Emacs dev team works pretty hard to make sure it has problem-free builds on a whole slew of platforms, so just following their instructions has a pretty good chance of working for you.It's worth the effort. Truly. Reading through its NEWS file, there's just tons and tons of new functionality. It's going to take me some time, maybe a few weekends, just to absorb it all.Personally, though, I think there are two features that by themselves justify the entire effort of upgrading: the Unicode and UTF-8 support, and the enhancedcommand.It's been a very long wait for Unicode and UTF-8 support, and now that I have it, I could never go back. There isn't much to say about it, except that it works. Seamlessly. It used to be hard to get international characters into and out of Emacs, because it had its own custom way of dealing with them. Now it's a snap.In fact — here, I'll show ya. If you type, it brings up the HELLO file, which contains greetings in a variety of languages. Here's some Chinese: 中文,普通话,汉语. Here's some Korean: 안녕하세요, 안녕하십니까. Here's some Russian: Здравствуйте!I'm not doing anything special; I'm just copying the strings out of the HELLO buffer and into my html buffer, and saving the file. I added the content-type header line in this HTML file, and all the characters just show up effortlessly in Firefox.If you can't see them in your browser, well... Firefox is free . Or it might be a font problem on your system. As far as I'm concerned, any problems you may have in viewing them is no longer the fault of my Emacs session, which makes me Happy. Speaking as a developer who needs to internationalize every program I write, I can't begin to tell you how useful it's been to have seamless editing of utf-8 encoded files for the past month.Right there, that feature alone is worth the upgrade.But wait, there's more... Even though Emacs 22 has a bunch of noteworthy and exciting new features, blah blah blah, I'm going to blithely ignore them all today and focus with single-minded zeal on just one feature. It has a teeny tiny entry in the NEWS file; it's barely mentioned, really. I'm sure it was a thousand times less work than the UTF-8 support, but even so, it might well be strong enough on its own to justify the (moderate) pain of upgrading from Emacs 21.Take a look at my examples and see if you agree!Emacs 22 sports an amazing new editing feature that's had me drooling in anticipation since I first heard about it, maybe six or eight months ago. As you can well imagine, that's a lot of drool.And what might the feature be, you ask? Well, they've enhancedto accept lisp expressions to be evaluated in the replacement string.That might not seem like a big deal, so let's run through some examples, from simple to very fancy.Have you ever wanted to change the case of certain letters in the replacement string for? It used to be a real pain; you either had to write a Lisp function or fix them all by hand. Now it's trivial.As a simple demonstration, let's say you have a list of names that you need capitalized, like so:It's a contrived example, since Emacs already has. Or you could use. But let's try it with the newevaluation feature to see how it works.It's just like a normal, but you'll prefix any lisp expressions in the replacement string with the sequence `' (i.e., a backslash and a comma). In this case, we match the whole word, and invoke the Emacs-Lisp function `' to capitalize the word we just matched:and we wind up with each word capitalized, just like we wanted:Unlike thecommands, which have hardwired behavior, thecommands give us tremendous flexibility. For instance, we could have capitalized the last letters of the names instead, by splitting each word into two regexps, with the second regexp matching just the last character. Then it's a simple matter to reconstruct the word with the last letter capitalized:..to get the reverse capitalization we wanted:For a somewhat more realistic example, let's say you've defined some "getter" functions in a Java class, like so:and your code reviewer wants you prepend the word "get" to each of them.Well, this is a classic refactoring situation ("rename method"), but you're going to have to invoke the refactoring manually for each method, and you may have hundreds of them.If these methods have been around a while, and they're being referenced by many external callers, then you're safest using a refactoring tool. You may even consider writing a one-off refactoring script, perhaps in Jython or Mozilla Rhino , that makes programmatic use of either your IDE's refactoring APIs or a lower-level tool such as ANTLR or JavaCC Whew! That's going to be a lot of work, no matter how you slice it. And if you've published your APIs externally, then you're screwed; all you can do isthe old names and hope people stop using them someday.But that's why you get your code reviews done early, right? In many real-world situations, you're performing a rename-method on a new class that has no external callers yet. And in those situations, Emacs 22 will get the job done far faster than a refactoring IDE can.In this case, we'd just do a straightforward replacement with capitalization, similar to the one in our last example, like so:Even if you do most of your coding in the comfy confines of a visual IDE, it can be awfully handy to keep Emacs around for your fine-grained text surgery.Now we can move on to some more interesting examples, so you can feel you got your money's worth out of today's blog entry. But first...Emacs regular expression syntax is very old, predating Perl 5's fancier regex syntax by almost a decade. Perl's regexp enhancements are the defacto standard, supported in virtually all major programming languages. Unfortunately, nobody has ever seen fit to retrofit poor Emacs with an alternate Perl-compatible regexp syntax, so Emacs regexps are now nonstandard and a bit awkward. Here are a few of the noteworthy differences:But Emacs still supports some of the constructs you've come to expect:There are also some Emacs-specific enhancements, such as matchers for entries in the mode-specific syntax tables. The Info pages have more details on Emacs regular expressions.If you plan to be more than a casual Emacs user, you should study the regexp syntax carefully, because there are many useful commands in Emacs that operate on regular expression matches. The better you know the Emacs-specific syntax, the more productive you'll be.I suppose before we move on to the next example, I should preemptively answer one of the most frequently asked questions about Emacs regexps.You use the key sequence. The C-q invokes the Emacs `' command, which basically says "insert the next character literally, without invoking any commands with it." C-j (i.e., control-j) is how a newline character is represented in Emacs.is a useful general-purpose Emacs command. Whenever you want to insert a character (in the minibuffer or a regular buffer), and it's just refusing to go in,will almost always do the trick.With our new replacement-with-evaluation feature, it becomes straightforward to create numbered lists. Emacs 22 has introduced a new backreferencing metacharacter, `', which counts the number of replacements we've done so far in the current command. So even without using any Lisp, we already have one way to make numbered lists.Let's see... we'll need a short list of words as an example. How about all the words inthat don't end in [a-z]? Easy enough to find out. We(only if you're on a Unix system, of course, and the location varies), and then. Ah, perfect — ourbuffer shows 32 matches:They're prefixed by their line number, but we can make that disappear during the replacement. Let's turn them into a numbered list. First copy the matches into a new, writable buffer, thento go to the top of the list, and then:Boom!Oooh, but only Computer Science students like lists numbered from zero. So let's(usingof course — everyone uses, but that's way too many keystrokes for something as common as Undo!) and use a tiny bit of Lisp to start the numbering at 1.It so happens that Emacs-Lisp defines a function called, which increments a number. So we can just wrap thein our replacement string with that function, like so:The result is just what we wanted:The lispfunction operates on numbers, not strings, so you might have expected it to barf with aerror. Our example works because themetacharacter returns a count of matches so far, which is a number, not a string value. In our next example, we'll have to do the conversion ourselves.We can use Lisp-code snippet similar to our previous one to renumber an existing list. Let's say we want to insert a word in a numbered list, like so:Easy to fix in Emacs22: place the cursor just after the word we inserted, and' with `'. The result:This time we used a numbered backreference (), which always returns a string. Don't be fooled by the fact that we appear to be matching a number: the regexp `' matches a string containing numeric digits, and we have to do a type conversion (using the Emacs-Lisp builtin function) if we want to increment it.I hope by now you're beginning to suspect that knowing a little Emacs-Lisp can help you immensely with your editing tasks. Believe it! (Not surprisingly, knowing a lot of Emacs-Lisp helps even more.)Let's say we have a list of 26 or fewer items, and we want to "number" it with A, B, and C rather than 1, 2, and 3.Well, shoot. The list we've been using has 32 items. I'd prefer a list of 26 (or so) for this example.Let's see... we can write a wee Lisp function to group the words inby their ending letters a-z:We just evaluate this snippet in ourbuffer (by typingafter the last paren). It crunches thebuffer and produces:Hmmm... nothing really promising. We only get 9 words if we combine the ones ending in "q" and "j". A minor tweak to our search function will show us words ending with a doubled letter:Evaluating it gives us:Looks like we have more options with this list. Maybe if we just take the ones with a count of 5 or less... looks like 26 of them. Perfect!We could write a little more code to extract the words matching our criteria, but it's clearly going to be fastest to eyeball it. So we callwith the regexp, and we get our list of 26 words:And what a fine bunch of words they are. Just try doing that exercise in Java or C++ sometime.In any case, now we have a list for our example, and we want to number it alphabetically. So we need to replace all the cruft up through each ':' with a counter converted to an alphabet character.As we saw in our little function that produced this word list, Emacs uses `?c' syntax to represent characters, and internally they're just ints. So we just add the character `?a' to our `\#' counter this time, to loop through the characters 'a' to 'z':And, ladies and gentlemen... behold!D'oh!!!! I forgot to convert the counter back to a character. Haha. Oops.After a quick C-/ to undo the operation, we can just change the replacement regexp to `', and we finally have our alphabetically-enumerated word list:Or we could get capital letters by usinginstead.Note thathas its own command history list, so you can just use up-arrow to fetch old regexps you've entered, and tweak them in place. Easier than re-entering them from scratch every time.So far we've used this amazing little new feature to generate (and renumber) various lists, and to change the capitalization of the replacement text on the fly (in two different ways). Both very practical and useful transformations.In our next example, we'll assume you're working on a Java-based Web application, because your company is too lame to let you use Ruby on Rails . Hypothetically speaking, of course.Suppose you have some JSP files containing references to various static images, e.g. , and you decide you want to change them to calls into Java code to fetch the image URLs as the page is composed. Soneeds to change to (say)Well, clearly no fancy-pants refactoring IDE on the planet is going to be able to help you with this. If you're an Eclipse or IntelliJ or Visual Studio user, get ready for some carpal tunnel while you manually change every instance.However, if you've followed the examples so far, you know it's trivial in Emacs 22:and they're all fixed in the blink of an eye.But you knew that by now. This example wasn't more complex than the others, just a little longer.It starts to get even more interesting if you permit side effects in your lisp expressions. That is to say, persistent changes to the world, whether it's Emacs variables, your buffer configuration, or even your filesystem. You have to be a bit more careful, but you can use the neweval feature as a powerful interactive scripting engine.Our last example will be opening files. Often you'll find yourself looking for files using the Unix `find' command in a shell. But what if you want to open the files it turned up?Again, it's a slightly contrived example, because it's already possible to use Unix shell commands and the "emacsclient" program to instruct Emacs to open the files you find. But it should suffice to show you what we mean by "side-effecting replacements".A simple example should work. Let's go to our installed emacs lisp directory. Mine's, which I found by looking at myvariable in mybuffer. There are various subdirectories, including, and others.To have Emacs open (say) all the elisp files beginning with the letter `x', we, cd to, and use the `find' command:If you select the lines naming the 8 files above, thenwill operate just in the selected region. Opening the selected files is then one easy command:The files are silently opened in the background when you execute this "replacement" command. We could alternately have usedto watch them opened noisily in the foreground, but when opening lots of files I personally prefer to open them in the background. (This also makes them appear at the bottom of your buffer-list.)Note that we used a new metacharacter here, `', which grabs the entire string that matched. That means we were able to omit the grouping parens. We also relied on the fact that Emacs regexps are, by default, anchored to the beginning and end of the line, so `' matches exactly one line, not counting the newline character. Pretty convenient!After the replacement, the lines in yourbuffer are replaced with the return values of the calls to, which in this case is just the name of the file. But we don't really care, since it's a shell buffer; we were doing it purely for the side effect and not for the replacement.Armed with your new-found knowledge,andshould become some of the most powerful tools in your editing toolchest. The more experience you have with Emacs regular expressions, and with Emacs Lisp, the more bang for your buck you'll get out of this enhancement.Oh, OK, fine. One laaaaaast example, because I just ran into it as I was putting in my final edits. You know how most browsers like to render