Piping Vim Buffer Through Unix Filters

( ! and !! Commands)

Everybody knows how pipes work at the command prompt. Text originates from some source, is processed via one or more filters and output goes either to the console display or is redirected to a file.

VI takes this same paradigm of pipes and filters and wraps it in a editor user interface in which the pipe is applied to editing buffer both as a source and as a destination. A VI pipe is thus can alter the buffer using standard Unix filters that instantly become a part of editor toolbox. This is an extremely elegant idea. The ability to pass nearly arbitrary chunks of text through any UNIX filter adds incredible flexibility at no "additional cost" in size or performance of the editor.

That was a stoke in genius in design on vi. And still remains one of the most advanced features vi(and by extension VIM) has. Which, unfortunately, few people understand and use to the full extent.

Pipes can be used both from command line commands:

ex commands

vi-mode commands.

With the vi filter command

:[address-range] ! external-command-name

you can process a range of lines with an external program. This enables you to do much more effective editing. For example, the command:

:1,$ ! indent

will beautify your program using standard Unix beautifier (indent). This is a classic example of using piping in vi. You can also create a macro for this operation using keystroke remapping. Instead of indent you can use any available batch beautifier most suitable for the language that you are using.

Without any filter, the command ! , prompts for the name of a UNIX command (which should be a filter), then passes selected lines through the filter, replacing those selected line in the vi buffer with the output of the filter command.

Actually the ex % operator is the easiest way to filter all the lines in your buffer, and this classic vi idiom should look like:

:%!indent

To edit the current line you need to use:

!!command

To edit the next paragraph:

!}!command

If you use any keystrokes that move cursor you need to use them so that they move cursor more than one line ( G , { } , ( ) , [[ ]] , + , - ). To repeat the effect, a number may precede either the exclamation mark or the text object. (For example, both !10+ and 10!+ would indicate the next ten lines.) Objects such as w do not work unless enough of them are specified so as to exceed a single line.

You can also use a slash ( / ) followed by a regular expression and a carriage return to specify the object. This takes the text up to the pattern as input to the command.

The entire sequence can be preceded by a number to repetitions:

20!!grep -v '^#'

or:

!20!grep -v '^#'

NOTE: To move one paragraph down you can use !}. That's a very convenient, often used idiom.

Among Unix filters that can be used I would like to mention AWK and Perl. Of course, any filter can be used, for example tr can be used for case conversion of deletion of some characters ( tr '[a-z]' '[A-Z]' ).

As most Unix/Linux sysadmin know some Perl along with bash it is natural choice that gradually replaces AWK (although AWK remains attractive for Python users because it is a more simple and still quite powerful language).

For example, to pipe all the buffer you need to specify the filer (which can be written by you Perl script) after ! command:

:1,$ ! perltidy

will beautify your program using perltidy . Beautifying your script is a classic example of using piping in vi. You can also create a macro using keystroke remapping. Instead of beautifier you can use any other filter. This way for example you can temporary remove all comments from the script as it simplifies understanding and then reread the file with comments back into the buffer:

To do the same for selection you need to provide boundary of selection to the command. The easiest way to do this is to use visual mode: v , V , or Ctrl-v commands to select the part of your script that you want to process. In this case boundary of your selection are defined by two mark '< and '> which represent upper and lower boundary correspondingly. They will be put on the command like automatically as soon as you enter : . For example:

'<,'>! perl -nle 'print "\#".$_'

(this operation is called "commenting out" and it can also be performed via a plugin, but it is a still a good illustration of using Perl filters with piping).

You can return to original text is you pipe worked incorrectly

Vim will run pass your selection to the script as STDIN and insert standard output in place of it. If you incorrectly overwrite your selection or the whole buffer you can restore it using undo command ( u ). As buffer is an "in-memory" mapping of the file, rereading file from the disk using r command also allows you to restore the original text.

Repeating previous pipe command

To repeat the previous pipe command, you need to type:

! object !

Shell can be used as a filter, which gives you an ability to replace shell command that you typed in the current line (lines) with their output. That's actually great, unappreciated and underutilized vi capability.

Shell can be used as a filter, which gives you an ability to replace shell command that you typed in the current line (lines) with their output.

Here is a relevant quote from Vim tutorial (Vim Color Editor HOW-TO (Vi Improved with syntax color highlighting) Vi Tutorial)

Create a line in your file containing just the word who and absolutely no other text. Put the cursor on this line, and press !! This command is analogous to dd , cc , or yy , but instead of deleting, changing, or yanking the current line, it filters the current line. When you press the second !, the cursor drops down to the lower left corner of the screen and a single ! is displayed, prompting you to enter the name of a filter. As the filter name, type sh and press the Return key. sh (the Bourne shell) is a filter! It reads standard input, does some processing of its input (that is, executes commands), and sends its output (the output of those commands) to standard output. Filtering the line containing who through sh causes the line containing who to be replaced with a list of the current users on the system - right in your file! Try repeating this process with date. That is, create a line containing nothing but the word date, then put the cursor on the line, and press !!sh and the Return key. The line containing date is replaced with the output of the date command. Put your cursor on the first line of the output of who. Count the number of lines. Suppose, for example, the number is six. Then select those six lines to be filtered through sort; press 6!!sort and the Return key. The six lines will be passed through sort, and sort's output replaces the original six lines. The filter command can only be used on complete lines, not on characters or words. Some other filter commands (here, < CR > means press Return): !/the < CR > sort < CR > : Sort from the current line up to and including the next line containing the

: Sort from the current line up to and including the next line containing the !1Ggrep the < CR > : Replace from the current line to and including Line 1 with just the lines that contain the

: Replace from the current line to and including Line 1 with just the lines that contain the !Gawk '{print $1}' < CR > : From the current line to the end of file, replace every line with just its first word.

For example, you need to insert into document or script you are editing the IP address and netmask of the server on which you are working. you can just execute ' ifconfig eth0 | grep Mask ' using ! command and get it directly in the place where it is needed. For example:

25 ! bash

You can use internal editor piping for a lot of interesting stuff. For example you can read list of files in the current directory into the buffer, convert then into some commands and then execute them in shell:

$ vim

:r! ls *.c

:%s/\(.*\).c/mv & \1.bla :w !sh :q!

or as one liner:

:r! ls *.c :%s/\(.*\).c/mv & \1.bla :w !sh :q!

You can format text without the fmt program using instead perl's Text::Wrap module (especially useful if you are working in Cygwin):

:% ! perl -00 -MText::Wrap -ne 'BEGIN{$Text::Wrap::columns=40} print wrap("\t","",$_)'

Numbering items with pattern matches in vi

:! type foo.html | perl -pe"BEGIN{$i=1;} ++$i if s:<foo>:<bar$i>:;" > bar.html

Use filtering with a tool like perl to get variable interpolation into search patterns, unless you are lucky enough to have compiled-in support for perl or other tools that allow you to do this.

Another quite useful and powerful Vim command is range command:

:range g[lobal][!]/pattern/cmd

The global commands work by first scanning through the [range] of of the lines and marking each line where a match occurs. In a second scan the [cmd] is executed for each marked line with its line number prepended. If a line is changed or deleted its mark disappears. The default for the [range] is the whole file.

Note: Non-Ex commands (normal mode commands) can be also executed from the command line using :norm[al]non-ex command mechanism.

In the previous section, we saw that we could execute shell commands from within Vim, causing Vim to be moved to a background process while the command executed. There are more practical use cases for this feature, mainly, having the ability to manipulate our content via external filters.

Let’s use as an example tr command. This command translates character set to another. For example we might need to convert some characters that appear in the text copied from the WEB to "normal" characters acceptable to particular scripting language interpreter.

If we selected visual block and types ":" after will we’d enter COMMAND-LINE mode with boundaries on the block already put into the command line. After that we can use tr command

:'<,'> ! tr '[:lower:]' '[:upper:]'

Top Visited Your browser does not support iframes. Switchboard Latest Past week Past month

[Nov 30, 2018] Filtering Text Through a Command

... ... In vi, text is filtered through a Unix command by typing an exclamation mark followed by any of vi's movement keystrokes that indicate a block of text, and then by the Unix command line to be executed. For example: !) command will pass the next sentence through command. There are a few unusual aspects of the way vi acts when you use this feature: The exclamation mark doesn't appear on your screen right away. When you type the keystroke(s) for the text object you want to filter, the exclamation mark appears at the bottom of the screen, but the character you type to reference the object does not.

Text blocks must be more than one line, so you can use only the keystrokes that would move more than one line ( G , { } , () , [[ ]] , + , - ). To repeat the effect, a number may precede either the exclamation mark or the text object. (For example, both !10+ and 10!+ would indicate the next 10 lines.) Objects such as w do not work unless enough of them are specified so as to exceed a single line. You can also use a slash ( / ) followed by a pattern and a carriage return to specify the object. This takes the text up to the pattern as input to the command.

, , , , , ). To repeat the effect, a number may precede either the exclamation mark or the text object. (For example, both and would indicate the next 10 lines.) Objects such as do not work unless enough of them are specified so as to exceed a single line. You can also use a slash ( ) followed by a pattern and a carriage return to specify the object. This takes the text up to the pattern as input to the command. Entire lines are affected. For example, if your cursor is in the middle of a line and you issue a command to go to the end of the next sentence, the entire lines containing the beginning and end of the sentence will be changed, not just the sentence itself. [32]

There is a special text object that can be used only with this command syntax: you can specify the current line by entering a second exclamation mark: !! command Remember that either the entire sequence or the text object can be preceded by a number to repeat the effect. For instance, to change lines 96 through 99 as in the previous example, you could position the cursor on line 96 and enter either: 4!!sort or: !4!sort As another example, assume you have a portion of text in a file that you want to change from lowercase to uppercase letters. You could process that portion with the tr command to change the case. In this example, the second sentence is the block of text that will be filtered through the command: One sentence before.With a screen editor you can scroll the page move the cursor, delete lines, insert characters, and more, while seeing the results of your edits as you make them. One sentence after. Keystrokes Results !) One sentence after. ~ ~ ~ ! An exclamation mark appears on the last line to prompt you for the Unix command. The ) indicates that a sentence is the unit of text to be filtered. tr '[:lower:]' '[:upper:]' One sentence before.WITH A SCREEN EDITOR YOU CAN SCROLL THE PAGE MOVE THE CURSOR, DELETE LINES, INSERT CHARACTERS, AND MORE, WHILE SEEING THE RESULTS OF YOUR EDITS AS YOU MAKE THEM. One sentence after. Enter the Unix command and press ENTER. The input is replaced by the output. To repeat the previous command, the syntax is: ! object ! It is sometimes useful to send sections of a coded document to nroff to be replaced by formatted output. (Or, when editing electronic mail, you might use the fmt program to "beautify" your text before sending the message.) Remember that the "original" input is replaced by the output. Fortunately, if there is a mistake-such as an error message being sent instead of the expected output-you can undo the command and restore the lines.

This is a *request* for a tip. I need to be able to pipe the output of a :blah ex command into the vim text buffer for editing. I wanted to do this many times for different reasons and could never find a way! I would just love to be able to do :hi --> textBuffer and examine the output at my own leasure scrolling up and down and using vim search commands on it. Same thing for :set all, and other things. Considering that cut and paste is horrible in windows, I can't for example do :set guioptions? then cut and paste! So I have to retype it, or cut and paste from the help manual. I really want to be able to pipe the output of ex commands into the text buffer. Can someone help me?

Yegappan, August 7, 2001 11:45 You can use the :redir command to redirect the output of an ex command to a register and then paste the contents of the register into a Vim buffer. For example: :redir @a

:set all

:redir END Now, register 'a' will have the output of the "set all" ex command. You

can paste this into a Vim buffer. You can also write a Vim function

to do the above. For more information, read :help redir Anonymous, August 7, 2001 14:13 Wow!!! That's awesome!! Exactly what I want!

mark@zieg.com, July 25, 2002 11:28 This may be obvious to experts, but it took me a very long time to figure it out, because Google searches on terms like 'pipe', 'buffer', 'shell', etc never brought it to my attention. However, you can pipe the contents of the file currently being edited (the current buffer) to a shell command, and replace the current file/buffer with the _output_ of that command, using this: :%! [cmd] ie, if you didn't know the :retab command (as for a long time I didn't), you could expand tabs using basic unix commands like ":%! expand -t 4". Wish I'd known this a long time ago, so I'm posting it here in the hopes that others might find it :-) aniou@root.pl, February 18, 2004 14:01 The answer is (for ex.): :read !ls ~ and :help :read for more info :-) Grateful, September 27, 2004 12:10 Thanks Anonymous and Yegappan, I've long wanted to do this too, but never known how. Great initiative Anonymous!

From the old edition of Unix Power Tools

30.37 Neatening Lines Have you made edits that left some of your lines too short or long? The fmt utility can clean that up. Here's an example. Let's say you're editing a file (email message, whatever) in vi and the lines aren't even. They look like this: This file is a mess with some short lines and some lines that are too long - like this one, which goes on and on for quite a while and etc. Let's see what 'fmt' does with it. You put your cursor on the first line and type (in command mode): 5!!fmt which means " filter 5 lines through fmt ." Then the lines will look like this: This file is a mess with some short lines and some lines that are too long - like this one, which goes on and on for quite a while and etc. Let's see what 'fmt' does with it. This is handiest for formatting paragraphs. Put your cursor on the first line of the paragraph and type (in command mode): !}fmt If you don't have any text in your file that needs to be kept as is, you can neaten the whole file at once by typing: :%!fmt There are a few different versions of fmt , some fancier than others. Most of the articles in the chapter about editing-related tools can be handy too. For example, recomment (35.4) reformats program comment blocks.

cut (35.14) can remove columns, fields, or shorten lines;

tr (35.11) can do other transformations. To neaten columns, try filtering through with the setup in article 35.22. In general, if the utility will read its standard input and write converted text to its standard output, you can use the utility as a vi filter. - JP

Google matched content

Softpanorama Recommended

Society

Quotes

Bulletin:

History:

Classic books:

Most popular humor pages:

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

Copyright © 1996-2020 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) in the author free time and without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine. FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes. This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree... You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info Disclaimer: The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: July 28, 2019