In the bowels of a business unit, a director got a great deal on a third party software package. He bought it, without talking to corporate IT, and then was upset when it couldn’t gracefully integrate with any of the corporate IT assets. Eager to throw good money after bad, the director hired his nephew’s consultancy to build an integration tool to make his new toy work.

A few months later, the users complained about performance, and somehow, fixing this thing became Jeff’s problem. The process was simple enough: slurp enterprise data out of a text file, and pass the data on to the third-party tool. It didn’t take Jeff long to figure out why it performed poorly:

Private Sub ProcessFile() ' prepare to do stuff Do Until blnLastTime = True Set fileReader = fso.OpenTextFile(strFileName) If fileReader.AtEndOfStream = True Then blnLastTime = True Else strTextLine = fileReader.ReadLine End If ' actually do stuff fileReader.Close Delete_Line (strFileName) Loop fileReader.Close End Sub Private Sub Delete_Line(strFile) Set fileReader = fso.OpenTextFile(strFile) If fso.FileExists(strFile & "2") Then fso.DeleteFile (strFile & "2") End If Set fileWriter = fso.CreateTextFile(strFile & 2) If fileReader.AtEndOfStream = False Then fileReader.ReadLine End If If fileReader.AtEndOfStream = False Then strLine = fileReader.ReadAll fileWriter.Write (strLine) End If fileReader.Close fileWriter.Close fso.DeleteFile strFile, True fso.CopyFile strFile & "2", strFile, True fso.DeleteFile strFile & 2, True End Sub

Start by opening a file “foo.txt”. Read a single line from the file. Send it to the third party app. Close the file. Open “foo.txt” again. Open another file, called “foo.txt2”. Read the first line from “foo.txt”, again. Throw that away. Read the remainder of “foo.txt”, and write it to “foo.txt2”. Copy “foo.txt2” back over “foo.txt”. Now, go back to the top of the loop and read a single line from “foo.txt” again.

So, for a 10,000 line file, this would perform 30,000 file open operations, write nearly 50 million lines, delete 20,000 files, and perform 10,000 copy operations. It didn’t take Jeff very long to rewrite this to simply read the file, one line at a time. The runtime dropped from a few hours to less than a minute.