Quitting Iris

Completely Unqualified

Getting to Work

Slow Going

More Code, Lots More

// the fucking alpha cpp compiler seems to fuck up the goddam type "LPITEMIDLIST", so to work

// around the fucking peice of shit compiler we pass the last param as an void *instead of a LPITEMIDLIST

* for idiots like MS-Access 2.0 who SetWindowPos( SWP_BOZO

* and blow away themselves on the shell, then lets

* just ignore their plea to be removed from the tray

All Done

This is the story of when I re-wrote the Lotus Notes Formula Engine. It's a pretty cool story, it turns out Vader is Luke's father and I get frozen in carbonite. No wait, that's something else. Enjoy!I've made my living writing software now since 1995. But it wasn't until 1999 that I became a hardcore engineer. That was the first time I quit my job at Iris Associates. Iris was the organization that created and developed Lotus Notes and I worked on the template team. Notes itself isn't really an application, it's more like an OS, and templates are the applications that Notes runs. Describing how Notes templates work is a difficult thing, there is really nothing else quite like it the software world, but I guess I'd describe Notes templates as sort of like web applications. As a template developer, you must deal with back-end storage issues, you are writing in scripting languages for both the back-end and front-end functionality and rendering stuff, it's sort of like working with ASP, HTML and JavaScript...sort of.While I was on the template team I developed the original Notes Welcome page, I did work on e-mail and a lot of work on some obscure administrative applications, among other things. At the time I quit I was working on new features for the R5 point releases. But I hated it. I hated pushing around pixels all day, negotiating with the Quickplace and Sametime teams who wanted tie-ins and functionality on the Welcome page. I frequently worked 80 hours a week, was stressed and was seriously burning out. Why did I work so hard at something I hated? I don't know, I think because I wanted desperately to get away from it. So I thought if I just worked harder, someone would be impressed enough to give me a shot working on something I wanted to work on. Something more geeky and computer sciencey, instead of the touchy-feely UI crap where marketing types argue with me all day about how they think it should work and look. But I think I just ended up trapping myself. Some months later I was told by another project leader that I was barred from moving to his team by my managers because I was too vital to the template team.UI stuff is hard, but not many people actually appreciate how difficult it is to do correctly. Everyone thinks they're an expert, but very few have actually spent any time thinking seriously about it, reading any article or books on the subject, watching users behind one way mirrors during usability tests and seeing how easily users get confused and distracted. So one day I'd had enough, I couldn't stand it anymore, I was tired of battling all day with people who thought their "little" tweak would have no repercussions whatsoever. Being a UI guy isn't just producing UIs, it also requires negotiating with lots of people, at least at Iris it did, which was the part I liked the least about the work. That's probably why Notes historically has such a maligned UI, because the culture there made it difficult to produce anything decent. So after a particularly frustrating run-in that I won't go into, I quit. I put in my two weeks and that was going to be that.Then a very friendly guy named Wai Ki Yip called me before my last day. He was a manager and engineer and he offered me a job on his team, the Language Services team. The Language Services team was responsible for the programming languages found in Notes and Domino: Formula Language, LotusScript, Java, and JavaScript. He wanted me to take over Formula Language, which was the original and most tightly integrated language in the product. Actually, unlike the other languages, it only existed in Notes and Domino and nowhere else. If you've used spreadsheets formulas, it kind of resembles that. There was no one who "owned" the language, no one who understood how its engine and compiler worked, and Wai Ki's group had only recently taken it over. It was pretty much an orphaned area of the product, but a very important area nonetheless. It was used for building view indexes, in Notes and Domino forms for field validation, dynamic rendering, application logic, in macros, in agents, and dozens of other places in the product. As a template developer I used Formula Language every day. For certain things it is highly productive, but for others it's completely unsuitable. Until Notes 4 it was the ONLY programming language in the product, and even though it was a simple language lacking even the most basic constructs that everyone takes for granted (like looping) it was possible to build very powerful applications using just that.Oddly, writing applications in Lotus Notes and Formula Language was the very first paid thing I did as a software developer. I started with Lotus Notes V3 in 1995 during my first college internship at a software engineering group at IBM in Charlotte. They used Notes internally and I was put to work on the internal Notes applications, maintaining existing ones and writing new ones as needed. Then I worked as a consultant, writing custom Notes applications for mostly Fortune 500 companies. I became so good at it (and sick of consulting) that I went to work at Iris to develop some of the templates that ship with the product. So it seemed very strange that I would get offered the job of taking over a language that only a few years before I was using just like millions of other customers, I'm still not sure how I got from one to the other.No one on the language services team, or even at Iris, understood the runtime engine internals completely, also known as "Compute" (the compile/decompiler was called "Compile", great names eh?). There was no documentation, the code was barely commented, many variables had one and two character names, wild pointer arithmetic was everywhere and not even the stored byte code format the compiler produced was documented. Bugs were piling up and there was no one to fix them. On top of that the engine was becoming a bottleneck for server performance, particularly for building view indexes. The engine needed to be rewritten from the ground up, it needed to be made maintainable, it needed to be faster, and it needed to be completely backwards compatible..So here was I was, offered this position that I clearly wasn't qualified for. I had no experience with language runtimes or compilers, I knew very little about C and didn't know anything about C++, I had never dealt with platform byte ordering and packing and all the other issues associated with writing something for eight different operating systems, I had never even used proper version control. But none of that mattered to me. It seemed to me like an amazing opportunity and I would be doing exactly the kind of stuff I enjoy most. I wouldn't be bothered all day by marketing folks asking could the forms be mauve, I wouldn't have to worry about the wording of some field label on some feature that no one even knew how to get to, I wouldn't have to deal with icons that were too Anglo-Saxon, too masculine or too feminine, I wouldn't have to worry that some important VP wanted a new button that would confuse 90% of users. I would just write code. So I said yes.To this day I don't know what possessed Wai Ki to offer me the job. It doesn't make sense really. I can't help but wonder if I was thrown into the fire just because they thought I couldn't do it, that I'd try it, fail miserably and go back to the template team. Maybe, or maybe he thought I would really be able to do it, maybe he saw something in me. All I know is that I was terrified. I knew the odds were far against me, and if I failed it would be because I wasn't good enough. I couldn't blame anybody but myself. But still I said yes, I was going to give it my best shot.I remember talking with friend that overheard a very senior engineer saying I didn't have any real chance of pulling it off, that the old engine was very optimized and mature and was nearly fast as could be reasonably expected. Another engineer who had done a little work with the engine told me I had a very difficult task ahead, and if I pulled it off it would be a miracle. This was the kind of stuff that really motivated me, I love it when people think I'm going to fail. It adds one more reason to do it, to prove wrong the people who don't believe in me.So I moved my Pee Wee Herman doll and tropical fish lights from the client development area to my new office in the server development area and I immediately got to work on the backlog bug queue. See, even though I was supposed to eventually rewrite the engine for the R6, the engine in R5 was buggy as shit. There were crashes and weird behaviors all the time, but since nobody "owned" the engine, no one fixed the bugs. The bugs often weren't reproducible and the code was tricky. It's not like there weren't people qualified to fix the bugs, but there simply weren't enough GOOD people to fix the bugs. And for most of the bugs, you had to spend a lot of time learning the engine before you could make heads or tails of the code, which as I said before was notoriously terse and hard to understand. Iris was a lot like most engineering organizations, most of the good work was done by handful of talented people, the others often produced barely functional crap that would need to be cleaned up someday. So since the all the good engineers were working on other bugs and features already, the engine got neglected. Plus no one wanted to do it.So I started on the bugs and I picked the easiest ones first. It was slow going as I didn't know C very well and I didn't know the Visual C++ debugger at all. Really it was pretty damn painful. But I spent my first couple of months just on the bugs, and I learned C and I learned the engine and I learned the joys of rampant pointer arithmetic and two letter variable names and no comments. I still remember those first weeks, it seemed like I looked at the code all day and dreamt about it all night. My brain seemed like it was rewiring itself so that it could understand what was going on. I learned slowly at first, then faster. Eventually I could actually look at the code and read what it was doing instead of slowly trying to figure it out it line by line.So I spent the first four months just learning and fixing. Sometimes I'd cause regressions, especially early on. At one point I made so many mistakes that the build team talked to Wai Ki about me, I was slowing them down and if I kept it up they would take away my code submission privileges. I'd have to get my work reviewed by someone else for each code submission. So I learned to get much more careful and thorough, and eventually I cleared out a rather large backlog of bugs without unleashing total chaos. I had accomplished step one.After I cleared all the bugs out, it was time to start rewriting the engine. The first thing I did was make clear documentation of the compiled format for the formula language byte code, down to the individual byte. Then I decided on the design for the new engine in my head, presented it to Wai Ki on a white board in my office explaining high level architecture and what it would accomplish. It was nothing like the old engine. Where the old engine was naive and slow, the new engine was going to be slick and efficient. And the new architecture would allow for all sorts of constructs that simply weren't possible in the old engine. For example an embarrassingly simple thing like looping wasn't possible with the old engine, it just wasn't. It went against its whole architecture. My new architecture was profoundly different and quite ambitious.Now you might think that I produced a bunch of design documents and specifications and presented them to the various senior engineers and architects, but I didn't. I remember being surprised by this myself. Even Wai Ki didn't have much to say about my design or how it should be implemented. The philosophy was that if I did those things, everyone would meddle with the design and nothing would get done. It's truly easier to ask forgiveness than to ask permission, not to mention things get done a lot faster if you just do them. It sounds weird, but that was big part of the culture at Iris. Just get it done and do it right, do whatever it takes. So that's the way I worked, I was just going to go headfirst into it, damn the torpedoes.Since most new areas of the product were being written in C++, I decide also to write the new engine in C++. I started reading Bjarne Stroustrup's The C++ Programming Language. I only got about one third through the book before I decided that I understood as much as I could without actually coding, and so I started coding. At first I didn't really know what I was doing, it was such slow going I probably only wrote about 1,000 lines of code the first month. I asked other engineers really dumb C++ questions, the kind of questions someone asks when they don't even know what's important. I would often get puzzled looks, not because the questions where so dumb, but because some guy who's rewriting the core language of a mature product like Lotus Notes should know the answers to those questions. But I slogged on. At one point I realized that I had written a large part of the code all wrong, that I was trying to use C++ in the wrong way (a very easy thing to do), and I had to throw away most of the code I had written.Near the beginning of the project, Wai Ki and I set milestones for my work. Actually, he just asked me the order I was going to do things and then how long each thing would take. I just made uneducated guesses at the tasks and their completion times and he wrote them down. For some reason I really thought I needed to meet that self-made schedule, I had said that's what I was going to do and dammit I was going to do it.At one point early on I became panicked, I thought not only there was no way my engine would be complete by the R6 feature deadline and the product would ship without it, I was scared to death that the new engine would never really work, that there was going to be some fatal flaw in my design that made it fundamentally incompatible with the old engine, or the new engine was going to be slower and hence useless. But through all this Wai Ki seemed to have unflagging confidence in me, he calmly told me that I just need to take my time, do it right and not get so stressed. At one point I was in his office talking about the all the work and how behind I was and how everything looked bleak, and the longer I talked the higher and louder the pitch of my voice got. He told me I was "freaking out," to which I replied, "Yes! YES! ....I'M FREAKING OUT!!!" But it was his calmness and confidence in me that gave me the courage to throw away large chunks of work when I saw it was bad. See, if I was panicked about it, then I'd have kept the shitty code around in a wrong-headed attempt to just get something built and worry about all the problems later, which is a very bad thing to do.So I kept at it. It took me nearly six months to produce about 8,000 lines of code. The new engine had a lot of architecture to be built, there was engine startup and shutdown, reading and validating the byte code, formula optimization analysis and building the computation structures, memory and resource management, interfacing with various contexts in the product, the error handling model, variable scope resolution, and making all the internals object oriented so the individual @function and operators could be written easily and clearly (and no pointer arithmetic, I abstracted it all away). During this time I made some very unconventional decisions, the kind that taken in isolation sound crazy, but when combined into a complete architecture make good sense. It took six months to write enough code so the engine could actually run a formula. I was a couple of months behind my arbitrary schedule but I finally had something. I remember the point at which there was nothing else to write to actually run the engine, it was a surreal moment, I felt like there was no way I actually could have reached that point, but somehow I did.Once the new engine was to a point where it could actually run, I needed to find something to test it out. I needed to see if it could actually run real formulas and if it was as fast as the old engine and could produce the same results. So I picked the most commonly run formulas I could find; the view formulas for the All Documents view in Notes mail. These were the formulas that got run for every mail message that gets delivered in Notes mail. I implemented all the operators and @Functions used in the formulas so it could run in the new engine.I still remember that day I turned it on and turned it loose on a Notes mail file, the day my new engine truly came to life and did something useful. I ran the new engine and the old against the same mail file, and compared the results. The computation results were byte for byte identical, and much to my amazement, the new engine was just barely slower than the old engine. This was very good news because the old engine had been tuned and tweaked years ago for optimal performance, even I had recently added to the old engine's performance during my bug fixing phase when I found some inefficiencies that had crept in. So here was my brand new untuned engine, hot off the presses and keeping up with the old, and I was a proud papa. I was so excited I could barely sleep that first night I got it running, because I knew the next day I was going to start profiling the new code and find its slow spots. I expected to get the new engine to be maybe 50% faster than the old, and I would have succeeded. I would have built something faster, I would have done something that extremely smart and well-respected engineers said couldn't be done.I profiled the engine, found the hotspots and destroyed them. They say you'll always be surprised by what the profiler shows, and I remember that I was. But one by one I picked off the hotspots, the easiest, most obvious ones first and then the trickier ones. And something amazing happened, the new engine wasn't just faster, it was a shitload faster (shitload is an engineering term meaning great amount). The measurable performance gain was 300% faster (I would later discover it was even faster when running in other contexts). I was very excited, I could barely believe what I had done. I checked and rechecked to make sure I was right. Everything pointed to the new engine being blazingly faster. It felt like I worked a miracle, I was giddy as school girl.It took me six months from starting coding to getting the engine to this point, to getting it to 8,000 lines of code that just had minimal functionality, the job wasn't even close to being complete. I had to rewrite the over 200 remaining @functions and operators before I could check the new engine into the daily build. I remember I had a big paper list of @Functions I worked from, and as I rewrote each one to work with the new engine I crossed it off my list. This part went so fast I barely remember anything from this two month period. I woke and coded and coded all day everyday. It was like I was on autopilot, my brain felt like a machine. Each @function I completed was like a hit of a drug and I was a junkie looking for the next fix. It's all a blur now, the code all runs together in my memory. It took me just over two months to complete, but when it was done I had written over 15,000 more lines of code. During the six months writing the first 8000 lines, I produced an average of around 60 lines a day. During that two month period my production had jumped to around 350 lines a day. Of course it was a lot easier to write that code, I didn't have to worry about architectural issues much, just individual @functions.Once I had the new engine completely rewritten and working on a side branch I ran it for about two weeks with it as my primary mail client. Then we ran it on several production Iris mail servers. This was a great test of code. At Iris we always ran experimental code on the production mail servers. You find so many bugs that way because it's a real world production environment. I was on call constantly, when the production servers went down I was usually the first on the scene debugging the problem. If it was my bug I'd fix it and rebuild and reload it on the server, if it was someone else's they'd fix it and put it into the daily build and I'd pick up the changes the following day. When everything was fairly stable, I checked-in the new engine code in late August of 2000. I was actually ahead of schedule. Lots of problems were found and I fixed them as soon as they came in. Some were crash bugs, some where incompatibilities with the old engine, some were just plain weirdness. But soon the problems slowed to a trickle and found myself with very little to do.That's when I started adding all the new stuff that I wanted to add to the engine. I remember I posted a question to a forum dedicated for professionals who worked with Notes and Domino (The Business Partner Technical Forum), and I asked what sorts of features they would want to see in the new engine. That question got over a thousand replies. Some of the requests where reasonable, some where stupid, some I just didn't understand. But I read each and every one and tried my best to consider them. In all when I was done adding all the new enhancements, the total code base had swelled to over 30,000 lines.And through all this the new formula engine continued to get tested. The new engine was pretty reliable from the get-go, it very rarely crashed and when it did I relentlessly hunted down the cause until it was found and fixed. After about the first six months there were pretty much no crash bugs at all, almost all the bugs that came in after that point were slight incompatibilities with the old engine. More often than not, the old engine had it wrong, it had some indefensible behavior that was clearly not what was intended and often the formulas that relied on these behaviors did so unintentionally. It's like two wrongs making a right, but regardless if the old engine was wrong or right, the new had to have the same broken behavior. In software we call that being "bugward compatible". It sucks, and I absolutely hated that I had to go in and purposely mess up my carefully crafted, elegant code but that's what I had to do. That's necessary for a truly stable and backwards compatible product.I remember when the Microsoft Windows source code got leaked on the net and somebody had searched all the code for expletives. There were tons of angry comments like this:and:I completely understand why the coders put in comments like that, because you feel like it's your job to ruin your own beautiful creation because someone else wasn't smart enough to use an API correctly, so you vent. I think I left at least one nasty comment like that on a bad day, but I felt like that much more than once.Notes and Domino 6 was eventually released more than a year later than originally anticipated. Had I known, my schedule probably wouldn't have been so aggressive, but it was a good thing for the new engine because it got so much more real world beta testing. I had already left IBM before the final release, but I kept tabs on what was going on and the new engine was a big success. While a couple of serious but rare backward compatibility bugs were found and fixed after release, the new engine was very solid, as far as I know it never crashes.The thing I really liked was getting feedback from people who coded Notes applications exclusively using Formula Language. There is a large population of developers who never bothered to learn more structured languages. They would tell me how all the new features made their job so much easier, and they could now build applications they couldn't before. Of course lots of people liked that I just made the language more powerful so they didn't have to use those other languages. I loved that my work got many of these people excited to upgrade to Notes 6 and that it played a part in making Notes 6 a big success. Many people make their living building Notes applications, and they bet everyday on us. I wanted to make those people right, I wanted them to be successful because they believed in us and the product, and I felt like I helped make that happen.Rewriting the Formula Engine was only one of the things I did during R6 development, I actually spent less than half my time on it. I also took over unread marks and developed the now patented unread mark replication system. Unread marks are a simple thing in concept, but in reality they are very difficult thing to do correctly, particularly in a distributed system. Before my work on them they were very much broken, with both flawed implementation and flawed design.I also spent many months during crunch time working on the Database Team fixing bugs and performance problems. These were some of the hardest bugs I had ever faced, frequently we had no reproducible scenario, just a memory and stack dump, sometimes not even that. But just because a bug wasn't reproducible didn't mean we could ignore it. I spent countless hours pouring over hundreds of thousands of lines of code, trying to find the code flaw that would later crash the system. For some reason I really enjoyed it, I learned a great deal about how to write and debug high performance server and database code. Just to see if I could do it, I set a goal for myself that every week I would try to fix more database bugs than any other member of the team. This was a private goal, not something shared with the team, but it was something to keep it really challenging for me. More often than not I met that goal, and the faster I got, the tougher the bugs the database team gave me. Eventually I got one bug that had been crashing servers for six months, but remained unfound. It had stymied at least three other engineers before me, but I don't give up easily. I probably spent 80 hours on that bug alone, but I found and fixed it. It was really a lot of fun, but I'm weird like that.My time at Iris during R6 was the best I've ever had in my professional career, almost every day was a new challenge and game to play. But ultimately IBM killed Iris. In late 2001 IBM folded Iris Associates and converted all the employees to IBM. All the energy the Iris building had was seeping out. Really it was more pushed out by the BS that IBM management kept trying to sell us, like that nothing important would change and you'd still be able to work unfettered. Yeah right, just ask anyone who still works there how much time they spend battling bureaucracy vs. writing code. But such is big corporate management, they don't even realize how much they're damaging things, they are essentially politicians not engineers. It's sad, but I'm grateful for the time I had at Iris anyway, if it was still there I'd go back in a second.Update: My follow-up can be found here