40. (with egg)

At Pivotal Labs last week we watched Linus Torvald's Google talk about Git, the Source Code Management (SCM) system he wrote and uses to manage the Linux kernel code.

I've watched it twice now and here are some thoughts, based on quotes and themes from the video.

"I Never Care About Just One File"

Linus stated that one of the reasons Git was wonderful for him is that, as a high level code maintainer, he needs to merge thousands of files at once. In fact, he stated that he never cares about just one file.

Not so for me. As an in-the-trenches developer, my whole life is caring about just one file, over and over again. When I merge, I care about each file because, since I work on small teams and with small codebases, there is a fairly high likelihood that my changes will collide with those from another developer.

"The Repository Must Be Decentralized.... You Must Have a Network of Trust"

Linus made the point that central repositories suck for large projects where the morons must not have commit access -- only the super privileged are allowed to commit code back to the repo. He argues that Git is better because it is a decentralized network of repositories -- there is no central master, only Some Dudes who have repositories. Usually there is Some Dude In Charge, like Linus, and everyone tends to pull code from them. To update the "master" code version, Some Dude In Charge pulls code from the repositories owned by Some Other Wicked Smart Dudes, who have most likely pulled code from Some Other Trusted Dudes (And One Gal), and so on. Thus, rather than limit access to just the hand-selected few, everyone has their own local copy of the repository, and the smart merge from the smart who merge from the smart, resulting in some kind of official or de facto version.

While I like the local copy of the repo idea, Pivotal does not work the way Linus describes... but Pivotal is weird, in a good way. We all have full commit rights. Our network of trust is everyone. The Dude In Charge is named Continuous Integration. CI makes the official versions. CI runs the tests. CI makes sure that the deploy process works. I'm sure that we could coerce Git into working in a centralized-like way, where it merges automatically from the individual developers and runs the builds, but I'm not sure if that would be forcing a square peg into a penguin-shaped hole.

"Some Companies Use Git And Don't Even Know It"

Linus described how developers at some companies use Git on their development machines, committing their changes and merging fellow developer's changes with Git, then pushing those changes to central SVN repos. He rather mocked this, but it actually sounds like a good solution: developers merge, so use the tool that's good at that. CI machines and deploy machines love centralized master repositories, so use that for those jobs.

"It Does Not Matter How Easy It Is To Branch, Only How Easy It Is to Merge"

Well said. I never thought about that before but he is completely right. I could never put my finger on why I never branch in SVN, even though it's practically 'free' and easy to do. Now it's obvious: who cares how easy it is to branch when merging sucks? Git is supposed to make merging incredibly easy because Git is content-aware rather than just file-aware... or something like that. I'll believe it when I see it, but if Git really does make merging highly divergent branches easy then I'll give it a try.

Joe's Take

I'd like to try Git, especially if it makes branching and merging those branches as easy as Linus suggests, but I don't think that Pivotal would get as much benefit out of it as large, distributed open source projects. A 'really big' project might have 10 developers, not thousands, and all must have commit rights. Our network of trust goes like this: if you are here, we trust you; if we don't trust you, you have to leave. And the idea of having to merge directly from my fellow developers sounds like a pain in the ass... why would I want to merge from 3 separate pairs when I can pull code from the central repo and be reasonably sure (thanks to CI) that it is clean and green? Hopefully I'll be able to answer those questions soon by using Git on a project.

I wrote the following in response to a coworker's questionnaire about continuous integration (CI):

CI's impact has been higher on some projects, while lower on others. If we are including past companies and projects, CI was extremely valuable when builds took a very long time to run on an individual's workstation (such as hours) and when CI tests things that are "hard" to test locally, such as deploys.

In general, CI becomes more important to me as the project becomes larger (resulting in longer test runs) and more complex (resulting in longer overall builds plus technical overhead/pain-in-the-ass factor). For those reasons I really like having CI set up at the beginning of a project in preparation for the impending time-consuming pain-in-the-ass.

Specifically for projects I've worked on, here's how CI has affected me:

• Previous project: Tests took a long time to run because of a huge Selenium suite, so offloading it CI was nice.
• Current project: Test suites, including Selenium, do not take very long so the time savings is not such a big deal.
• Both: Deploy testing is invaluable. We have a complex deploy process at and I sleep better at night knowing that CI is exercising this every time it runs successfully. If CI did nothing but deploy testing then it would be worth it.

Where broken builds have hurt me:

• For both projects a consistently red CI usually meant timing issues in Selenium. On a previous project these were especially annoying, since the build took over 30 minutes. In both projects we have solved these, but they were rarely if ever indicative of a "real" problem -- instead, the we had to make the testing framework handle the timing issues. The end result was that I was "not allowed" to do deploys because our deploy process, if you follow it, relies on a successfully CI run. This is not a criticism of the process: I wish everyone would strictly follow the CI-reliant deploy process because it brings order to chaos and will save you from shooting yourself in the foot by deploying bad code.

"I'll take CI if I can get it, but I don't really care THAT much whether it's there or not".

• Sometimes. It's a continuum. When a project has no deploy process and test take 1 minute run, and the team is 1 pair, then CI is a sledgehammer hitting a nail. For projects larger than then my-workstation-can-quickly-handle-it situation, with long, complex test suites, complex deploy process, CI is very valuable.

Installing Rails on Ubuntu and Windows XP at the same time under Parallels

Now that I have my Ubuntu Linux and Windows XP virtual machines running in Parallels (see Street cred? No: Geek cred (Part 1 of 2)), it's time to set them up to run Ruby on Rails. Why? You know... because.

Again, I'm not going to go into excruciating detail, but instead reference the sites that helped me with this and the problems I faced. Luckily there were only a few.

Windows XP + Ruby on Rails == Easy

Here I pretty much followed the instructions on the main Ruby on Rails site. As much as the Rails community loooooooooves their Macs, getting Rails up and running on Windows is much, much easier and faster than any other OS. No ports, no compiling, no apt-get install voodoo. It's just double-click and next-next-next-next-next...

• download and install using the ruby windows .exe installer. Make sure you select "enable gems" when asked!
• download and install mysql
• gem install rails -y
• gem install mysql -y
• gem install mongrel_cluster -y

Ubuntu Linux + Rails = Funkadillio

The guide to installing Ruby on Rails on Ubuntu Dapper Drake at Urban Puddle was a very good guide, but I still had troubles. The issues had the same theme: I needed development/compilation tools, which I had not installed as part of the Ubuntu install process. Here are the other things I needed to do:

Ruby Dev Library

I got the following error when trying to gem install mysql -y and gem install mongrel_cluster -y:

extconf.rb:1:in 'require': no such file to load - - mkmf (LoadError)

Solution: install ruby1.8-dev:

sudo apt-get install ruby1.8-dev

make And gcc

The mysql and mongrel gems also complained about make and gcc not being found:

make
sh: make: command not found

and

make: gcc: Command not found

If you get this, install them both:

sudo apt-get install make
sudo apt-get install gcc

Mysql Client Dev Library

I ended up installing this, too, but I'm not sure if it actually fixed anything.

sudo apt-get install libmysqlclient12-dev

That's it so far. I'm planning on setting up the Ubuntu image as a Capistrano deploy target and treating it as a test server.

I'm finally getting around to playing with Parallels Desktop for Mac, specifically the free release candidates for the next version. My geek-meter is rising with every moment. Here are some of the cool things and problems I found along the way. I'm not going to detail a complete install guide, but I will reference the ones I used and give credit where credit is due.

Windows XP on Parallels

The Parallels Release Candidate includes this nice feature: if you already have a Windows installation on your Mac from using Boot Camp, Parallels will let you run it in a virtual machine. This gives you the best of both worlds: a true hardware-accelerated Windows install for games (why else?) that you can also fire up in Parallels when you want to see if your CSS works in IE6 how badly your CSS is broken in IE6.

Problems with Windows XP on Parallels

Parallels wouldn't start my Boot Camp image immediately, instead giving me this error:

Parallels Desktop cannot find necessary drivers to configure your Boot Camp partition. Please refer to Help> Parallels Desktop Help>Using Boot Camp Windows XP Installation for troubleshooting*

... and so I did. Help> Parallels Desktop Help>Using Boot Camp Windows XP Installation suggested the following:

1. Boot into Windows XP via Boot Camp
2. Insert Microsoft Windows XP installation disc which was used for this Windows XP installation.
3. Locate the folder i368 on the disc and open it. Copy the following files: driver.cab and any of spN.cab files (sp1.cab or sp2.cab, etc depending on service package installed) to the folder: C:\WIndows\Driver Cache\i386.*

... and so I did. The virtual machine worked fine the next time I ran it, no problems.

Ubuntu Linux on Parallels

Nothing very exciting here. When creating the virtual machine, I created a "custom" installation of "Linux/Other Linux kernel 2.6"; this worked fine (mostly... keep reading.) Download yourself an ISO disk image from Ubuntu's download site or bittorrent one. I'm using Dapper Drake (6.06 with Long Term Support). When prompted to "Insert Linux Kernel 2.6 installation CD," choose "More Options" and specify the location of your downloaded ISO.

Problems with Ubuntu Linux on Parallels

I had to install it twice for some reason, which means that I had to delete the first virtual machine I crated and start from scratch. There was some kind of fatal error that I didn't pay attention to at the time, and thus I cannot tell you what it was; now unhelpful of me.

Next up: setting up the virtual machines for Ruby on Rails development.

*Text is most likely © Parallels. Hopefully they will be nice and let me help people help themselves and also sell more of their product.

Does you're company have one of these?

It's a super fancy digital whiteboard/printer! I've worked for 2 companies with these things. It was supposed to be a killer app, so to speak, since many of us have had the same thought: wouldn't it be great if we could get a print-out or digital copy of our whiteboard diagrams? Imagine that! So some company invented this contraption with a printer built right in, which could also scan the drawings and save them as digital images. And guess what?

Nobody used it. Not even as a regular whiteboard. Why? Several reasons:

• It was never plugged in.
• Nobody knew how to use it.
• It was small: about 1/4 the size of a "real" whiteboard around which several people could gather around.
• Dry-erase markers acted more like semi-permanent markers, smearing and marring the writing service.
• There was something scary about it. I can't explain it.
• Did I mention that it was never plugged in and nobody knew how to use it?

Note that while I am bashing the digital whiteboard, I'm not bashing the plotter, which does rock, though occasionally print The World's Largest Email when you forget to change your target printer.

Don't Erase That!

At Pivotal Labs we make high-tech digital copies of our whiteboard drawing and notes all the time: we use a digital camera. Before erasing a whiteboard that I want to use, I fetch my Canon PowerShot SD450 Digital Elph, a teeny-tiny 5-megapixel camera, which I have in my backpack at all times (please don't jump me if you see me on the street.) Then, after taking the pic, I run over to the HP all-in-one color printer we have, pop the Elph's SD card into the memory card slot, and print the pic right then and there. When it's done, I tape the print next to the whiteboard and write the date on it. There. Documented. High tech, yet simple. This also works great for marathon UI mockup sessions and you want to reuse the whiteboard: snap a pic, print it right away, and tape it up. And digital copies? 5MP worth!

... because it's where I work. We're still working on content.

http://blog.pivotalsf.com

Today, as we do every 3rd Thursday, Pivotal Labs has our Retrospective, a beer-aided romp though the last 3 weeks of Smilies, Quizzicals, and Frownies: that is, the Good, the Questionable, and the Bad.

Retrospectives are a core practice of many Agile teams, since they represent a dedicated time and place to discuss those exact topics: core practices. We try (and often fail) to stick to non-technical issues, but hey, we're geeks, and discussions often dissolve into passionate PC vs. Mac vs. Linux debates and the virtues of specific keyboard mappings. But, I digress. Here's a quick narrative about our Retrospectives and what we try to accomplish.

On day 1, the day after the last Retrospective, we place a large, sticky-backed sheet of paper against the Whoo-Ha wall (don't ask). This sheet has 3 columns: one marked with a large Smiley face, one with a quizzical/not-happy-but-not-sad face, and the last column topped with a frowning face. Over the next 3 weeks, as various successes, happy events, questionable activities, and frustrations arise, we write them down in the appropriate column. For us, perfectionists that we are, the Frownie column fills up fast and occasionally requires extra paper. Quizzical is close behind, also filling its space. The Smiley column usually has a sorry few examples of good events, and not because they don't happen, but because we are so self-critical that we feel the need to fix any problem that arises. On retrospective day, folks tend to dredge their memories for those happy bits, say, for example, the launch of our largest customer project in history; oh yeah, that was kinda nice!

And now, finally, it's Thursday, at 4:00pm, and 3 weeks since the last Retrospective... beer run! The nearest Tenderloin liquor store owner loves us.

Upon returning with several cases of brewskies, the Moderator, whom was usually reluctantly elected at that day's stand-up meeting, is trying to get the show on the road. Tops are popped and we're under way. The Moderator reads through the list of ToDo items resulting from the last Retrospective: have those assigned specific tasks followed up on them? Have items assigned to the group as whole been addressed? Discussion enssues.

Now, on to the Big Retrospective Paper: do we start with the Smilies or the Frownies? The Moderator decides. Either way, the Smiley topics go fast, with clapping and laughing involved. Everyone's happy!

Now, the Frownies and Quizzicals. In a company of elite geeks, everyone has an opinion, even if they don't voice it loudly. When I moderate, I try to call on folks that I notice do not speak up much, but it's hard -- assertive and opinionated attendants can dominate the conversation. We've made a lot of progress in this area lately and the results are promising: more voices heard. We spend a lot of time on the Frownie and Quizzical topics: when should we hold stand-up, and what should happen when people are late? What is the best way to roll clients out of our continuous integration environment after they spread their wings and fly off with their own development teams? What's the best way to deal with the fact that developers with laptops "pillage" workstations of monitors, keyboard, mice, etc.? And how about stand-up again? Oh yeah, and about the stand-up meeting...

It's both an science and an art to figure out how much time to spend on each issue, especially those that affect the entire group. Some people just want to move on, while others are far more invested in the same issue and wish to really hash it out. For the most part the group and Moderator combine to decide when to finish a discussion (at least for the time being) and address the next. In the end, every issue raised has a ToDo item directly addressing it, with someone assigned.

After the last item is addressed, everyone feels great on several fronts: We talked about a lot of deep topics! The meeting is over! Beer! It's definitely a group-hug moment.

Now, if we can just figure out a way to get everyone to show up at the stand-up meeting on time...

How's this for a New Year's resolution: Get in shape! Nope. Eat healthier! Nada. Call my grandma more often! Well, yes, but that's not the point. Here it is:

Switch to a super-ergonomic keyboard.

Most geeks know that repetitive stress injuries (RSI) are a big risk for us, especially software developers who literally type and type and click and click all the live long day. But recently it has dawned on me just how big a risk it is: more and more developers at our small consulting company have RSI problems, ranging from minor annoyances to complete debilitation.

Yo, P: Yes, I'm talking about you.

On Monday, a fellow developer had to stop typing for the week, doctor's orders. And for a coder, that's an issue, since coding-by-thought technology is seriously lacking. This caused me to do some quick RSI-inflicted-peeps math:

• We have about 20 people in our office:
• At least 6 have RSI problems that I know of, which means there might be more.
• 1 has been under doctor's orders not to type for the last year, and another has been told to keep of the keyboard all week.

These folk's ages are, as far as I can remember: early 40s, 39, 34, 32, 27 (no typing!), 23 (no typing!).

23? 27? Oh crap.

I'm 31 years old.

Right now, I have no wrist, arm, or hand problems, save for the psychosemantic tingling I feel right now, but I've decided not to mess around with my software ditch-digging tools. Quite a few people I know swear by the Kinesis Contoured Keyboard, which is a pretty crazy looking device, but is optimized to baby your hands. I'm not looking forward to learning a new key layout, but I'll deal.

Now that kids are growing up in the Internet age, maybe parents will ad a new nag to their arsenal: floss your teeth, put on sunscreen, and use an ergo keyboard.

As a side note, since we practice pair programming at our office, we can absorb RSI-inflicted developers since they don't have RSI-of-the-brain.

NOTE: You might have been redirected here. Don't worry! I've moved my active blog to 40. (with egg)

UPDATE: fixed my bad code...

Here follows my summary of "Why there is no Smalltalk-like [or Java-like] IDE for Ruby," compiled from the over 100 responses the threads on comp.lang.ruby and comp.lang.smalltalk,and the email thread on the Ruby on Rails mailing list . I will do my best not to bodge everything up. Interestingly, many comments on the newsgroups and most of the mailing list revolved around the IDEs are Good vs IDEs are Evil debate, which I might summarize at a later time.

I'll do my best with this! Be gentle.

Code Completion, Refactoring, etc.

It seems that the difficulty of Ruby code completion, refactoring,and pretty much every issue can be summed up as this: Ruby has no abuild-in concept of a live runtime "image" at development time, nostatic typing and method argument/return typing that clearly defines objects and methods, and no compiled artifacts that can be mined for data.

Smalltalk vs. Java

In a very, very small nutshell:

Smalltalk, like Ruby, is loosely-typed and (some say) compiled at runtime, but "runtime" is always happening. Smalltalk is alwayslive, always in a particular state at development time; the concept of"runtime" rather melts away, as there is nothing to really contrastagainst it. The state of the entire Smalltalk environment is saved in an ever-evolving "image" that can be mined for ample data to perform refactorings, code completion, and such.

Java, unlike Ruby, is a compiled, statically typed language, and these two features give an IDE ample data about Java objects at development time.

Challenge #1: Language differences between Ruby and Smalltalk

Ruby and Smalltalk are different beasts, even though they share many language constructs and philosophies, most notably loose typing, blocks, and the ability to "reopen" classes and instances at runtime to add methods, etc. But Smalltalk knows about it's development environment (link):

The Smalltalk language 'knows about' its development environment at quite a deep level. The fact that Ruby does not have any built-in knowledge of an environment makes it relatively hard work to make the language work 'seamlessly' in an IDE - I speak as someone who is trying to do just that :)

In other words, Smalltalk is tightly coupled to it's development environment. Smalltalk IDEs keep track of the state of things by keeping a constantly-evolving "image" of the code, which contains an enormously amount of metadata about the state of classes, methods, etc. For example, take the following example Ruby code, which changes a class named Shoe if a condition is met:

class Shoe
...
end

class ShoeTweaker
   def tweak_shoe
      if (self.i_feel_like_it)
         class << Shoe
            def laces #reopen class Shoe and change it!
               ...
            end
         end
      end
   end
end

If this code were saved in a Smalltalk-like IDE, this possible manipulation of the Shoe class would be stored in an "image", and the IDE (and the Smalltalk runtime environment itself) would forevermore know about this possible change to Shoe. Obviously, Ruby has no such facility built into the language itself. But, as pointed out in in the newsgroup thread, there is nothing preventing an IDE maker from providing this functionality (link):

... [In Smalltalk] methods and classes are /always/ added at runtime -- just as in Ruby.

The difference is that Smalltalk doesn't initialize itself by reading and executing a sequence of class- and method-creation expressions. The definitions already existed when your restarted your image. If you define new classes or methods, /then/ these are compiled at runtime and added to the image, so that if you save the image, |they will be saved too.

It makes Smalltalk /look/ as if it's like (say) C++ -- with a list of classes and methods which "just exist" and you browse over them in the IDE.... It would certainly be possible to create something similar for Ruby, but first it would have to have some way of preserving the state of the /entire/ computation between runs.

Hmm... an image-saving IDE would get around Ruby's interpreted nature, a very common theme pointing to why it's so hard to create a good IDE for Ruby. In Ruby, there are no compiled files or bytecodes to mine for metadata about Classes. The entire object tree is recreated as the script is executed. That said, the ruby Interactive Ruby Shell (irb) gets around this somehow, and if the irb's state could be saved, or always running, or if the editor in an IDE was the irb... This might be a great opportunity to implement a Smalltalk-like image/model paradigm for a Ruby IDE, but not make the same mistake choice :) that Smalltalk made to tightly couple the development image to the language and runtime itself. This might even be better (link):

... there are positive points about decoupling the programming from the environment. ... it is quite possible to introduce behavior that may have seemed neat this time last week but does not seem anything like as good now...

Challenge #2: Language differences between Ruby and Java

This one seems pretty easy to understand:

• Ruby is loosely-typed, while Java is statically-typed
• Ruby is interpreted at runtime while Java is compiled

As for the compiled-nature issue, reference the previously written stuff about Smalltalk images for info about how Smalltalk got around this. But, there are some interesting details about how Java IDEs have an advantage (link):

... In Java the IDEs use the front end of a compiler to build an abstract model of your program. The refactorings are actually applied to the model in a provably correct way, and then the changes are reflected back in the code. Having things like method generators, method_missing and dynamic structs makes this impossible to do without being inside a running program that can be inspected, which just won't work for a ruby IDE unless you turn it into a live memory type environment.

Java's statically-typed nature makes the world a relatively easy place to live in for IDEs: A class has X methods, implements Y interfaces, and inherits from Z classes. Method signatures are typed as well: in Java, lookUpNames(List names) takes an object of type List, while Ruby's look_up_names(names) takes... who knows? As long as that object don't blow up if messages are sent to it later, then we're happy. This makes code completion a little hard. Nobody had an easy answer for this (or even addressed it) but I'm sure there are some brute-force ways of implementing this, such as "guessing" Classes, or having the developer pick the Class once and remember it, or simply remembering a linked-list of method calls ('person' was once followed by 'name' and also by 'address', so those will be in the list of possible method calls...)

I'm sure I'm leaving stuff out, and if there are glairing mistakes I'll update this later. Hopefully those who were confused (like me) about why Ruby IDE development is so hard will find this useful.

1 Comments (from old blog):

At 10/29/2006 9:57 PM, Chad Woolley said…

There was a thread I saw recently where someone proposed that the unit tests could be leveraged to help exercise code in the IDE, and help determine the dynamic runtime state of the program. This is interesting, but even with 100% code coverage, you still couldn't ensure that every logic path through your program is covered (Unlike EMMA for java, which does block-level coverage metrics, rcov still only does line-level, and even that imperfectly).

Also, it seems like it would help a lot if IDEs were smart enough to find the areas where dynamic code might cause problems with refactoring (instance_eval, method_missing, etc), and either attempt to automatically handle them, or at least point them out to you and let you handle them.

-- Chad

Pop quiz: your browsing some ruby code (which you often do in your spare time) and see the following:

    party_like[:its] = "1999"
    puts party_like[:its]

What will be put to the console when these lines of ruby execute? You might be saying that since party_like seems to be some kind of Hash or a Hash-like-thingie, you should get...

    #=> 1999

At least 99% of the time you'd be right, unless the code is actually the cookies method in a Rails Controller:

    cookies[:its] = "1999"
    puts cookies[:its]

    #=> nil

BUWAHHHHHH? Cookies isn't a Hash, although the cookies method does return a CookieJar object, which extends Hash. CookieJar mangles the expected Hash methods of [] and []= with these gotchas:

• The CookieJar object does not represent one set of cookies, it represents 2 sets: the incoming cookies from the browser, and the outgoing cookies that will be sent back to the client
• cookies[:key]gets you the incoming cookies from the client
• cookies[:key]= value sets outgoing cookies that will be sent back to the client
• ... all of which mean that cookies[:its] = "1999" will not allow you to retrieve that value with cookies[:its].

But wait, it gets more confusing. In a test you do the following:

    def test_cookies
      @request.cookies[:its] = "1999"
       get :index
    end

And in your controller:

    class CookiesController < ApplicationController
      def index
        puts cookies[:its]           #=> nil
        puts cookies.inspect    #=> {:its=>"1999"}
        render :text => ""
      end
    end

The cookies.inspect confusingly returns {:its=>"1999"}, which seems impossible: I put the value in and get nil when I ask for it, but if I inspect the object to see what's up, it's there! Right? Not, not really. The incoming cookies are displayed, not the outgoing. In a typical test-fail-debug scenario, this is a head-scratcher.

For the love of SANITY: do not override methods on core objects to do completely unexpected stuff, and especially don't mangle the expected functionality of operators! I realize that with languages like ruby you can play God, overriding and changing the implementation of anything you please, but some things are sacred: +,-, [], []=, etc. + should add stuff. - should remove stuff. And objects that have [] and []= should let you get stuff with [key] and set them with [key]=value.

---

1 Comments (from old blog):

At 10/13/2006 5:33 PM, Joshua Jarman said…

I have a slightly different take on the cookies object. It is a hash, with a wormhole inside connecting it to the last and next browser requests. Essentially it doesn't exist in the now. ;-)

Cookies are decoupled. When you set the value it starts a chain of events. The server needs to send the cookie to the browser and the browser has to return the cookie. Then you can read the value.

    cookies[:its] = "1999"

    *server response
    *browser request

      p cookies[:its]
      #=> "1999"

You can use "app" in tests (or the console) to simulate user sessions and to send and receive cookies, which will allow you to get your values back out and check them.

Example:
http://clarkware.com/cgi/blosxom/2006/04/04

So maybe more twisted and misunderstood then broken.

Cheers,
Josh

NOTE: You might have been redirected here. Don't worry! I've moved my active blog to 40. (with egg)

Normally I don't spam/link post, but this is too important not to shout out to the world: Ruby development, with debugging support, and soon with Rails support, in... wait for it... wait for it...Visual Studio 2005! Check out Steel here:

If you talk to any of my coworkers, they will tell you that I am an Eclipse zealot. I love Eclipse. I love and understand it's way of thinking, and I am the most productive with this IDE. Well, at least I used to be when I was doing full time Java development with Eclipse's Java IDE, which is actually a massive Eclipse plugin. Now I'm a full time Ruby on Rails developer using the painfully poor Ruby/Rails plugins for Eclipse: RDT and RadRails. RDT and RadRails are developed by Some Dudes who are busy either working at real jobs that don't include developing free IDEs for me, or studying for finals and being excited that they are old enough to drink beer. Point being that the IDE support for Ruby and/or Rails is stone age at this point, and 99% of the features I use for Ruby development are not RDT/RadRails features at all, but default Eclipse feature or features from other plugins, such as the HTML, JavaScript, and CSS support from the excellent J2EE Standard Tools project.

If Steel can bring to Ruby/Ruby on Rails development even a small portion of productivity gaining features that I sorely miss from Eclipse's Java IDE, then I will burn my Eclipse Fan Club membership card and send the ashes to Erich Gamma, then buy Visual Studio for my workplace with my own money.

Can you tell I'm excited about this? I'm switching back and forth between writing this post and reading the Sapphire In Steel home page. I've just read the About Us page and I'm getting more insight into why these guys "get it". Here are some snippits:

has spent the last twenty years...

has programmed in languages ranging from Delphi and Java to C# and Smalltalk...

since 1988...

Ah, there it is -- These are Old Dudes! I love Old Dudes! And I really love Old Dudes Who Know Smalltalk! I was nurtured, sculpted, and brainwashed by Old Dudes Who Know Smalltalk from my very first day as a professional programmer, and they universally "get it". Young whipper-snappers out there, take note: if you ever here some Old Dude say the words "in Smalltalk you could blah blah blah" or "In VisualWorks you could yada yada", spend as much time with this person as possible. You will learn more from them about software development than the Young Dude who only wears black and thinks that the bash shell is "too bloated".

And what does "get it" mean? Maybe I'll get into that some other time (it will be ugly, as this is one are where I am very opinionated), but the important thing is this: these guys don't come from the school of web scripting hackery in vi, they come from the land of building real enterprise applications, where real tool support is appreciated. And at this point in the Ruby IDE game, I'd place my bets on them to produce the first a truly usefully development tool.

---

1 Comments (from old blog):

At 9/30/2006 8:18 AM, Torv said…

Well, finally you get it ;)

www.squeak.org

Right after I posted Unit Testing: Leave No Trace vs. Schmutz, our team went to the opposite extreme regarding test data: we decided to load all test data (that is, all fixtures) before each test method executed. This took advantage of Rails fixture "feature" of deleting all data from the target fixture's table prior to that fixture's data being loaded into the table. Using this technique, we include all fixtures, thus deleting all Schmutz in the database prior to each test run.

At first, I was against this due to my Leave No Trace Rules philosophy, but then I started thinking about what we, as developers are guaranteeing to the tests, what the "contract" is for them... what is The Promise. I realized that The Promise is not strictly that we will start them with a clean database, but that we will start them from a known starting point. Based on that, deleting-then-loading all data is not really so different from starting from a clean database, because all tests start from the same known state.

I discovered something interesting about Ruby on Rails fixtures. Fixtures, for those who don't know, are how Rails loads test data for unit and functional testing. YAML files are loaded with sample data, which is loaded into your database when each test executes. If you want to load "users" and "addresses" data, you indicate such by stating "fixtures :users, :addresses" in your tests. Pretty handy!

I would expect this data to be cleared from the database at test teardown time, leaving the DB in a pristine state for the next test but this is not the case. Instead, the target fixture's data is deleted, then reloaded at test setup time, while teardown does not clear this data, leaving schmutz -- left over test data.

When it comes to test data, I have always practiced a "leave no trace" philosophy, so my inclination is to disagree with how the Rails folks implemented this: teardown should not leave schmutz.

And before you say anything, transactional fixtures don't fix this. More on that later.

Let's say we're running a test suite consisting of UserTest and UploadTest, each with a bunch of test methods. UserTest loads "fixtures :users, :addresses" while UploadTest loads"fixtures :files". If UserTest executes first, it's last test method will reload users and address, potentially doing a bunch of stuff with those, and then leave it behind. There will be users and addresses polluting the database, potentially interfering with UploadTest, or not, or sometimes, or never or... who knows? Schmutz! Schmutz!

I have heard some arguments for what I will affectionately term the "schmutzy philosophy":

• If you leave schmutz behind, you can look at the database and see the affects your test had upon the data.
• If schmutz messes up future tests, it is actually exposing an untested area of the system.
• In the real world, functionality does not exist in a vacuum, so it's better to test with schmutz around.

And my favorite ...

• Clearing all the test data is hard. Eventually you load tons of it, and your tests create a bunch more, then they mangle it, and you can't keep track of it all, and so we give up.

None of these arguments are strong enough for me to embrace the schmutzy testing philosophy. Unit testing in isolation has more benefits than any random side effects schmutz might reveal. And as test suites grow and change, schmutz's side affects will be constantly changing, and often muddle the true intent of your unit tests -- I've seen it many times before. Are the features of "save user preferences" truly broken, or is there just a bunch of random junk in the database messing up the results? You won't be able to see the true bugs from the side effect problems.

Ok, now for transactional fixtures. From Clarkware:

Transactional fixtures use database transactions to isolate tests. Rather than deleting and re-inserting fixtures for each test method, transactional fixtures are loaded once at the beginning of the test case. The fixture data in the test database is restored to its original state after each test by doing a transaction rollback.

This still leaves schmutz! It just leaves clean schmutz, which is kind of like clean dirt -- not clean. It seems like transactional fixtures could have solved this problem: perform everything within the transaction: the data inserts, the test's data manipulations, everything, and perform a ROLLBACK in teardown, thus rolling back the inserts as well. Transactional fixtures focus on performance, not on clean tests.

What are your opinions? Does testing with a schmutzy database have merits?

---

3 Comments (from old blog):

At 1/10/2006 12:01 PM, Chris Nolan.ca said…

I have no problem with it. I don't do anything with the test db except run the tests. Each of my tests have the appropriate fixtures loaded so it doesn't matter to me what is left behind.

---

At 8/23/2006 5:47 AM, James Mead said…

I totally agree with you, Joseph. In fact this is one of the many reasons we are avoiding the use of fixtures altogether. You might find Mocha gives you another way to approach unit testing.

---

At 8/31/2006 8:20 AM, Casey Helbling said…

Also be aware of using fixtures, acts_as_enumerated, observers and an enumerated type in an association. You can run into problems when the enumeration fixture data isnt loaded yet but the observer loads the model with the association (trying to use the enum). This is terribly annoying and even loading all fixtures in your test case wont fix it. And to Chris's point, I too load individual fixtures in each test but it is sometimes dificult to know if you add a relationship that depends on a new fixture what other fixtures are required. The arguement could be made that the testing isnt being done at the right granularity but that is different story. Also be aware of problems with rails transactions in general and save points.

http://dev.rubyonrails.org/ticket/5457

Transactions in general in rails do NOT behave as you expect!

Week one as a professional agile engineer is over and my head is hurting. Week 1 at any new job is going to be tiring but I think (hey, because it’s happening to me) that it’s even more so when working in an agile software development environment. Pair programming is an intimate thing regardless of how well people know each other, and the dynamics are particularly funky when you’re the new guy. There is definitely a “probing period” – which agile practices do these folks adhere to vigilantly? Which practices do they feel are less important? How do they align with my agile ideology? Do they like curry, and if so, how hot?

Actually, I wish I was an objective observer, watching my own interactions with people I don’t know, because I’m sure I’d be laughing. Agile people, because they want to work closely with others and recognize that interpersonal interactions are important, peculiar, and even fragile, tend to tread gentle steps when suggesting alternative solutions to problems. This tends not to be as much the case after the familiarity barrier is breached.

All in all it was a great week and I learned a lot about Web application design and development. I’m especially impressed by CSS and the flexibility it affords a developer, but I see a potential for a spaghetti mess if developers aren’t careful about reuse, refactoring, and cleanup.

Oh yeah, and the Google Maps API rocks. There are even some “undocumented features” that helps developers find the best map zoom and center point when working with multiple locations on the map. Good stuff.

I've updated the site's sidebar on the right-hand side of the page with a "Random Blue Top 10 Card" section. Click there and see one of the 74 (so far) "Top 10" cards that I've accumulated since I've been using blue 8x5 index cards to keep myself organized. I'll explain.

For as long as I can remember, I've loved things that organize stuff: tool boxes, personal organizes, PDAs, wallets, backpacks, even those clear plastic compartmentized boxes that fisherman use for tackle. I really like the idea of being organized, but, as it turns out, I really hate actually staying organized. Personal organizers are bulky to carry around and I get tired to opening them, finding the right page, refilling them, etc. PDAs are still clunky, with interfaces that still don't satisfy me. I doubt I'll ever faithfully use a PDA until I can safely stuff it in my back pocket without worry that it will break, or until they are integrated into a flip styled cell phone without losing their usability.

So one day a few years back I was talking to my mother about this problem, and she mentioned that she had become a "list person". She leaves lists for herself all over the place, although she now forgets to take her lists with her, or what the items on the list really mean, and other such age-related fun that I can't wait to experience.

I remembered this conversation one day while wandering around a small mom-and-pop stationary store in San Francisco, and on a whim I bought a pack of blue, lined, 8x5 inch index cards. Over the next few weeks I developed a pretty low-tech to-do list system: I would carry a few cards around, bound with a small binder clip. I'd draw little sections with lists of tasks I'd need to do, and cross them off when I finished them. The most important thing I would do is force myself to write down on the card everything that came up in meetings or conversations that I wanted to follow up upon. Traditionally this has been a bad trait of mine: I forget the little things that I need to do for people, especially when I get bombarded with tasks from every different direction. By writing down everything, even if I've already finished the task, I get the pleasure of crossing it off the list when it's done, which effectively crosses it off of my conscious, too.

Several times people noted that the cards looked like those used by David Letterman for his Top 10 List every night, so naturally I started labeling my cards as my own personal Top 10.

Creating a new Top 10 is a pleasurable ritual that I try to do every Monday. I get a fresh card, write the date on both of the top corners, and write "Top 10" in the middle in some funky text: sometimes it's boxy, sometimes rounded, sometimes normal, but I take my time because it's cathartic for me to dedicate myself, once again, to keeping track of my thoughts.