Field Annotations Unexpectedly Rock

I’d like to introduce you to an unexpected effect Saff and I ran into the other night while working on JUnit: annotations on fields are much more powerful than annotations on classes or methods.

Suites

JUnit 4 introduced the use of annotations to tag methods that were intended to be executed as tests. We extended this idea to annotating classes to tell JUnit to run them in a non-standard manner. For example, the old JUnit test suite functionality now looks like this:

@RunWith(Suite.class)
@SuiteClasses({
  AssumptionTest.class,
  ...
})
public class AllTests {}

I always found this syntax ugly and awkward, but I didn’t have a better alternative that was declarative, easy to write, and easy to read.

Annotated Fields: Rules

A recent innovation in JUnit is the introduction of rules. We noticed that may people who implemented custom test runners (like the Suite runner above) we over-served by having to subclass one of JUnit’s existing runners. The extraneous detail made learning how to write runners harder than it needed to be and most requirements from custom runner could be met with just a few entry points. All customer runner writers need is a way to insert their code into the flow of control that results in a test running.

We created rules, annotated fields that are called before and after a test is run. Here is a rule that writes to the console when a test is about to run:

@Rule public MethodRule logger= new TestWatchman() {
  public void starting(FrameworkMethod method) {
    System.out.println(method.getName());
  };
};

We were surprised by how flexible rules turned out to be. People began writing new rules for all sorts of unanticipated purposes. We had stumbled on a really useful API.

Categories: Take 1

For many years people have been asking JUnit to support categories, being able to choose subsets of tests according to a user-generated classification scheme. I thought (and still do) that this is properly the job of the IDE, but since we had implemented everything else higher priority we gave categories a try.

@Category(Fast.class)
public class B {
  @Test
  public void c() {}
}
@RunWith(Categories.class)
@IncludeCategory(Fast.class)
@SuiteClasses({ A.class, B.class, C.class })
public class FastSuite {}

By tagging the class B as belonging to the Fast category, the FastSuite, tagged to include Fast tests, will run the tests in B but not A or C.

The API is ugly but functional. As we got into the twisty little corners of the implementation, though, I had a hard time following what was going on (the true measure of a design is whether the main flow remains clear after all the corner cases have been accounted for). I was ready to quit programming for the life of an itinerant goat cheese maker when I thought, “Maybe the problem isn’t the implementation, it’s the API!”

Categories: Take 2

Putting our experience with rules to use, Saff suggested we try annotating fields instead of classes. Categories are an example of the more general concept of a filter:

@RunWith(Suite.class)
public class FastSuite {
   @SuiteClasses
   public Classes classes= new Listed(A.class, B.class, C.class);

   @FilterWith
   Filter filter= CategoryFilter.include(Fast.class);
}

We transformed both class annotations into field annotations, one to tell the suite which classes to run and one to tell the suite how to filter out unwanted tests.

Here’s the magic part. As soon as we had written this we could imagine several other ways of filtering tests–by recency of failure, by name, and so on. Implementing these changes requires no modification of the Suite runner. In fact, these changes don’t require any modification of the core of JUnit at all.

Likewise, using an annotated field opened up possibilities for retrieving the classes to run as part of a suite. Johannes Link’s handy ClasspathSuite can be recast as an alternative implementation of Classes above, along with a host of other ways to find the classes to run.

Taking a Step Back

I’m a big fan of declarative expression in programming languages. Annotations are superior to convention (like naming all test methods “test…”) for declaring intent because they can be checked by the compiler. The addition of parameters to annotations makes them even more expressive. However, annotations attached to elements fully determined at compile time (like classes and methods) are fundamentally limited.

The problem is the coupling between intention and implementation. An annotation attached to a class specifies both intention (i.e. “run this class as a test in a certain way”) with an implementation (“this exact class right here”). Conflating intention and implementation limits the flexibility and power of annotations.

Annotations on fields separates intention and implementation. The intention is attached to the field while the implementation is specified by the value of the field, which can vary at runtime. It is this separation, and the flexibility it buys, that is the secret behind rules and this proposed new test suite semantics and syntax.

I’m still getting used to annotations and what they are capable of. I know I’ll be looking for ways to annotate fields in the future instead of classes or methods if I have the option.

The Responsive Design Project so far: January ‘10

In the Beginning

The second program I ever tried to write was a copy of the Star Trek simulator I had seen at the Lawrence Hall of Science. I had gotten through the BASIC language workbooks in a weekend. I had written a Roman numeral printer with no problems. How hard could Star Trek be?

I can remember my feelings sitting down at the teletype: anticipation, excitement, puzzlement, frustration, despair. I had no idea what to type. I knew the statements of the language, no problem. What I didn’t understand was design. I had no idea what elements there could be above the level of statements or how those elements could be related to produce the behavior I was imagining.

That day I just gave up and walked away. Over the next decade I slowly got a sense of what elements there could be in a program and how they could relate to each other. I acquired an aesthetic sense. I could take pleasure in a good design and feel discomfort in the presence of a poor one. Eventually I could take a problem and design software to address it.

But something was still missing. I couldn’t explain what I did when I designed. Little pieces like refactorings I learned, with a lot of collaboration, to articulate, but I couldn’t escape the nagging sense that design had hidden structure, like the plumbing and wires in the walls, structure that was critical to really understanding the process and product of design.

Revealing that hidden world of design is the goal of the Responsive Design Project. I want to understand and articulate the deep structure of software design. While the project has only just begun, I have some practical results I’d like to take you through briefly in this introductory article. And I’ll share with you the result that most disturbs and excites me:

• Our illusion of control over software design is a dangerous conceit best abandoned.

In the Slightly Later Beginning

In 2005 I was invited to sit on a panel celebrating the 25th anniversary of the publication of Ed Yourdon and Larry Constantine’s Structured Design, the book that introduced the terms coupling and cohesion. I’d had a copy in my library since I’d used it as a college textbook, but, since I had used it as a college textbook, I hadn’t really read it. As I studied it in preparation for the panel, I realized that many of the questions I had about software design had been answered two decades before. This motivated me to make my informal study of design more structured and rigorous, and to communicate the results to a new generation of programmers. This was the official genesis of the Responsive Design Project.

Design is good. Design is central to effective software development. Programmers can add features steadily to well-designed software. Programmers can easily test well-designed software. Well-designed software is easy to tune for better performance. Most of the hard problems in programming turn out to be design problems.

But design has a dark side. While there isn’t a single best design for any system, there are many poor designs for that same system. The more experience a programmer has, the more design ideas he knows. The temptation is to put these design ideas in the system now because you just know you’ll need them eventually. Over-designing early leads to delaying feedback from real usage of the system, makes adding features more complicated, and makes adapting the design more difficult. By the same token, under-designing makes adding features more complicated, increases defects, and makes adapting the design more difficult.

The word “responsive” in the name of the project reflects this need for balance. Some design needs to be done in advance of coding, but over the life of the project most design will be done in response to the changing needs of the system and the changing and growing understanding of the developers.

The project is based on three parallel tracks of inquiry:

• Quantitative. Measure real designs and see how they behave “by the numbers.”
• Introspective. Carefully watch my own practice of design and draw lessons from the patterns I find.
• Pedagogical. Teach software design and draw lessons from the students and their subsequent practice.

The Tour

Here is a quick overview of the lessons I have learned so far. Each topic deserves an essay of its own, or even a book chapter (say, you don’t know anyone who publishes technical books, do you?).

Beneficially Relating Elements. Designs are made up of elements that relate to each other in beneficial ways. Looked at actively, designing is creating and deleting elements, creating and deleting relationships, and increasing the benefit of existing relationships. Elements contains elements and so on down (and up).

Design is Fractal. There is no fundamental difference between implementation, design, and architecture. They are all a matter of beneficially relating elements. Learn to bounce between levels of abstraction. Most design takes place at a single level of abstraction but big advances often come as a result of rearranging the levels.

Safe Steps. Resolve the tension between efficiency and safety in favor of safety, then learn to take small, safe steps fast enough that from the outside it looks like you are flying.

Isolate Changes. Before making a change, isolate the area to be changed from the rest of the system so you can change an entire element at a time. For example, before changing a part of a procedure, extract the area to be changed into its own procedure. Make the change, then inline the changed sub-procedure if appropriate.

Embrace Ambiguity. Half done with a change and not sure what to do next? Stop and deploy (you can do this if you are working in safe steps). Complete the change when you know what to do.

Cultivate Confidence. Master your tools. Your feeling of mastery will improve your cognition.

Cultivate Humility. Try tools or techniques you aren’t comfortable with. Being aware of your limitations will improve your effectiveness.

Exploit Symmetries. Divide similar elements into identical parts and different parts.

Trust Succession. Design simply today, knowing you can add complexity needed tomorrow.

Design Is a Team Sport. The social side of design is as important as the technical side. Learn the skills needed to communicate and understand designs.

Play with Words. Recasting a design using a different metaphor can revolutionize your understanding.

Play with Pictures. Explain a design using hand-drawn pictures. Ask others to do the same. Design benefits from your whole mind: symbolic, verbal, visual, kinesthetic, conscious and unconscious.

Clear Strategies. All design changes use one of four strategies:

• Leap. Make the new design and use it immediately.
• Parallel. Make the new design and run both new and old simultaneously.
• Stepping Stone. Create a platform to bring the desired new design within reach.
• Simplification. Only implement part of the design now and add complexity bit-by-bit later.

Inside or Outside. Change the interface or the implementation but not both at the same time.

Toss It. Checkpoint often. If you get confused, learn what you can and revert. Now. Beware the sunk cost fallacy.

Keep It. If the system is working and creating value, keep it and improve it.

Start over. Sometimes the constraints of the problem plus the constraints of the existing system are just too much to handle at one time. I prefer incremental improvement, but sometimes it’s best to just wipe the slate clean and start over.

Both. Faced with design alternatives without a clear winner, do it every way. Just coding each alternative for an hour is more productive than arguing for days about which is better in theory, and a lot more satisfying.

Suit Design to Needs. A design strategy intended to maximize feedback from rapid changes looks very different from a design strategy intended to increase value by reducing cost. Both are appropriate at different phases of a system’s lifecycle.

Use Tension. There will always be tension between designing better and delivering sooner. Learn to fuel your creativity from this tension.

Conclusion

Many metrics in design are power-law distributed. For example, if you graph a histogram of the cyclomatic complexity of the methods in a large system on a log-log graph you get a straight line. In other words, there are many very simple methods and a few very complicated ones.

This same distribution is found in many measurements from nature—the intensity of avalanches, for example. This leads me to the most exciting and disturbing lesson I have learned about design so far:

• Design is not a rational process.

Much as we like to think we are making design decisions, much of the structure of our systems emerges from the nature of the problem and the nature of the tools we use. Our designs shape us as much as we shape our designs. Insisting that we as designers are “in control” is counter-productive, leaving us wasting energy trying to act against nature. The challenge is to embrace responsibility and at the same time take an appropriate role in designing. Learning to achieve this balance is the next goal of the Responsive Design Project.

Why NoSQL Now?

Why have NoSQL database reached the tipping point now? Almost twenty years ago I lived through the attack of the killer object databases. While they had some lovely technical superiorities (and I still make part of my living helping maintain a Gemstone/Smalltalk application), relational databases were able to beat object databases in the market. It wasn’t time.

Now, though, seemingly suddenly, alternative data models are all the rage. While my narcissistic programer self would love to believe this had something to do with technical superiority, experience argues otherwise. Instead, if you want to understand technical change it’s more effective to follow the money.

I was talking to a developer of a cloud-deployed application (hi Patrick, love that BrowserMob.com!) and our design discussions quickly focused on money. SimpleDB is great for high-transaction rate storage, but it’s too dang expensive for reporting. A special-purpose database just for reporting makes sense. It’s worth extra programming to reduce operational costs (the classic capex/opex tradeoff familiar to telecom engineers).

Some Made Up Numbers

Here are some made up numbers (thanks @jkordyback) to illustrate the dynamic (adjust the numbers for your actual situation). Say you have a database from a commercial vendor. It costs you annually:

$50K (license) + $1.5K (electricity) + $1K (capital) = $52.5K

If you need more performance it makes sense to get beefier hardware:

$50K + $2K + $3K = $55K

The performance advantages of an alternative data storage paradigm (column-oriented, document-oriented, key-value, map-reduce) don’t justify the additional cost and complexity.

Eliminate the license and the cost of electricity becomes a huge percentage of the cost of a database. (EC2 is basically a really complicated way of charging for electricity.) Any technical advantage that reduces energy usage turns directly into profit. Your database now costs you:

$0 + $1.5K + $1K = $2.5K

Buying beefier hardware is a giant bump:

$0 + $2K +$3K = $5K

What if you can avoid the hardware upgrade by shifting to an alternative database? Factor in internet-scale applications so you’re multiplying all your costs by 100 or 1000. The engineering required to shift to a different store or to keep multiply stores in sync vanish in comparison to the operational expense savings (1000 servers for illustration):

$0 + $1500K + $1000K = $2500K

Improving performance with hardware:

$0 + $2000K + $3000K = $5000K

Versus switching to a different store:

$0 + $1500K + $3000K + $500K (engineering cost) = $4000K

Implications

There are several things that catch my eye in this picture. One is that when I was going to school we were always taught, “In the olden days of computing, computers were expensive and programmers were cheap. Now it’s the reverse. Therefore…” We are back to the future. At internet scale, programmers are (sometimes) cheap compared to the cost of electricity. That’s a pretty fundamental assumption to change. I’m sure we haven’t fully digested the implications.

Another is that the technical advantages of alternative stores translate directly into economic advantage. If I was a big database vendor, I’d be diversifying away from reliance on big-iron licenses, say by buying a hardware company (oh…) or running a variety of storage models on my cloud (double oh…) Again, I’m sure we haven’t fully digested the implications of this shift.

In spite of the roughness of the numbers above, based on this I feel justified in my gut feel that the row-oriented relational model we’ve lived with for 30 years is about to shatter. Look for opex optimization to become an increasingly important topic for engineers. Look for vendors, both software and services, to deliver further opex improvements. Where it goes from there isn’t clear, but it certainly will be interesting. The time has come.

The “financial” “models” above are just thinking tools to look at trends. I’d love to see some real numbers and trends to validate the qualitative conclusions I’ve already jumped to.

Write Your Own Fan Letter

During an exchange on the lean startups mailing list, one of the participants was having a hard time articulating the value of the vision he had for his proposed site. It was clear that he was passionate about what he wanted to build, that he was completely convinced about its value. The problem was that he couldn’t communicate the value to others.

I’ve been in a similar position many times, having a vision of where things were going next and how cool it could be but not able to communicate it clearly. I know three things about this situation 1) it’s incredibly frustrating for me, 2) it’s at least as frustrating for the people I’m trying to talk to, especially when I won’t shut up, and 3) I’m often wrong.

I’ve tried a variety of tactics to get out of being stuck. Just making the dang thing is one good way to clarify the situation. I wondered, though, if there was a less wasteful way for him to clarify his vision (I’m a geek and I don’t mind failing when I make something because making it is so much fun).

I borrowed an idea I’ve used often, speaking back from the future, and massaged it a bit: write a fan letter to yourself, written in the voice of a deliriously satisfied user. Here was my example:

Visions start out in your head but they don’t matter until they touch someone else. Finding a path by which visions can create joy is one of the many challenges of entrepreneurship. The purpose of writing your own fan letter is to prepare your mind to bridge the gap between vision and effect. When you can imagine how one person’s life is changed by your vision, you are ready to move forward.

Not My Money Any More: Lessons from Poker 4

Here is my first lesson from poker. I take that back. My first lesson from poker is when two strangers ask if they can join the game, lose a little all night, then offer to cut the cards for $100 as the game is breaking up, count the deck before and then after they turn up the ace.

Okay, so this was my second lesson from poker: as soon as it leaves my hand it’s not my money any more. The day after the game I was incredibly frustrated. I’d lost and lost all night long. I kept rebuying and losing, rebuying and losing. All I could think about was when was the next game so I could win my money back.

Over the course of the next week I became obsessed by that thought: I was going to win my money back. At the same time, a part of me was standing off to one side say, “Excuse me, hello, this doesn’t make any sense.” At some point I figured out how to articulate what was wrong with my thinking–it wasn’t my money. It wasn’t my money as soon as someone else won the hand. In fact, it wasn’t my money as soon as I put my bet in. I gave my money to the game, the game gave the money to the winner, and that was that. I couldn’t win my money back because it wasn’t my money.

Peace. Sometimes obsession works like that. You figure out what’s wrong with your thinking and the cycle just stops. This was one of those times. It wasn’t my money. I couldn’t get it back, because it wasn’t mine. The next time I played, I was just playing. I started with $20 and went from there.

Many of you will have recognized the Sunk Cost Fallacy. Money already bet is gone, sunk. It makes no sense to factor it into your decision making. I knew about the Sunk Cost Fallacy, but I wasn’t spotting it in my own thinking.

The same situation comes up all the time in programming. You have a long refactoring session, every step of which is perfectly safe. Then you run the tests. A test breaks. What should you do? What should you do if the error isn’t obvious? What should you do if the error isn’t obvious and you have been refactoring for a minute? An hour? A day? Does the amount of time invested have anything to do with how you proceed?

My experience is that the right thing to do in a situation where I don’t know what I did to break the tests is immediately return to my last known green state and re-run the tests. Then I begin refactoring again and run the dang tests every step. But, and here’s the irrational part, the longer I’ve been refactoring, the more tempted I am to press on. That’s the Sunk Cost Fallacy at work.

The longer I’ve been refactoring, the less well I remember all the steps, the harder it will be to spot the error, the greater the value of just starting over. The harder it is to start over (emotionally), the more valuable it is to start over.

It’s not my time any more. Have I been refactoring for a minute? An hour? A day? Doesn’t matter. It’s not my time any more. I’ve given it to the program. It’s gone. What I should do now should not be influenced by the magnitude of time I already have invested. It’s not my time any more.

As I was writing this I occurred to me that I’ve been guilty of a variation on the Sunk Cost Fallacy with JUnit. If you’ve been reading this blog for any length of time, you’ve heard me whining about not having found a business model to turn the success of JUnit into revenue. The faulty reasoning goes like this: JUnit has created all of this value in the world (billions of dollars by my envelope calculations) and I haven’t received any of it.

So what? If JUnit hadn’t created any value at all so far but was still in the same position, what should I do? Exactly the same as if it had created a million dollars of value or a billion. It’s the Sunk Benefit Fallacy. What’s past is done. It’s not my benefit. All that matters is the current situation. Any energy or thought I expend on the past will only muddle my thinking.

So here’s my resolution. I will think about the current state of developer testing, figure out the trends, and get ahead of the curve. That’s how I will make a living from JUnit. Oh, and I’ll be sure to count the deck, or better yet, not gamble with strangers.

The Pleasure of Pattern: Lessons from Poker 3

Studying poker has helped me crack a long-standing (maybe 20-year-old) puzzle for me: why are so many programmers musicians? I’ve sang and played guitar and various other stringed instruments since I was eight and I started programming at 13, so I’ve lived this dual life for a long time, but I’ve never understood it. I’ve heard explanations like “music is mathematical and programming is mathematical”, but I was never satisfied with them because they didn’t match my experience. I don’t think “mathlich” when I’m programming. Here are the three threads–programming, music, and poker–and how they came together for me.

Programming

One of the transformative experiences of my young life came in high school when Mr. Burishkin (hi Dave!) got mad at our algebra class (this was fairly common) and assigned us all 40 trig identities in the book. Usually we’d get half or a third of the exercises in a given set. Here was a big pile of work I couldn’t get out of easily.

For once I did the right thing and decided I would just plow through the whole stack. The first few were hard. I had to sweat to figure out how solve them. Each one posed unique challenges. At this rate the assignment was going to take me hours. Somewhere around half way through the assignment, though, a door opened in my mind. I’d look at a problem and see it in terms of the problems I’d already solved–I can split this in two, transform part 1 into this and part 2 into that and I’m done. I blew through the rest of the assignment in minutes, eager to solve more.

Not could I solve the problems quickly but I enjoyed the process. Every time I transformed the chaos of the original problem into a pattern I knew how to handle, it was intensely satisfying.

I have the same feeling of satisfaction with many programming-related tasks–when I see a big refactoring in terms of a succession of smaller refactorings, when I see a big task in terms of independently verifiable sub-tasks, when I see a big feature in terms of a succession of features each of which delivers value–at all these times I feel good.

Music

For thirty years I’ve played off and on with my musical partner Curtis Wright. We play American folk music. While I’ve enjoyed playing with many people, Curtis and I share a deeper level of musical communication than I’ve experienced with anyone else.

We don’t plan song endings. We end every song differently every time. Here’s how it works. When we’re really “on”, I will hear him play some interesting phrase somewhere in the middle of a song and something will click in my mind. There’s no explicit communication at that moment though–no nod, no eye contact, nothing. When we get to the end of the song, I will play that phrase as a tag and he will too, in perfect time. It is the most exhilarating feeling because we’re so completely at risk. If we chose different phrases to play, it would sound terrible and we’d look like fools. When we re-invent the ending and pull it off, though, it’s absolutely electric, both to play and to hear.

This is the most extreme example I have of matching patterns in music, but music is full of patterns–what makes a phrase, what makes a chord, what rhythm or rhythms are going on, what style should this piece be played in, backing up an unfamiliar tune by ear, and on and on. I enjoy music because I enjoy those moments of discovering the patterns in a piece. When I see a pattern, I feel satisfied. Woodshedding (practicing physical skills) is necessary to be able to take advantage of patterns when I see them, but I don’t enjoy it the way some people do. Experiencing music, for me, is experiencing patterns.

Poker

When I started studying poker I didn’t have any particular expectations, I was just interested. One of the basic problems in poker is having a more accurate belief about your opponent’s hand than they have about your hand. Their patterns of betting give you clues. At some point you say, “He must have just gotten a second queen,” because how he is playing doesn’t make sense otherwise. When I’ve guessed right, I noticed that I have a moment of the same kind of emotional pattern matching satisfaction that I have when I’m playing music or programming. That’s when I made the connection–this feeling is familiar, it happens when I program and when I play music.

There’s my answer: talent for music and programming occur together because accomplishment in each relies on enjoying seeing patterns. See a pattern, feel good, look for more patterns.

The bigger the stakes and the bigger the pattern, the greater the satisfaction. I won’t play poker for much money because I’m not any good at it. However, I experience more satisfaction when I see big patterns in programs (what some people call “architecture”) or big patterns in music (the so-called “long phrase”).

I suspect that my chaotic early life left me with a brain wired to crave moments of order and feel good when I find them. That and the innate ability to see patterns led me to activities where I got frequent mental rewards. I’ve just identified this connection, so I haven’t done any background studies into the neuro-science of pattern matching. From my experience I would guess that the same kind of dopamine release that accompanies other pleasurable activities happens in my brain when I match a pattern, musical or programmatical. I’d also like to see an fMRI to understand the neural activation patterns at such times and see if programming, music, and poker all really have similar physical neurostructures.

My partner Cynthia Andres is also a pattern matcher, but she matches patterns of human behavior. She has a good sense of what people are thinking or feeling based on cues that are totally invisible to me. I wonder if there are any studies correlating skill at reading people with skill at music. Are psychologists musicians the way programmers are musicians, or are musical and programming pattern matching similar in a way that musical and social pattern matching aren’t?

Another question this brings up for me is the evolutionary psychology of this kind of pattern matching. What is the evolutionary advantage of being able to match abstract patterns? Assuming it increases evolutionary fitness, what is the cost of it that kept it from becoming universal (I suppose my kids could provide a long list in answer to this question)?

Variance: Lessons from Poker 2

One of the great things about styles of software development is that two people can hold absolutely contradictory opinions and they can both be right. Context is critical to what makes sense. (Hence the futility of searching for “best” practices.)

I was thinking about this today when thinking about our style of development for JUnit. I had tweeted that Saff and I were making “aesthetic” improvements. That’s how we work–we will add a few features, then spend a (seemingly) long time “cleaning up”, establishing symmetries, normalizing names, extracting and inlining components. At some point, we see the opportunity to add new features and progress is rapid.

Our style of development on JUnit delivers uneven results when viewed from the outside. We can go months without a new feature. Sometimes when we add features, though, it feels like we’re wearing jet packs.

One of the concepts in poker is variance, the magnitude of the swings suffered simply because of the effects of chance. Different strategies result in different amounts of variance. One strategy might result in small but steady winnings. Another would result in larger winnings but larger swings. The strategy with greater variance would only be appropriate for an emotionally stable person with a large bankroll.

Our JUnit strategy is high variance. High variance is not appropriate for all projects. If you’re making external commitments, you should work in a more predictable  way. Building and maintaining trust is more important than making occasional blazing progress.

We have the equivalent of a large bankroll because we make no external commitments beyond promising to continue improving JUnit. We enjoy improving the internals of JUnit as much as we enjoy releasing new features, so we aren’t bothered when there is a long gap between features.

How should you develop? As always, the answer begins with “it depends”. One of the factors it depends on is how much variance you are willing to support.

Manipulation: A Matter of Intent

Since I was thinking recently about negotiation, I thought I would set down my thoughts on manipulation. This was in important issue for me to get clear because I grew up in such an intensely manipulative family. As an adult I caused a lot of damage being manipulative before figuring out there was a difference between manipulation and assertion. Keeping the distinction clear is important to effective negotiation.

Manipulation can result in a short term gain–you do what I want you to do. The problem with manipulation is the long-term effects. People aren’t stupid, not forever. If I manipulate you, you end up trusting me less. If we are in an important relationship, this lack of trust is costly. That’s the utilitarian argument against manipulation–it’s too expensive. An additional cost of manipulation is the personal cost–make a habit of manipulation and you come to see other people as objects instead of people. Since they stubbornly remain people, the gap between perception and reality is a significant overhead. Or you can rely on morality, that manipulation is simply wrong. This has the advantage that when you are tempted to manipulate, there is no short-/long-term tradeoff to tempt you.

I once asked a well-known figure in the software world, someone I respect and trust, about manipulation. He told me it was a matter of interpretation, much like the difference between and breeze and a draft. In both cases air is moving, the difference is up to the person experiencing the movement.

This definition is completely aresponsible (that is, lacking in responsibility). The problem with the definition is that it takes manipulation completely out of my control, but that doesn’t match my experience. If I talk about a topic, sometimes people take it as helpful and sometimes they take it as manipulation, but their reaction isn’t random. My thinking in discussing the topic influences their reaction, even though it doesn’t control their reaction.

Intent

What am I trying to achieve when I talk to you about, say, TDD? Am I trying to get you to try TDD so I can feel good about myself when you do what I advocate? Am I trying to get paid by your boss because I’m a consultant and I want to be seen as effective? Am I trying to positively influence my craft of software development so in twenty years I can look back with satisfaction on my career? These are all reasons I’ve advocated ideas, and they are all manipulative. What they all have in common is they are all self-centered. They are about me, not you.

Sometimes, though, when I’m talking about a programming topic, I’m just talking about that topic with others who really care. We each learn something. We might act differently as a result of our conversation, we might not, but no one will be offended either way. The conversation is a sharing of experience between people who care. The result is new knowledge, perspective, and deeper relationships.

Manipulation is intending for others to make less than a free and informed decision. If I have cars for sale (to pick another random example), I can sell them in a clear and direct way–help you understand your needs and resources and help you choose the best match between them. I can also sell in a manipulative way–try to get you to buy the car that makes me the most money. The flow of the conversation in each case might be similar. Many of the words and phrases might be identical. It’s the intent that’s difference.

Clues

How can you tell when you’re on the receiving end of manipulation? Intent becomes clear during conflict. Let’s say I’m telling you about TDD. You say, “All those tests would be too expensive to write.” How do I respond? If I’m being manipulative I’ll contradict you (“No they aren’t”), I’ll try to make you feel guilty or incompetent (“if you were a craftsperson you’d always right tests”), I’ll try to get you to ignore yourself (“C’mon, just try it.”) Would a friend do any of these things, someone you trust? No.

The question I have a harder time with is how to know when I’m being manipulative. At first I thought manipulation was just how everyone acted, and I interpreted everyone else’s behavior on that basis even when they weren’t being manipulative. Eventually, with a lot of helpful (i.e. painful) feedback, I started to learn that I could trust other people to make their own decisions and I would still be safe.

I still have the habit of a lifetime. When conflict becomes apparent, I know that I tend towards manipulation so I watch my own reactions. Do I feel angry? Frustrated? Fearful? The early signs of these emotions can be subtle–a twitching muscle, tightness around my mouth, my patterns of eye contact. These emotions tell me that I’m not trusting the other party to make their own decision. Unless I consciously choose otherwise, my arsenal of manipulative techniques will come next.

When I’m aware of myself and I encounter conflict, I remind myself that I can trust others to make their own decisions. I make a point of asking questions to understand their point of view. Where does it come from? What experiences led to it? What motivations lie behind it?

Maybe this sounds like a soft technique. Sometimes it is, but sometimes it isn’t. I can choose to be clear and direct even when in the presence of someone I don’t understand at all. I can try to understand without agreeing with them. Fortunately, I’m seldom in the presence of stark, raving lunacy. More often (maybe I should say “always”) there is a difference of opinion, as in the case of the cost of tests. Both parties have reasons for what they think. Maybe there is still a worthwhile exchange, maybe there isn’t, but I can always choose to help the other person make a free and informed decision.

I’ll come back to this again and again: we are, all of us, surrounded with unidentified, valuable possibilities every day. If I maintain the intent of helping others make free and informed decisions and maintain my own intent to make free and informed decisions, much of the time there is the possibility of a valuable exchange. If this is one of the times when there isn’t, okay. Move on. In any case I will have learned something by really listening and I will have given the other party the opportunity to learn something from me. There really isn’t any way to lose, unless I panic and pick up the puppet strings, strings that aren’t really attached to anything.

Information is Money: Lessons from Poker 1

I’ve been unknowingly following good advice for keeping my brain fresh into middle age: studying topics that challenge my thinking. Podcasts during chores are a great time to study. The difficult concepts I listen to shake me up and keep my brain moving.

My latest obsession (my son’s word) is poker. I started playing with friends a couple of years ago, but I recently began actually studying. I’m still a beginner but I’ve noticed that my business thinking is changing based on what I’ve learned. I’ll try to relate the changes as I experience them.

Here’s the first. I was talking with a new client about how we could get started with our relationship. We were looking for something small to do first and we settled on having me present a webinar on Responsive Design. He asked how much it would cost for a one-hour presentation with followup email for questions. I quoted him a price. His response, “Oh, that’s cheap!”

Here’s where poker thinking helped. Before studying poker I would have started kicking myself–why oh why didn’t I ask for more? Simply put, I didn’t know what the market was for webinars. I couldn’t have known to ask for more, but that wouldn’t have stopped me from self-recrimination.

Rather than be frustrated, though, with my poker hat on, I was satisfied with the transaction. No need to kick myself. I had made a bet by offering my time. In return I got clear information–I could charge much more. The information I received was worth far more than the money I “lost” by not asking for more. I was out two hours (at a “reduced” rate), but I was up the future increase in earnings.

Studying poker has helped me establish the monetary value of information much more clearly than I ever have before. In poker, it’s common to trade money for information. By getting used to such exchanges, I am more relaxed in business.

Oh, and if you’d like a webinar, I’m available. My bid will be more than it would have been a month ago, but we can talk.

On Negotiation

I’m involved in several negotiations, so I thought this was a good moment to remind myself what I believe about negotiation. My fundamental belief is that negotiations are a value-creating process (or at least can be), not just a transaction cost or (worse) a power struggle. Here’s how I try to create value in negotiations.

First a little background. My mother’s side of the family were hustlers. They settled on car sales as the family business. This gave everyone ample opportunity to exercise the skills with negotiation to gain power over other people. In this model, the payoff of negotiations is getting someone else to do something they didn’t want to do.

I spent many hours as a kid sitting in the corner of my grandfather’s office, ostensibly playing but in reality paying attention to him doing deals. This gave me a solid grounding in the mechanics of dealing. Sometimes while negotiating I have a sense of what’s going to happen next that has been valuable to me. That’s the good part of having learned from an expert negotiator. Watching all the short-sighted power games was the bad part.

Negotiation as a power struggle comes at great personal cost. I watched people in my family destroyed by their addiction to power and I experienced the cost of that addiction to the people around them. Even if power negotiation “worked”, it wouldn’t be worth it, but it doesn’t work. In the long haul, you keep running into the same people over and over, even in a profession as large as ours. The potential gains from leaving the last negotiation with everyone satisfied is much greater than the immediate gain of a better deal that leaves regrets in its wake.

When I studied economics I learned that negotiation was a transaction cost. Negotiation was something you did before you exchanged value. Minimizing the cost of negotiation was the way to create the most value.

Once I got into business for myself and started negotiating I discovered a couple of things. First, as a child I had absorbed the lessons of negotiation as power struggle and I had to actively work to change the beliefs underlying those lessons. I had the advantage that I just wasn’t that good a negotiator at first, so the power games just didn’t work for me. Lucky for me I had to find another way. Second is that negotiation isn’t just a cost because nobody knows what they need.

I found this hard to believe. How could it be that business people don’t know what they need? Money now or more money later? Risk or reward? Time or money? What’s the problem to be solved? The initial positions people (including me) staked out at the beginning of a negotiation frequently changed with respect to these fundamental questions. I learned to appreciate the learning that accompanied negotiation. Rather than try to conclude negotiations as quickly as possible, I shifted to trying to learn as much as possible and help my negotiating partner learn as much as possible.

I’ve come to see negotiation as a compassionate act, a gift. By negotiating I am giving the other party a chance to learn about themselves at the same time that I am learning about myself.

Just because I use a word like “compassion” doesn’t imply softness. The toughest negotiators are the ones I learn the most from, if that toughness comes from the right intentions.

Holding to positive intentions is the hardest part of negotiation for me. A part of me would still enjoy “putting one over” on my negotiating partner. When I spot the warning signs of this attitude, I back up and reset my goals.

I have a pathological fear of conflict, so it’s a bit strange that I enjoy negotiation. Sometimes in negotiations I stop communicating. However, at my best I’ve learned to appreciate how much I will learn about myself and how much I’ll be able to help my partner.

Having written this much, I’m struck by how much more there is to write: encouraging creativity, lateral moves, shifting risk, The Only Game, and succession. However, time is short for today so I’ll close by saying that we are all of us surrounded by opportunities. Sometimes we can see them by ourselves, but sometimes we need the help of others. Negotiation is a candle that, when lit together, illuminates the possibilities around us.