// Internet Duct Tape

How I Learned to Shoot the Engineers and Ship the Product (by guest blogger Andy Lawrence)

Posted in Programming and Software Development, Technology by Guest Blogger on April 20, 2007

This post is by a guest blogger.

Andy Lawrence aka andyxl is the Head of the School of Physics and a Professor of Astronomy at the University of Edinburgh, located at the Royal Observatory Edinburgh (ROE) on Blackford Hill. He work on quasars, observational cosmology, big, big sky surveys, and the Virtual Observatory.

I am a scientist. I come across a lot of engineers. They are alien life forms, but I have had a symbiotic relationship with them. I found myself getting confused trying to understand the differences between us, until I realised that behaviour depends on environment. You can see how we are different in three ways:

  • general differences in behaviour,
  • working on hardware projects,
  • and working on software projects

The latter is where it gets weird.

General Behaviour

Left to themselves, scientists are obsessed with truth, and engineers worry only about what works. Scientists are impractical purists. Stuff you build — a lab lash up, a sprawling C program — is ragged and messy. It doesn’t matter because you are going to change it soon. You don’t stop until you figure out whats really going on. Engineers are pragmatic. It doesn’t matter whether the theory is actually correct in any deep sense. They worry and puzzle until they get something that works. Then they stop, polish, and admire.

I once listened to an engineer talking about grain silos. He’d done a simulation showing that when you set it up so and so, and jiggle, the big and little bits segregate. My brow furrowed, “Doesn’t that seem to violate the second law of thermodynamics?”

He stared back blankly.

“So?”, he said.

[Answers on a ten pound note please]

Hardware Projects

When it comes to hardware projects the scientist becomes a customer. The purist/pragmatic hats get swapped from the scientist to the engineer.

I am an astronomer. Unlike my condensed matter chums, we don’t potter in the lab. Our work is done with big instruments on even bigger telescopes, that we share as national or international facilities. These things are highly engineered machines and they have to work right. The scientists come up with the concepts – e.g. “what we need for the next step in big cosmological redshift surveys is a multi-fibre spectrograph, only it has to work in the infrared.” Then they work with a team of engineers to design it; then it actually gets built by engineers, technicians, industrial contracts, etc. Normally there is a “Project Scientist” inside the team who makes sure the engineering group delivers what the scientists wanted. So essentially s/he is a kind of commissioning customer.

VISTA Infrared camera under construction in the lab

So does this mean the Project Scientist has to stop the engineers from cutting corners? It’s normally the opposite. The scientist is getting impatient; he wants his new toy. It may be the Scientist’s toy, but its the Engineer’s baby.

“Ahh”, say the engineers, “we need to do more tests.”

“Oh, and wait a minute, if we open up the cryostat and use a different type of wire the read noise should go down.”

“Well,” they say, “it could pass the CDR we suppose; but there is a 5% chance it will fail at altitude. We should really take it apart and re-align. We should change the baffling while we are at it…”

There comes this point in every Project Scientist’s career where you learn the same lesson that managers learn in industrial teams. Eventually you have to shoot the engineers and ship the product. Yell “enough!” and get that f***er up the mountain.

Software Projects

When getting a piece of kit made, the scientist is no longer a purist. This isn’t her work, it is just something she needs so she can do her work. For the engineers, when building that piece of kit, this is their work, and they are very proud. They are still basically practical folk, but the lure of the technically sweet, and their pride in their work, overcomes their judgement. Now when we come to software projects, the engineers seem to shoot straight past the purist stage and land in a misty realm of philosophy.

I am not talking about normal software development – the code that keeps everyday systems working. Here the engineers are pretty hard headed and practical. I am talking about innovative software projects – difficult datamining tools, ambitious grid infrastructures — roughly the software equivalent of building some new type of spectrograph. Talented software engineers love this stuff. It’s even more fun than transaction software for banks. Already we are in the “oooh, I could make this even better” stage. But it gets worse.

“Well,” they say, “there is whole new technology emerging.” (OWL, REST, Web 3.0, take your pick)

“We should change the whole architecture and re-engineer. In fact,” they say, their eyes glistening, “when you think about it, it makes you realise there’s a whole different approach, I mean for users and for us. It’s difficult to explain.”

“but its kinda, well. Hmm. I need to think about this.”

Think??!! You’re not supposed to think! You’re supposed to implement! You’re an engineer!

Cool engineer John Taylor undertaking research in Moscow

But the scientist is hooked as well. When he wants a piece of hardware, he states what he wants — it needs to do A and B; it needs to fit next to that other gizmo; it mustn’t flex more than X when its tipped. Then he crosses his arms and sticks to his guns.

When he wants a new software tool, he is a bit woolier, but he knows all about this iterative development stuff, and expects he will sharpen his requirements as he sees the prototypes emerge. But he sees the first prototype he says, “oh, now hang on that gives me an idea. Nah, we got this all wrong, what we want is more like blah. In fact, thats really strange, you know, I read this blog the other day, and it made a very interesting point…”

This is why most software projects fail. Not because the software industry is incompetent or immature in its methodologies. Not because software is just too intrinsically hard. Nope.

They fail because this stuff is just too damned interesting with too many ways to do things and too many things to do.

Andy blogs regularly at The e-Astronomer. Here are a few other examples of his work:

29 Responses

Subscribe to comments with RSS.

  1. engtech said, on April 20, 2007 at 1:53 am

    I really enjoyed that. Having worked both in hardware development and software development (or rather the software/apps side of hardware) I see a lot of myself in there.

    And it’s true, software programming is very seductive because you don’t have the same real world constraints as hardware. Anything is possible, so you usually end up doing more than is necessary.

    Implementing a new technology without fully knowing its pitfalls is a classic example. I got caught bad by that with the whole XML as a universal data format hype a few years ago.

  2. Mr Angry said, on April 20, 2007 at 4:49 pm

    Excellent conclusion. Plus I really loved the exchange:

    “Doesn’t that seem to violate the second law of thermodynamics?”

    “So?”

    That sums up the differences perfectly.

  3. Some Engineer said, on April 20, 2007 at 6:04 pm

    I admit that opposing a smart scientist to a stupid engineer can be funny. But please realize that there are smart engineers and, dare I say, stupid scientists as well. I’m pretty sure that all non-dum engineers know the laws of thermodynamics pretty well. After all they are essentially very simple.

  4. Ulluru said, on April 20, 2007 at 6:22 pm

    amen brother!

  5. engtech said, on April 20, 2007 at 7:32 pm

    @Some Engineer: yup, this is taking things to the extreme to prove a point, but I think it’s a valid generalization.

    I also see the divide between engineering hardware and programming software. With so much programming you don’t have to worry about constraints like memory or speed unless it stands out as a problem, whereas with hardware you’re always thinking about the size of the circuit, power consumption, critical timing paths as well as just getting things to work.

  6. andyxl said, on April 20, 2007 at 11:47 pm

    To Some Engineer : err … chip on shoulder somewhere ? Nowhere did I say scientists are smart and engineers are dumb. Just differently smart and differently dumb. I can tell you for sure that I have seen plenty of projects with top-rank engineers and second rate scientists, where the engineers need a lot of patience..

  7. Alec Resnick said, on April 20, 2007 at 11:49 pm

    Could someone explain why it violates the second law? It seems pretty intuitive: if you had a cylinder, put some big rocks in, put some little rocks in, and then shook it side to side, the small rocks would fall between the big rocks and segregate.

    Just because something becomes more ordered “naturally” doesn’t mean that the second law is violated. It just means there’s an energy sink somewhere, right? So for instance, if I take oil and water, shake them up, and then let them sit, they separate. Is this a reasonable analogy? Here, the molecular forces (polar/nonpolar) exert force to separate them. In the silo, gravity exerts a force in a similar capacity.

    That was my guess–anyone care to clarify? Thanks!

  8. engtech said, on April 21, 2007 at 12:39 am

    @Alec: I believe Andy is only accepting answers written on the back of ten pound notes.

  9. Chris said, on April 21, 2007 at 2:26 am

    Re: jiggling segregating matter of different sizes “violating the second law of thermodynamics” – no, it doesn’t. I’m assuming that you intended to imply that you were joking about the violation and the joke in the statement is the engineer not getting your “joke”. If you were serious about the (erroneous) violation then the alas joke’s on you.

    Vibration sorting of particulate matter of different sizes is a fairly common technique – larger matter moves to the top as smaller matter fills in the spaces below. This can easily be proven with a simple bucket of gravel and some vigorous shaking (a good experiment for children, who tend to assume that heavy things will head to the bottom). The second law is upheld through the expenditure of energy used to effect the sort.

    That all said, I’ll assume we’re laughing at the engineer for his uninformed “so?” at the patently silly question you fired off to him, yeah?

  10. andyxl said, on April 21, 2007 at 3:49 am

    Well.. I am impressed with engtech readers, as yes indeedy there is no violation of the second law because its not a closed system, and energy is being put in. To get the point of the story, its irrelevant whether its correct that the second law was being violated or not. The point is that the physicist thinks this matters, and the engineer thinks it doesn’t.

    In the true event this refers to, the egg was on my face, because I didn’t mean it as a joke, and moments later was smiting my forehead and feeling foolish, because OF COURSE the second law isn’t being violated ! But the engineer’s stance was the same before and after. All that matters is that it works, duh.

    So, no, it’s not a joke about how engineers are stupid because they don’t understand thermodynamics (they do); and its not a joke about how engineers are stupid because they don’t get physicist’s jokes; it is possibly a joke about how dumb physicists are because they don’t live in the real world; but mostly its not a joke “on” anybody at all. Its a description of two modes of thinking, each with their weaknesses.

    But I think I am seeing which tribe has the inferiority complex, reading invisible insults. Jeez.

  11. John Botscharow said, on April 21, 2007 at 6:36 am

    I want to comment not on the science or technology in your post, but on your semantics – your choice of words in the title. I think, given the events this week at Virginia Tech, a school that trains ENGINEERS, and by a professor no less, is bordering on tasteless. Even in Edinburgh, I assume you are aware of what happened. Talking about shooting enginees so soon after so many prospective engineers were senselessy massacred strikes me as insensitive at best. Please show some of that intelligence that I know you have, and think before you leap next time.

  12. andyxl said, on April 21, 2007 at 1:34 pm

    To John Botscharrow : well, the post was written and passed to engtech before the massacre, and loaded up several days after the massacre. As you are an educated man, I am sure you are not suggesting that the occurrence of these words in the title of an obscure blogpost will have caused any mental distress to someone. But you might have meant that we should all be more responsible day by day in our tendency to casually use aggressive language, which could incrementally encourage the culture of violence which is all-pervasive in our culture. This would be a good point.

    Of course we should avoid straining at gnats when there is an elephant in the room, and I hope you will join one of the campaigns to ban handguns in the USA, or at least reform gun laws in an intelligent way – for example at http://www.csgv.org/index.cfm

  13. [...] Posted by andyxl under science , Life , blogging  Yesterday I had the honour of providing a guest post on the popular //engtech site, which I have had on my own feedlist for a while. The guest post has [...]

  14. engtech said, on April 21, 2007 at 10:11 pm

    @andyxl: complete agreement. This article had been sitting in my inbox for days before the shootings.

    It only hit me sometime after the post had been live for 14 hours that there that the phrase “shoot the engineers” could be misinterpreted to be somehow involved with the Georgia Tech incident.

    If anything I’d blame the entire media sideshow around the tragedy for making us think that everything is somehow related to that isolated incident. Glorifying and sensationalizing the actions of a sick individual will only create more sick individuals who want the attention.

  15. rbonini said, on April 22, 2007 at 1:29 am

    I agree completely. Especially when it comes to porting to a new tech (.net 3.0 etc). I find my self itching to that for ALL the projects I’m working on.

    And I always, as a rule, end up doing more than I’m suppost to.

    Thats why I stich to software: the sky really is the limit

  16. Top Posts « WordPress.com said, on April 22, 2007 at 3:58 am

    [...] How I Learned to Shoot the Engineers and Ship the Product (by guest blogger Andy Lawrence) This post is by a guest blogger. Andy Lawrence aka andyxl is the Head of the School of Physics and a Professor of […] [...]

  17. engtech said, on April 22, 2007 at 7:57 am

    Interesting, Scott Adams of Dilbert is getting hit with same “publishing in advance” problem as the title of this post.

  18. Stephen said, on April 23, 2007 at 8:02 pm

    Maybe, because i don’t immerse myself in the news, but i find absolutely nothing offensive here. What i do find offensive, is that for years and years the US flag has been flying at half mast. That means it can’t be lowered. We didn’t fly at half mast for Ford – it was already there. That’s just disrespectful. I’m glad Ford was Republican. I have a child young enough that he will be shocked when it starts being flow full mast. I don’t expect that until, what, 2009?

    I disagree that most software projects fail because they are interesting. It makes for a good story. Most software projects die for one of three main reasons. First, the customer may change their mind on what they want, or add requirements endlessly. This can be fixed by frequent deployment. Second, some large projects end up with some hard to find flaw, and deployment doesn’t happen because it can’t make it through testing. This can usually be fixed by frequent deployment. Often, a software project fails simply because the contractor underbid by a factor of two or three. They wouldn’t get the contract if they didn’t underbid. This can be fixed by having the customer estimate the project. You do that by having a programmer do the estimate who won’t be on the development team, and then (and this is key), listen to this programmer. Once in a long while a big software project will fail because it’s really hard. In 30 years, i’ve experienced this just once.

    When i, or most of my software peers, see that, “oh, the customer probably wants this other thing”, we don’t just launch into it. We view it as a change of requirements, and discuss the implications with the customer. Sometimes it gets implemented. Sometimes it goes into an enhancements phase. Sometimes it’s dropped. Only if the change is so small that the schedule is not impacted does it go in without comment. And this minor additional discipline does not make the work any less fun. There’s an infinite amount of work to do – it doesn’t all have to be done now.

    All this said, given an arbitrary deadline, there is a time spent vs. quality of implementation tradeoff. The customer is generally blissfully unaware of this tradeoff. Well, it’s hard to measure, isn’t it?

  19. David Hutchinson said, on April 23, 2007 at 8:44 pm

    Everything you say is true, except that’s not why software projects fail.

    They fail because its best practice… Overall.

    Individually projects may be screw-ups or may not, but that’s not the point, approx 2 trillion dollars will be spent on IT this year, I *think* 40% of that is software development (the 40% is from memory so not so sure), statistically that is a lot of projects.

    Currently 15% of projects fail (see chaos report), which is better than it used to be at 25%.

    However it costs, say 10 times more to completely work out everything you need to know about a project to guarantee success, or abort before starting.

    This is what the good folks over on the NASA space shuttle project do, because of the lives and money at stake. All safety critical projects are run this way.

    If this is the result you want you get 10% of the working projects per year (and they take many times longer), instead of 85% of the projects.

    Bizarely and unintuitively what we do now is best practice.

    (I have an article I started about this some weeks ago, unfortunatly real life stepped in, so I haven’t finished it.)

  20. [...] Lawrence wrote on this blog : This is why most software projects fail. Not because the software industry is incompetent or [...]

  21. Ste said, on April 23, 2007 at 11:03 pm

    I have nothing particularly clever to say, good post anyway

  22. James said, on April 25, 2007 at 4:18 am

    Man, there are some touchy people. As a former scientist, a current software engineer, and a Virginia Tech alumnus I have to say: that was pretty funny! But to throw in my $.02 I have to disagree with the very last line: I’ve never seen a project fail because the engineers find it too interesting and there are too many ways to do things. True, engineers never think the project is done because they know where the flaws are, but if they’re that excited it’s a good, successful project. In failed ones no one understands the requirements, it’s not clear what the goal is, no one has the resources to do what they’re suppose to do, and morale is low. If you have to beg the engineers to stop working it’s a successful project. On a failed project most of them quit half way through.

  23. John Botscharow said, on April 28, 2007 at 7:31 am

    I missed the posting date. The post showed up in my tagsurfer the day after the massacre. But, I still feel comments about killing anyone, even in jest, are best avoided. We live in an age of senseless violence, where our children seem to think that killing is a valid solution to their problems. We are both educated people and it is, IMHO, up to us to force society to set higher standards for itself. Too often it seems that the standard is the lowest common denominator. I am a theologian by training and so I look at issues from a somewhat different perspective than perhaps you do. It is my contention that we Christians are missing the point of Jesus’ message. He demanded a higher standard of conduct, but our supposedly Christian society, at least that is what some politicians claim we are, keeps lowering the morial standards and using their distorted Christianity as the rationale.

    Jesus said those who live by the sword shall die by the sword. He did not qualify that statement in any way shape or form. Violence, whether “legal” as in a war or illegal, only begets more violence. We, the hated intellectuals of the world, need to take the higher ground and show our children that there is a better way.

    I know your post was intended to not be taken seriously in terms of resorting to violence. I am just suggesting you think about alternative forms of humor that do not inadvertently condone the senseless violence that is so common in our society today. I am the father of three children, one of whom will be going to college in a few years, What happened at VT has me frightened for my daughter’s safety. It is a sad commentary on our society that even the halls of academia are no longer a sanctuary from the insanity!

  24. Baron VC » links for 2007-05-09 said, on May 10, 2007 at 12:21 am

    [...] How I Learned to Shoot the Engineers and Ship the Product (by guest blogger Andy Lawrence) « //engt… This is really a good piece that anybody that dabbles in programming should read because we end up chasing the stupidest “requirements” sometimes. (tags: programming) [...]

  25. Tom Stoll said, on August 21, 2007 at 11:39 am

    As an engineer who has worked both sides(hardware and software) I would have to agree that I do tend to get “lost” in software. And yes I have given a “so” answer to a client, it works doesn’t it? thats what you wanted right? so you think I’m violating the x law of thermodynamics, I’m reasonable certain that I’m not. Please I would love to see you prove that I am! Id be famous! then maybe I would get a real salary with benefits. I started programming when I was 8, by the age of 12 I was published, I’m now 37 and my last job I realized that the janitor was making more money and had better benefits and better hours. The last thing I need is some client telling me that my work violates some idea he had of how and engineer does his job.
    I recently worked with a longtime friend, I got him a job as my boss(he’s not an engineer he has a science background, but he is a great project manager) For the first few weeks of the project he followed managements directions precisely. Nothing got done.
    One day I said to him “You can’t do it that way”
    He said “but thats the way I was told to do it!”
    “look” I said, “Do it this way and it will work”
    with frustration he replied, “but they told me to do it this way”
    My reply to this, “The trick is to do it the right way, then tell them you did it their way and that everything worked great! Then management will smile pat themselves on the back, give themselves great big paychecks and forget your name.”
    We did it my way and thats exactly the outcome. Scientist worry about the why Engineers about the how.
    Scientist ask ,”Why do we get that outcome?” Engineer asks, “How do we get that outcome?”

  26. engtech said, on August 23, 2007 at 10:47 pm

    @Tom Stoll:

    “The trick is to do it the right way, then tell them you did it their way and that everything worked great! Then management will smile pat themselves on the back, give themselves great big paychecks and forget your name.”

    Sssh! That’s my secret too. The art of happiness at work for me seems to be padding the jobs they need for me to get done, so that I have time to work on the jobs that really need to get done.

  27. Eidolon Night said, on August 31, 2007 at 2:03 pm

    The last software engineer that I shadowed told me, “You’re better of quickly creating a simple program that does exactly what you need than if you were to spend a long time creating a program that does way more than you need it to do.” The lesson is to get version 1.0 out of the way and then add features at leisure and a little at a time. You also have to prioritize what the user wants/needs.

  28. engtech said, on September 05, 2007 at 12:45 am

    @Eidolon Night:

    Joel Spolsky calls that “hallway usability testing” — the idea is that you have to get your app in front of users as soon as possible to find out if your going in the right direction.

    http://www.joelonsoftware.com/articles/UsabilityTestingwithMorae.html
    http://www.joelonsoftware.com/articles/fog0000000043.html

  29. [...] I wrote a couple of posts on this subject some time back – on my own blog, and as a guest on \engtech. Anybody who has had any experience with instrumentation or software projects has enormous respect [...]


Comments are closed.

Follow

Get every new post delivered to your Inbox.

Join 285 other followers

%d bloggers like this: