The cost of failure

[swatts]

Hello My Sweets

This article is a plea! Sadly the people who really would benefit from listening to this plea will probably not read it, but it may add power to you my dear readers.

PLEASE PLEASE PLEASE DEAR CUSTOMER IF YOU SPEND $$$$$$$ ON SOFTWARE, DON'T SPEND 0.000$ ON THE HARDWARE IT WILL LIVE ON!

Most LabVIEW programmers in the western world charge between $50 and $250 an hour. Most software that is written will cost in excess of $10k.

As discussed many times in this blog a well designed program will cost less to maintain than a badly designed one. Taking software maintenance out of the loop, let's reflect on some of the uses of our systems and how saving money on the hardware can be a bad thing?

Production Test

A reasonable percentage of our systems will be part of the production environment, generally these projects are costed on a payback basis, this payback is generally ❤️ years (if you can get the payback <2 years you are almost certain to get the job!). For most industry sectors the expected life time for a test system will be >5 years.

The cost of failure here can be as follows......

• Inability to produce
• Reduced capacity to produce
• Disruption in the product flow
• Poor test quality
• Lateness
• Cost of repair/support

Monitoring

We are looking for accurate measurements here, generally these projects are justified on a pre-existing failures and retrospectively costed on the cost of putting the failure right. They usually sit on the machines for years.

The consequences of failure here could be...

• Machine Damage
• Machine downtime
• Poor decision making

Control

Here we usually need lack of down-time and responsiveness, these projects are often new requirements or replacement systems. The costing therefore is a pretty standard process improvement or cost saving.

In addition to the costs discussed earlier we have......

• Difficulty tuning
• Poor machine capability
• Destruction of the thing being controlled
• Safety

In summary bespoke software is expensive, LabVIEW is usually written to do high value and important things. The cost of failure for the customer is high!

For us the real cost of failure is that the project fails and is not used, this can result in loss of business and reputation. i.e. damned expensive!

SO WHY THE HELL IS SO MUCH OF MY SOFTWARE BEING PUT ON $200 DELL VOSTROS!!!

By way of example here's a case study...

We provided the software to control a vacuum furnace designed by a bespoke machine manufacturer to a large Aerospace company, the total cost of the system was approximately $350k, our software services were probably <10% of the total cost. To save money they used second hand vacuum equipments, scrimped on the sensors, heating elements and lid control. The prospects for the job were that we could modernise vacuum furnace design and sell these on, we could also replace the aerospace companies 4 vacuum furnaces. The system never worked correctly in all the areas where money was "saved", we suffered additional software costs struggling to get the system working. Unsurprisingly none of the prospective work surfaced.

On top of this consider the concept of rare resource, in most companies the guys and girls who write the bespoke software are usually bogged down with far too much work. Isn't it just good business to ensure this resource is used efficiently and believe me trying to get cheap-ass hardware to work is not efficient!!!

Starting the week in feisty mood.

Lots of love

Steve

[crelf]

One of the yuckiest thing I hear when hardware is selected is "don't worry, we'll fix it in software". A cheap machine vision camera lense, USB DAQ devices from a vendor with no reputation, second-hand sensors that "worked fine on the last project we used them on in the late 70s". Not only dosuch assertions drive up the engineering services hours, they disrupt the overall quality of the system, and often mean we don't get to our goal.

[#3]

When I was in sales, I had more than one sales-lead buy a competitors product, because it was so much cheaper. Several months later they would become a customer after they had wasted

ridicules amounts of time trying to get the cheaper hardware to do what it was sold as doing. This is the time in my life where I paid heed to the phrase "You get what you pay for".

[swatts]

One additional point I didn't make because the article was a getting a bit dry for my taste, was about speccing the hardware to the initial requirements. So for example the initial requirements for the project describe 8 digital outputs, so the hardware ordered will have 8 digital outputs right?

NO NO NO!! I get really twitchy if there are no spare slots, no spare channels etc etc. This is because the initial requirements are never complete.

[Ian_Billingsley]

Very well put Steve. On a small to medium project we generally work on doubling the original specs defined number of ditital I/o. It's amazing how many times this works out to be about right to still leave around 20% free for later expansion.

There is nothing more frustrating than running out of I/o, when typically is so cheap to plan for at the front end of the design.

[drjdpowell]

More than once I’ve been given computers that are “hand-me-down” office machines.   Not good enough for running Office anymore; hey, lets run our X-ray machine with it!

[swatts]

The worst example we have experienced is where IT have stipulated that no more PCs will be bought (I'm shaking my head as I write this), we got given an old Compaq server held together by ductape. Utter madness where a decent machine for the job would have cost $700.

[Ian_Billingsley]

I think this stems from managers being faced with a requirement to shave something off a quotation.

Faced with a bill of materials that they don't really understand they see "pc" and think it's good target for saving. The reality is of course completely the opposite. As we know, the best way to save money is to have new, reliable hardware that is fit for purpose.

[swatts]

Ian_Billingsley wrote:

                       

  As we know, the best way to save money is to have new, reliable hardware that is fit for purpose.

                   

Heh! way to sum up my article in 21 words Ian

[Thoric]

So we're all complaining about the same thing. But what's the solution?

How do we convincingly argue against these ill-considered but well-meaning ideas to reduce costs without hurting the customer's feelings or looking like a spoilt child by demanding bigger and better?

[swatts]

It all equates to cost, so at the risk of sounding stroppy to a customer we now explain to them that yes we're happy for them to use xxx cheapoDaq running on a Dell Cheapo but these are the associated and additional support/development costs. The risk argument is also pretty powerful.

When we do a quote now we stipulate the hardware platform and the relevant equipment up front, this gets all the discussions out of the way prior to any money being spent.

This has been a lesson learnt from experience tho'.

[Ian_Billingsley]

I totally agree. It's all about highlighting the risk.

On the pc issue, we have found that not even describing it as a pc really helps keep IT at bay. The result is usually that IT refuse to allow it on the network but sometimes this is the lesser of two evils.

What strategies do you use for getting your own specified pc in the door and persuading IT to allow it access to the network?

[Thoric]

Establishing the hardware choices upfront is quite sensible, yes, and insisting on risk-based cost additions when poorer hardware is substituted seems sensible when you can do that. I expect that requires a degree of trust between you and the customer, normally based on at least one successfully delivered contract.

So now I wonder about new leads. I see a difficulty with new customers with whom one is yet to establish any trust. If they've already planned the hardware and offer to hire you for the software and you immediately start trying to convince them that they've made bad hardware choices, so much so that you'll be adding additional costs to your quote to cover the associated risk; isn't there a good chance they'll just go with someone else, someone who's being less 'difficult'?

How do you mitigate the possibility of losing work whilst balancing the need to educate the customer on the risk/cost issues?

[swatts]

First off we try and talk to IT to get them onside.

I confess I have been known to call computers Instrumentation Interfaces on quotes, the downside of that is that it is quite nice for IT to look after the PCs for you.

Usually our strategy is to talk about cost, so if you want it on a 1980s ZX Spectrum it will cost considerably more than running on a modern $700 PC.

Just to clarify 1 point, I'm not anti-Dell, but I really hate Vostros! A decent Dell costs $200 more than a cheap one, this can be less than an hours work for some companies.

[swatts]

I have always regretted those jobs, the best you can do is to carefully explain the consequences (it's not so much being difficult, it can be done in a diplomatic way). If the predicted consequences come to pass then you need to involve the customer (i.e. don't just battle through). So assuming you're successful but it has just been harder than expected, you need to explain why and detail all the extra hours put in. This has the added benefit of showing that you know what you are talking about. If the project fails your name will be mud anyway, it's difficult to forge any sort of win from this.

Sometimes walking away from unreasonable customers and poor work is very good business. Some of the best business decisions I have made have been whether to sack a customer or not.

[AristosQueue (NI)]

You have no idea how much it pains me when we give our interns the hand-me-down machines for the summer. I watch them fight with just-non-quite enough memory or build times that encourage them to take two lunches during the day, and other problems. And then I look at the cost of a new machine and compare it to the salary we're paying them. Even for those three months of summer, I find it hard to believe that we aren't losing more than we're gaining.

I know it seems wasteful to dump PCs that are four years old and still functioning fully. I don't know where those PCs should go, but I haven't seen anyone that I would happily drop them on. I was going to give my old laptop to my niece going off to college, but after evaluating it, I decided I was doing more harm than good in offering it and got her a new model. This is a computer that is going to have to survive for four years of college, and it will bog down over time, but there's not going to be a good opportunity to stop in the middle of the semester to change over. Or it breaks, leaving a college student without a critical tool.

Over and over, I look at the not-quite-up-to-snuff hardware and I just can't justify it in my personal experience. It doesn't take much to convince me that my experience is fairly universal.

Even the quarter million dollar supercomputers that I worked on in the late 1990s at Hewlett Packard were cheap compared to the engineering salaries of the people using them.

Don't skimp on the hardware. It isn't worth it.

[justACS]

Ian_Billingsley wrote:

                       

The result is usually that IT refuse to allow it on the network but sometimes this is the lesser of two evils.

                   

Keeping it off the general network isn't a lesser evil, it's a goal.  If the IT guy sees your computer on his network, he's going to try to maintain it.  That rarely ends well.

[justACS]

Thoric wrote:

                       

So now I wonder about new leads. I see a difficulty with new customers with whom one is yet to establish any trust. If they've already planned the hardware and offer to hire you for the software and you immediately start trying to convince them that they've made bad hardware choices, so much so that you'll be adding additional costs to your quote to cover the associated risk; isn't there a good chance they'll just go with someone else, someone who's being less 'difficult'?

Yes.  But you have to protect yourself.  I worked with a fellow around the turn of the century who referred to this as "customer self-selection."  There are certain issues where, if the customer would rather walk away than work with you, you're better off.  I got burned a couple of times when I was on my own, and it was always because I compromised when I knew better.

[swatts]

Extremely well put Aristos, part of the issue here is cheap (free) labour, so issues with hardware often come from companies that have spun out of universities or have leadership straight from that environment. Any issues you just throw a student at it and in the end the cost of failure is time. I come from a production environment and the cost of failure is $$$$$$$$ and it accrues really quickly. Upside is when you understand this, financially justifying the cost of hardware becomes far easier. I guess it might be a service to list ways to justify project costs in another article.

[swatts]

Completely agree Mr Smith but my company has a priviliged position in that the person quoting for a job generally has to do it. Sadly sometimes customers are thrust upon you and you have little option. Especially if you are kept out of initial negotiations.

The outcome is often poor and the quality of software suffers too. All round it's a situation that needs managing.

[crossrulz]

I'm glad you guys are mentioning the time of the engineer as part of the expense.  I have had countless "discussions" where somebody wants to use an analog input as a counter (as an example of trying to use the tool you have instead of getting the right tool).  So they saved $100 by using a different device.  But if it costs me 2 hours to get it to work (I'm using the bottom end of Steve's rate range), that is a break even.  And it will likely take me more between 8 and 40, or more, depending on the kludge I have to make.

[swatts]

Don't underestimate your cost, if your company was to hire you out, what is their external rate?.

That's the cost you should be using to estimate the cost to do the work. Because that's the cost they base their accounting on.

[crossrulz]

swatts wrote:

                       

Don't underestimate your cost, if your company was to hire you out, what is their external rate?.

                   

I did that one time with my old company.  That $100 would be gone in 45 minutes of my time.  And if you include the person I'm arguing with (just to get the stupid thing on order), 20 minutes.  Oh, and don't forget the schedule impact as well.  That's a week I could have been doing something else that was pushed out.

[swatts]

That my friend is your true worth then. Hold onto that number and consider what else is added to it by your company. It's an interesting exercise if nothing else..

[swatts]

Oh and everyone should LIKE my article because I'm teetering on the edge of breaking my record (5)

Woot personal best exceeded, thanks for the ego food all!

[crossrulz]

Fully agree.  My first example was just using the lowest numbers you supplied with the $50-$250 charge rate.  But using bill rates is sometimes depressing, especially when it is 3-5X your salary.  But it makes manager back down really fast.

[FabiolaDelaCueva]

Steve, as usual, another great article.

Customers sometimes are being penny smart and pound foolish and I agree that the best way to illustrate that their numbers don't add up is to add the time & cost associated with the porposed equipment.

I had a customer call once requesting my help upgrading an old system. I had already helped them move from LabVIEW 4 (?) on Windows 95 to LabVIEW 2009 and Windows XP several years ago. Now it was time to move to a Windows 7 machine and their old DAQ board did not fit any of the new computers. I found a new DAQ board that would fit just right in. The only change needed in their system was to rename the device to "Dev 1" in MAX... They called me later with issues, I couldn't understand why their board was listed as a VISA device. After a several minutes of scratching my head, the customer revealed that his purchasing guy thought my suggestion was too expensive and bought a PCI to USB converter (no brand name, found somewher in the internet)!!!!!  and they wanted me to make it work. I told the guy, even if I could, this would mean re-writing DAQmx. The time spent on this call would have already paid for the board and the couple of minutes that it takes to call it "Dev 1"... Nonsense!

Like others have already said, sometimes it is better to walk away from a situation like this than sticking around. I think it is perfectly fine to ask a customer to find someone else who will be a better fit for them.

[swatts]

I've pretty much had the exact same experience and I think it is very common, I think it has to do with some weird accounting thing where the hardware is immediately classed as an asset, but the software comes out of overhead. I've always found it strange that manufacturing companies are pretty much run by accounts (or nowadays IT). Something is definitely pressuring companies into these counter-intuitive practices.

[FTI_Newton]

This lack of vision is common, not just to the type of hardware needed for software, but to all engineering endeavours I've been involved with. I test a lot more mechanical systems (a lot of rotating machinery) , and the "cost-saving" decisions take a huge bite out of my time not just for programming and interfacing, but limit functionality and increase maintenance.

For example, a choice made to save 100$ on a particular coupling pair now costs us 6 hours per test unit changeover. That is not just in cost of the techs time to swap it over (or a senior engineer when a tech isnt' available), but the delay in test schedule. It adds up to a LOT.

[swatts]

You're right about the commonality of this, in fact it's a nice surprise when it isn't the case!

The old adage in the 1980's and 90s was you won't get sacked for specifying HP. I think much the same about NI these days for hardware (hence why I invest so much of my time and effort doing this sort of thing).

A case study I was given yesterday....

A large research organisation in the UK specified a PXI rack, DAQ card and rack mounted computer from another vendor from NI (lets call them BadStink Inc). The system was put together by the vendor and taken to the customer, where it transpired that the system drivers don't work reliably with Windows 8.1 (that was loaded by the vendor!). 2 Months down the line the project reached a crisis point and kit was borrowed from NI, 2 weeks later all software written, jobs all done!

The price difference was about $1500 for the 2 systems. I'd be interested in the cost of the 2 months struggle to get it going.

I know that sounds a bit like a advert for NI, but it's fact, I've experienced it several times myself.

[crelf]

swatts wrote:

                       

The old adage in the 1980's and 90s was you won't get sacked for specifying HP. I think much the same about NI these days for hardware...

                   

As an NI Select Alliance Partner, this will probably come as no surprise, we like to use NI gear. And it comes down to something super important: it works. That said, there are certainly holes in the NI product offerings where the requirements of a system force us to stray outside of the NI suite, and we use cards from other vendors (PXI, cPCI, etc), and I've found the improvements to relaibility in those to have skyrocketed in the last 15 years. I'm not saying that all are made equal, but I have much less anxiety using non-NI hardware in NI-based systems where I need to.

[swatts]

We tend to pass on the hardware costs, so for us it is the hassle factor when something doesn't work that eats into our earnings. We tend to spec. NI, Agilent, Adam if given the choice, most times we don't get a choice tho'.

The biggest pain I have at the moment is people speccing equipment that is built to a proprietary standard (NMEA2000 as an example). You then have to pay a stupid amount of money to just do basic communication work. Infuriating!

[AristosQueue (NI)]

swatts wrote:

The old adage in the 1980's and 90s was you won't get sacked for specifying HP. I think much the same about NI these days for hardware

...

I know that sounds a bit like a advert for NI, but it's fact, I've experienced it several times myself.                   

An advert? Really? I read it more as a guilt trip to get me to go fix that CAR I was thinking of deferring this morning. 🙂

[swatts]

I'm getting a little worried about myself, my Rome presentation is tending towards a love letter to LabVIEW. Think I need to lower my medication!

Get back to work Mr M!