If you fixed a lot of bugs, that means ... you fixed a lot of bugs

by Volker Weber

You may have noticed quite a few posts on Planet Lotus about Notes/Domino 8.5.1, indicating that beta participants are now officially released from their NDA*. There are conditions, for instance a mandatory disclaimer that this is beta code and may or may not be representative of the final product.

The unwritten rule seems to be that you also are supposed to write nice things about it. That rule was broken by Ben the other day, when he wrote I wish Lotus knew quality assurance better. You have to understand that Ben makes a living off the fact that IBM (and also their competition) can't be bothered to fix their outstanding bugs in rendering rich text, for years on end. Read his post, and the comments.

Charles also wrote over the weekend about My (his) impressions of Lotus Notes and Domino 8.5.1 beta. His post acknowledges that there is progress so that Domino Designer finally becomes usable and that he is no longer suffering from bad performance. This is on the other hand quite a statement about the quality of the released product.

This is the flip side of any progress. If you squished a lot of bugs to improve quality, there were a lot of bugs to begin with. And there are most likely quite a few more. Killing a hundred mosquitos a night does not say anything about the remaining ones. There are of course (yet) unknown bugs, but there is also a list of known bugs, which won't be fixed, because in the world of software a product has to ship even if it isn't finished.

If there is one software product out there which can claim to be of good quality, it's probably TeX. Quoting from Wikipedia:

Knuth has kept a very detailed log of all the bugs he has corrected and changes he has made in the program since 1982; as of 2008, the list contains 427 entries, not including the version modification that should be done after his death as the final change in TeX. Donald Knuth offers monetary awards to people who find and report a bug in TeX. The award per bug started at $2.56 (one "hexadecimal dollar") and doubled every year until it was frozen at its current value of $327.68. Knuth, however, has lost relatively little money as there have been very few bugs claimed. In addition, recipients have been known to frame their check as proof that they found a bug in TeX rather than cashing it.

One of my colleagues at c't owns one of these checks.

ad *) I have participated in quite a few beta programs which would not even allow me to acknowledge that a product was being developed.


8.5.0 was a disappointment to many of us -- particularly when by the time it shipped we were seeing notable improvements in the early code drops on 8.5.1. The sad truth is that it shipped on schedule instead of when it was ready. What happened to Orson Wells in his elderly bloated self standing up and declaring "We will sell no wine, before its time." (admittedly he was pitching crappy wine for a U.S. TV audience -- but the line was good).

Fixing hundreds of bugs is a good thing. No matter how you pitch it or spin it, it's a good thing. More important than how many bugs you fix though, is what the overall impact is for the end user experience. That's the real key with 8.5.1. Using it feels better. What used to be epoxy and pancake makeup joining Notes to Eclipse now feels like a fairly good reconstructive surgery job. There's still a few scars (and face-lift is still long overdue), but it doesn't feel like the whole thing is about to come un___ted. It's been months now since I've seen one of those weird meaningless eclipse error boxes that says nothing useful and gives you no real choices (the user interface equivalent of being urinated on by a drunk in a pub).

What's most important, is that I think I could deploy 8.5.1 to a large user base (provided they had at least 1gb of ram) and many of them would like it -- and the rest at least wouldn't try to track me down and kick me to death.

Progress is progress.

Andrew Pollack, 2009-09-07

"The unwritten rule seems to be that you also are supposed to write nice things about it."

I don't believe that was what the issue was about. It was that there was no disclaimer on the initial blog post.

Also as long as he is posting the bugs in the beta forum so the developers can work on them it should be ok.

Simon O'Doherty, 2009-09-07

Simon, I was somewhat reminded of this comment. It may explain why these things are never being fixed, or -- reading from the comments at Ben's -- are even considered worth fixing before the next quality release. IBM must love Ben very much, to time and again give him the business opportunity to work around these perpetual issues.

Volker Weber, 2009-09-07

If a user finds bugs in the product they can be reported through the PMR system or through LDD forums. You don't have to be in the beta program to do this.

I am not sure if Ben does this at all, or uses a different method. But just posting them on your blog is not the best way to go about getting an issue fixed.

I did try chasing up two issues he reported on his blog but never got anything to work off good enough to open an SPR with development.

Simon O'Doherty, 2009-09-07

I believe the unwritten rule is a reality and I do accept it. The closed beta has been quite "open" this time and for this to work I think it is best if you only speak positive. Otherwise next time the betas will be closed again. The final version will probably have enough issues left for criticism.

Henning Heinz, 2009-09-07

"But just posting them on your blog is not the best way to go about getting an issue fixed."

Actually, I think that's a pretty good way of getting someone to address a bug. Don't know how it is now, but when I worked on N&D, there were many thousands of open bugs, anything that gets a bug to rise to the top helps it get fixed.

Damien Katz, 2009-09-07

Simon, the PMR system is a nice way to report bugs, describe them, make them reproducible to IBM. But it seems to be not the best way to get a fix for the bug - once everything is done to create an SPR, you're lost as a customer. The next time you hear from it is when it's fixed (if you remember the SPR number).
We've had an issue with crashing Notes clients while rendering a corrupted RT fields - the body of mails. Instead of fixing the rendering issue (NotesBuddy e.g. shows the corrupted mail bodies correctly) there has been a discussion for months who corrupted the body field. The issue raised when ND8.0 came out. I'm still working on it.
I know what Ben's talking about.

Steffen Pelz, 2009-09-07

Steffen, Simon is right about that bugs need to be entered into the PMR system, no doubt. Damien is also right that bugs need to be kicked up in the the priority list in order to be fixed. There are always more bugs than developers to fix them. That is the hard part to acknowledge: you finally found a bug that is important to you, and the vendor does concur, but it isn't being fixed. That is where you lose the customer. If it is a big loss, the vendor will kick up the bug in the prio list to have a fix, sometimes a private one.

If you are a small customer, you might as well go look elsewhere.

Volker Weber, 2009-09-07

Posting bugs in my blog may not be optimal from IBM's point of view, but remember that many of the rendering bugs I have pointed out were present in R5, and several were ones I pointed out to IBM during the R5 beta. I might not mind if IBM just did poorly, as it offers me a chance to write products to do things better, but when they leave in bugs that deter people from using the product, or the product area, I object.

If these bugs were new in 8.5.1, I would not focus on them as it is a beta. When these are regression bugs, or bugs that were never really fixed from ND7/ND8/ND8.5, does it still seem best to throw them into the queue with thousands of other issues.

This seems very much like the argument vendors have with vulnerabilities in their software. They would rather the people who discover them reported them quietly, but once it became clear that such reports didn't lead to fixes, the people involved in such work stopped being quiet. I am tired of being quiet.

Ben Langhinrichs, 2009-09-07

Volker, of course we reported that issue via PMR system. We've had CritSit-Rounds every week, we became Premium Support customers in the meanwhile. May be the company is not the biggest customer, but a remarkable one for sure. That didn't change the length of the discussion though...
At least premium support ensures to inform you when bugs you reported are fixed in a new version.

We'll have to work for years to repair IBM's and Lotus' image in the company though.

Steffen Pelz, 2009-09-07

Steffen, it is indeed a remarkable one. :-)

Volker Weber, 2009-09-07

"Steffen, Simon is right about that bugs need to be entered into the PMR system, no doubt. Damien is also right that bugs need to be kicked up in the the priority list in order to be fixed."

When we enter an SPR an APAR is created online which anyone can read. If you have a support contract you can open a PMR and ask for a customer report to be put against the SPR. This is what helps in pushing it up the list.

Sure if you have done that part then please feel free to blog about it. :) It is not uncommon for us to have a number of reports come after someone has posted about the SPR they logged.

But just blogging about it isn't going to get an SPR unless someone is reading the blogs. I and others do read quite a few blogs/tweets/etc. I create SPRs when there is enough information to work off.

@Steffen: 'We've had an issue with crashing Notes clients while "
Yea, this is to do a lot with the level of your support contract and client fixes. :/ It is not something I have power over. But certainly bring it up with your PSM (I'm betting you already have ;).

For Non-Premium Support customers the way to track SPRs is via the APAR system (and if you don't like that, best put an ideajam up ;).

Simon O'Doherty, 2009-09-07

Simon, so what do you suggest should be done with bugs that have been known since R5?

Volker Weber, 2009-09-07

I think Simon answered that. Open PMRs and have them put against the existing SPRs. This is a good filter to ensure that the purported bugs actually exist and aren't a factor of the environment in which they are being reported. It also ensures that the person thinks enough of the issue to ensure that IBM's processes to get it fixed are followed, since some seem to think that bugs are an opportunity to promote themselves.

Ed Brill, 2009-09-07

Oh, a snipe at Ben. Not a good target.

Volker Weber, 2009-09-07

@Ed - Given the level of attention this bug has generated inside and outside of IBM, versus the level of attention of other bugs I have reported (including a silent crash that prevents me from using Notes 8.5.1 CD8 at all in Standard mode most of the time - zero attention at all to that as a Design Partner), I'd have to say I picked the appropriate venue. The problem with PMRs is simple - everybody from top to bottom agrees that a business case needs to be made, and the bigger the business behind the business case, the better. But how do you handle a problem that represents risk? I have watched companies taken down completely by not managing risk, but it doesn't work well on a business case analysis. I posted a scenario which described exactly how a company could make a multimillion dollar mistake due to this bug, but that company isn't going to put their weight behind a PMR - they are going to sue, and the deepest pockets in the room belong to IBM, who has known about this particular bug for quite a long time. Risk is vital, but it doesn't belong in the same category as "features that will make a company add seats".

But all this aside, of course I am promoting my software using this. There are real customers who have real businesses who don't have your vested interest in Notes/Domino, they have an interest in their own business, and my software fixes bugs that IBM won't fix. Do I wish I were in that business? NO! I want to be in the business of adding value (as with Midas), not in the business of fixing known, documented, obvious bugs. I want IBM to fix the damn bugs so that I can be in the business of adding value again. Notes 8.5.1 is a huge, wonderful, awesome release that will please many, many people, but I don't give a damn whether there are 100,000 bugs or 10, if those ten are ones that I know will prevent people from using the software. I know how to distinguish between trivial bugs and serious ones, and it is not always by adding up the number of seats that fixing a single bug will bring. IBM is myopic about that, and it is easy to bring down a Cyclops (Greek myth, not XMen). I don't want to see IBM go that way.

Ben Langhinrichs, 2009-09-07

One last question for Ed or others who think I am simply promoting myself: if all I cared about were selling my software, wouldn't it make more sense for me to wait a couple of months and talk about the bugs when IBM had no clear chance to fix them? It would make a more compelling case for using my software than to have IBM fix the bugs.

Ben Langhinrichs, 2009-09-07

Ben, I don't think you can find a customer who would not want this to be fixed for good.

Volker Weber, 2009-09-07

My point is the assertion that these bugs have existed in released versions of Notes. If that is the case, there should be PMRs and SPRs. Triage isn't only about business case; SPRs earn higher weight if there are more PMRs logged against them. If at some point you have done so for 8.5 or prior, then my gratitude for funneling into the system. And as well I appreciate you reporting them in the 8.5.1 beta cycle. And as I said on my blog, I know we still won't get to every issue in this cycle. Having said all that, attention is good but triage takes into account many factors.

Ed Brill, 2009-09-08

@Ed: Sounds to me that you're missing the point that the way PMRs/SPRs "work" is part of the problem. I don't think it's appropriate to ask business partners to "run around and have customers open PMRs/SPRs" - if this leads to an insufficient number of SPRs I'd say it'd be worth while rethinking the PMR/SPR-system when it comes to business partners. Ben as an example imo speaks for a representative number of customers which could give any single PMR (at least in the area if richtext) more (if not sufficient) weight - I am pretty sure that this could be taken into account in the existing PMR system, as it is already done today for other reasons/customers/or business partners even.

Blaming Ben for not doing something he'd otherwise have to pay for (become a support "customer") or spend his unpaid time on, is imo pretty much pointing into the wrong direction. If the support structure is still the same as it was years ago, where customers pay for a fixed number of incidents / per incident they report, or have to be premium support customer for quite a substantial amount of money, this clearly isn't Bens' or any customers' homework but IBMs'.

In order to ease the problem a bit, I could imagine it to be pretty helpful if the PMR system were tied to something IdeaJam-like, which would be available to at least all customers on active maintenance (not just support customers) and business partners, too. That way, PMRs could be promoted/demoted easily, giving IBM what it needs = continuous feedback, but not at customer or business part expense.

Please accept my apologies if support structures have changed over the years - the last I remember is that one has to pay for giving feedback to IBM (let alone personal relationships with clearly a many great people inside IBM). Having to pay for feedback can imo only lead to the following:

a.) You won't get many SPRs for things people would like to have fixed / improved but live with because they don't want to pay for reporting it
b.) Living with inabilities of a software piles up over time into "I'm tired of the many inabilities" - (yes, in addition to fantastic progress of N/D, IBM is also clearly adressing quite a number of such inabilities these days, but your above comment leaves little room for hope for at least a glimpse of cultural change)
c.) Being tired leads to loss of traction in the overall market
(and d.): people blog about inabilities as it's for free)

If there is a way today, through which "I" can report problems without additional cost to my business, do let me know.

p.s. why wouldn't IBM check for whether PMRs/SPRs are fixed in a new release, but demand opening more PMRs/SPRs from support customers?

Florian Vogler, 2009-09-08

Isn't this completely backwards? Finding a bug and documenting it requires work. Why would you pay for that instead of being paid?

Volker Weber, 2009-09-08

My experience with some bugs I've reported during the Public Beta of Notes/Domino 8.5:
I've reported some of them in PB2 and found them in the release version.
I've found other bugs in the release version of 8.5 and reported them in the Design Partner Forum after 8.5 went gold.
In most cases IBMs answer was "open a PMR" or I never got an answer in the DPP.
Because I have no support contract, I am/was not able to open such PMR and because my customers were not using the appropriate version I found the bug in, they were not interested to open a PMR or have no time/resources to take care about it.

Well, found some of these bugs still in 8.5.1 ...

I agree with Florian that there should be another process to report bugs for BPs and DPPs.

Christian Henseler, 2009-09-08

If reporting a bug “the proper way” is more complex / time-consuming / expensive / annoying than posting a ’blog entry, then the process has failed. Thus it’s no surprise that individuals post these reports “in the wild.”

Ben Poole, 2009-09-08

Sadly, most bugs are known bugs -- and there is no longer enough of a negative stigma for software companies around bugs. If I have a bug in a bit of code I've written, I am embarrassed by it. Most developers are. It's only when aggregated into huge development teams and lead by managers who's personal pride isn't tied to the outcome that bugs are allowed to fester as part of the normal routine.

IBM, like most other big software companies, needs to decrease its comfort level with "acceptable bugs".

Andrew Pollack, 2009-09-08

Reporting a bug is more complex, time-consuming, expensive and annoying. It is a PITA. (I'm right in the process -- reporting an issue with the C API's REGNewUser() in combination with ID Vault; I wish I hadn't started...)

Jan-Piet Mens, 2009-09-08

i found a minor bug in the domino 8.5 server installer but in lack of other places to report it i posted on my blog and on the notes.net forum.

its a trial product so i dont exept any support though , but a errors@lotus.ibm.com would be a start at least we know its ending up in the right department

Flemming Riis, 2009-09-08

I have refused to use any release of R8 to date because of the level of bugs and the overall lack of quality. 8.5.1 was billed as a bug-killing release, so I tried it out to see for myself. I like what I see. A lot of the more egregious bugs are gone. Some of the new features actually work. That's not to say there aren't plenty of bugs still sitting there, or lots of half-implemented features that Lotus support has historically claimed are "working as designed" or "no problem found".

The PMR and SPR process is a gigantic black hole. Stuff goes in and you never know whether (or if) anything is coming out. I don't think customers should have to pay an extra fee for the privilege of doing IBM's QA work for them.

Charles Robinson, 2009-09-08

"If reporting a bug “the proper way” is more complex / time-consuming / expensive / annoying than posting a ’blog entry, then the process has failed. Thus it’s no surprise that individuals post these reports “in the wild.”

Reporting a bug the proper way via a process you don't control will ALWAYS be more uncomfortable (ie difficult?) than opening a text editor and free-forming your report via a blog entry when you're in the mood. Not knocking anyone, but that's just how it is. Reporting and tracking PMRs could be done better, but trying to make it as friendly as a blog posting is unrealistic.

Mike McPoyle, 2009-09-08

I get that blogging is easier than reporting a PMR. Does that make it the right approach?

Do you all suggest that your users who have issues with the IT that you implement create blog entries rather than call the help desk or open incident reports? Easier is not always better for the collective good. Part of the reason you pay for support is to ensure that the troubleshooting and triage process funnels up into something meaningful and scales.

Ed Brill, 2009-09-08

Mike, I’m saying that’s how things lie, not necessarily how they should be.

As a coder myself, I understand the need to extract a baseline of information as part of a bug report in order to make it reproducible and so forth. But the PMR / SPR processes are pretty grim as reporting systems go, it has to be said.

Ben Poole, 2009-09-08

@Ed - I wouldn't suggest that they blog it. I'd suggest that they fill out a simple website form which gives them a place to add relevant details. I know this is far less controlled than IBM would like, and it has its own issues, but the process is simply slanted way too far in the direction of IBM's convenience. The current system is confusing, difficult and requires paying to report the bug (as far as I know). I am already spending a lot of valuable time diagnosing and tracking down the bug. Why should all the effort fall on me? Why shouldn't IBM give business partners an easy way to enter bugs, and do some of the QA work themselves?

Ben Langhinrichs, 2009-09-08

I'd like to see something similar to IdeaJam, where bugs are visible, ranked, and I have quick access to my PMRs and where they sit.

I always liked the old system of releases, where you could see what stage a release was at, and when it was scheduled to ship (I can't quite recall, but I think there were 5 stages or something). That example is simplistic, but open and useful. I'd like to see something similar regarding PMRs.

Ideally, I could see a PMR, see how many related reports were logged against it, the external severity that the caller assigned to each of the reports, and the internal severity assigned to the bug by IBM. Additionally, there would be a targeted release for the fix (if available).

Registered IBM users could browse PMRs and mod up/down the significance of issues, similar to /. or IdeaJam. This would be a huge help. This month I went through a PMR with IBM regarding the design task, only to find a link on a blog post and an existing PMR...my technician didn't even suggest the related PMR.

The whole concept is that openness promotes better code, puts increased visibility on bugs that customers want fixed, and turns around to become a data mining tool to help mitigate some PMR calls before they get to the point where a customer picks up the phone.

Mike McPoyle, 2009-09-08

I think reported bugs which end up being valid should earn free support call incidents as payment.

If I find a bug, reporting through the onerous channels, and it is an actual bug -- then I should get compensated with a few supported incident passes.

Of course, I don't use support at IBM (ever) but generally speaking this wouldn't be be a bad way to go.

Better -- support can then internally bill QA for each of them.

Andrew Pollack, 2009-09-08

Triage often times does not work. It's good in theory but the process to often lacks common sense.

Felix Binsack, 2009-09-08

@Ed - "Part of the reason you pay for support is to ensure that the troubleshooting and triage process funnels up into something meaningful and scales."

I agree that's how it should be. But we're telling you that from the customer side of the equation it doesn't work like that. As a customer paying for support I have opened SPR's. Now what? There is no commitment expressed or implied to actually fixing the problems, and I have zero visibility into the process and no way to affect it. So as far as I'm concerned I might as well not even bother because it goes into a black hole.

It's a classic Catch 22. You can't fix what you don't know about, and we can't be bothered to tell you what's broken because we have no expectation of it being fixed.

Make opening a PMR as easy as blogging and close the feedback loop and I think you would find people doing it more often. A large percentage of problems are never reported through official channels because it is simply too difficult and there is no measurable return on that investment.

Charles Robinson, 2009-09-09

The best idea in that discussion is to open the SPR list to (not only paying) customers to put weight in it - otherwise you won't be able to see the real need for a bugfix. If someone plans to use a functionality and knows that it's buggy, he won't start at all. PMRs can only be successful if you have a reproducible scenario with an application you already have. So these apps that are not made put no weight to SPRs that won't be fixed as there is no weight... nice circle.
On the other hand: @Ed, it's pretty obvious that RT rendering while showing mails should cause the client to crash. Such an error doesn't have to be weighted, it has to be fixed. It's really that simple.
What bothers me is that - spoken with the idea of paying customers to find bugs - it is a business case: more bugs means more customers who are tired of it and spend money to be able to report them. To get them fixed. This model simply won't work.

Steffen Pelz, 2009-09-10

It's obvious that RT errors should NOT crash the client I meant. Of course. ;)

Steffen Pelz, 2009-09-10

Old vowe.net archive pages

I explain difficult concepts in simple ways. For free, and for money. Clue procurement and bullshit detection.


Paypal vowe