New poll: How would you like your beta test?
by Volker Weber
This time I need to calibrate my opinion. Assume you are beta testing your own software. How would you like the feedback from your testers?
- No comments on missing features
- No comments on performance
- All of the above
- Honest to the bone
Here are the results:
Comments
Ich hab zwar letzteres gewählt, aber wenn die Featureentwicklung abgeschlossen ist, dann ist Feedback (speziell zur Beta) über fehlende Features witzlos. Das könnte man dann eher als Feedback zu fehlenden Features des Programm insgesamt sehen, welches wiederum wichtig ist. Wenn es speziell um die Beta geht, können die Entwickler nur mit den Schultern zucken und sagen: "Diesmal nich - schau mer mal bei der nächsten Version!"
Und wenn die Arbeit an der Performance noch gar nicht begonnen hat, dann ist Feedback zur Performance genauso witzlos, weil wirklich jeder weiß, dass das Ding noch saulahm ist und viel zu viel Ressourcen braucht.
Dementsprechend muss man an dich zurückfragen, was in deinen Augen in den von mir beschriebenen Fällen die Feedbacks zu den Features und der Perfomance bringen sollen.
Martin, ich will die Abstimmung nicht mit meiner Meinung beeinflussen. Es soll jeder für sich selbst entscheiden, was er für richtig hält.
Dann hoff ich mal, dass ich sie mit meinem Kommentar nicht zu stark beeinflusse ;-)
Aber ich werd nochmal fragen, wenn die Umfrage abgeschlossen ist, in Ordnung?
I went for "Honest to the bone" but there are several things being dealt with in a beta release that should be taken into account and handled differently.
First you need to filter out the actual bugs. By the time you get to beta this is what the programme managers are really after.
Feature requests. You are always going to get feature requests. Some will be simple tweaks that could be put into the release, others will have to wait. But you can still take them on board.
General Whinging. Hey, some one is providing you feedback!! Listen to it. You might just categorise it under "Looney" but at least listen to it first.
Managing the data you get into these major categories can be difficult, but it needs to be done. It's better if you can get the testers to filter out duplicates, or if you are interested, post AOLs so you can see how widespread the problem is. Categorise, find duplicates and prioritise.
Ultimately too much feedback is better than not enough.
Zu spät, Martin. Im vierten Kommentar stehts schon drin. :-)
Hmmm :) I think your survey should also include friendly options like "Focus on new features" or "High praise & patience"... But yes, if it was my software, I'd prefer the tough truth.
With regards to the beta we're talking about here: I'm a slow adopter. Despite (really) having access to the code for quite a while, I have not touched it and am quietly lurking and observing until all the expected release problems are sorted out, like they usually do by release x.02 .
I do think the performance issues should be highlighted and addressed asap. It is a concern as it painfully reminds me of Notes 4 and 5 (remember how much rejoicing there was when Notes 6 and 7 speeded up significantly?) I don't think performance issues should be considered a little beta bug.
Features I'm less worried about. Those are the things that will get fixed as people test it. If it was my software, and I knew I was still going to code more features in, I'd rather people just get on with the testing of existing features than keep hammering on what is still to come.
Martin, mit Feedback über fehlende Features hätten die Entwickler aber wenigstens die Chance, die Entscheidung nochmal zu überdenken, ob die Version auch aus Kundensicht "fertig" ist oder vielleicht doch nicht. Die entscheiden ja nicht unwesentlich mit am zukünftigen Erfolg des Produktes.
Das gilt besonders, wenn ein Produkt derart mit Vorschusslorbeeren kommt wie dasjenige, das wohl der Auslöser für diese Umfrage gewesen sein dürfte. Ein "schau mer mal" kann man sich hier vielleicht nicht mehr leisten.
Und im Allgemeinen wird "Beta" heutzutage eigentlich nicht mit "Feature Complete" gleichgesetzt. Das wäre beim Release Candidate der Fall.
Adelaida, I am actually trying to get a general opinion on how you would like a beta feedback. That is why I wrote:
Assume you are beta testing your own software.
I know. But your post has a history, so I put my answer in context. You are welcome to replace the product reference with Product XYZ (in fact, please do) - my answer would still be valid if it was my own product.
Indeed, it does have a history. That is why I need to calibrate my own opinion. So far it's only my own, which is as good as anybody else's.
I won't get 200 and more people to come out of the closet and state their ideas. But I can easily get them to push a button.
I voted "Honest".
The thing is, I don't think it is (or should be) a matter of what I want in terms of feedback. I think it is a matter of what I want users to do to help classify the feedback so it is most useful.
The main form for the Notes 8 beta forum presents four choices: "Problem", "Suggestion", "Comment", "Question". There are additional categorizations, but they're not really the right ones.
If they added a second and third level of categorization that is dependent on the tope level, then users could very easily help IBM filter things.
I.e., ...
For "Problem"
. 2nd level is: Download issue, Installation bug, Functionality bug, Usability bug, Documentation bug, Performance issue, Other
. 3rd level is: Server, Eclipse client, Legacy client, Admin client, Desginer client
For "Suggestion"
. 2nd level is: Change functionality of a feature, Change UI of a feature, Request a new feature, Other
. 3rd level same as above
- For "Comment":
2nd level is: Server, Client, Mail, Composite apps, Templates, Notes Dev, Admin, Other
3rd level: ?
- For "Question":
. 2nd level same as for "Comment"
. 3rd level is: For Lotus, For testers, For all
If beta users were simply asked to go through a slightly more structured feedback process like this, I doubt that anyone would have a problem with it. There would be no need for so-called "rules". People within IBM who need to concentrate on bugs could easily concentrate on bugs. People within IBM who want to look at feature requests could easily find them -- either now, or in the future when they have some bandwidth.
And obviously, while I used IBM in my answer, this was just as an example. The categories I suggested are what I would set up a beta feedback form myself, if I had a software product that had an eclipse client, legacy client, server, designer, and admin components ;-)
if I had a software product that had an eclipse client, legacy client, server, designer, and admin components
LOL, Richard. I appreciate the thoughts you have put into this, but I am not confident that people would want to fill out all that information. They will often mix many topics into one comment, so you would have a hard time filing it correctly.
I dunno, being part of a small shop that develops software, how much feedback could I reasonably read thru? If I had a categorized feedback list, "Bug" would be read much before "Performance Issue" or "Missing Feature".
From my own development experience, we usually know which features are missing and which parts of the code run slow. It would be more useful to me to report things that don't function correctly. If a program takes 30 minutes to load, it's not a performance issue, something is broken.
I think it all boils down to IBM's internal process -- someone there should be acting in a triage role, sorting out the feedback they need right now, sending it to the right people, and documenting the rest for later/never.
Without that role, I can understand wanting a filtered feedback during the beta.
And I can understand why a small company might not be able to afford the resources for that role. But we're talking about a major product from IBM here. I would think that resources would not be an issue.
Tony, 30 minutes would be an easy answer. But what about 3 minutes? Or 1 minute? Acceptable or not? You have to ask your customers. Same goes for features. What you deem important may not be what your customer deems important.
This has all but killed Groove. Version 2 slowed down your machine enough so that you did not want to run it in the background. When version 3 came out, people had stopped trying. Same goes for features. Sharing a file system folder was very high on the wish list, but was not implemented before 3.
Dave, Tony did the right thing answering for himself and not for IBM.
One of the most problematic aspects of a any beta program is that most people don't have a clue about how to write up a good bug report. My experience is that this is true even for the majority of experienced IT professionals. Most people really need to be led through the process, and better yet, they need to see some examples of good bug reports to help them understand the level of detail that is necessary.
But yes, I agree that that a lot of people would not want to fill out that info, still... I don't think that many will just walk away because of a couple of radio buttons. The most diligent testers probably will at least try to fill out the fields with the most appropriate choices, and they are the ones whose feedback is going to be most helpful. At worst, some people will just click the radio buttons without bothering to think about the right choices... but that will show through quickly to anyone who reads the report. A mismatch between the fields and the content can effectively serving as a way to triage out a lot of reports that aren't going to be helpful anyhow because, when it comes down to it, they just say "feature X didn't work for me".
Richard, we agree on the bug report.
However, do not confuse well organized people with skillful people. I know quite a few people who would be good administrators and terrible engineers. You are very thoughtful and diligent, and I won't hold that against you. But you are more of an exception than a rule.
Richard - there are groups that are testing the Notes 8 client that are doing exactly what you suggest. That is just not true of the public beta
For the whole beta test you of course want all the information you can get. But you may want to ask different questions to different audiences, e.g. your internal users, your existing customers, some prospects or all people that are interested. You may also want to target the different audience at different times in the development cycle.
There may also be other channels to provide feedback besides the beta program.
Third there may be a marketing aspect too. You may know up front that some of the feedback from some audience will contradict with other input. So why open this up in public even before you have a chance to prove the benefits of your decision. You may never be able to make everyone happy.
So the options in your poll may not really fit to the decisions one has to make in a real life situation.
