The Los Altos Workshop on Software Testing

LAWST HANDBOOK                      

LAWST is Hosted By Cem Kaner and Brian Lawrence

Introduction

The Los Altos Workshop on Software Testing (LAWST) is a periodic gathering of software testing professionals. Our goals are to share good practices and to develop insights into the strengths and weaknesses of testing practices. In many ways, we are engaging in "bottom up" improvement of the field, solving specific problems and learning about and improving the larger picture in the process. LAWST is hosted by Cem Kaner and Brian Lawrence. It is a facilitated meeting, where public notes are taken by a scribe. The LAWST structure is intentionally different from traditional conferences, which provide short blocks of time for rigidly scheduled presentations. At those meetings, presentations are often too short to go into any depth or to allow time for detailed questioning and discussion. The result is that many great practices have been presented at conferences but have not diffused into the general community. Our approach at LAWST is to narrow the discussion to one or two concrete, narrow topics and to explore them in detail. The format is discussion rather than presentation and the discussion of individual points or examples continues until the group as a whole is satisfied that that discussion is exhausted. Our basic format is:

1. Topic selection. Before the meeting, we circulate different proposals for meeting topics. Everyone on LAWST-L (the LAWST mailing list) who thinks they will participate in the next LAWST is encouraged to submit and discuss proposals. Ultimately, Cem and Brian determine that one topic has garnered widespread interest and announce that as the meeting topic. Sometimes we have two or three announced topics, but discussion of the first topic will be given the time it needs in the meeting, even if that uses up the time needed for the second or third topic.
2. Imvitation. Cem and Brian invite people to attend LAWST. Up to 20 people may attend, though we are currently planning to keep the attendance at about 15, counting the facilitator and recorder. Attendees at the immediately previous LAWST meeting have first claim on attendance at the current LAWST. Notice of the invitation is published on LAWST-L. People who did not attend the immediately preceding LAWST are not automatically invited when the notice is posted to LAWST-L. Rather, they are subject to the discretionary invitations made by Cem and Brian after feedback from the first wave of invitees.
3. Evening Social. On the Friday night before LAWST, some attendees get together at a local restaurant, typically from about 6 p.m. through closing time (perhaps 10 or 11 p.m.). Currently, these sessions are organized by Cem. Any present or past LAWST attendee is welcome to come to this event. Spouses, children, significant others, consulting clients, pointy-haired bosses, and the family dog are all welcome. When LAWST is held in Los Altos, the restaurant will be Chevy's at El Camino and Rengstorff in Mountain View (across the street from Golden Gate U's Los Altos campus) unless attendees are informed otherwise.
4. Check-In. The meeting starts promptly at 9:00 a.m. The first hour (approximately) involves a round-the-room check-in and a brief introduction to the meeting topic and process.
5. Topical Discussion. The first topic is discussed, then the second, etc. For each topic, we follow the following structure:
6. Focusing Memo. Commonly, one person has circulated a document that is intended to focus the discussion. Participants at LAWST will receive this document (typically from Cem) in advance and will have read it before arriving at the meeting. Participants who have not read it in advance will read it during the meeting and will not speak on the topic until they have read the entire memo.
7. Opening Presentation. The topic of discussion may have been suggested by one person who is particularly knowledgeable or particularly interested in it. Or a particularly knowledgeable person might volunteer (and be approved by Cem and Brian) to focus the discussion even though he didn't propose it. In either case, this person is the most likely author of the Focusing Memo (if there is one) and is also the Opening Presenter (if there is one). This person is allocated up to one hour (and will probably require much less) to introduce the issues and frame the topic. The Opening Presenter is entitled to assume that all attendees have read the Focusing Memo.
8. Practice Stories and Technique Presentations. Any participant may volunteer a story on the subject from her personal experience. Any participant may volunteer to explain or describe a technique that is immediately relevant to the subject. Stories and techniques are selected from the list of volunteered ones and ordered by the facilitator, with input from the participants. Normally (but not necessarily), presentation will be of things that worked, rather than of failures. The selection should reflect a consensus of the group. The ordering is done by the facilitator and reflects his personal judgment as to the most effective ordering of material in the meeting.
9. Story/Technique-Focused Discussion. As each story or technique is presented, participants may question or debate with the presenter, including presenting alternative techniques or solutions, requesting or challenging or providing contrary data. The ground rule here is this: everything is focused on developing insight into the specific practice or technique or failure that is being presented. This is not a time for general discussion. For example, if Participant 1 is presenting Technique 1 and Participant 2 presents an alternative Technique 2, this is done as a means of helping people understand the strengths and weaknesses of Technique 1, and not as a means of helping people come to understand Technique 2. This ground rule will be enforced by the facilitator. Another example: during this discussion, some general principles or problems will be raised. These will be noted by the Recorder, but will not be discussed until later. (Anyone can say, at any time, "Put this on the general principles chart for later.") The discussion of the practice or technique continues until the facilitator (probably guided by the group) decides that this sub-topic is exhausted. There is no pre-set time limit. The discussion is expected and intended to continue through a period during which attendees are slowly and perhaps painfully working through some concepts, and even though this will often be a period that feels uncomfortable or unproductive for some participants. The point of allowing the discussion to extend is that many of the group's most creative ideas will develop during these awkward but intense periods.
10. Discussion of Broader Issues. After some number of practices, techniques and failures have been discussed, the facilitator will shift the focus of the meeting to a more general discussion of the topic. General issues raised before will be noted and may be discussed at this time, along with any other issues on topic that are raised by any participant. Duration of this discussion is controlled by the facilitator. As with the narrower discussions of examples, subject to the discretion of the facilitator, this discussion will often be allowed to run past the time of facile contributions into an awkward period during which people are thinking and presenting half-formed new ideas or are exploring underlying significant differences with each other.
11. Summary. One attendee (in the Los Altos sessions hosted by Brian and Cem, this will be Cem) will prepare and present a summary that captures the key points, practices, and conclusions made on the topic. Ideally, the summary will be in the form of a list of clear, short statements. Participants may discuss any item on this list and the list will be revised to take into account points from the discussion.
12. Voting. If there is time and if the discussion has advanced to the point at which voting is sensible, participants will vote on each summary point. We expect to discover that several conclusions are shared by all attendees, that several are shared only by a few attendees, and that some are held only by the one person who suggested them.
13. Publication. Any attendee at LAWST may publish any aspect of the LAWST discussions, so long as the publication properly acknowledges the contribution of LAWST (see below).

Principles

We hold these to be guiding principles for LAWST:

• LAWST is an intimate forum for experienced software testing professionals to share their knowledge and experiences with one another in a setting of mutual respect.

• Discussion is rooted in participants’ actual experiences. "Experiences" include empirical research (which is subject to methodological examination and attack by participants.) Theoretical contributions are welcome to the extent that they inform the discussion of a specific practice or technique.

• We explore testing experiences for similarities and differences in approaches, processes, techniques, tools, and philosophies.

• We discuss and critique ideas about testing in a healthy atmosphere of professional cooperation and constructive exchanges.

• When possible, we crystallize conclusions, rules, and techniques. The group may or may not agree with these conclusions, but it should normally agree that a given statement fairly characterizes a conclusion, practice, technique or failure description that was made during the meeting.

• We share the results of the workshop with the rest of the industry.
• Any attendee at a meeting may publish any of the discussion at the meeting. Practices, techniques, and lessons learned are publishable by any of us. However, if one participant has developed a particular technique or lesson, no other participant should publish this without appropriately acknowledging the inventor. Additionally, no attendee shall publish anything from any LAWST without crediting each attendee at that meeting.

• LAWST meetings will often explore unknown territory in the software testing domain. When we set out, we may not know where the discussion might lead to. It may lead nowhere. This in itself is informative and appropriate to publish. If we can identify the issues that leave us in deadlock, frustration and confusion, we might be able to use that list to invite others, better informed on these issues, to share experiences with us later. Progress might require a few meetings.
• The LAWST process is not the exclusive property of Cem and Brian. We have developed this process as a tool to foster the advancement of the state of the practice in software testing and software development. Anyone, anywhere, can set up their own meetings that use this process. We do not assert a trademark in LAWST and we encourage others to use the name to identify workshops that they intend and believe to follow the LAWST model. Cem and Brian specifically volunteer to help others set up LAWST meetings and under certain clearly non-profit circumstances, we will provide this service for free (expecting only reimbursement of out-of-pocket expenses). Contact us for details.

Topics of Discussion

Each LAWST has a specific topic of discussion which is normally proposed and agreed upon via email prior to the workshop. Topics are re-confirmed during agenda definition at the outset of each workshop. LAWST topics have included:

• Automating testing

• Defining requirements for test automation (a seminar)

• Documentation of test cases

• Maintaining test cases

• Creating test cases that are maintainable for testing localized products

• Deciding when a test case should not be automated

• A framework for testing oracles

Our intent has been to define topics narrowly but we have often discovered that a narrow-looking topic is broader than anticipated. We are still learning how to decide whether to narrow the topic on the fly or to carry forward with the broader topic. In our experience, broader topics are less amenable to precise summary and voting. LAWST (Los Altos) has looked largely on issues that in some way involve or bear on test automation and it will probably continue with this loose filter on topics (though the group can adopt any topic for any meeting). We expect other LAWSTs to form (probably in other cities) and we expect these to focus on other classes of issues in software testing.

Attendees

LAWST is by invitation only. It is not an official function of any organization and it is not affiliated with any organization. Cem and Brian are jointly responsible for inviting people to LAWST. Ordinarily, attendees of the most recent LAWST are automatically invited to the next one. Other previous attendees may also be invited to the next LAWST. Any attendee is welcome to suggest a new person for invitation to LAWST. Cem and Brian make the final decision as to who is invited. LAWST meetings are limited to a specific number of people, typically around 15. The invitation list closes as soon as it looks likely that 15 people will sign up. Please do not sign up if you don’t believe you can attend. The typical profile of a LAWST attendee is a practitioner in software testing with enough experience to both share knowledge and gain understanding by attending. Zealots and bullshitters are not welcome. We have very few open seats at LAWST. If you accept an invitation to LAWST, you are expected to show up. Anyone who accepts an invitation to attend LAWST and then fails to attend without advising Cem or Brian in advance will be dropped from LAWST-L and not be invited to another LAWST unless there was good cause for both the failure to attend and the failure to advise. Brian and Cem will exercise discretion and sympathy in this, but this general policy should be clear, and is necessary.

Cost

There is no registration fee. LAWST is hosted by Cem and Brian. You pay your own travel, hotel, food, and entertainment expenses. The meeting room and photocopying bill are on Cem. Facilitation is provided pro bono by Brian Lawrence.

Format of the Meeting

The typical meeting format was described above. However, we may create other meeting formats, such as seminars, for other specific purposes. Seminars may be opened to a broader audience and may not be facilitated and may not be offered for free. Even though we can imagine seminars that are specifically LAWST-topic-related (imagine a seminar on a specific automation technique, or a follow-up on requirements, or a seminar on progress metrics), attendance at a seminar does not put you in the automatically-invited queue for the next LAWST and failure to attend does not lose your place in the queue.

Publication

If we achieve our goal, the resulting work will be very interesting to the general testing community. Each of us will probably have our own take on what was learned. Participants agree to the following:

1. Any of us can publish the results as we see them. None of us is the official reporter of the meeting unless we decide at the meeting that we want a reporter.

1. Any materials presented to the meeting or developed at the meeting may be posted to any of our web sites. That is, if I write a paper, you can put it on your web site. If you write a paper, I can put it on my web site. If we make flipchart notes, those can go up on the web sites too. None of us has exclusive control over this material.

1. Any publication of the material from this meeting lists all attendees as contributors to the ideas published.

1. Articles will be circulated to LAWST attendees before being published. Circulation will be via posting to LAWST-L.

1. Any attendee may request that her name be removed from the list of attendees identified on a specific paper.

If you have information which you consider proprietary or otherwise shouldn’t be disclosed in light of these publication rules, don’t reveal that information to the group.

Mailing List--LAWST-L

LAWST-L is a mailing list that all attendees of LAWST (Los Altos) are invited to join. Meeting announcements are made by posting to LAWST-L, as are selection-of-topic discussions. If you attended a LAWST, and you have not signed up for LAWST-L, you will not be notified of any LAWST papers or meetings unless your name is III. The members of LAWST-L will determine over time whether broader discussions are appropriate on this list. People who have been invited to attend their first LAWST and have accepted the invitation may join LAWST-L. However, anyone who accepts an invitation but then fails to show up will be dropped without notice from LAWST-L.

Miscellaneous Activities

Any LAWST attendee may distribute any documents at LAWST, subject to the usual bounds of good taste. Attendees may engage in recruiting and other networking as part of their attendance at LAWST and may even post notice of their recruiting (etc) in the LAWST meeting room, but any recruiting or other business-conducting during the meeting itself is highly inappropriate and unwelcome.