Free Test Conference Norway, 2010 - feedback

I have been speaking at Free Test conference in Trondheim, Norway last week. Even though we had not so many participants (~50 people vs. ~70 last year), which is probably due to the recession w/w, the atmosphere was good, and people really liked to get more knowledge. My talk was about Root Cause analysis - dealing with problems not symptoms. It dealt with an enhanced way for performing root cause analysis, using the 5ys and cause effect diagrams methods, enabling the outcome to include priorities in it, thus helping us pick the right improvement route to reduce chances for the problem to occur again (you may find it also on the testing experience magazine, volume 4, January 2009).

Other speakers I heard were Michael Bolton, Canada, who spoke about Median and the things be gain vs. the things we loose if we are over heating this median. Asha Jyothi Venkat Meruvu from India, spoke about the TestLink, open source tool for test management, and how it interacts with other tools (like Bugzilla, etc'). I also went to an interesting talk by Mike Scot, UK, who spoke of the Testify open source tool for quick infrastructure for test automation with Fit/Fitnesse, Java, etc'. The topic name was: 'Testify - One button test driven development tooling'. Mike presented the package he built quickly, and seems that it is easy to maintain and get things on ones test automation going fast.

Martin Gijsen, Netherlands, gave a nice view over the evolution of test automation, with clear examples and the right touch for business keyword driven approach. I guess KDT is here to stay...

Bernt Marius Johnsen, Norway, spoke of random generation of data, used with specific grammar of SQL sentences, to 'bomb' a few DB products (Like MySQL, Apache Derby and PostgreSQL) to find defects - which they do in his report.

Henri van de Scheur, who was the program chair, together with his dedicated organization team, worked a lot to make this conference a reality, and have succeeded in my eyes. Data, the organizer supporter deserve a lot of thank you on this, and I hope they will support the conference next year as well.

So, in general it was good choice of topics, and I had a lot of interest in those presentations, and it was fun as well! meeting interesting people with the same passion for test tooling, automation, and testing in general...

I am waiting for what next year is going to bring us...

Hiring New testers - How do they think?

As I was looking on other things, I stumbled on James A. Whittaker's blog on testers framework of thinking (http://googletesting.blogspot.com/), which I liked very much, also referred to Eric Jacobson (http://www.testthisblog.com/2010/01/test-this-light-switch.html) on the topic.

I have interviewed a few hundred testers myself in my professional life, and believe the insights are unique and gets your attention.

I usually ask the candidate to describe how she/he will test a commonly used device, thus not having to know specifications so much, and being able to focus on the approach, strategy, and scenarios. I like to see systematic thinking, and then on the other hand also unsystematic thinking. I believe that I have learned as an exploratory tester, that when you break you systematic thinking, bugs start to pop up.

Some good questions I have heard and used in past interviews are:
1) Which is more important positive or Negative testing?
analysis: in this question, I couldn't care less about what is being said, as they both are important, but I would like to hear the approach and how the candidate explains that to me.

2) If you had only half the time for testing what you have defined as in scope, how would you act?
analysis: in this case, I would like to hear the candidate find the way to identify, prioritize, improvise, be creative and approach this bad situation - which in many cases is a reality - and explain that to me. I accept questions and a few assumptions, so that the candidate would feel comfortable, but not too much.

3) If you had to provide quick feedback on the quality of a product, what will you do?
analysis: in this case, I try to put the candidate in pressure, as there is no time. I speak fast as well, and do not wait but only 3-5 seconds until I ask : well? and again, well? Creativity is important, improvisation as well, methods and good skills will get nice ideas come out from a guy who knows testing, and experienced challenging situations.

If we can identify testers thinking patterns and frameworks according to what we see in the interview, and even guess what approach they are going to take (after establishing the pattern), my question would be if we can identify and forecast their performance? and to what extent?

We all want to hire good potential testers, that would stay long at our company and fit our team profile, maybe further analysis will help us do that with a questioner or facilitated interview directed just at that?

We'll see...

In Agile Teams - A Plumber is a Plumber, and a Painter is a Painter

One of the things that Agile discipline encourage, and preach is that everybody should be equal on the Agile team.

Well, I guess I would not want my plumber to paint me a painting, nor do I want my painter to fix my pluming... I would not want the plumber to fix the roof of my house, or the roof builder to touch my pluming... etc'. a Plumber - even though knows how to fix a roof - will never be as good as someone who is a professional, with long years of experience, and methodological and theoretical knowledge in that area.

A developer - under this assumption - can test, but will not test as good as a professional tester on an Agile team. The same happens when a tester will develop software on the same Agile team. Yes, he can develop code, but no, not as good as a professional developer.

It is the same also on classical development models: testers are doing automation, but automation is writing code, testing it in low level as well (unit test). We - testers - are good and professional in testing. In Quality Control. Not in developing complex code.

So, the focus in Agile teams should be about communicating and bringing the best knowledge of each one of the team members on the Agile team to be visible to the other members. His/her knowledge can help the team go through the challenges that the company is facing, while others in the team can learn more about it.

Brussels Course: Adding Buisness Value Increasing ROI in Testing

Last Week I was at SWIFT, giving a course on Adding Business Value Increasing ROI in Testing. It was held in La-Hulpe area in Brussels, Belgium. I did this course with my German partner D&H.

The main course objective is to show and discuss how to optimize 3 major areas: the testing processes, the risk handling (risk based testing) and the effectiveness and efficiency of us managing our team of professionals.

We have discussed this and a lot of other good topics of discussion came like: how TPI(r) relates to Agile projects? Will some of the measurements and metrics work in an Agile world? etc'. In short, the idea behind improvement is not specific for a V model or Waterfall model, but has concepts that fit all models. But, nevertheless, one needs to adjust the tools to the right situation (as testing is of course context dependent).

We had 3 groups discussing: interfaces, Back Office and Network.

Also discussed dashboards and how they play a unique role in today's world both to find trends and identify risks (which can lead to improvement of our processes) and as management has a focus on testing and risks (see pic).

12th Year CONQUEST Nuremberg, Germany 2009

When many conferences around the world stop their operations due to the financial crisis, the Conquest (by iSQI) has not. With its 12th year in a row, the program was with the right mix, and still sponsors came. Conquest is the Quality Engineering in Software Technology, and is organized by iSQI/Germany.

iSQI organized it's 12th year Conquest testing conference and this time in Nuremberg. This year, India was the iSQI conference partner (last time it was Israel), and 5 companies were present with the delegations from India and came to present what they can offer. Meetings were held during all day Thur/Fri.

Tutorials were held on the topics: Test Management, Agile Software Development, Software Requirements, Load & Performance with high quality and international speakers and with live discussions.

Exhibitors that were present included also tsg, D&H, which presented unique courses - instructor led or via their virtual learning environment.

SIGIST Israel - Performance & Load Meeting - 26 May, Crown-Plaza

In the last SIGiST meeting on performance and load, there were ~50 professionals from 20 and more companies that came to hear about the unique methods exercised by experts. 3 speakers presented the topic.

On hte first presentation, we heard Ariel S from Comverse, that was given the task to establish the performance and Load department and expertise across the company. Ariel spoke about what is it performance and load profession, what challenges we are facing in this technical field of expertise.

Also, we had a discussion on the importance of this profession or area of expertise in the testing field, and which skills are needed from a professional exercising it.

Ariel finished saying that such a 'superman' does not exist, but the thing is that performance and load engineers must be heterogeneous in their knowledge and skills in order to perform a good job.

On the second presentation we heard Elad S. from NESS, who presented a performance and load project of monitoring and control system for the army. He spoke about the benefits that the performance and load test can create, what is the downside of NOT doing it, and not doing it in the right time - e.g. at the last time of the project.

Elad, stressed out the KPIs that are suggested to monitor and analyzed in order to isolate and identify the bottlenecks and problems of the system. Saving 60% of the HW of the project, making more than 97% of the transactions stand in the time standards and requirements, and making the system support in hundred of clients instead of less than 200 - all due to good performance and load test project that identified them.

Elad presented the difference a performance and load project can do to a product/project from the client perspective - the difference between working with the system to not working with it at all - money down the drain... and another statistics...

The last speaker, Nachum D, spoke of the business aspects of performance and load. How it is the business driver as well as other functional testing efforts. How we can identify areas of weak performance and load ahead of time, looking for them in the architecture and design documents already - very early in the SDLC.

"The search for the problems from the data collected, is the real challenge of the performance and load engineer", said Dimmer. He presented a few methods for collecting data using tools, and collecting thin data, and important data that will be used for analysis by the performance and load engineer. The aspects presented by Dimmer were for a Web application project, done with a client in the UK.

in short, the meeting was very professional, people asked good questions offline, and the community got a chance to say: we believe performance and load is important, and should be discussed more.

* SIGiST Israel is initiating a working group, to start reviewing performance and load methods and approaches, and will try to produce a white paper to be published on the SIGiST website (www.sigist.org.il)

* SIGiST is presenting a performance and load seminar a the SIGiST Israel 2009 Conference, 26-29 July 2009, with Mieke Gevers - Performance and Load in Action. More details in www.sigist.org.il/minisites/sigist.index.html

Test Management under Crisis

We are in a situation where the market is closing on us fast with a lot of demands on product quality, reducing costs, improving business value. Test managers will have to cope in various ways, and develop new strategies and tactics.

The main challenges we are faced with in my opinion are: managing our processes better, taking calculated risks (based on facts and trends), and managing the human factor better - we are dependant on good people in times like we have more than in regular times.

That can be achieved only via innovative and initiative effort by test managers, gaining knowledge in how to leverage their testing and also to add more value to the business.

Learning new ways to improve our test processes, is a good start. On that we may focus on TPI or TMM or TMMi, CTP or other methods that exist in the market, and are proactive proven.

On risk based testing, and managing our risks better, we should look for risk management knowledge - it is very much related and similar in guidelines and concepts.

The last topic, is maybe the most challenging one - people focus. How to drive good people to have high moral in times like we have, improve their technical abilities, their dedication, so they will be able to push our products quality froward.

There is no magic we can do, but to learn more about the motivation factors, and know more about the future technical challenges of our organization (product wise). Make sure we have been in at least 1 soft skills course, maybe take a consultant/coach to take us through some of the high bumps, and get on our way. It is a lot of trial and error, but errors are difficult since we are dealing with people.

High executives have a better position in making strong demands now, and test managers must come behind those demands, support them, and show what they can do.

SIGiST Israel Year Conference 2008

Great show again.

For 4 days, SIGiST have put a testing show that exceeded last year by far. With more than 230 participants, 51 companies, 7 tutorials (1-2 days course), the Israeli forum for Testing - SIGiST Israel - have organized the biggest independent testing professional conference in Israel so far.

There were 2 days of professional courses, like performance & load in action, TPI, offshore testing risks and approaches, how to write test cases from requirements phase to end of test design, data base testing, and more. more than 85 people participated!

On the next 2 days, there were 3 parallel tracks on various topics: test management, test design techniques, test automation, test methodology.

Interesting sessions that made a unique appearance and interest in my eyes were of Debi Zylbermann, of Dr. Avi Ofer about metrics, and Dakar Shalom about BPT. you will be able to read about those on www.sigist.org.il in a few days probably.

Another improvement in this conference is that conference participants will be able to download any lecture - as they were audio recorded in sync with slides. Using a Webcast tool by SELA Group, a unique effort was done recording 28 hours of presentations - covering all track sessions. 28 hours of recording were done during the last 2 days, with 22-24 presentations to choose from.

Others, non participants, like SIGiST members and any person interested, will have to pay some amount. I was assured by SELA that SIGiST members will by asked to pay a symbolic amount only for that great service.

Now if you were in the conference, and missed the presentation you heard later on about good things, you can download and hear/see it with speakers voice and Q&A session at the end! What a great improvement for the conference services, and the testing community.

Awards: best presentation award was won by Danny Kovach, expert in Agile/Scrum for his presentation about the role of QA engineer and QA manager in an Agile world. Among the 3 best presentations were also Debi Zylbermann and Michael Stahl.

Sponsors were : QualiSystems, Aqua SW, Microsoft, RadView, iSQI, ITCB, SELA Group. Supporting organizations were: D&H, Testing Experience (magazine), Testing & Finance, Conquest, Gasq.

Report - Security Technical Meeting, At Sogeti, Netherlands

At the Security Technical Meeting of Sogeti in the Netherlands, I presented a topic: "The Main Challenges of Testing Today", and discussed 6 main challenges that needs to be addressed:

Technical Perspective:

1) Adding Value to the Business - we need to measure the right things, and aim with those measurements to fit business and management needs. We need also to speak the language of risk for the risk-takers to understand us, and to get clients and business involved with the testing group earlier in the life cycle.

2) Test Automation Huge systems and systems of systems pose a situation where we cannot cover a lot in our manual testing anymore. Will test automation need a boost? I believe so. What shall be the future of TA - will Model Based Testing catch up? Is TA going to be only a design issue?- Those questions were discussed.

3) Requirements Oriented Testing The testing world of today is moving into being more 'engineering' like profession. In order to do that, we shall be forced into becoming more proficient in requirements creating and analysis. How would that change our: skills? Tools? Estimation and budget? ROI? Systems quality? and maybe... lifestyle?

Profession Perspective:

1) Lack of unity and standard We should be more professional, reading more literature, learning more techniques and methods, adopting methods that have worked well in the field. We should certify and educate out test engineers, and invest in their knowledge.

2) Testing: is it a Profession Yet?We do not agree yet on the testing career path, or on the testing promotion levels both as a manager and as a professional. We should promote the TBOK, We should develop ourselves in: application domain, technical aspects, testing skills, and communication skills. Plus, as multidisciplinary test engineer is the tester of tomorrow, we must learn more domains in the engineering life cycle like: project management, product management, system architecture, etc'.

3) People as the Dominant factor of successful testing projects People have always been the dominant factor for our projects success. Since tomorrow's testers are going to be with much higher skills and knowledge (that is the demand of the market/business/clients), we should invest more in people: moral, education, certification, management - all these have to change in order to keep the knowledge (people...) in our companies. Outsourcing will also have a big weight and a factor in this investment.

Alon Linetzki
Best-Testing

Testing Experience Magazine

I would like to draw your attention to a new magazine for testers.It will be issued 4 per year (for free). Attached you will find a short introduction by José M. Díaz Delgado, Editor of the magazine "testing experience":

Dear ladies and gentlemen,We have for a long time hatched the idea of a magazine covering the daily experiences of testers and providing a platform for test professionals on the topics and trends around their fields of interest have. “testing experience” is intended to be both: a magazine and a platform for discussion.“testing experience” is the challenge of producing a high-quality magazine for professional testers made by and issued for people involved in testing.

In the first edition, we rely on a lot of experienced and well-known authors with attention-grabbing articles like Tom Glib, Patricia McQuaid, Erik van Veenendaal, Mike Smith, Alon Linetzki, Rex Black, Derk-Jan de Grood, Satoshi Masuda etc.

We hope to get the support of the global testing community by asking them to subscribe to the magazine and to make the magazine’s link known to their contacts.Please find the first issue on the link: www.testingexperience.com/testingexperience01_08.pdf If you’d like to get the “testing experience” as printed edition please subscribe on http://www.testingexperience.com/subscribe.php.

Please don’t hesitate to contact us, if you have any recommendation, article to send or just to unsubscribe! "

The first issue of the magazine had more than 13,000 downloads, and I congratulate Jose for his efforts, vision and motivation for creating this magazine. Good luck in the future!

Alon Linetzki
Best-Testing

Challenges of testing Down-scaling of systems

I once came across an unexplainable phenomena. We where in the middle of a POC, very tight schedule, and it was evening Thuresday. We decided to leave the server 'on', and not inject any transactions or events, and come back Saturday evening to continue working on it. When, after only a few hours later (11 or 12 I believe), we found out that the system carshed!

That was the fist time I saw a big system fall so hard, when not dealing with anything, just in idle state. regardless to say, that we have included from that day forward an 'Idle state test' in the regression of every release.

It brings us to the point of asking do we know to test downscaling systems? We always ask how to test up scaling ones, but the downscaling is a big issue as well. Systems are 'used' to high communication and high volume of events, and are exercising daemons, loggers, and other means to make sure everything is 'alive and kicking', but seldom do we see big systems testing trying to simulate small scale traffic.

What other issues are to take into consideration in downscaling testing?
- traffic
- buffers
- log mechanism
- memory
- synchronous things that should happen vs. asynchronous ones
- shooting 'by requests' processes and or quesries and or reports after long time

Add your own commenting this.

Alon Linetzki
Best-Testing

Testers Knowledge Accommulation & Tracking

We are putting too little effort in knowing what knowledge we have, and what knowledge we should have in the future to cope with emerging technologies.

As systems get more complex, and we are facing systems of systems, and as the knowledge we are required to know is growing (demand of the companies as a direct demand from the systems), we are about to witness a knowledge crises with testers if we do not respond. The future tester, as I wrote in one of my presentations to a European conference, is going to be more multidisciplinary, and know more of different topics (i.e. project management, product management, risk management, system architecture and design, etc'). So in order to be that person, we should educate ourselves, and be prepared.

Having said that, you have many courses existing today on all areas. It is a matter of budget and priorities for the the manager where to send his/her team to.

I use a simple matrix to track the knowledge of my team of testers, and it helps me also know where I have an "exposure" or a risk in the future test design, infrastructure administration or anything else I choose to track and direct the training and education efforts to:

The matrix helps me to identify the testers who learn faster, the testers that have little knowledge about something, and the exposure I have in dealing with complex test design and execution. It brings out the best out of people that can learn individually, and let them have further topics of knowledge to study on. They have to present what they do once in a while, and that brings more responsibility into their work, and of course satisfaction.

I can also identify problems using the matrix: personal gaps of knowledge (i.e. Ruth is no expert in any topic), and project gaps of abilities (i.e. no one has done test design more than once on Appl#2).

Professional knowledge is power, and we testers should realize that lack of that knowledge is our failure to face the future of our profession.

Alon Linetzki

Best-Testing

Legitimacy to Test An Application Domain

12 years ago when I have trained a group of testers for one of the banks, a guy stood up in the middle of my testing course, and told me: "Do you have an economics degree?". "No." I have answered, and he continued: "Do you have any formal education in Finance at all?". "No." I have answered, "not formal". Then he said:" Then why should I give you a job testing my financial systems?".

That was a legitimate question, and on the break I have answer that "I do not have a financial degree or any formal finance background, but I have tested such systems (as they have) in two different banks in the last 2 years. That answer satisfied him well.

My point is that we are asked to test complex application domain systems, and sometimes (or many times) without the domain knowledge. Is that legitimate from a company to hire only testers with domain knowledge? or it is not? Can we test well enough, while not possessing the domain knowledge?

Let's take as example. A health care machine like CT, that has an analysis module analyzing the brain slices (pictures) - is that OK to test without being a doctor? I mean the user is a doctor, and we always take the user point of view in our end to end testing.

And, it experience is accounted for, like in my example, than when can we have the first experience?

Life tought me that the simple dynamics is that within a team there is at least one person that knows alot on the application domain, and a few others that learn from him/her. That is the case in most projects I have encountered. Some companies would not hire a tester, but rather hire someone that possesses the skills of the specific applcation domain (i.g. flight engineer, pilot, etc') and train him/her in testing skills.

I guess to summarize the discussion - it depends on the application domain and on the complexity of the system or type of system we must test. The fact that lives are at stake (mission-critic systems) could matter as well for making a decision as a test manager of that company.

Who says testers needs to know programming?

In the last couple of talks I had with a few testing colleagues of mine, I found out that some persisted that testers will be trained in the future with programming skills as a must in their portfolio. They presented a job activity request that testers will be engaged in code inspection, in root cause analysis of problems in the code and finding patterns of problems in the future. All of that plus test automation in code level.

Are we or should we go into that direction? Maybe: we [testers] are known as people who like to know more, want to find how to break things and build them better [improving processes]. But, the simple truth is that we are also interested in systems thinking and systems view, and NOT going into code is something I saw attracting testers into the profession many times in my career.

Testers are specializing in systems testing; they are familiar with the system architecture, requirements, and studied the processes and end to end scenarios that are possible from client perspective - all clients profiles are relevant not just the ones that are most common. So, I can say that these testers are not less professionals, representing the client perspective, are capable or testing the product with its many aspects.

Do we need to go into a future where every tester needs to know programming? I do not believe so. But, we should have some testers do that – I suggest they shall be the ones having the most technical tendency. Some others with a more process orientation, and wide system view should keep on doing mainly what they have been doing so far – kick the hell out of the system’s processes.

So, to summarize our discussion (or is it just a beginning...), in my eyes we shall probably need two kinds of testers in the future:

Technical approach – a tester that would have to know code, be involved in a more close relationship with the developers, knows how to read code, and try to influence developers in testing better (with their code).

Customer-like approach – a tester that would have to know the product on a super customer level, knows the different architecture and design aspects, the end to end possible scenarios or customer stories, that the customer might exercise with the product.

Both approaches may be going hand in hand, enableing us to get from detection to prevention when exercising the testing profession.