A Quick Intro to Model-Based Testing

1801 days ago, 684 views
PowerPoint PPT Presentation

Presentation Transcript

Slide 1

A Quick Intro to Model-Based Testing 45,000 tests in 45 minutes or less! Harry Robinson, harryr@microsoft.com Test Architect Microsoft Engineering Excellence Group

Slide 2

The Villain of this Piece CurrentWindow = WFndWndC("Calculator", "SciCalc") WSetWndPosSiz(CurrentWindow, 88, 116, 260, 260) WMenuSelect("@2\@2") CurrentWindow = WFndWndC("Calculator", "SciCalc") WSetWndPosSiz(CurrentWindow, 88, 116, 480, 317) WButtonClick("@32") WButtonClick("@27") WButtonClick("@38") WButtonClick("@58") WMenuSelect("@2\@1") CurrentWindow = WFndWndC("Calculator", "SciCalc") WSetWndPosSiz(CurrentWindow, 88, 116, 260, 260) WMenuSelect("@2\@1") Play "{Click 330, 131, Left}" WToolbarButtonClk("@2", "@6") WToolbarButtonClk("@2", "@7") WToolbarButtonClk("@1", "@1") Awe-motivating Unchanging Indecipherable

Slide 3

Traditional Automated Testing Imagine that this projector is the product you are trying.

Slide 4

Traditional Automated Testing Typically, analyzers mechanize by making static scripts.

Slide 5

Traditional Automated Testing sufficiently given time, these scripts will cover the conduct.

Slide 6

Traditional Automated Testing But what happens when the product's conduct changes?

Slide 7

So What's a Model? A model is a portrayal of a framework's conduct. Models are less difficult than the frameworks they portray. Models help us comprehend and foresee the framework's conduct.

Slide 8

Model-Based Testing Now, envision that the top projector is your model.

Slide 9

Model-Based Testing The model produces tests to cover the conduct.

Slide 10

Model-Based Testing … and when the conduct changes…

Slide 11

Model-Based Testing … so do the tests.

Slide 12

Calculator sufficiently familiar sufficiently simple sufficiently complex Hard to test well

Slide 13

Exploratory Testing the Calculator Start Scientific Enter a Number Standard … gee … we should see … I Start the mini-computer, and the adding machine begins running - OK. I select Scientific , and the adding machine goes to Scientific mode. - Good. I Enter a Number , and the number shows up in the show – Yup. I select Standard , and the number cruncher does a reversal to Standard mode - OK. …

Slide 14

Scripting the Calculator Testing Step Action Result 1 Start Calculator ought to run 2 Scientific Calculator in Scientific mode 3 Enter Number Number in the show 4 Standard Calculator in Standard mode … How could you have been able to you think to attempt these activities in a specific order? Furthermore, how could you have been able to you know you'd wind up with these outcomes?

Slide 15

You Have a Model in your Head NotRunning Start ? Logical Standard Enter Number

Slide 16

You Could Hand-Draw Your Model But you wouldn't have any desire.

Slide 17

You Could Hand-Trace Test Sequences 1 Start Scientific Enter Number Standard … But you wouldn't have any desire to do that either.

Slide 18

Let the Machine Do It for You !

Slide 19

But first … we should discuss adding machine's conduct.

Slide 20

Let's Start Simply

Slide 21

And Get a Bit More Complex

Slide 22

And Even More Complex

Slide 23

What Does This Model Care About? Application status: Running Not Running Mode status: Standard Scientific? Show status: Empty Not Empty

Slide 24

Find the Rules of your Model (Start) Natural Language If the Calculator is Not Running then the client can execute 'Begin'. At the point when the client executes 'Begin', the Calculator goes to "Running" mode C# if (AppStatus == AppStatusValues.NotRunning) a.Add("start") AppStatus = AppStatusValues.Running;

Slide 25

Find the Rules of your Model (Standard) fascinating Natural Language If the Calculator is Running then the client can execute 'Standard'. At the point when the client executes 'Standard', the Calculator goes to "Standard" mode and the show is cleared. C# if (AppStatus == AppStatusValues.Running) a.Add("standard"); if (ModeStatus == ModeStatusValues.Scientific) DisplayStatus = DisplayStatusValues.Empty; ModeStatus = ModeStatusValues.Standard;

Slide 26

Let the Machine Do It for You 1 Start Scientific Stop Start Standard Stop Clear Stop Start Scientific Clear Enter Number Standard Clear … !

Slide 27

So we should produce a couple of thousand tests … demo

Slide 28

Executing the Test Actions, pt 1 while ( (strRecord = StreamToRead.ReadLine()) != invalid) { string[] individualWords = strRecord.Trim().Split(whitespace.ToCharArray(),strRecord.Length); switch( individualWords[1]) { <perform the actions> } }

Slide 29

Executing the Test Actions, pt 2 switch( individualWords[1]) { case "start": Process.Start("calc"); break; case "stop": SendKeys.SendWait("%{f4}"); break; case "standard": SendKeys.SendWait("%vt"); SetForegroundWindow(FindWindow(null, "Calculator")); break; … <process the actions>

Slide 30

A Brief Filk on Test Oracling If you wish to prevail As an analyzer you have To consider all matters cryptic. Confirming is extreme; Monkey tests aren't sufficient; (Though the accidents are regularly stupendous!)

Slide 31

Executing the Test Actions, pt 3/duplicate show substance to the clipboard SendKeys.SendWait("^c"); if ((individualWords[4] == "Empty") && (GetClipboardContent() != "0")) { Console.WriteLine(" mismatch"); } if ((individualWords[4] == "NotEmpty") && (GetClipboardContent() == "0")) { Console.WriteLine(" mismatch"); } <oracle the result>

Slide 32

So how about we run a couple of dozen tests … demo

Slide 33

Doing Cool Stuff with Models

Slide 34

Beeline Choose any 2 hubs in the way Find the most brief way between them Execute the joined "easy route" way Evaluate the outcomes and rehash

Slide 35

That Was The Year That Wasn't bug Start, Minimize, Stop, Start, Restore, Date

Slide 36

A 84-stage bug repro arrangement begin about ok_about no_title doubleclick seconds reestablish seconds doubleclick date about ok_about reestablish gmt boost doubleclick date seconds date stop begin stop begin stop begin seconds date reestablish about ok_about no_title doubleclick computerized doubleclick no_title doubleclick no_title doubleclick seconds reestablish doubleclick gmt simple amplify date advanced minimize reestablish minimize stop begin reestablish advanced date minimize stop begin expand gmt computerized reestablish doubleclick about ok_about augment computerized advanced seconds simple about ok_about about ok_about minimize stop begin reestablish date

Slide 37

Reducing the Sequence: Initial way length: 84 stages Shortcut endeavor 2 : repro succession: 83 stages Shortcut endeavor 3 : repro grouping: 64 stages Shortcut endeavor 4 : repro succession: 37 stages Shortcut endeavor 5 : repro arrangement: 11 stages Shortcut endeavor 7 : repro arrangement: 9 stages Shortcut endeavor 20 : repro arrangement: 8 stages Shortcut endeavor 29 : repro arrangement: 6 stages

Slide 38

# Repro Steps Over Time

Slide 39

Why Does Model-Based Testing Work? framework under test intricacy demonstrate speed "… I feel that under 10 percent of most projects' code is particular to the application. Moreover, that 10 percent is frequently the most straightforward 10 percent. Thusly, it is not absurd to construct a model program to use as a prophet." –Boris Beizer, Black Box Testing, p.63

Slide 40

Benefits of Model-Based Testing Easy experiment support Reduced costs/more tests Can run distinctive tests on 1000s of machines Early bug identification Increased bug tally Time reserve funds Time to address greater test issues Improved analyzer work fulfillment Start mechanized testing from Version 0.1

Slide 41

Obstacles to Model-Based Testing Comfort calculate This is not your folks' test mechanization Skill sets Need analyzers who can plan Expectations Models can be a critical forthright venture Will never get every one of the bugs Metrics Bad measurements: bug tallies, number of experiments Better measurements: spec scope, code scope

Slide 42

Tools Used in this discussion: C# (is free) Notepad (is pretty much free) WinSTDtoDOT (was composed by a companion)

Slide 43

Acknowledgments Michael Corning Rory Clark Wolfram Schulte Mike Barnett Margus Veanes Wolfgang Grieskamp

Slide 44

To Learn More Model-based testing site: www.model-based-testing.org Books: "Discovery Testing : Techniques for Functional Testing of Software and Systems" by Boris Beizer "Testing Object-Oriented Systems: Models, Patterns, and Tools" by Robert Binder A genuine model-based testing device: Spec#

Slide 45

much appreciated!