Testing Overview

The most important thing you're not doing

Readings

Bugs -- Not just for breakfast

Break this page!

Can you break this calculator? It takes a date in the form YYYY-MM-DD after 1980-01-01, calculates the number of days since 1980-01-01 and then calculates the year from those days. Focus on breaking the calculation, not the date validation.

Enter valid date:
Days since 1980:
Result:

Why We Don't Test

  • Client wants new code now!
  • Writing and running tests will delay delivery
  • Even more delay if tests fail!

Root causes

  • By end of iteration, no time to test well.
  • Manual testing is boring.
  • It gets worse as the project progresses.

Solution: Automate running tests

Testing frameworks

Continuous Integration Servers

  • Why?
    • Running tests still takes time
    • Easy to forget to run them
  • Solution: continuous integration server
    • A separate machine that runs all tests
    • Runs automatically on every commit to master

Types of Testing

  • User tests: detect what is and isn't working with users
  • Acceptance tests: define just-in-time requirements for each user story
  • Unit tests: define and verify intended behavior of every unit of code
  • End-to-end (E2E) tests: detect inter-module communication failure
  • Stress tests: detect scale-up issues

Acceptance Tests

  • Client and developers define acceptance tests for each user story
    • current iteration stories only!
  • Typically a new product will eventually end up with dozens to a hundreds
  • Many tools exist to make these more readable by clients

Unit Tests

  • Written by developers
  • Meant to test code units (functions, methods, classes)
  • Should be numerous, fast, automated
  • Typically call code functions directly, without starting any server or native application

End-to-end Tests

  • Written by developers
  • Test interactions between modules and services
  • Typically start a server or native app and interact with its user interface
  • Slower than unit tests

End-to-End Failure → Missing Unit Test

  • If an end-to-test failed when component A called component B then either
    • A sent bad data to B, or A failed to handle a valid response from B → Add unit tests on A
    • B failed to handle valid data from A or B returned bad data to A → Add unit tests on B

Stress Tests

  • Test ability to handle many users, tons of data, hostile attacks, ...
  • Many tools to simulate users, generate fake data

Becoming Better Testers

  • Start small
  • Add tests for every bug that got deployed

Bugs and Tests

  • When a bug happens, don't fix it right away
  • Write a unit test that reliably reproduces the bug
    • If you can't, you don't know what the bug is
  • Now write code to pass the test and fix the bug

Tests Have to be Constantly Updated

  • Why?
    • UI changes break UI tests
    • Database structure changes break unit tests
    • Live data changes break integration tests
  • Solutions:

TDD: Test-Driven Development

  • First, write code to test for the new functionality
  • Run the tests. Verify they fail as expected
  • Code until tests pass
  • Deploy and celebrate
  • Repeat

Thanks to Hakim El Hattab for RevealJS