Best Practices for Testing

From Perl QA

Jump to: navigation, search

Contents

[edit] Purpose

This page should contain the best practices for tests used in CPAN packages. The purpose of these practices should be the following:

  • Help the first-time author to avoid beginner mistakes
  • Help the seasoned author detect errors
  • Allow the person who installs the package a chance to
    • See that all relevant tests are passing
    • Check - if they wish - that the author-only tests are passing
    • Give feedback to the author if one or more tests fail
  • ...

[edit] Audience

The practices here are primarily for module authors, and for authors of modules that support the different aspects of module authoring.

Please follow the discussions on the perl-qa@perl.org mailing list, and list any new or improved best practices (including it's reasoning) below.

[edit] Current Best Practices

[edit] Goals

  • just make CPAN "just work"
  • work toward raising the overall quality of CPAN
  • help new authors "do the right thing"

In general, automated tests on CPAN...

[edit] Should not:

  • require user input
  • require an account/subscription to some external service
  • assume network connectivity
  • assume connected hardware
  • take too long
  • require modules which are not prereqs to normal operations

[edit] Should:

  • verify that the module compiles
  • verify that it functions properly on the given version/platform
  • list testing prereqs in META.yml under test_requires[1]

[1] Though META.yml doesn't have that yet.

[edit] Test Filesystem Organization

During the Oslo QA Hackathon 2008, it was agreed upon that there can exist an 'xt' directory that contains tests to only be run under certain circumstances. Module::Install::ExtraTests was written to help run these under the right conditions. So far, these are:

  • ./xt/author - run when the tests are being run in an author's working copy
  • ./xt/smoke - run when the dist is being smoked (AUTOMATED_TESTING=1)
  • ./xt/release - run during "make disttest"

[edit] TODO

Design and document practices for meta-testing, author-testing, extra testing, and how cpan-testers (and other smoke systems) identify/support/feed these tests ($ENV{AUTOMATED_TESTING} and maybe also $ENV{PERL_AUTHOR_TESTING}.)

[edit] Related

Retrieved from "http://perl-qa.hexten.net/wiki/index.php/Best_Practices_for_Testing"
Personal tools
Mongers