Best Practices for Testing

From Perl QA
Revision as of 16:44, 21 February 2011 by Gabimuc (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Contents

Purpose

This page should contain the best practices for tests used in CPAN packages. The purpose of these practices should be the following:

  • Help the first-time author to avoid beginner mistakes
  • Help the seasoned author detect errors
  • Allow the person who installs the package a chance to
    • See that all relevant tests are passing
    • Check - if they wish - that the author-only tests are passing
    • Give feedback to the author if one or more tests fail
  • ...

Audience

The practices here are primarily for module authors, and for authors of modules that support the different aspects of module authoring.

Please follow the discussions on the perl-qa@perl.org mailing list, and list any new or improved best practices (including it's reasoning) below.

Current Best Practices

Goals

  • just make CPAN "just work"
  • work toward raising the overall quality of CPAN
  • help new authors "do the right thing"

In general, automated tests on CPAN...

Should not:

  • require user input
  • require an account/subscription to some external service
  • assume network connectivity
  • assume connected hardware
  • take too long
  • require modules which are not prereqs to normal operations

Should:

  • verify that the module compiles
  • verify that it functions properly on the given version/platform
  • list testing prereqs in META.yml under test_requires[1]

[1] Though META.yml doesn't have that yet.

Test Filesystem Organization

If you have .pm files in your module's distribution that are used only for tests, put them in 't/lib' and add the line

use lib 't/lib';

to each test's file. This should work regardless of OS because 'lib' takes unix-style paths regardless of OS and can take a path relative to the current working directory. (The cwd is (by default) the root of the module's distribution when tests are run.)


During the Oslo QA Hackathon 2008, it was agreed upon that there can exist an 'xt' directory that contains tests to only be run under certain circumstances. Module::Install::ExtraTests was written to help run these under the right conditions. So far, these are:

  • ./xt/author - run when the tests are being run in an author's working copy
  • ./xt/smoke - run when the dist is being smoked (AUTOMATED_TESTING=1)
  • ./xt/release - run during "make disttest"

TODO

Design and document practices for meta-testing, author-testing, extra testing, and how cpan-testers (and other smoke systems) identify/support/feed these tests ($ENV{AUTOMATED_TESTING} and maybe also $ENV{PERL_AUTHOR_TESTING}.)

Related

Personal tools