If I've finally convinced you with my continual reminders to run some tests on your website, there are a lot of decisions to be made.  The planning stage of any testing effort is just as, if not more, critical than the actual test itself.  Ultimately the goal is to collect data that can help you make your website or service better and the best way to do that is to carefully plan out the who, what, when, where, and why of your test.

ClickZ has a post on doing just that:  Four Steps to More Effective Web Site Testing, written by Bryan Eisenberg.  He presents us with a very informative excerpt from a white paper on A/B testing that makes the post more than worth your time.  In addition, the four steps Eisenberg suggests are:

1. Define Your Goals:  What type of change do you want to make?  Conversion?  Sign-ups?  Purchases?
2. Who Are Your Customers?:  Who are they?  What type of person visits your website?
3. Creative:  What's the message?  How will the customer you defined in the previous step react to different types of creative?
4. Test Time

As Eisenberg points out, skipping steps usually leads to more cost and more time required to do testing.  With some clear planning and a few answers to the questions he asked, I think you can save yourself the hassle of creating variations to test that do absolutely nothing or, worse yet, never actually make it into the test.

All good points that I should work into my testing motto:  Plan, plan, and plan some more… then test, test, test, and test again!