Sampling and pop-up surveys
Polls and surveys created with ViewsFlash are usually published on the web site in places where they can be seen and voted on as often as possible. There are times, however, where you may want to be as discreet as possible, such as when collecting marketing data. For example, you may want to ask visitors what they think about this section of the site and use that information to improve it. Or, you may want to know why they pressed the Cancel button instead of the Order Now button. You could even ask them, perhaps offering them an incentive, to provide you with personal information for demographic analysis or profiling.
In all of these uses, it may not be desirable, at least at the outset, to force every visitor to answer these questions. You can always invite them, by offering them a "take a survey" link or banner. Another effective method involves having a survey, or a survey invitation, pop up for some, but not all, visitors. ViewsFlash provides the capabilities needed to both pop up a survey, and most importantly, to present the survey to a statistically valid sample of visitors.
Sampling on the Internet has often been shortchanged, without sufficient care being paid to its fundamentals. First, ViewsFlash helps you decide how many samples you need to have a satisfactory confidence level, paying due attention to correcting sample size for small populations, which can affect the results of sites or sections of sites that are not heavily visited yet -- which is precisely one of the better times to ask people for feedback, for example. Second, ViewsFlash is careful to present the survey only once to each visitor, and most importantly, gives every visitor the same chance of being chosen as any other visitor, thus fulfilling the fundamental premise required for a representative sample.
Other software and services often miscalculate the sample size required when the number of visitors is small, typically claiming that size doesn't matter. More dangerously, they also present a survey every Nth page view, which will result in a sample heavily biased towards the views of frequent visitors, who will be much more likely to be chosen for the survey. While the views and habits of frequent visitors are important, any survey that tries to understand those views from all visitors to the site must be able to give equal consideration to infrequent visitors, whose views may be just as important to the growth of the site. Worse, not being aware of the significance of this sampling methodology can cause you to make decisions based on information that can be completely wrong, as the views and profiles of frequent visitors are likely to be significantly different from those visitors who are just trying out the site or use it not very often.
Recapping, ViewsFlash's two objectives are to provide a sufficiently large sample, and to provide a representative random sample. The Calculate button on the Define Sample page calculates the correct sample size, given what you know about visitors. For situations where the number of visitors is small, this calculator can give very different answers from other sample size calculations which ignore the correction required for small populations. For example, here are the recommended sample sizes to be 95% confident that the answers to a question will be within 5% of the answers if everybody had been surveyed instead of asking only the people in the sample.
|Sample size, corrected for population size
|Sample size without correction (others)
As expected, the correction is negligible for large numbers of visitors, but when the number of visitors is small the correction is very significant. If you only expect 100 visitors, the standard answer of 384 can only mean that you must interview everybody, whereas you only need to ask 80 people. For 500 visitors, only 217 interviews are needed, as opposite to 384, and so on.
How to do it, step by step
1 The Define Sample screen first asks to specify the Sampling Method. There are three choices:
Don't sample. This is the default setting, which disables these options.
Fast Sampling presents every visitor with the survey in a pop-up window once, until the desired number of completed surveys have been collected or the Publication ending date is reached. This is a terrific way to get a representative sample of a desired size, provided that the site does not exhibit a pattern of different use during the time it will take to collect the sample. Since this is not usually the case, this method should be used only when gathering the information as fast as possible is more important than having results that are representative of visitors' views over a given period, such as a week. For example, after installing a new video capability on the front page which is technically challenging, it would be useful to know what the next 1,000 people think about it so that it can be removed if it is providing a less than satisfactory user experience.
The first two fields, collect __ responses and pop up the survey every __ th visitor, can be filled in by hand, or the can be calculated by filling in the other fields in this section and pressing the Calculate button. Here's how that works. You chose the confidence level (90, 95, or 99%) for your results from the pull down menu, and the degree of precision for your answers as plus or minus a percentage. Then you estimate how many unique visitors will be viewing the page where the pop-up survey can appear during the period when the survey will be Published. You also estimate how many people will be actually completing the survey. When you press the Calculate button, the first two fields will be filled with the corresponding values. ViewsFlash will use the every Nth visitor number as its criteria for deciding who should be presented with a survey and who should not.
3 Press Submit to set the options you select.
Schedule the survey for a period of time that is representative of the life cycle of your site's content. For example, a stock market site could run a survey for one 24-hour week day, as visitors each day of the week should be the same as any other weekday. A news site could be treated as a daily site or a weekly site, if its content changes significantly for weekend programming. Other sites may be steady-state sites, where there is no reason to believe that visitors change from one day to another. The period of time should also be long enough to make sure that there will be enough visits to the hosting page so as to provide the desired number of surveys; analyzing web logs for that page is one way to ascertain this information.
Monitoring the survey
Once the survey is up and running, visitors will be presented with the survey and some of them will be responding. You can get a good idea of how the survey is progressing by using the View Log command on the toolbar at the left of the screen. This log is updated periodically as voting progresses, and is updated with a final report when the Publication ending date is reached, or the requested number of surveys are completed, whichever happens first.
Changing settings after going live
You can hasten the end of a survey by changing its Publication ending date. Sampling begins anew whenever you change Publication dates.
You can also change the Sampling parameters while the survey is in progress. For example, if you are not gathering surveys fast enough, you can change the sampling rate (every Nth visitor) to accelerate the pace at which they are gathered. Some of the statistics in the log will be reset, but sampling will not be affected.
When used for sampling, the ViewsFlash server is required to respond to every page where a survey may pop up. This response is very fast, but the amount of work required can be huge. For example, installing this on the front page of a portal site could result in millions of hits per day and incapacitate even a powerful ViewsFlash server. On a 300 MHZ Intel machine, ViewsFlash completes these requests in under 10 milliseconds CPU time. Use this as a guideline for where to put these surveys. In a very large site, ViewsFlash should be run in a distributed configuration; see Architecture.