Which sampling provides a known non zero chance of selection for each population element?

Introduction

2.7.2 Sampling techniques

There are two main types of samples: probability and nonprobability samples. Nonprobability samples are cases where you do not know of every unique member of the population in question [i.e., the entire user group in our case]. Another way to describe it is when every member of the population does not have an equal chance of being invited to participate. Probability samples are when you do know of every unique member of the population and therefore each has a probabilistic chance of being invited for the sample [e.g., 100 users of a product, each has a 1/100 chance of being invited]. Here's a taste of a couple of common nonprobability sampling techniques.

Convenience sampling. This is the most common nonprobability sample. You might send invitations to people in your company, students from a school you're affiliated with, the city you live in, and so on. It's referred to as “convenience” sampling because unless the targeted user group is truly limited to those people, it is likely introducing some bias to recruit just a particular slice of the population.

Snowball sampling. This is a type of convenience sampling in which those participants invited invite other participants and so on to create a pyramid effect.

With probability sampling, you can choose a more scientific way to sample because you know the number and characteristics of the true population. For instance, a particular product has a small contingent of users. Some of the common techniques include the following.

Simple random sampling. This is a known population of users from which you take a random sample via some means, such as a program or application, an Excel formula, or a simple “pick out of a hat” lottery.

Stratified sampling. You assign everyone in the population to a specific [but meaningful] group and then take another probability sample within each category. The users are therefore chosen randomly, but there is a representation from each “strata.” Note that the nonprobability sampling method that correlates to stratified sampling is called quota sampling. It means that despite not knowing the true population, you divide the users you know about into groups and still try to get some representation from that group [usually via convenience sampling].

Systematic random sampling [also known as just systematic sampling]. The idea here is that you list all of the users in no particular order [in fact, it should not be in any logical order] and then pick every Nth user. You define what N is; for instance, if there are 1000 users total and the goal is to invite 500 to participate, you'd simply take every second person from the list and invite them.

Multistage sampling. This takes different samples in different ways to eliminate bias. For example, you may do a random sample, then stratified, and so on.

Chances are that the number and characteristics of users for a product you're testing may not be entirely known, especially for Web sites, and still further for Web sites that don't require users to register or create an account [and thus can't be tracked]. If using a generic participant panel, it's likely a convenience sample and is not necessary to worry about probability sampling techniques. However, in cases where you're providing a recruiting service with part of a customer or user list from which to invite people, you may want to use one of the probability sampling techniques discussed. For example, if there are 10,000 customers on a customer list and it needs to be whittled down to a 1000 person sample for study invitations, you might use a simple random or stratified sampling method to get a representative sample of customers.

If recruiting by posting on the Web via message boards, forums, or in newspaper ads, rather than using email invitations, it's likely a snowball sample as people might forward on the information to others that they know. Just be aware that this comes with a self-selection bias. This type of bias is where the participants who choose to participate may have certain outstanding and overrepresentative characteristics, such as being the ones who particularly love and/or hate the Web site enough to participate. One way to minimize this bias is to use a phased launching strategy [breaking up the study launch into multiple groups].

TIP

Check out //stattrek.com/Reading/Sampling.aspx for references to some good books on sampling methods and the biases associated with them.

Nonprobability sampling describes any method for collecting survey data which does not utilize a full probability sampling design. Nonprobability samples are usually cheaper and easier to collect than probability samples. However, there are a number of drawbacks. Such methods can be prone to selection bias, and standard design-based methods of inference cannot be used to ensure approximately unbiased estimators of population quantities or to provide associated measures of precision. In this article, some of the more common methods of nonprobability sampling, quota sampling in particular, are introduced. Their advantages and disadvantages are discussed, and a formal framework for assessing the validity of inferences from nonprobability samples is described.

2 Methodology

The majority of the literature rests on a foundation of empirical methodology. Attention has focused more on individuals than on institutions and systems as units of analysis.

There are various modes of data gathering. Surveys of probability and nonprobability samples compose the major mode. Three variants have been used. One consists of self-administered questionnaires of elementary and secondary school students. The use of such instruments for young children has been questioned. A second variant consists of personal interviews, either face to face or by telephone. Some of these surveys employ a structured format while others are more semi-structured. Investigators using either mode typically employ statistical analysis. Other modes of data collection include intensive interviews, sometimes combined with participant observation and projective techniques, with a small number of subjects [Coles 1987]. Analysis in this instance is usually qualitative in nature. Although not typically classified under the heading of political socialization, analytic biographies of political figures—employing a variety of data sources—also constitute a mode of data gathering. Studies based on experiments are relatively rare.

Most investigations are cross-sectional, that is, the subjects were studied only once. However, a few over-time studies of the same individuals exist, thereby enabling more insight into developmental patterns. Some studies stretch from pre to full adulthood and involve either special populations or more general ones. Other longitudinal projects, especially those assessing the impact of educational institutions and political events, cover a shorter period of time.

Replicated surveys of the same populations provide historical trend data. Longitudinal studies of pre-adult populations are now available in a number of countries via national and international archives. Similarly, replicated surveys of adult birth cohorts are widely available. These surveys facilitate analyses devoted to assessing the probable impact of preadult experiences as cohorts age.

Sampling Methods

Sampling, the process of choosing the subset of people who will represent your population of users, is a complex topic that is described only briefly here. The two major types of sampling are probability and nonprobability [Bailey, 1994; Levy & Lemeshow, 1999; Robson, 2002]. In probability sampling, the probability of selection of each participant is known. In nonprobability sampling, the interviewer does not know the probability that a person will be chosen from the population. Probability sampling is expensive and time-consuming and may not even be possible because there is no complete list of everyone in a population. For many interview studies, you are likely to be dealing with nonprobability samples where you can use one or a combination of the following approaches [Bailey, 1994; Robson, 2002]:

Quota sampling. You try to obtain participants in relative proportion to their presence in the population. You might, for example, try to get participants in proportion to a distribution of age ranges.

Dimensional sampling. You try to include participants who fit the critical dimensions of your study [time spent as an architect or engineer, time using a particular product, experience with a set of software tools].

Convenience sampling. You choose anyone who meets some basic screening criteria. Many samples in UCD are convenience samples that can be biased in subtle ways. For example, the easiest people to find might be users from favorite companies that are generally evangelists of your product. You might end up with a “positivity bias” if you use participants from your favorite companies.

Purposive sampling. You choose people by interest, qualifications, or typicality [they fit a general profile of the types of participants who would be typical users of a product]. Samples that meet the specific goals of the study are sought out. For example, if you are trying to understand how experts in a particular field work on complex projects, you might seek out the “best of the best” and use them for your interviews.

Snowball sampling. You identify one good participant [based on your user profile or persona] who is then asked to name other potential participants, and so on. Snowball sample is useful when there is some difficulty in identifying members of a population. For example, if you are looking for cosmologists who use complex visualization tools, you might find one and then ask him or her about any friends or colleagues in the field who might want to be interviewed.

Extreme samples. You want people who are nontraditional or who have some exceptional knowledge that will provide an extreme or out-of-the-box perspective.

Extreme Input Can Be Useful

The use of “extremes” in user research can provide inspiration [Jansen et al., 2013] and help you understand the limits of a system. In addition to extreme samples of users, you can also explore extreme data sets that are large and dirty [something that usability research often ignores in small-scale testing] and extreme scenarios that highlight risks and rare, but critical, usage patterns.

Heterogeneous samples. You select the widest range of people possible on the dimensions of greatest interest [e.g., you might choose people from many industries, countries, genders, and experience ranges].

For any type of user research, it is important to be explicit about your sampling method and its limitations and biases.

Developing Web Surveys

2 Models in Descriptive and Analytic Uses of Surveys

Estimating a finite population mean or total is an example of a descriptive use of a survey: the characteristics of a particular finite population are of interest. In much social science research, survey data are used for analytic purposes: investigating relationships between factors and testing sociological theories. Data from the US National Crime Victimization Survey may be used to estimate the robbery rate in 1999 [descriptive], or they may be used to investigate a hypothesized relationship between routine activities and likelihood of victimization [analytic]. In the former case, the population of inference is definite and conceivably measurable through a census. In the latter, the population of inference is conceptual; the investigator may well be interested in predicting the likelihood of victimization of a future person with given demographic and routine activity variables.

Smith [1994] argued that design-based inference is the appropriate paradigm for official descriptive statistics based on probability samples. Part of his justification for this position was the work of Hansen et al. [1983], who provided an example in which small deviations from an assumed model led to large biases in inference. Brewer [1999] summarized work on design-based and model-based estimation for estimating population totals and concluded that a model-assisted generalized regression estimator [see Särndal et al. 1992], used with design-based inference, captures the best features of both approaches. Models must of course always be used for inference in nonprobability samples [see Sample Surveys: Nonprobability Sampling]; they may also be desirable in probability samples that are too small to allow the central limit theorem to be applied for inference.

Lohr [1999 Chap. 11] distinguished between obtaining descriptive official statistics and uncovering a ‘universal truth’ in an analytic use of a survey. Returning to the depression example, the investigator might be interested in the relationship between depression score [y] and variables such as marital status, financial resources, number of chronic health problems, and ability to care for oneself. In this case, the investigator would be interested in testing a theory that would be assumed to hold not just for the particular population of 10,000 women but for other populations as well, and should be making inferential statements about the βs in model [2]. The quantity for inference in the design-based setting is bp, the least squares estimate of β that would be obtained if the xis and yi were known for all 10,000 persons in the finite population. The quantity bp would rarely be of primary interest to the investigator, though, since it is merely a summary statistic for this particular finite population. In social research, models are generally motivated by theories, and a model-based analysis allows these theories to be tested empirically.

The generalized least squares estimator of β, β̭LS, would be the estimator of choice under a pure model-based approach because of its optimality properties under the proposed model. This estimator is, however, sensitive to model misspecification. An alternative, which achieves a degree of robustness to the model at the expense of a possibly higher variance, is to use the design-based estimator of bp. If the proposed stochastic model is indeed generating the finite population and if certain regularity conditions are met, an estimator that is consistent for estimating bp will also be consistent for estimating β. Under this scenario, a design-based estimate for bp also estimates the quantity of primary interest β and has the advantage of being less sensitive to model misspecification.

Regardless of philosophical differences on other matters of inference, it is generally agreed that two aspects of descriptive statistics require the use of models. All methods currently used to adjust for nonresponse [see Nonsampling Errors] employ models to relate nonrespondents to respondents, although the models are not necessarily testable. In small area estimation, sample sizes in some subpopulations of interest are too small to allow estimates of sufficient precision; models are used to relate such subpopulations to similar subpopulations and to useful covariates.

Selection Criteria Provide the Ingredients for Sample Designs

Cases can be selected on the basis of one or more of six criteria:

Availability

Fulfilling a size quota

Random [or known probability] selection

Case characteristics

Presence in specific enumeration units

Presence along transects or at specific map coordinates

All samples that utilize random [or known probability] selection are called probability samples. If one does not employ random selection, one produces one of four different forms of nonprobability samples.

Quantitative Research

For sociologists, one of the main purposes of quantitative research is to make statements with some degree of confidence about sizable human groups. Such statements typically accept the hypothetico-deductive premises of regularity and recurrence seen to be present in the natural and physical sciences. Quantitative research often begins with the researcher selecting the population [or universe] to be studied, based on the nature of the questions to be answered. The selected universe could be all the citizens of a designated nation, laborers in a sweatshop, or the homeless in a large urban center. It could be women chief executive officers of Fortune 500 corporations, or children raised in homes with same-gender parents. If the universe is large, the researcher might wish to select a manageable-sized sample from which to gather data. The sample could be selected in such a way that every unit in the universe has an equal likelihood of being selected [a probability sample]. Or the sample could be structured in a way that guarantees the inclusion of certain subsectors [e.g., genders, ages, locations, ethnic groups] of the universe. An initial sample could be selected on the basis of convenience [a nonprobability sample] and then expanded by “snowball” referrals to additional units of the sample. The manner in which the sample is selected directly affects the confidence with which findings from the sample can be generalized statistically to the universe. A researcher can introduce additional degrees of sophistication by engaging in time-series sampling, paired- comparative sampling, and longitudinal panel studies.

The hypothetico-deductive approach calls for examining statements of purported invariant relationships [hypotheses] between antecedent [independent] variables and consequent [dependent] variables, presented so that the relationship is capable of falsification. In order to be capable of falsification, such statements must be nontautological and without empirically continuous antecedents or consequents, and they must specify the relationships between the antecedents and consequents [e.g., as necessary conditions or sufficient conditions]. If, after the data have been collected and examined, the hypotheses have not been falsified, it can be stated that the data demonstrate the hypotheses. But it cannot be stated that the hypotheses have been “proved.” “Proving” invariance requires evidence that has yet to be gathered.

In order for a statement of invariance to be examined, the variables in the statement must be operationalized [i.e., made capable of observation and measurement]. If, for example, the statement refers to domestic violence, democracy, or intelligence, instructions must be provided for how data are to be gathered as evidence of domestic violence, democracy, or intelligence. Furthermore, the case must be made that the data obtained by following the instructions are valid indicators [i.e., actually provide evidence] of domestic violence, democracy, etc. This case must be made rhetorically, because it cannot be made empirically.

Comparisons are at the heart of sociological research: comparisons of sociological phenomena at two points in time, comparisons of dependent variables following changes in independent variables, and quasi-experiments in which treatment effects on dependent variables are compared with control effects on comparable dependent variables while accounting for experimenter effects. Quantitative research tends to focus on structural agency [rather than human agency] and hence recurrent comparability. The goal of such research is to demonstrate empirically supported sociological relationships, often in sizable human groups. In order to gather systematic information from large numbers of people, sociological researchers frequently conduct surveys for which they prepare data-gathering instruments, such as questionnaires [for written responses] and interview schedules [for oral responses]. The questions asked in these instruments may be close ended [with limited fixed choices] or open ended. The major advantage of close-ended questions is the ease of final tabulation. The major advantage of open-ended questions is the possible acquisition of useful unanticipated information. If replies to open-ended questions are to be used systematically, they must be coded. Raters who code the open-ended answers must be trained so as to minimize differences in their coding patterns and to maximize their interrater reliability.

Prior to using questionnaires or interview-schedules in the field, it is essential to pretest them with a sample of respondents who will not be included in the final survey. Pretests may uncover problems of question clarity and accuracy. Because the phrasing of questions can alter respondents' answers, efforts are usually made to ask each respondent identically worded questions. At times, however, wording equivalency may be more important than wording identity. In a survey of sexual behavior, for example, alternative words were substituted for certain sexual practices so that respondents from different social backgrounds could understand what sexual practices were being referenced. Designers of questionnaires and interview schedules can prepare scales [such as Guttman scales, Likert scales, or Thurston scales] to measure relative degrees of attitudes or practices. Interview schedules require trained interviewers to gather data in face-to-face conversations or over the telephone. One advantage of using interview schedules rather than questionnaires is the possibility of asking respondents to answer follow-up questions for clarification or additional details. Advantages of using questionnaires generally include greater speed and lower cost of data collection.

Reliability [the extent to which questionnaires or interviews give the same results in repeated trials] is desirable in survey instruments. Asking the same question in different ways can test a procedure's reliability. In a survey of sexual behavior, for example, differently worded questions were asked at different points in the interview about the number and gender of the respondent's recent sexual partners. Afterward, the respondents' replies to each of the different questions were compared for consistency, and hence for procedure reliability. Another technique for testing a survey's reliability is to reinterview a random selection of already interviewed subjects. The reinterview can achieve two ends: it can establish the fact that the subjects were indeed interviewed, and it can identify differences between the ways subjects originally answered the questions and the ways they answered the questions the second time. The narrowness of differences between the first and second answers would be a measure of the survey instrument's reliability. Arranging for questions to be asked in multiple languages requires special care. A useful technique is to have one translator convert the English questions into Spanish [for example] and a different translator convert the Spanish questions back into English. Comparing the original English with the “back-translated” English could identify words or phrases needing to be retranslated.

The heart of the sociological enterprise involves concepts and the measurement of concepts. Over the years, increasingly sophisticated statistical procedures have refined researchers' abilities to analyze data. But sociology's fundamental philosophical problems remain the same: How are social collectivities accurately defined and measured, and how are their causal relationships established?