Danvers, MA
May 1994
Recently survey researchers have been examining a variety of list- assisted techniques to improve the cooperation rates in telephone surveys. Declining response rates along with the need to reduce costs have prompted researchers to search for alternatives or supplements to random-digit- dialing (RDD) surveys. In this paper, we present the results from two surveys that used list-assisted randomly-selected telephone numbers. The surveys were conducted in fifty neighborhoods in Indianapolis during 1993.
Our analysis focuses on two issues. First, we analyze the differences in cooperation rates between samples when presurvey letters were sent and when they were not sent. Second, we examine the demographic characteristics of the neighborhoods based on the 1990 decennial census to determine the impact of differential cooperation on the demographic representativeness of the samples.
The cooperation rates in neighborhoods that received presurvey letters were higher than both an RDD sample and in neighborhoods that were not sent letters. There are mixed results from a comparison of the demographic characteristics of the neighborhoods and the demographic characteristics of the various samples. Overall, the presurvey letters did not yield samples that were closer to census proportions than those that did not receive letters.
INTRODUCTION
Recently survey researchers have been examining a variety of list-assisted techniques to improve the cooperation rates in random number telephone surveys. Declining response rates along with the need to reduce costs have prompted researchers to search for alternatives or supplements to random- digit-dialing (RDD) surveys. In this paper, we present the results from two surveys that used list-assisted randomly-selected telephone numbers. The surveys were conducted in fifty neighborhoods in Indianapolis during 1993.
Typically, list-assisted samples are used to reduce costs and increase response rates. In this project, list assisted samples were used primarily because the geocoding of the households was a critical part of the analysis. The project focused on neighborhood differences and the need to accurately locate the households within neighborhoods was crucial to the project design.
Two alternatives were considered. The first alternative was an RDD sample based on the exchanges listed for each neighborhood. In Indianapolis, exchanges are primarily focused in large but identifiable areas. An RDD sample would allow all households a chance for inclusion in the study but it had significant drawbacks. First, an RDD sample would require asking households for either their addresses or information that would allow the researchers to geocode the household. We expected significant item nonresponse to this question because we were sampling central cities areas. Second, we would have to screen early in the questionnaire on the household location, and we expected this would significantly increase noncooperation. An RDD sample did not appear to be a cost-effective sampling method.
The chosen alternative was to purchase a sample from a sampling company that contained the addresses for the households. The sample purchased was based on White Pages listings for Indianapolis. The researcher was able to specify the geographic boundaries for each neighborhood and the samples of telephone numbers for each neighborhood were provided to the Center for Survey Research. The listed sample, too, had significant drawbacks. Households who moved to the neighborhoods recently or have unlisted numbers would not be included in the sample.
Tucker et al (1992) report on other problems with list-assisted samples such as duplicate listings. In our processing of the samples, the addresses were checked for duplicates. Very few duplicates were found. Tucker et al (1992) also reported that households whose mail is addressed to post office boxes are often missing from list-assisted samples. No estimate of these number is available for the neighborhoods in the study. Our only significant problem was missing apartment numbers but, in general, the lists were fairly accurate.
This paper reports on two surveys. In the first survey, we supplemented the neighborhood listed samples with an RDD sample based on the exchanges in the listed samples. In general, the RDD results were disappointing for the reasons listed above - lower cooperation rates, many respondents refused to give their address or even a general location for their households, and among those who did provide geographic information, a very high percentage were outside the target neighborhoods.
The purchased sample allowed us to send presurvey letters to the households. We expected that a letter sent from an Indiana University research center located in Indianapolis would improve cooperation rates. Traugott et al (1987) found that presurvey letters improved response rates (13.4 percentage points and 8.5 percentage points in two experimental treatments; Table 3). Subsequently, Traugott and Goldstein (1993) replicate parts of the survey earlier survey but found the impact of the presurvey letter on the response rate to be substantially less (3.6 percentage points). Both studies also examined the effects of the differential response rates between letter and no-letter groups on two substantive questions. The results might be summarized as providing no clear indication of the overall effect of the differences between the groups. The differences depend partially on the type of question (Traugott et al, 1987).
In our study, we took a different approach to evaluating the impact of the presurvey letters. We sent presurvey letters to all households in some neighborhoods, to about half of the households in other neighborhoods. In some neighborhoods, no letters were sent. We examine the differences between the sample distributions (both weighted and unweighted) and 1990 decennial census distributions on three demographic characteristics - gender, race, and home-ownership - for each neighborhood. In addition, we examine types of neighborhoods to determine if there exists differential impact of presurvey letters on response rates and on the sample distributions.
SURVEY PROCEDURES
The first survey was conducted in February and March, 1993. The target area was 13 neighborhoods in south Indianapolis. Approximately 100 respondents were interviewed in each neighborhood. The sample of names and addresses was purchased from Survey Sampling. It included persons listed in the white pages in the targeted neighborhoods. We sent personalized letters to the person listed at each address. The letter described the survey and listed both an 800 and a local telephone number to call to obtain additional information about the survey. At the same time, an RDD sample using the exchanges in the listed sample was conducted. Within multi-adult households, a respondent was selected using a random number. Because of random selection, the person who received the letter was not necessarily selected to be interviewed. The CATI questionnaire was programmed to provide the name and address of the person to whom the letter was sent, but the interviewers did not use that information unless needed to convert a reluctant informant or respondent. The interviews averaged about 16 minutes and focused on community policing issues.
The second survey was conducted from June through August, 1993. Thirty- seven neighborhoods from Marion County were selected for the study. Marion County contains the city of Indianapolis and several small municipalities. The number of respondents in each neighborhood varied from 75 to 150. A similar listed sample was purchased from Survey Sampling. We sent presurvey letters to all persons in the sample in eight neighborhoods. In seven neighborhoods, we sent a letter to a randomly-selected one-half of the list. In the remaining neighborhoods, we did not send letters. We used the same mailing procedures and the same within-household random selection procedure as in the first survey. Again, we programmed the name and address of the person to whom the letter was sent in the CATI instrument for the interviewers, but they did not use the information unless needed. The interviews averaged about 25 minutes and focused on community conditions. The topics of both surveys were relatively similar.
The 1990 decennial data for each neighborhood was obtained from the STF1A data file. The neighborhoods in the first survey represented approximately ten percent of the Indianapolis population. The neighborhoods for the first survey were selected because they represented specific police beats in south Indianapolis where community policing was in place. The second survey's had two foci - 1) to assess conditions in Indianapolis neighborhoods as experienced and perceived by neighborhood residents, and 2) to provide a baseline against which to measure changes over time as various public and private programs are implemented. The neighborhoods for the second survey covered the entire city. As a result, there is some overlap between the two surveys.
DATA ANALYSIS
In this paper, we will use the term "cooperation rate" to evaluate the effectiveness of the presurvey letters. Cooperation rate is defined as a percentage calculated by dividing the number of refusals by the sum of the refusals and completed interviews. We use this indicator because it most directly captures the impact of the presurvey letter and not other survey procedures that affect the "response rate." For example, we do not include in the calculation those cases classified as "persistently unavailable" or the cases where an answering machine message indicated a household telephone number but we never interviewed anyone in the household.
We interpret the data in Table 1 to indicate that the presurvey letters improved the cooperation rate in the first survey. In the listed sample, about 73 percent cooperated by doing the interview. In the RDD sample, only about 60 percent cooperated. This percentage is consistent with other RDD studies conducted in Indianapolis. The cooperation rates differed substantially among the neighborhoods. The lowest cooperation rate was 66.3 percent (6 percentage points higher than the RDD sample), and the highest rate was 80.8 percent. These rates are much higher than our previous Indianapolis RDD studies.
We suspected the higher response rate in the listed samples might also be affected by the sample selection. Samples drawn from those persons who agreed to have their names listed in the White Pages may be more likely to cooperate with telephone interviewers than those who choose unlisted numbers. We also wondered if the presurvey letters were cost efficient. That is, could we reduce survey costs with little impact on the overall cooperation rate by not sending the letters? If those who were listed in the White Pages were more cooperative, then there might be little impact of the presurvey letters.
The more recent research by Traugott and Goldstein (1993) indicated less impact of presurvey letters than happened in their 1985 project (Traugott et al, 1987). In our second survey, we attempted to determine the impact of the letters on cooperation rates. Due to the size of the study, the entire sample was not prepared before the study began. Groups of neighborhoods were setup as needed for calling. Early in the study, all households received letters. Next, we sent letters to one-half of the sample in selected neighborhoods. After preliminary analysis of the impact of the letter in these neighborhoods, and comparing our results with other research, we decided to not send letters for the last group of neighborhoods. Table 2 shows the distributions of cooperation rates for the neighborhoods in the second survey.
It appears that most letters were delivered as addressed. No more than 20 letters (less than 10 percent) were returned as undeliverable in each neighborhood except neighborhood E05 where many addresses did not have apartment numbers. The cooperation rate in E05 was about equal to the overall cooperation rate for the letter neighborhoods, so there is no indication that many households did not receive the letters.
The data in Table 2 indicate that the presurvey letter improved cooperation rates by about 9 percentage points overall. The analysis of neighborhood differences indicates that the improvement is not consistent across neighborhoods. In the neighborhoods where a random half-sample of households received a presurvey letter, the cooperation rate is generally higher among those who received the letter. Only in Neighborhood E08 was there a higher cooperation rate among those who did not receive a presurvey letter. An equally interesting result is that some neighborhoods where we did not send letters had higher cooperation rates than those where we did send letters. These results indicate that presurvey letters are important but may not needed in all instances. The characteristics of the neighborhoods may determine the relative impact of presurvey letters.
We discontinued sending letters during the study because our initial analysis of the neighborhoods where we sent letters to one-half the sample indicated there was minimal improvement in cooperation. When we made the decision to stop sending letters, we felt that the cost of preparing and sending them was not cost efficient. We did this analysis before all calling was completed in the test neighborhoods, so we could only estimate the percentage of the households still in process that would become completed interviews. In retrospect, we probably overestimated the cooperation rate among that group. We should have continued sending letters if we wanted to maintain higher cooperation rates.
Another measure of the impact of the presurvey letters are the
demographic characteristics of the samples and the neighborhoods. In Table 3, we divide
the neighborhoods into five types - low income minority, low income mixed races, low
income white, higher income mixed races, and higher income white. In the table are the
percentages that indicate how much the sample composition on each characteristic differs
from the census distribution. The cooperation rate for each neighborhood is also included.
Table 3 demonstrates the variability in the differences between sample and census distributions. In most neighborhoods, and especially in the low income neighborhoods, the male proportion of the sample is substantially less than the census proportion. For example, in Neighborhood E07, the sample had 32.7 percent male which is 12.9 percent less than the census proportion. In the higher income neighborhoods, the gender differences are smaller. In most neighborhoods, our sample contained a higher proportion of whites than the census proportion. In many of the mixed-race neighborhoods, the sample proportion of white respondents is substantially higher than their census proportion. (Small differences would be expected in single race neighborhoods.) A high proportion of owner-occupied housing usually indicates greater neighborhood stability. The proportion of respondents who live in owner-occupied housing is larger in all neighborhoods. The focus of the survey may have appeared more appropriate for long-term residents and the saliency of the survey issues may have been greater for long-term residents, so the differences are not surprising.
The impact of both the presurvey letter and the neighborhood cooperation rate on demographic compositions of the samples is not clear from Table 3. Some neighborhoods with 80 percent cooperation rates (S11 and S30) had a much smaller proportion of males in the samples than in the census. The male composition of the samples in the lower income neighborhoods that received the letters were not much different from those that did not receive a letter.
Table 4 summarizes the distributions of the samples by neighborhood type. In the high income neighborhoods, the cooperation rate is substantially improved when a letter is sent. There is relatively little impact in lower income neighborhoods.
Tables 5 and 6 are the same as Tables 3 and 4 but the respondents are weighted by the number of adults in the household. (Only the neighborhoods from the second survey are included.) In general, this weighting decreases the differences between the sample and the census proportions. For example, in Neighborhood E07, the male undersample was 12.9 percent with unweighted numbers and 9.7 percents when weights were used. Similar changes were found between the sample and the census distributions in the proportion that are white or owner-occupied. In a few neighborhoods, the differences increased.
Tables 7 through 10 contain the results of multiple regressions used to predict the various dependent variables in this paper. In these tables, the demographic compositions of the neighborhoods based on the 1990 census are used as independent variables. The letter and 1/2 letter are dummy variables representing whether the neighborhood was sent presurvey letters to all, one- half, or none of the sample. Police 2 is a dummy variable that measures the impact of the different times of the year that the surveys were conducted.
The analysis in Tables 7 through 10 focuses on the impact of the letter and the cooperation rate on differences between the sample and the census proportions. Table 7 indicates that sending a letter to the entire sample is the best predictor of higher cooperation rates. Overall, none of the other demographic characteristics of the neighborhoods are good predictors of response rates. This is not surprising because we expect personal characteristics rather than the composition of an area explain differential response rates.
In Table 8, the dependent variable is the difference between the percent male in the sample and in the census for each neighborhood. Most values are negative, so a positive coefficient indicates that the variable helped to reduce the differences. The dependent variable is constrained by the proportion male measured by the census, so a full range of values is not possible. The coefficients provide some mixed information. Increased cooperation rates in neighborhoods indicates a better match on the proportion male between the sample and the census. But, the neighborhoods where the letters were sent to all households produced larger gaps between the sample and census proportions. Most letters were addressed to men, so we speculate that men may have received enough information in the letters to tell them they were not interested in the survey topics.
The coefficients for the letter variables and the cooperation rate are not significant in Tables 9 and 10. The coefficients are in the expected direction - letters reduce the difference between the sample and census proportions of whites. The cooperation rate appears to have little impact on reducing the differences for either variable. The apparently contradictory coefficients for letters when predicting percent owner-occupied is a mystery.
The multiple regression tables were not intended to predict fully the impact of the letters on cooperation rates and demographic distributions. Rather, they were an attempt to summarize the wide variety of outcomes that were shown on Tables 3 through 7. Overall, they do not provide definitive numbers on the impact of presurvey letters on cooperation rates or distributions.
This research focused on the impact of presurvey letters on cooperation rates and the composition of demographic characteristics of samples drawn from fifty neighborhoods in Indianapolis. Overall, the results indicate that presurvey letters improve cooperation rates. We cannot determine from the results that the letters are effective in producing more representative samples, at least as measured by the demographic characteristics of the neighborhoods. From this analysis, we can conclude that the letters help with those who are likely to cooperate anyway (women, homeowners, whites), and might actually increase differential noncooperation and produce data that are not representative of the population they are assumed to be drawn from.