Structure and Function

The Institute comprises five administrative departments: Study Direction, Sampling, Field, Programming/IT, and Data Processing. Study Directors have overall administrative responsibility for projects. They develop effective organizational work plans and coordinate the departmental efforts. Survey project teams are usually composed of at least one representative from each department. To guarantee maximal input from senior staff, team members report both to their department heads and to the project Study Director. ISR emphasizes staff involvement in early and thorough planning, constant monitoring, regular communication, and attention to detail, augmented by current technology. It is the responsibility of the Study Directors and department heads to maintain open lines of communication for comprehensive and constructive discussion of all aspects of the project. These strategies have been applied by ISR for nearly 40 years to produce high quality data and client satisfaction.

1. Study Direction

ISR's Study Directors are social and behavioral scientists with a wide range of professional skills and interests, substantial expertise in statistical and sampling methods, and extensive survey and evaluation research experience. Advanced graduate students from relevant fields periodically assist with tactical execution of projects.

Our experienced staff contribute to the following to research projects:

  • proposal writing (including the new 424 electronic submission process for government funding)
  • survey and evaluation research design
  • project planning, budgeting, and management
  • qualitative and quantitative methods
  • statistical modeling
  • sampling
  • interviewer screening, hiring, and training
  • oral presentations
  • graphics and report writing

In addition, after data are collected and cleaned, ISR's study direction staff is capable of producing data analyses required by a specific project from qualitative analysis to complex multivariate procedures. The types of analysis used on past projects include an assortment of qualitative analyses of topical interest, as well as a host of modern quantitative techniques. The analysis plan for every study is prepared with a focus on the client’s research questions and implemented only after careful consideration of the factors that affected the data collection effort and the eventual use of the results. Thus, analyses conducted by the Institute begin near the inception of the project and ultimately represent thorough and well-planned explorations of the data.

2. Sampling

The foundation of any scientifically conducted survey is a sound sampling strategy. ISR devotes considerable attention to sample design and sampling methods and to balancing statistical considerations with the many practical limitations dictated by field costs. ISR has experience in computing sampling errors and in devising weighting and imputation schemes to facilitate analysis of data.

  • [-] Survey Weighting

    Most of ISR’s samples are based on complex designs (i.e., using both clustering and stratification). As such, statistical tests based on assumptions of simple random sampling are not appropriate for analysis in complex designs. Consequently, we adjust sampling errors accordingly. Depending on the complexity of the analysis and the sampling design, sampling errors can be computed based either on paired selections or successive difference procedures (Kish, 1965) or on balanced repeated replications (Kish and Frankel, 1970). ISR also uses SUDAAN, a software package based on the Taylor series linearization method.

    Variance estimates can be computed for entire populations or for subgroups of the population. In addition to the variance estimates, ISR reports intraclass correlations; confidence intervals; and design effects, which are ratios of the actual variance obtained to a hypothetical variance estimate of a simple random sample of the same size.

    However, since estimates of the design effects for any one variable are themselves subject to sampling error (the variance of the variance), it is suggested that groups of key variables be formed and mean design effects be computed for each group. Using these mean figures, variances can be adjusted to correct for the design of the study. Hence, standard computer programs which compute variance and confidence intervals, using formulas based on simple random sampling, can be corrected by using ISR's estimates of the design effects.

    All estimates should reflect the probabilities of selection. This means that all interviews should be weighted by the number of eligible respondents represented by each interview, as well as by any other weights used to correct for differential nonresponse. The calculation of sampling errors incorporates these weights, and, as a result, the sampling errors reflect the effects not only of the stratified cluster sampling but also of the weighting scheme. The Sampling Department at ISR also has extensive experience in the imputation of survey data using a wide variety of strategies.

    In developing sampling designs, ISR makes extensive use of US Census data. In addition, we use the Census Bureau's TIGER/Line files, which, in combination with a mapping software system, enable fast, cost-efficient, and accurate computer-generation of maps for any location in the country. Other applications of this integrated system are thematic map generation, address matching, and the linking of demographic and housing data to geographic data.
  • [-] Sampling Errors

    ISR recommends that survey data be weighted to reflect variations in probabilities of selection as well as nonresponse and other factors which cause the sample and population distributions to differ. Although the national household sample and most telephone surveys are designed to produce self-weighting samples of households, variations on the self-weighting design are inevitable. The first source of variation in sampling probabilities arises when certain subpopulations are to be oversampled, usually on variables such as age, sex, race, and location. The second source of variation is the usual requirement that only one person in a sample household be interviewed, although several eligible respondents may reside in the household. The actual probability of selection of an individual is inversely proportional to the number of eligible respondents in the household. Whenever ISR samples are selected by probability methods, the probability of selection can be computed for every household and for every individual. Thus, sampling weights, which are inversely proportional to the overall probability of selection, can be assigned to each survey respondent.

    ISR recommends that the sum of weights assigned to respondents be set equal to the best estimate of the size of the study population. This is accomplished in several steps. First the weights just described, reflecting the probabilities of selection, are computed. Then screening and interview nonresponse weights are calculated. These weights are multiplied by the previous set of weights and summed. These sums are usually computed separately for demographic subgroups, which are often important to study outcomes.

    Highly accurate Census estimates of the population sizes of these subgroups are then obtained. The final weight adjustment, the poststratification weight, is computed as the ratio of the proportion of the population in a given subgroup to the proportion of the weighted sample in the subgroup. Each subgroup ratio is the poststratification weight for all respondents in the given demographic subgroup. In sum, the final weight allocated to each survey respondent is the product of the four components described above – the sampling weight, the screening nonresponse weight, the interview nonresponse weight, and the poststratification weight.

3. Field Department

The Field Department collects data and manages the data collection staff. In-house personnel include the Field Director, Field Administrators, Manager of the Center for Telephone Interviewing (CTI), CTI Supervisors and Monitors, tracing staff, and various support staff. The 60-station CTI hires and trains telephone interviewers, mostly Temple University undergraduate students, for each new study. Many of them are cross-trained to work on multiple studies and sometimes in more than one capacity. Off-campus staff consists of field coordinators and interviewers across the country. Usually, these personnel are brought to our offices in Philadelphia for multi-day trainings on each project.

  • [-] Interviewing and Supervisory Staff

    Coordinators – Interviewers with proven interviewing performance, leadership qualities, sound judgment, and success in completing challenging and sensitive assignments are designated as Coordinators. They assist Field Administrators in the selection, training, supervising, and rating of interviewers. Coordinators are the strong link between the in-house and field staffs.

    Supervisors and Monitors – These individuals are CATI interviewers who have demonstrated a strong work ethic and dedication. Their primary task is to observe the CATI interviewers at work both visually and audibly and to record these observations. These observations are then shared with the interviewer in a constructive manner designed to improve the interviewers’ skills and confidence. Supervisors also act as monitors, but have the added responsibility of managing the CTI activity, including sample flow and interviewer attendance and behavior.

    Field Interviewers – Over the years, the Institute has trained a national staff of field interviewers, resident in their respective communities throughout the coterminous United States. During each study, every interviewer's work is reviewed and, at the conclusion of the study, the interviewer is rated for field performance and quality of data collected. Ratings become part of the interviewer's file; only those interviewers whose work meets ISR’s standards are retained for additional work. This results in maintaining a high quality field staff.

    Telephone Interviewers – The CTI maintains a staff of telephone interviewers, mostly from the ranks of Temple University’s vast undergraduate pool. Many of these interviewers work for the Institute in an on-going basis and are trained to conduct interviews for each new study in the CTI. Each interviewer is consistently monitored and given performance reviews to ensure that the data collected remain of the highest quality. Because the pool is so large, very stringent retention criteria can be applied.
  • [-] Computer-Assisted Personal and Telephone Interviewing (CAPI & CATI)

    ISR's field staff conducts computer-assisted telephone interviewing (CATI) studies and computer-assisted personal interviewing (CAPI) studies. In fact, ISR fielded one of the first national CAPI surveys in July 1992, when over 300 interviewers conducted more than 15,000 CAPI interviews.

    The laptops and CATI software handle the most complex questionnaires with ease, freeing the interviewer to concentrate on proper use of visual aids and, most importantly, appropriate rapport with the respondent. CAPI/CATI virtually eliminates interviewer error by handling logic checks, skip patterns, and flagging missing or invalid data during the interview itself. In comparison with paper instruments, CAPI/CATI allows greater control and flexibility in skip patterns and complex question sequences, minimizes interviewer error, permits relatively rapid questionnaire revision and coding, and facilitates the generation of field progress reports. In field studies using CAPI, data are uploaded daily.
  • [-] Administrative Pretests

    Pretesting is an integral part of the instrument design process. Cognitive interviewing and focus groups may be employed when special attention must be paid to questionnaire development issues, while more standard pretests provide information concerning the mechanics of the questionnaire, the clarity of instructions for the interviewer and respondent, and clues as to whether or not the questionnaire measures what it purports to measure. Pretests reveal factors that could affect the response rate and data quality and inform us about the reliability, validity, and length of instruments. Pretests of CAPI and CATI instruments test the program in real-life interviewing situations and reveal errors that may exist in the program.

    A debriefing session follows each pretest, during which interviewers report the general and specific problems encountered by interviewers and respondents. In consultation with the clients, revisions are made between each of the pretests, with the final pretest instruments virtually identical to those that will be used for the study.

    Frequently, ISR translates questionnaires and other study materials, usually into Spanish. Our translators independently perform direct translations, meet to reconcile discrepancies, and agree upon a version that is a functional, idiomatic translation of the original. Harmonization may be required for different dialects. Spanish questionnaires are also pretested, as described above.
  • [-] Interviewer Training

    Effective training methods and training materials development are part of every field study. Our programs include home study, preconference practice interviewing (“mock” interviews among the trainees), and direct interpersonal training. Training conferences are tailored to the needs of the project and are from one to five days long. Attending conferences are a Study Director, a Field Administrator, and Coordinators or CTI Supervisors, with support from field, programming, and sampling management.

    A detailed, study-specific instruction manual, which describes the background and purpose of the study, question-by-question specifications, and procedures to be followed, is used in addition to other training materials. For CATI and CAPI projects, an additional manual, the Computer Guide, is used as a training tool and serves as a valuable resource for interviewers while working in the field or the CTI

    Training covers all aspects of an interviewer's assignment, including (as appropriate) the listing of housing units, contacting households, respondent selection, refusal conversion, nondirective probing, verbatim recording, completion of forms and reports, and transmission of work. For longitudinal studies, interviewers are also taught successful strategies for locating respondents. For CATI/CAPI projects, interviewers are taught the basics of the CASES software through presentations, demonstrations and hands on exercises. As part of every training, items in the questionnaire are reviewed and objectives of questions explained. Interviewers actively participate in the training by conducting practice and mock interviews, and receive immediate feedback from the trainers.
  • [-] Call Reporting and Callbacks

    To obtain a high completion rate and to reduce bias, ISR follows an unlimited call back rule in most studies. This requires interviewers to make personal visits or phone calls on different days and hours until screening information is obtained and the selected respondent is contacted and interviewed or found to be ineligible. Every sample unit has a unique call report form to be completed by the interviewer. The interviewer records the date and time of every contact and attempted contact, and the code for the result of that visit or call.
  • [-] Quality Control and Verification

    Interviewers’ completed work is reviewed in the Data Processing Department’s check-in procedure. An essential feature of ISR's quality control procedures is the validation of data. For most studies, this involves a letter of thanks (usually with the promised incentive) to each respondent as soon as a completed interview is recorded at ISR. The letter contains verification questions about the length of the interview, the manner in which it was conducted, the respondent's reaction to the interview and the interviewer, and some factual questions from the questionnaire

    If there is any reason to question the validity of the interview or the integrity of the interviewer, every respondent in that interviewer's assignment will be contacted. If necessary, interviews or reinterviews are conducted by another interviewer.
4. Programming and Information Technology (IT)

The Programming Department manages and maintains the IT infrastructure of the Institute. The programming staff, using a variety of computing platforms and software, process data, design and maintain databases, develop logical tools for data collection and post data collection activities. In addition, this department oversees and manages the flow of electronic information throughout the Institute, such as e-mail (which resides on ISR-specific servers), student payroll information, and various financial and business development files.

  • [-] Data Collection Equipment and Software

    Computer Assisted Telephone Interviewing (CATI):
    CATI surveys are administered on Sun Microsystems servers running the SOLARIS™ (UNIX) operating system. Interviewers connect to our CATI systems from ISR'S Center for Telephone Interviewing (CTI) using their own customized login and password. The login system controls interviewer access, allowing access to the studies for which each interviewer is authorized. Once logged into a specific study, the interviewer will be directed only to appropriate cases. For instance, only certain interviewers speak Spanish and only certain interviewers are approved for refusal conversion. Once the interviewers are logged into a study, the survey instrument is administered using the Computer Assisted Survey Execution System™ (CASES) software developed by The University of California, Berkeley.

    Computer Assisted Personal Interviewing (CAPI):
    The CASES™ software also runs on laptops running Windows™. For CAPI and mixed mode studies, interviewers conduct face-to-face interviews using a laptop. Upon completion, the interviewer transmits encrypted data to ISR to be stored on our server. The transfer process may also download a revised instrument or a database with updated study information. All files and data transmitted are encrypted to the FIPS 140-2 data encryption standards prior to transmission.

    Web:
    ISR maintains a WEB server running the LINUX operating system and APACHE™ Software. We employ the Perseus™ package as a front-end survey tool to develop the Hyper Text Markup Language (HTML). Each web respondent is assigned a unique ID and password for security and identification purposes as well as for preventing duplicate interviews. All data collected in web surveys are stored with the same level of security as data collected in person or by telephone.

    Paper And Pencil (PAPI) / Mail Surveys:
    Paper questionnaires - either self- or interviewer-administered - are keyed by data entry staff into a computerized data entry instrument using Direct Data Entry (DDE) to ensure quality control. These data entry instruments are written using CASES™ software and reside on the SOLARIS™ servers.

    Interactive Voice Response (IVR) Interviewing:
    ISR maintains the hardware and software necessary to administer IVR instruments over the telephone. An IVR survey uses pre-recorded scripts to ask questions of the respondent, and the respondent answers either by voice or using the phone's keypad to select responses. We use TELESAGE™ software running on a Windows™ server.

    Facilities:
    ISR's CTI includes three separate telephone rooms for a total of over 60 CATI stations. Some stations are equipped with internet access to internet sites used to trace Respondents.

    Data Processing:
    ISR uses the MYSQL™ database for storing production information, producing mailings, and generating reports. Scripting is done in UNIX shell, PERL, and PHP. Data analysis is done using SPSS™ and SUDAAN™. All data are backed up regularly and stored in a secure location.
  • [-] Security

    All of ISRs servers reside behind Temple University's firewall. All access from outside of the University is denied by default. The servers are stored in a locked room, which is continually monitored by video camera.

    The Unix/Solaris servers are further protected by "TCP Wrappers" that by default deny all access to all services. ISR explicitly allows access only to identified computers from within our own domain. Any temporary access that might be needed is configured on an as-needed basis, carefully monitored and revoked immediately once the need has been satisfied.

    All interviewers who connect to administer CATI interviews have been assigned a unique login and password to log in to the interviewing system. Every time any access is made to any case on any study, this access is logged in a database, thus ensuring all information about contact is recorded and available to the interviewer. (This database includes no personally identifying information). The interviewers cannot access any completed cases or any other collected data. Staff access to the data is carefully controlled by login permissions, as well.

    The MYSQL™ databases are stored on a separate Solaris server. Each study has its own database. Staff working on a particular study have access only to that study's information. The login/password for each user determines levels of permission and access to each database. User identity also determines what data within a database can be accessed.

    Personally identifiable respondent data reside only on the server. It is never stored on an individual PC, sent via email, or transferred internally or externally on any sort of removable storage. The only exceptions to this rule occur when conducting a CAPI study, which by its nature necessitates that minimal amounts of identifying information are loaded onto the laptops the interviewers use in the field. These data are stored separately from any data collected in the interview process and are protected by multiple layers of passwords and access control. When data are delivered to a client, they are delivered password encrypted, with identifying information separated from collected data.

    ISR, working together with our clients, determine at what point all data from a particular study will be destroyed. We sanitize/decommission hard drives using "CSC Pro Hard Drive Workstation" model DOD-55220-22M that has an erase function meeting DOD-5220 government erase specifications.
5. Data Processing

ISR's Data Processing Department processes completed questionnaires, call report and screening forms, and other data collection forms. Check-in procedures, edit and quality control, coding, data entry and verification, and data cleaning are designed to improve the accuracy, reliability, and validity of the data. Record keeping and filing systems are maintained so that the physical raw data are readily retrievable at any time by authorized personnel. In-house personnel include the Manager of Data Processing, CATI Programmers, Data Processing Supervisors, and other support staff.

  • [-] Check-In

    The check-in system, an integral part of the processing of every in-person or self-administered interview study at ISR, monitors receipt of study materials from the field so that close control may be maintained over individual interviewers' workflow and overall study completion rates.

    The check-in system is initiated with identifying information on each sample unit. When questionnaires and call reports are received either manually or electronically, the check-in file is updated to reflect the current or final disposition of each case. Weekly completion reports are generated with breakdowns needed for assessing study progress, in the aggregate or within particular target subpopulations.
  • [-] Edit and Quality Control

    After check-in, completed interviews are reviewed by a member of the editing staff who works closely with the Study Direction and Field Departments. Early in the field work period, feedback is provided to the Study Director on any problems that may require clarification of interviewers' instructions.

    In reviewing a completed questionnaire, the editor may find that certain responses need clarification or that the interviewer has misunderstood instructions or has omitted essential questions. In such cases, an explanatory memo is sent to the interviewer with copies of the questionnaire pages where the problem occurred. Questionnaires are then updated with the retrieved missing information. Throughout a study, an evaluation form is assembled for each interviewer. At the end of the study, these evaluation forms are one of the measures used in determining an interviewer's overall performance rating. These ratings are used to maintain a pool of suitable top-performing interviewers for subsequent studies.

    Historically, data collection and data processing have represented separate and independent tasks. With CAPI/CATI technology, however, these tasks mesh to become a unified process. The study specific program guides the interviewing, as well as the data entry, coding, cleaning, and data file creation tasks. The integration of field and data processing efforts, and the resultant strengthening of quality control, is one of the chief benefits of computer assisted survey methodology.
  • [-] Coding

    ISR's coding section has had experience in constructing and using codes for a wide range of subjects. Areas experienced have included classification of fertility-related variables, diseases, drugs, religions, occupations, geographic locations, and others. ISR's university setting provides access to a variety of resources that may be drawn upon in constructing codes pertaining to specialized fields of knowledge.

    Plans for coding study data begin with input from the Data Processing staff during questionnaire construction. To assure that the data will be processed in the most useful form, the Principal Investigator is consulted about plans for analyzing the data before decisions about coding are made and the coding manual is prepared.

    Preliminary codes for open-end questions are developed before the start of coding using responses expected by the Principal Investigator or Study Director and responses given in questionnaire pretesting. Codes are further refined with additional responses from a sample of completed questionnaires, and drafted codes are tested on another sample of questionnaires. These codes with frequencies of occurrence are given to the Study Director or Principal Investigator for suggestions and approval.
  • [-] Data Confidentiality

    ISR ensures that the information provided by respondents is kept completely confidential. Throughout data collection, processing, and analysis, information that could identify a respondent is kept separate from data supplied by the respondent. Each study participant is assigned a unique ID number that appears on the data file in place of personally identifying information. In the case of paper surveys, a cover tear sheet is removed immediately from completed survey and kept in a separate file. The ID number then becomes the identifier for all material relating to a particular respondent. Even though identifying data and questionnaire data are stored separately, both sets of files are restricted to password-only access (or lock-and-key access, in the case of paper) from the System Administrator or Study Director. All ISR data collection personnel sign an assurance of confidentiality and are trained in the execution of this responsibility.

    Analyses and reports present aggregate data only, never information associated with any individual. At the end of each study, clean data files and tapes are created that include the edited data file and sequential ID, but no identifying information.
  • [-] Data Cleaning

    Data cleaning operations begin as soon as data are keyed. The Data Processing staff develop study-specific checking and updating programs that screen for invalid codes and logical inconsistencies. Marginal frequencies are reviewed to make sure that the cleaning programs are detecting all errors. Cleaning of a data file continues until the marginals are free of invalid codes and logical inconsistencies