Design Elements

Design Elements

Research designs are often divided up in terms of experiments, quasi-experiments, simulations, and field studies, with field studies further divided into surveys, ethnographies and archival studies. This is useful but mixes up design issues with data sources.

Another approach is to construct a typology based on the underlying design elements, such as whether the study includes an intervention, whether it collects data at multiple points in time, and what kind of variance or diversity there is in the variables.

Studies Involving Interventions

	Post Only	Pre- and Post-
Treatment only	Stupid Example. Give every student a workshop on taking SATs. Then have them take the SAT. If scores are high, conclude workshop was effective. Problems. How do you even know what "high" is unless you have a group to compare against.	Bad quasi-experiment Example. Give students SAT test. Give the students a workshop on taking SATs. Give students new SAT test. If significant improvement, conclude workshop was effective. Comment. At least you have something to compare against -- their own past performance. But you don't if they would have improved anyway.
Treatment and Control	Quasi-experiment Example. Make SAT workshop available for those who want it. Take SAT. If scores significantly higher for those taking workshop, conclude workshop was effective Problems. The students electing to take the workshop may have been the ones who were going to score higher on the SAT anyway.	Quasi-experiment Example. Give each student SAT test. Make workshop available for those who want it. Alls students take SAT again. If improvement in scores is greater for those taking workshop, conclude workshop was effective Problems. Those who elect to take workshop are more motivated. They were going to improve more than others anyway. Note. There doesn't have to be a deliberate intervention here. This is point of commonality with observational studies.
Randomized Assignment to Treatment and Control Groups	True Experiment Example. Randomly assign some students to take SAT workshop (others take different workshop) Administer SAT to all students. If those taking workshop score higher, conclude workshop is effective. Problems. By chance, one group might have had better learners than the other. The likelihood of this gets vanishingly small as sample size increases.	Really Good True Experiment Example (Ancova design). Administer SAT to all students. Randomly assign some students to take SAT workshop. Re-administer SAT to all students. If those taking workshop improve even more than others, conclude workshop is effective. Problems. Ultimately, you can't control for everything. If it were possible, you'd rather it was the same students in both control/treatment groups. But again, with large samples, sampling error gets very small

Observational Studies

	Cross-Sectional	Lagged Cross-Sectional	Repeated Measures
Sample on the independent variable	Stupid Example. You want to know what problems women face in the workplace, so you ask women to tell you about problems they have encountered. Problems. The outcomes might have been the same for men. No variance on gender, so can't tell.	Not even worth talking about
Sample on the dependent variable	Stupid Example. You want to know what makes a great leader. You select a sample of great leaders and measure a large series of traits to see what great leaders have in common. Problems. Traits in common may also be shared by non-great leaders. e.g., learning that all great leaders have two eyes and a nose doesn't really tell you anything	Not even worth talking about
Obtain sample with adequate variance in the independent and dependent variables	Standard Research Example 1 (Cross-Sectional). Worker happiness and productivity. Measure worker happiness and productivity in the same survey. If positively correlated, conclude that happiness makes workers productive Problems. How do you know the direction of causality? How do you know a third variable, such as office decor, isn't determining both happiness and productivity? Example 2 (faux longitudinal). Measure testosterone and dominance behavior at T1. Measure both again at T2. If increase in testosterone is associated with increase in dominance, conclude testosterone causes dominance Problems. Is this any better than cross-sectionally correlating testosterone and dominance at either time period? No. The unit of observation is change, and this is observed cross-sectionally.	Lagged Cross-Sectional Example 1 (lagged cross-sectional). Measure nutrition intake 12 hours prior to administering SAT test. If those who ate better also score higher, conclude that nutrition affects performance. Problems. Direction of causality no longer a problem, but no controls for individual differences. E.g, smarter students might be more likely to eat well and do better on test.	Lagged Pre/Post Example 1 (natural experiment). Administer SAT test. Measure nutrition intake in next 12 hours. Re-administer SAT test. If those who ate better improve their score more than those who ate poorly, conclude that nutrition affects performance. Problems. Those who ate better might have done so deliberately to score higher on test. Example 2 (lagged pre-post). At T1, administer SAT test and at same time measure ambitiousness. At T2, re-administer SAT test. If the more ambitious students improve more, conclude that ambition affects performance. Problems. Suppose the more ambitious students were more likely to be women. Is it ambition or gender that is accounting for the results?

Visits: