Data collection, reliability and validity
In a minute
Good statistics start before any calculation: honest, precise questions, the right variables and the right data. You also group numerical data so a χ² test works, adjust the when you estimate , and judge whether a method is reliable and valid.
You'll be able to
- Write survey questions that are unbiased, precise and structured, and spot ones that are not.
- Choose the variables and the data that answer a question, and say why.
- Group numerical data for a χ² test so every is greater than , and justify the grouping.
- Find the degrees of freedom for a when parameters are estimated from the data.
- Define and , and use test–retest, parallel forms, content and criterion-related checks.
Before you start3 quick questions on what this builds on
Revise first: 2.6 Fitting and testing models, 4.1 Sampling, data quality and outliers, 4.2 Data presentation and 4.11 Chi-squared tests and the t-test
Designing surveys and questionnaires
A survey is only as good as its questions. A badly worded question collects answers that don't show what you wanted to know, however carefully you analyse them.
Check every question for four things: bias, how personal it is, its structure and its precision.
Unstructured questions suit a small pilot survey, to learn what answers people give. A large survey needs structured questions, because their answers go straight into a table you can analyse and test.
Try it first, then reveal one step at a time.
- (i) is biased: "Don't you agree" leads, and "too high" names one side. Rewrite: "How would you rate the café's prices? Very low / Low / About right / High / Very high".
- (ii) is personal, and family income isn't needed to judge the café. Leave it out. If spending matters, ask: "How much did you spend at the café last week? Nothing / Less than €5 / €5 to less than €10 / €10 or more".
- (iii) is imprecise: "regularly" means daily to one student and monthly to another. Rewrite: "On how many school days last week did you buy food at the café? 0 / 1 / 2 / 3 / 4 / 5".
- (iv) is unstructured. It is fine in a pilot, but its answers are hard to count. Rewrite: "Overall, how satisfied are you with the café? Very dissatisfied / Dissatisfied / Neutral / Satisfied / Very satisfied".
- (i) is biased; (ii) is personal and not needed; (iii) is imprecise; (iv) is unstructured. Each becomes a neutral, precise question with consistent fixed answer choices.
Try it first, then reveal one step at a time.
- The choices overlap: a teenager who sleeps exactly hours fits two boxes. Use boundaries written with "less than".
- They don't cover every answer: someone who sleeps or hours has no box. Add an open class at each end.
- The question is imprecise: school nights or weekends, and over what period? State it.
- Better: "On a typical school night last week, how many hours did you sleep? Less than 6 / 6 to less than 7 / 7 to less than 8 / 8 to less than 9 / 9 or more".
- Now every amount of sleep fits exactly one box, so the counts go straight into a frequency table.
- The choices overlap and do not cover every value, and the time period is not stated. Use classes that do not overlap, with "less than" boundaries and open end classes, and name the time period.
Choosing variables and data
Real data sets hold far more variables than you need. Start from the question: it tells you which variable is the response, and which ones might explain it.
1. Keep a variable if it could plausibly affect the response and you have enough trustworthy values of it.
2. Drop identifiers, such as names and ID numbers, and variables with no link to the question.
3. With several numerical candidates that look linearly related to the response, find between each one and the response (4.4). Compare the size of , not its sign.
Choosing the data matters just as much. Use data from the population the question is about, measured in the same way under the same conditions. Leave out values that belong to a different situation, and say what you left out and why.
No analysis can fix data from a with . If the data don't represent the population, collect them again.
Try it first, then reveal one step at a time.
- The response is price. The owner's name is an identifier with no link to price, so drop it.
- GDC linear regression of price on each variable: floor area , distance , age .
- Compare sizes: , so floor area has the strongest linear relationship with price.
- Interpret: bigger flats cost more. Distance also matters (flats nearer a station cost more), but age has almost no linear link with price here.
Try it first, then reveal one step at a time.
- The question is about weekdays, so Saturday and Sunday belong to a different situation. Leave them out.
- Thursday's time came from a one-off road closure, not from the timetable. It isn't typical, so leave it out and report that you did.
- Use the five remaining weekday times: .
- Interpret: a typical weekday 8 am journey now takes about minutes. All values would give minutes, mixing in quiet weekends and a road closure.
Grouping numerical data for a χ² test
A χ² test works with counts in categories. Numerical data, such as ages, masses or the number of successes, must be grouped first.
The test can only be trusted when every expected frequency is large enough. In IB questions every expected frequency must be greater than .
Justify a grouping in words: the merged classes are adjacent on the number scale, so the new class is still one interval, and every expected frequency is now greater than . Merge as few classes as you can, so you keep detail.
Try it first, then reveal one step at a time.
- Expected frequency row total column total . Column totals: cardio , weights , yoga .
- Row 60 and over (total ): , , but for yoga. That is not greater than . Every other expected frequency is greater than .
- Ages are numerical, so merge the oldest group with the adjacent group 45–59 into "45 and over". Observed: (total ). Expected: , , .
- The first two rows give expected and . Every expected frequency is now greater than .
- The new table is , so .
Try it first, then reveal one step at a time.
- : the number germinating per pot follows . : it does not follow .
- Expected frequencies are for : , , , , , .
- Classes and are below , and even together () they are too small. Merge them with the adjacent class into "2 or fewer": expected , observed .
- Four categories remain. The claim gives , so nothing is estimated: .
- GDC goodness of fit with observed , expected and df : , p-value .
- , so reject . There is evidence at the level that germination doesn't follow the claimed model: far more pots than expected had or fewer seeds germinate.
Degrees of freedom when parameters are estimated
In 4.11 the claimed distribution was fully given, so for categories. Often you choose a model but don't know its parameters, so you estimate them from the same data you then test.
Each estimated parameter pulls the expected frequencies closer to the data. So each one costs a degree of freedom.
Here is the number of categories after any merging, and is the number of parameters estimated from the data.
Binomial : is known from the set-up, and estimating gives . Normal : estimating both and gives . A parameter stated in the question, such as a claim, is not estimated, so it costs nothing.
To estimate for a binomial model, divide the total number of successes by the total number of trials. Call the estimate ("p hat").
Try it first, then reveal one step at a time.
- Total correct: . Total questions: . So .
- Expected frequencies, with : (keep the full values on your GDC).
- Class has expected frequency , which is not greater than . Merge it with the adjacent class into "4 or more": expected , observed .
- Five categories remain, and one parameter, , was estimated: .
- A GDC test with df gives a p-value of . There is no evidence against the binomial model, so it is a reasonable model for these data.
Try it first, then reveal one step at a time.
- : apple mass follows a normal distribution. : apple mass does not follow a normal distribution.
- Expected frequency with . For example, . All six: , each greater than .
- Both and were estimated from the data: .
- GDC goodness of fit with the observed counts, the unrounded expected counts and df : , p-value .
- , so reject . There is evidence at the level that apple masses are not normally distributed: fewer apples than expected lie between and g, and more weigh g or over.
- Typing df instead would give a p-value of , and you would wrongly fail to reject .
Reliability and validity
A bathroom scale that always reads kg too heavy gives the same reading every morning. It is consistent, but it is wrong. Consistency and correctness are different questions.
The scale is reliable but not valid, so a method can be reliable without being valid. A method that gives very different results each time can't measure anything accurately, so poor reliability also limits validity.
To check reliability, measure the same people twice and find between the two sets of scores. A strong positive correlation suggests the method is reliable.
Test–retest needs a gap long enough that people don't just remember their answers, but short enough that what is measured hasn't changed. Parallel forms avoids the memory problem, but two truly equivalent versions are hard to write.
A question may give a value of to compare with. If not, describe the correlation as strong, moderate or weak, and say what that suggests.
Try it first, then reveal one step at a time.
- Same patients, same test, two different times: this is a test–retest check.
- GDC linear regression with the first scores as and the second as : .
- is a strong positive correlation: patients who scored high the first time scored high again.
- So the balance test appears reliable. That alone doesn't make it valid: it could consistently measure something other than balance.
Try it first, then reveal one step at a time.
- Two equivalent versions taken by the same students: a parallel forms check.
- With the same quiz twice, students might remember the words and score higher the second time. Different words avoid that.
- GDC linear regression with A as and B as : .
- A strong positive correlation: the two versions rank students in much the same order. The quiz appears reliable, and either version can be used.
Testing validity
Reliability is checked by repeating. Validity is checked by asking whether the test measures the right thing. You need to know two ways.
The criterion can be measured at the same time, such as a quick new test against a longer trusted one. It can also come later, such as an entrance test against final grades.
Try it first, then reveal one step at a time.
- The rules of the road also cover speed limits, right of way, parking and hazards. The test samples only one part.
- So it lacks content validity: a learner could score highly while knowing nothing about right of way.
- It may still be reliable: a learner would probably get a similar score again. Reliability doesn't fix a validity problem.
- Improve it: list every topic in the rules of the road, then write questions on each topic, roughly in proportion to its importance. Ask experienced instructors to check the coverage.
- The test lacks content validity, because it covers only road signs. Add questions on every topic in the rules of the road.
Try it first, then reveal one step at a time.
- The new test is compared with an established measure of the same thing: a criterion-related validity check, with the treadmill test as the criterion.
- GDC linear regression with step score as and VO₂ max as : .
- is a strong positive correlation: athletes who score high on the step test also have a high VO₂ max.
- So the step test appears to have criterion-related validity for athletes like these. With only athletes, the club should check it on a larger sample before relying on it.
Common mistakes
- Giving a decision with no comparison or no context. Write the comparison, such as , then the decision about and what it means.
- Merging the expected frequencies but not the observed ones, or finding df from the original number of classes.
- Using when parameters were estimated from the data. Subtract one more for each estimated parameter.
- Saying a test is valid because it is reliable. A consistent method can still measure the wrong thing.
- Giving a vague validity criticism such as "the sample is too small". Link it to what is measured: a self-rated fitness score may not reflect actual fitness.
- Truncating or a p-value, such as for . Keep the GDC value and round to 3 s.f. at the end.
Exam tips
- Describe a way to improve reliability or validity: name one concrete change and link it to the word asked. Reliability: repeat the measurement and average the results, or use a larger random sample. Validity: measure the intended thing directly, cover every part of it, or compare with an established measure.
- Justify a grouping: say which classes you merged, that they are adjacent, and that every expected frequency is now greater than .
- Write down the degrees of freedom with the reason, such as " categories, parameters estimated, so df ".
- State the conclusion of a χ² test and justify it: the p-value comparison, the decision about , and its meaning in context. A decision with no reason scores nothing.
Key takeaways
Practise this
10 past-paper questions on Sampling and data collection in the AI HL bank, each with a mark scheme and a worked explanation.