4.12 · AHL · HL onlyPaper 1Paper 2Paper 3

Data collection, reliability and validity

In a minute

Good statistics start before any calculation: honest, precise questions, the right variables and the right data. You also group numerical data so a χ² test works, adjust the when you estimate , and judge whether a method is reliable and valid.

You'll be able to

  • Write survey questions that are unbiased, precise and structured, and spot ones that are not.
  • Choose the variables and the data that answer a question, and say why.
  • Group numerical data for a χ² test so every is greater than 55, and justify the grouping.
  • Find the degrees of freedom for a when parameters are estimated from the data.
  • Define and , and use test–retest, parallel forms, content and criterion-related checks.
Before you start3 quick questions on what this builds on
Check yourself
A model gives a category probability 0.150.15. Find the expected frequency of that category in 240240 observations.
Check yourself
Write down the degrees of freedom for a on a with 33 rows and 44 columns.
Check yourself
A reporter interviews the first 3030 people who walk past her desk in a train station. Which sampling method is this?

Revise first: 2.6 Fitting and testing models, 4.1 Sampling, data quality and outliers, 4.2 Data presentation and 4.11 Chi-squared tests and the t-test

Designing surveys and questionnaires

A survey is only as good as its questions. A badly worded question collects answers that don't show what you wanted to know, however carefully you analyse them.

Check every question for four things: bias, how personal it is, its structure and its precision.

Unstructured questions suit a small pilot survey, to learn what answers people give. A large survey needs structured questions, because their answers go straight into a table you can analyse and test.

Worked exampleA student council drafts four questions about the school café. (i) Don't you agree that the café's prices are too high? (ii) What is your family's yearly income? (iii) Do you eat at the café regularly? (iv) What do you think of the café? Name the problem with each question and rewrite it.

Try it first, then reveal one step at a time.

  1. (i) is biased: "Don't you agree" leads, and "too high" names one side. Rewrite: "How would you rate the café's prices? Very low / Low / About right / High / Very high".
Worked exampleA researcher asks 400400 teenagers "How much sleep do you get?" with the choices 4–6 hours, 6–8 hours and 8–10 hours. Explain the problems with this question and design better answer choices.

Try it first, then reveal one step at a time.

  1. The choices overlap: a teenager who sleeps exactly 66 hours fits two boxes. Use boundaries written with "less than".
Check yourself
A town council surveys residents about a planned skate park. Which question is unbiased?
Check yourself
A survey asks "How long was your journey to school this morning, in minutes?" Which set of answer choices is structured best? Set A: 0–10, 10–20, 20–30, 30 or more. Set B: less than 10, 10 to less than 20, 20 to less than 30, 30 or more. Set C: short, medium, long. Set D: 0–9, 10–19, 20–29.

Choosing variables and data

Real data sets hold far more variables than you need. Start from the question: it tells you which variable is the response, and which ones might explain it.

1. Keep a variable if it could plausibly affect the response and you have enough trustworthy values of it.

2. Drop identifiers, such as names and ID numbers, and variables with no link to the question.

3. With several numerical candidates that look linearly related to the response, find rr between each one and the response (4.4). Compare the size of rr, not its sign.

Choosing the data matters just as much. Use data from the population the question is about, measured in the same way under the same conditions. Leave out values that belong to a different situation, and say what you left out and why.

No analysis can fix data from a with . If the data don't represent the population, collect them again.

Worked exampleAn estate agent records 88 flats in one city. Price (€ thousand): 212,348,265,410,189,302,375,240212, 348, 265, 410, 189, 302, 375, 240. Floor area (m²): 53,79,66,98,47,81,88,5453, 79, 66, 98, 47, 81, 88, 54. Distance to the nearest station (km): 1.8,0.6,2.2,0.9,2.6,1.4,0.8,2.41.8, 0.6, 2.2, 0.9, 2.6, 1.4, 0.8, 2.4. Age of building (years): 35,12,8,40,23,15,5,3035, 12, 8, 40, 23, 15, 5, 30. The agent also has each owner's name. Which variable is most useful for predicting price with a linear model? Use your calculator to find rr for each candidate, and give the chosen rr to 3 s.f.

Try it first, then reveal one step at a time.

  1. The response is price. The owner's name is an identifier with no link to price, so drop it.
Worked exampleA city wants the typical weekday journey time of the 8 am bus on route 12 since a new timetable started. Times (minutes): Mon 34.234.2, Tue 36.836.8, Wed 35.535.5, Thu 57.957.9 (a burst water main closed a road), Fri 36.136.1, Sat 22.422.4, Sun 20.920.9, Mon 33.433.4. Choose the appropriate data and find the mean journey time.

Try it first, then reveal one step at a time.

  1. The question is about weekdays, so Saturday and Sunday belong to a different situation. Leave them out.
Check yourself
A school compares 6060 students' final exam marks with three variables. Hours of revision: r=0.64r = 0.64. Hours on social media per week: r=−0.79r = -0.79. Distance from home to school: r=0.08r = 0.08. Which variable has the strongest linear relationship with the mark?
Check yourself
A researcher asks whether daily screen time affects sleep for 14 to 16-year-olds. She has a national data set covering every age. Which data should she analyse?

Grouping numerical data for a χ² test

A χ² test works with counts in categories. Numerical data, such as ages, masses or the number of successes, must be grouped first.

The test can only be trusted when every expected frequency is large enough. In IB questions every expected frequency must be greater than 55.

Justify a grouping in words: the merged classes are adjacent on the number scale, so the new class is still one interval, and every expected frequency is now greater than 55. Merge as few classes as you can, so you keep detail.

Worked exampleA gym asks 131131 members their age and favourite class. Age 18–29: cardio 2121, weights 1515, yoga 1111. Age 30–44: cardio 1717, weights 2020, yoga 99. Age 45–59: cardio 88, weights 1111, yoga 44. Age 60 and over: cardio 22, weights 77, yoga 66. The gym will test whether favourite class is independent of age group. Show that the table must be regrouped, regroup it, and write down the degrees of freedom.

Try it first, then reveal one step at a time.

  1. Expected frequency == row total ×\times column total ÷131\div 131. Column totals: cardio 4848, weights 5353, yoga 3030.
Worked exampleA seed packet claims 70%70\% of its seeds germinate. A gardener plants 55 seeds in each of 8080 pots and counts how many germinate in each pot. Number germinating: 00 in 11 pot, 11 in 44 pots, 22 in 1818, 33 in 2323, 44 in 2121, 55 in 1313. Assume seeds germinate independently. Test the claim with a goodness of fit test at the 5%5\% level, using a binomial model. Give the p-value to 3 s.f.

Try it first, then reveal one step at a time.

  1. H0\mathrm{H}_0: the number germinating per pot follows B(5,0.7)\mathrm{B}(5, 0.7). H1\mathrm{H}_1: it does not follow B(5,0.7)\mathrm{B}(5, 0.7).
Check yourself
A goodness of fit test with no estimated parameters has 66 classes of a numerical variable. In order, their expected frequencies are 3.2,11.5,18.7,14.9,6.1,2.63.2, 11.5, 18.7, 14.9, 6.1, 2.6. Merge each small class with its neighbour, then write down the degrees of freedom.
Check yourself
Why do you merge a small class with an adjacent class, rather than with any other small class?

Degrees of freedom when parameters are estimated

In 4.11 the claimed distribution was fully given, so df=n−1\text{df} = n - 1 for nn categories. Often you choose a model but don't know its parameters, so you estimate them from the same data you then test.

Each estimated parameter pulls the expected frequencies closer to the data. So each one costs a degree of freedom.

Learn this
df=k−1−m\text{df} = k - 1 - m

Here kk is the number of categories after any merging, and mm is the number of parameters estimated from the data.

Binomial B(n,p)\mathrm{B}(n, p): nn is known from the set-up, and estimating pp gives m=1m = 1. Normal N(μ,σ2)\mathrm{N}(\mu, \sigma^{2}): estimating both μ\mu and σ\sigma gives m=2m = 2. A parameter stated in the question, such as a claim, is not estimated, so it costs nothing.

To estimate pp for a binomial model, divide the total number of successes by the total number of trials. Call the estimate p^\hat{p} ("p hat").

Worked exampleA quiz app gives each user 55 hard bonus questions. For 120120 users, the number answered correctly was: 00 by 1111 users, 11 by 2828, 22 by 4040, 33 by 2929, 44 by 1010, 55 by 22. A binomial model is fitted, with pp estimated from the data. Estimate pp, find the expected frequencies, and write down the degrees of freedom for a goodness of fit test.

Try it first, then reveal one step at a time.

  1. Total correct: 0×11+1×28+2×40+3×29+4×10+5×2=2450 \times 11 + 1 \times 28 + 2 \times 40 + 3 \times 29 + 4 \times 10 + 5 \times 2 = 245. Total questions: 5×120=6005 \times 120 = 600. So p^=245600=0.40833…\hat{p} = \frac{245}{600} = 0.40833\ldots.
Worked exampleAn orchard weighs 150150 apples. Mass mm (g): m<160m < 160: 99; 160≤m<170160 \le m < 170: 2525; 170≤m<180170 \le m < 180: 4040; 180≤m<190180 \le m < 190: 3434; 190≤m<200190 \le m < 200: 2323; m≥200m \ge 200: 1919. From the raw data, the mean is estimated as 181.4181.4 g and the standard deviation as 13.213.2 g. Test at the 10%10\% level whether a normal model fits the masses. Give the p-value to 3 s.f.

Try it first, then reveal one step at a time.

  1. H0\mathrm{H}_0: apple mass follows a normal distribution. H1\mathrm{H}_1: apple mass does not follow a normal distribution.
Check yourself
A normal model is fitted to reaction times, with the mean and standard deviation both estimated from the data. After merging there are 77 categories. Write down the degrees of freedom.
Check yourself
A factory says 15%15\% of its light bulbs fail within a year. The model B(4,0.15)\mathrm{B}(4, 0.15) is tested on boxes of 44 bulbs, with 44 categories after merging. What are the degrees of freedom?

Reliability and validity

A bathroom scale that always reads 22 kg too heavy gives the same reading every morning. It is consistent, but it is wrong. Consistency and correctness are different questions.

The scale is reliable but not valid, so a method can be reliable without being valid. A method that gives very different results each time can't measure anything accurately, so poor reliability also limits validity.

Three targets. Left: hits scattered all over the target, labelled not reliable and not valid. Middle: hits tightly grouped but away from the centre, labelled reliable but not valid. Right: hits tightly grouped at the centre, labelled reliable and valid. not reliable not valid reliable not valid reliable valid
Reliability is how tightly the results group. Validity is whether they hit what you meant to measure.

To check reliability, measure the same people twice and find rr between the two sets of scores. A strong positive correlation suggests the method is reliable.

Test–retest needs a gap long enough that people don't just remember their answers, but short enough that what is measured hasn't changed. Parallel forms avoids the memory problem, but two truly equivalent versions are hard to write.

A question may give a value of rr to compare with. If not, describe the correlation as strong, moderate or weak, and say what that suggests.

Worked exampleA physiotherapist scores 88 patients on a balance test out of 5050, then repeats the same test two weeks later. First: 43,36,47,41,33,45,37,4943, 36, 47, 41, 33, 45, 37, 49. Second: 42,37,46,44,34,47,35,4842, 37, 46, 44, 34, 47, 35, 48. Name the reliability check, find rr to 3 s.f. and comment on the reliability of the balance test.

Try it first, then reveal one step at a time.

  1. Same patients, same test, two different times: this is a test–retest check.
Worked exampleA language teacher writes two versions, A and B, of a 6060-word vocabulary quiz, using different words of the same difficulty. Ten students take both on the same day. A: 47,53,34,58,41,39,50,43,56,3647, 53, 34, 58, 41, 39, 50, 43, 56, 36. B: 45,55,37,54,42,35,52,40,57,3945, 55, 37, 54, 42, 35, 52, 40, 57, 39. Name the reliability check and say why the teacher used it rather than test–retest. Find rr to 3 s.f. and comment.

Try it first, then reveal one step at a time.

  1. Two equivalent versions taken by the same students: a parallel forms check.
Check yourself
A gym gives 1212 members a grip-strength test on Monday and the same test the following Monday, then finds the correlation between the two sets of results. Which check is this?
Check yourself
Six children take a reading test, then the same test a month later. First: 42,55,38,61,47,5042, 55, 38, 61, 47, 50. Second: 45,53,40,63,44,5245, 53, 40, 63, 44, 52. Find rr to 3 s.f.

Testing validity

Reliability is checked by repeating. Validity is checked by asking whether the test measures the right thing. You need to know two ways.

The criterion can be measured at the same time, such as a quick new test against a longer trusted one. It can also come later, such as an entrance test against final grades.

Worked exampleA driving school's 4040-question theory test is meant to show whether learners know the rules of the road. All 4040 questions are about road signs. Comment on the test's validity and suggest an improvement.

Try it first, then reveal one step at a time.

  1. The rules of the road also cover speed limits, right of way, parking and hazards. The test samples only one part.
Worked exampleA sports club designs a 55-minute step test as a quick fitness score. To check it, 88 athletes take the step test and a laboratory treadmill test of VO₂ max (ml/kg/min), the trusted measure of fitness. Step test: 62,48,75,55,81,40,68,5862, 48, 75, 55, 81, 40, 68, 58. VO₂ max: 44.1,38.5,47.9,45.6,52.3,33.0,43.2,41.844.1, 38.5, 47.9, 45.6, 52.3, 33.0, 43.2, 41.8. Name the validity check, find rr to 3 s.f. and comment.

Try it first, then reveal one step at a time.

  1. The new test is compared with an established measure of the same thing: a criterion-related validity check, with the treadmill test as the criterion.
Check yourself
A company compares the scores on its new 1010-minute job aptitude test with the same employees' performance ratings from their managers a year later. What does this check?
Check yourself
A kitchen scale always reads 5050 g more than the true mass. Which statement is correct?

Common mistakes

  • Giving a decision with no comparison or no context. Write the comparison, such as 0.0613<0.100.0613 < 0.10, then the decision about H0\mathrm{H}_0 and what it means.
  • Merging the expected frequencies but not the observed ones, or finding df from the original number of classes.
  • Using df=k−1\text{df} = k - 1 when parameters were estimated from the data. Subtract one more for each estimated parameter.
  • Saying a test is valid because it is reliable. A consistent method can still measure the wrong thing.
  • Giving a vague validity criticism such as "the sample is too small". Link it to what is measured: a self-rated fitness score may not reflect actual fitness.
  • Truncating rr or a p-value, such as 0.9200.920 for 0.92071…0.92071\ldots. Keep the GDC value and round to 3 s.f. at the end.

Exam tips

  • Describe a way to improve reliability or validity: name one concrete change and link it to the word asked. Reliability: repeat the measurement and average the results, or use a larger random sample. Validity: measure the intended thing directly, cover every part of it, or compare with an established measure.
  • Justify a grouping: say which classes you merged, that they are adjacent, and that every expected frequency is now greater than 55.
  • Write down the degrees of freedom with the reason, such as "66 categories, 22 parameters estimated, so df =6−1−2=3= 6 - 1 - 2 = 3".
  • State the conclusion of a χ² test and justify it: the p-value comparison, the decision about H0\mathrm{H}_0, and its meaning in context. A decision with no reason scores nothing.

Key takeaways

Learn thisGood survey questions are unbiased (neutral wording) and precise (one meaning, a stated time period). Structured answer choices don't overlap and cover every possible answer. Ask personal questions only if needed, and anonymously.
Learn thisChoose variables linked to the question. For a linear model, compare candidates by the size of rr, not its sign. Use only data from the population and conditions the question is about, and say what you left out.
Learn thisIn IB χ² questions every expected frequency must be greater than 55. Merge a small class with an adjacent class, observed and expected together, until every expected frequency is greater than 55, then count the categories again.
Learn thisGoodness of fit: df=k−1−m\text{df} = k - 1 - m, with kk categories after merging and mm parameters estimated from the data. Binomial pp estimated: m=1m = 1; normal μ\mu and σ\sigma estimated: m=2m = 2; parameters given in the question: m=0m = 0.
Learn thisReliability: the same results when the method is repeated under the same conditions. Validity: the method measures what it claims to. Reliable does not mean valid.
Learn thisReliability checks: test–retest (same test, same people, two times) and parallel forms (two equivalent versions, same people). A strong positive correlation between the two sets of scores suggests reliability.
Learn thisValidity checks: content (the test covers every part of what it should) and criterion-related (a strong correlation, in the expected direction, with an established measure of the same thing).

Practise this

10 past-paper questions on Sampling and data collection in the AI HL bank, each with a mark scheme and a worked explanation.

Practise Sampling and data collection →