TakeMeter

Staff solution — a fictional running forum.

---

Unit 5

What This Does

It sorts posts from a running forum into three kinds: analysis, hot_take, and reaction. The point is to tell an argument from an assertion — a post that gives you a checkable reason to believe it, versus one that just sounds certain — and to keep both of those separate from someone reporting how their morning went. 205 posts, hand-labelled, fine-tuned on distilbert-base-uncased.

Label Taxonomy

analysis

Makes an argument backed by a specific, checkable fact that carries the argument.

"Pulled the splits from the last five editions. The winning time has moved by 40 seconds total in ten years, but the 100th-place time is four minutes faster. The depth changed, the front didn't."

"Compared the two plans properly instead of guessing. The 5-day one has 38 miles a week with two quality days; the 6-day has 41 with the same two. The extra day is four easy miles, not more work."

hot_take

A confident general claim with no support offered — even if it mentions a number.

"Heart rate zones are astrology for runners."

"Times are down about 3% since the shoe rules changed, so obviously the shoes are the whole story and the sport has a problem."

reaction

An immediate personal response to something that just happened. Feeling rather than argument.

"Sub 20. SUB TWENTY. Nineteen fifty-eight and I'm sat on a wall not entirely sure it happened."

"Got up at 5, drove ninety minutes, and it's cancelled. Standing in the car park in the rain reading the email."

The hardest boundary

analysis vs hot_take, and it isn't close. Both are opinions about running stated with confidence; the only thing separating them is what the facts are doing.

The decision rule: a post is analysis only if the checkable fact carries the argument. If the number is decoration on a conclusion the writer had already reached, it is a hot_take with a statistic in it.

The worked pair:

"Super shoes have ruined racing, times are down 3% since 2017." → hot_take. The 3% is a flourish; "have ruined racing" was the position before the number arrived.

"Times are down 3% since 2017, and the shoe rule changed in 2017. That's the most likely single cause." → analysis. Take the number away and there is no claim left.

I wrote this rule after Milestone 2 and I still applied it inconsistently. That is not a throwaway admission — it is the thing unit 6 found twice, from two directions.

The Dataset

Source: a running forum. 205 posts, collected by hand, kept whole.

Label	Count	Share
reaction	74	36.1%
hot_take	67	32.7%
analysis	64	31.2%

No label above 70%, comfortably — the file is close to balanced by construction, because I went looking for thin labels rather than taking what came.

How I labelled it. 20 posts cold, no tools, marked cold in the note column — those were first and they are the honest sample. The rest by hand against the definitions. I did not use a model to pre-label: on a distinction this fine I'd have been correcting its boundary calls all the way through, and its boundary is exactly the thing I was trying to pin down.

One thing I did deliberately that shaped everything after it. I made sure the data would break the shortcut. Several analysis posts contain no digits at all and argue from a named comparison; several hot_take posts quote a percentage or a year. If I'd let "has a number" and "is analysis" line up, a model could score well having learned nothing about my taxonomy, and I'd never have known.

Three hard cases

1. "Three of us ran the same session on the same day, different shoes, within 10 seconds of each other over 5 miles. I'm not saying shoes do nothing, but whatever they do was smaller than the noise between three people."

Three runners on one day is barely evidence. But the observation is checkable and the conclusion is drawn from it rather than decorated with it, and the writer states their own limits. Called it analysis — and I'd note the rule says nothing about sample size, which is a gap.

2. "Every injury thread on this forum is the same injury and the same three replies."

Reads like an observation, and in principle you could count it. But no count is given — the writer is asserting a pattern, not showing one. Called it hot_take. This one I got wrong in the staff check, which is in the Agreement Report.

3. "Just realised the shoes I set every PB in this year are the ones I bought by accident in the wrong size."

A shoe observation, so it pulls toward the shoe debate and hot_take. But it is the writer reporting their own realisation, now, with no general claim attached. Called it reaction — and this is the one where I think my rules genuinely don't say, because rule 2 assumes the choice is between a general claim and a personal report, and this is a personal report about a topic that is usually a general claim.

The Training Run

Setting	Value
Base model	distilbert-base-uncased
Epochs	3
Learning rate	2e-5
Batch size	16
Seed	42
Split	70/15/15 stratified — 143 train, 31 val, 31 test

Nothing changed from the defaults. Hyperparameters are vocabulary this week and I had no diagnosis pointing at any of them; changing one to look busy would have made the unit 6 comparison harder to read, not easier.

Worth noting for next week: 31 test posts means about 10 per label. One misclassification moves a per-label F1 by roughly 0.05. I wrote my criteria knowing that and I still don't think I allowed enough for it.

How I Used AI

1. I asked Claude to generate eight posts sitting exactly on my analysis/hot_take boundary, then tried to label them with my own definitions. I could not cleanly place three of them, and all three had the same shape: a real statistic attached to a conclusion that clearly came first. That is what produced the "does the fact carry the argument?" wording — the rule came out of failing the test, not out of writing a rule.

2. I pasted my five criteria in and asked it to tell me, for each target, one reason it might turn out too easy and one reason too hard. For criterion 5 it pointed out that a model getting everything right would score a gap of zero and fail. I thought that was a hypothetical. It happened, on seed 7, and it is the revision in criteria.md.

Disclosure: no pre-labelling. Every one of the 205 rows was labelled by me.

---

Unit 6

Baseline vs. Trained

python baseline.py --trained results.json, on the 31 held-out posts in test_split.csv — the same posts, both models.

Measure	Baseline	Trained	Difference
Overall accuracy	0.323	0.839	+0.516
Macro F1	0.209	0.839	+0.630
F1 — analysis	0.462	0.842	+0.381
F1 — hot_take	0.000	0.857	+0.857
F1 — reaction	0.167	0.818	+0.652

What I predicted before I looked: "It'll do alright on reaction because those posts announce themselves, and it'll struggle to tell analysis from hot_take, because that distinction is about what the facts are doing rather than what words are in the post."

Half right, and wrong in an interesting way.

What the gap actually means. The baseline scores 0.000 on hot_take — not "poorly", zero. It never once identified one correctly. Reading the predictions, it puts almost everything opinionated into analysis, because a hot take and an analysis are about the same subjects in the same register and the only thing separating them is a relationship between the claim and the evidence. Zero-shot entailment cannot see that from three sentences of definition.

So the labelling bought the one distinction the task is actually about. And it undercut my prediction: I guessed reaction would be the baseline's easy case, and it scored 0.167 there while managing 0.462 on analysis. It was never finding analysis — it was defaulting to it.

Run Log — Before

Three seeds, from results_three_seeds_before.json. Device: cpu (torch 2.13.0, MAX_LENGTH = 128).

Criterion	Target	Seed 42	Seed 7	Seed 2024	Verdict
1. Overall accuracy	≥ 0.70 all seeds	0.806	0.968	0.903	MET
2. Every label F1	≥ 0.70 all seeds	0.750 (min)	0.947	0.824 (min)	MET
3. Beats baseline	≥ +10 pts	+0.516	—	—	MET
4. analysis↔hot_take confusions	≤ 4 of ~30	3	1	3	MET
5. Confidence gap (thirds)	≥ 0.15 all seeds	+0.500	+0.100	+0.300	MISSED

Accuracy: 0.806, 0.968, 0.903 — a 16.1-point spread. Nothing changed between those three runs except which posts landed in the test split. Seed 7 got 30 of 31 right; seed 42 got 25.

Confusion matrix

Seed 42:

true \ predicted	analysis	hot_take	reaction
analysis	9	0	1
hot_take	3	6	1
reaction	1	0	10

Seed 2024, same shape:

true \ predicted	analysis	hot_take	reaction
analysis	10	0	0
hot_take	3	7	0
reaction	0	0	11

My biggest off-diagonal number, and what it means: 3 real hot_take posts called analysis, in both of those seeds. And analysis called hot_take zero times in either — in fact zero times across all three before-seeds. That is not confusion, it is a one-way lean: the model over-assigns analysis and never once makes the opposite mistake. Something is telling it that an opinionated post with a fact in it is an argument.

Agreement Report

30 posts, labelled by me under the definitions in data/staff_taxonomy.md — the staff rules, not my own — then compared with python agreement.py.

Agreement rate: 25 / 30 = 83%

Disagreements

1.

The post: "Three of us ran the same session on the same day, different shoes, within 10 seconds of each other over 5 miles. I'm not saying shoes do nothing, but whatever they do was smaller than the noise between three people."

Staff said / I said: analysis / hot_take

My call, and why: I'll argue mine. Rule 1 asks whether a checkable fact carries the argument. Three runners on one morning is an anecdote, and the writer's own hedge — "I'm not saying shoes do nothing" — signals they know it doesn't carry. Staff read the observation as load-bearing; I read the hedge as an admission it isn't. Neither reading contradicts the written rule, which is silent on how much evidence counts as evidence.

Which pile: (c) — the rule is ambiguous here and my reading is defensible.

2.

The post: "Times are down about 3% since the shoe rules changed, so obviously the shoes are the whole story and the sport has a problem."

Staff said / I said: hot_take / analysis

My call, and why: Staff is right and I have no defence. This is the worked example in the rules almost word for word. "So obviously" does the work; the 3% is decoration. I saw a percentage and stopped reading.

Which pile: (a) — the rule covered it and I applied it loosely.

3.

The post: "Track sessions are wasted on anyone running slower than 8 minute miles. You'd get more out of just running more."

Staff said / I said: hot_take / analysis

My call, and why: Staff is right. "8 minute miles" is a threshold inside the assertion, not evidence for it. Same mistake as case 2, eleven posts later.

Which pile: (a).

4.

The post: "Every injury thread on this forum is the same injury and the same three replies."

Staff said / I said: hot_take / analysis

My call, and why: Staff is right, and this one is the most interesting. It is checkable in principle — you could go and count. But the writer didn't, and rule 1 is about what the post does, not what someone could do afterwards. I labelled the same shape of post hot_take in my own 205 and analysis here.

Which pile: (a), though it points at a genuine gap: my rule says "checkable" without saying "and shown".

5.

The post: "Just realised the shoes I set every PB in this year are the ones I bought by accident in the wrong size."

Staff said / I said: reaction / hot_take

My call, and why: I'll concede but not fully. Staff's reading is better: it reports the writer's own realisation, now, which is rule 2. I read the dry undercutting of shoe discourse as an implied general claim. On reflection an implied claim isn't a claim.

Which pile: (b) — the rule doesn't say what to do with a personal report that is obviously commenting on a general debate.

What the pattern in my disagreements tells me: three of the five are the same error in the same direction — I called a hot_take an analysis because it contained a number. That is exactly the error the confusion matrix shows the model making, in exactly the same direction, in every before-seed.

I did not go looking for that. The matrix and the agreement check are independent instruments and they found the same thing, which is the strongest evidence in this submission.

Verdicts and Diagnoses

#	Criterion	Verdict	How I decided
1	Accuracy ≥ 0.70 all seeds	MET	0.806 / 0.968 / 0.903. Lowest is 10 points clear.
2	Every label F1 ≥ 0.70 all seeds	MET	Worst single label-seed is hot_take at 0.750 on seed 42.
3	Beats baseline by ≥ 10 pts	MET	+51.6 points on the seed-42 split. Not close.
4	analysis↔hot_take ≤ 4	MET	3 / 1 / 3, all under the target.
5	Confidence gap ≥ 0.15 all seeds	MISSED	+0.500 / +0.100 / +0.300. Seed 7 fails it, so the target doesn't hold.

Diagnosis — criterion 5

Cause: the criterion, not the model. The criterion compares how often the most-confident third of predictions is right against the least-confident third. On seed 7 the model got 30 of 31 right — so the least-confident third contains one error, and both thirds score near 100%. The gap collapses to 0.100 not because confidence is uninformative but because there is almost nothing left to be wrong about.

That is a measurement problem, and a nasty one: the criterion gets harder to meet the better the model does. A perfect run would score a gap of exactly 0.00 and fail outright. No change to the model fixes that — a better model fails it harder.

So the revision in criteria.md compares mean confidence on wrong answers against right ones, which stays defined however few errors there are. Scored that way the before-run is +0.070 / +0.078 / +0.051 against a 0.05 floor — it passes, narrowly, on all three.

Two wordings, two verdicts, and I'm recording both. The original is MISSED and stays MISSED; the revision would have been MET. I'm not claiming the revision as a pass — it exists so that next time there is one answer instead of two, and the fact that it flips the verdict is exactly why the original line stays visible above it.

The finding that isn't a criterion

Four of my five criteria are MET, and I don't think that's good news. The brief warned about this and I did it anyway: my targets were safe. Criterion 1's floor of 0.70 was cleared by 10 points on the worst seed; criterion 3's 10-point margin was cleared by 52.

The number that actually taught me something isn't in the table. Accuracy swung 16.1 points across three seeds — 0.806 to 0.968 — on identical data and identical settings, with only the split changing. Any of my criteria measured on one seed would have been a coin toss, and a submission reporting seed 7 alone would have claimed a 97%-accurate classifier. If I'd written a criterion about stability rather than about level, I'd have had a target worth missing.

The diagnosis the two instruments agree on

Not a criterion miss, but the real problem with the system. The matrix shows hot_take → analysis three times in seed 42 and seed 2024, once in seed 7, and the reverse zero times in any of them. The agreement check shows me making the same directional error on 3 of 5 disagreements. The model learned the shortcut I taught it: a post with a number in it is an argument. I know I taught it that, because I made the same mistake on somebody else's posts an hour earlier.

That is a labelling problem, not a model problem, and it points at exactly one fix.

The Improvement

What I changed: I re-applied rule 1 to every row currently labelled analysis and asked one question — does a checkable fact carry this, or is a pattern merely asserted? Six rows moved from analysis to hot_take. Nothing else changed: same taxonomy, same definitions, same settings, same seeds, not one reaction row touched.

The six are the ones that assert a pattern without ever showing it — "Almost all of them land in January", "Half the records up there have been beaten twice", "Our fastest 10K times are all from the same three courses". Every one is disagreement 4 in a different costume.

Label counts after: reaction 74, hot_take 73, analysis 58.

Why I picked it: two independent instruments pointed at the same boundary in the same direction. Adding examples would have taught the model more of the same confusion, and changing epochs would have been changing a number because I could. The diagnosis named my labels, so I fixed my labels.

Run Log — After

From results_three_seeds_after.json, same three seeds, same device.

Criterion	Target	Seed 42	Seed 7	Seed 2024	Verdict
1. Overall accuracy	≥ 0.70 all seeds	0.839	0.806	0.839	MET
2. Every label F1	≥ 0.70 all seeds	0.818 (min)	0.714 (min)	0.783 (min)	MET
3. Beats baseline	≥ +10 pts	+0.516	—	—	MET
4. analysis↔hot_take confusions	≤ 4 of ~30	1	4	3	MET
5. Confidence gap (thirds)	≥ 0.15 all seeds	+0.200	+0.400	+0.400	MET

Did it help?

It depends entirely on what I was measuring, and that is the finding.

The headline number got worse. Mean accuracy across three seeds fell from 0.892 to 0.828. If I had one number to report, the relabel made my model worse and I'd have stopped here.

The measurement got dramatically better. The spread across seeds collapsed from 16.1 points to 3.2 — 0.839 / 0.806 / 0.839. Before the change I could not have told you what my model's accuracy was; now three independent draws agree to within three points. I traded about six points of average accuracy for a number I can believe, and I'd make that trade again.

Some of the old accuracy was the shortcut working. The analysis class was partly "posts with numbers", which is an easy thing to learn and scored well on splits where that heuristic happened to hold. Seed 7's 30-of-31 is what that looks like when the draw is kind. Taking the shortcut away cost accuracy because the shortcut had been earning some.

Criterion 5 flipped to MET, and I don't trust it. The gap is now 0.200 to 0.400 on all three seeds — but only because the model is less accurate, so the least-confident third has real errors in it again. The criterion was satisfied by making the model worse. That is a damning thing to be able to say about a criterion, and it justifies the revision better than my revision note did. Under the revised wording — confidence on wrong answers against right ones — seed 42 comes in at +0.044 against a 0.05 floor and it would be MISSED. Two wordings, opposite verdicts, again.

Criterion 2 nearly broke and I want that on the record. analysis on seed 7 came in at 0.714 against a 0.70 floor. Moving six rows out made analysis the smallest class at 58, so it gets about 9 test posts and each one is worth roughly 0.11 of an F1. One more misclassification and this is MISSED. It passed, and it passed by less than one post.

What's Still Broken

Criterion 2 is one post from failing. 58 analysis rows means ~9 in the test split. What I'd do is collect 30 more analysis posts — specifically ones that argue from a named comparison with no digits in them, since that is the shape my model is worst at and the shape my corpus has fewest of. I stopped because that is a new labelling session, not a change to an existing one, and the rule this week is one change.

My criteria measure level, not stability, and level was the less useful thing. The single most informative number I produced — the 16.1-point spread — corresponds to no criterion at all. Next time one of the five would be "the spread across three seeds is under 5 points", which before the improvement I'd have missed badly and after it I'd have met at 3.2.

Criterion 5 is still wrong in both directions. Under the original wording it is failed by a good model and passed by a worse one. Under my revision it flips the other way on this data. I don't think either wording is right yet, and the honest position is that I don't currently know how to measure "confidence means something" on 31 test posts. That is worth saying plainly rather than picking whichever wording flatters the run.

The rule still doesn't say "and shown". Disagreement 4 found it, my six relabelled rows are all instances of it, and I fixed the rows without fixing the sentence. The definition in label_definitions.txt still reads "checkable", which is what let me label the same shape two ways in the first place. That is a one-line fix I deliberately did not make this week, because changing a definition mid-test would have made the before and after incomparable.

What the model learned versus what I meant it to. I meant analysis to be "the fact carries the argument". What it learned first was "the post contains a number", and the whole visible history of this project is that shortcut being found and partly removed — the one-way confusion matrix, the three identical disagreements, the accuracy drop when I took it away. It is partly removed and not gone: seed 7 still put 4 real analysis posts into hot_take after the fix, which is the same boundary failing in the other direction now that analysis is the thin class.