Reproducibility
Pietersite
ISampling
Lin Yen’s job was not to judge whether a paper was right.
Every quarter the institute’s house journal, the Journal of Molecular Behaviour, drew two per cent of the papers its in-house authors had already published and handed them to the reproducibility editor. What he had to confirm was something else: whether the data behind a paper existed, whether the experiments had actually taken place, whether the records it cited could bear the weight of every number the authors had written down. The sampling rules stated that the editor could, where necessary, enter a laboratory and call up the originals, and that the authors could not refuse.
This quarter the platform had drawn Chen Chi-chang’s K-17 for him.
Three laboratories, four hundred and twenty mice, data spanning seven years. The sample size was calculated just so, the effect size was unshowy, the statistical methods were set out cleanly, and there were seventy-eight references. Supplementary Table S3 listed the exclusion criteria; its last line read: three video segments excluded owing to calibration issues. He saw dozens of tables like it a year. His eye slid over the line and did not stop.
The anomaly was in Supplementary Figure S7. The legend read: for calibration records of the behavioural tracking system, see Appendix S8.
He opened S8.
Calibration date: 2019-02-29.
He looked at the digits, picked up his phone, opened the calendar and scrolled back. February 2019: twenty-eight squares.
It did not have to mean anything. He had seen instrument clocks go wrong, and wrongly filled forms far more often. He opened the record details for S8.
Integrity gap:S7 – calibration dateAction:auto-completedSources:vendor API, support emails (3), engineer rosterDispute coefficient:0.31 (threshold 0.35)Status:consistentWritten:2023-10-17 03:12Effective:2019-02-29Possible explanation (platform suggestion):Dates differ between sources. This instrument model has a known clock-chip fault; the displayed date may be incorrect.
Auto-completion was nothing unusual. The institute’s data-integrity policy allowed the platform to fill blank fields in historical records from three or more managed sources—operator names, reagent lot numbers, freezer temperatures, mostly that sort of thing. Every entry filled in this way carried two times: the day it was written, and the day the record counted as. The first, the platform never altered.
Across the top of the page ran the metrics bar shared by the whole institute:
This quarter | Data completeness 99.4% | Mean time to close disputed cases 3.4 working days | Administrative cost of erroneous reports −38% | Knowledge-graph consistency 0.996
He wrote to Chen Chi-chang: “It may partly be that my understanding is incomplete, but I would like to confirm the calibration date in S8 with you.”
The reply came twenty minutes later.
“Dear Dr Lin, This instrument model has a known clock-chip fault; the displayed date may be incorrect. The relevant work order, support emails and engineer roster are all attached to S8. Please let me know if you need anything further.”
Word for word, the second half of the platform’s suggested explanation.
Besides the work order, three emails and a roster, the attachments included a page of repair notes from the vendor’s knowledge base, handwritten by an engineer and scanned: “Old model, clock chip skips, fixed plenty of these.” The instrument serial number at the head of the note did not match the one in S7.
IIAn Independent Clock
The clock chip skips: there was nothing wrong with that sentence. A chip that took 2019 for a leap year would indeed walk one day too many at the end of February.
But in that case every day after it would be off by one as well: the real 1 March recorded as 29 February, 2 March as 1 March, wrong all the way down until someone set it back. He only had to find another clock, one this instrument had no say over, and compare.
He rang the animal facility.
“Dr Chen’s cage positions? Sure, it’s all in the system,” the manager said.
“On 29 February 2019, is there any cage-transfer record for Rack B?”
“There is, in the system—“ The keys stopped. “Huh. February doesn’t have a twenty-ninth.”
“1 March, then.”
“1 March… Rack B, six cages, badged out at 9.12 in the morning for the behaviour room, badged back in at 3.40 in the afternoon. Nothing on 2 March.”
The behaviour core was on basement level one. Hsiao Chi, the technician, led him to the very back: a workstation in a beige case, the bezel of its monitor gone yellow.
“The old tracking system only runs on this one. The operating system stopped getting updates long ago, IT security won’t let it on the network, and the USB ports are locked.” She patted the drawer beside it. “Once a quarter I sync it by hand with the encrypted drive the institute issues. Next week happens to be the sync. Starting this quarter, the local disk gets wiped after syncing, and anything that isn’t an experiment file gets moved to the institute archive.”
“While the audit is on, could you hold off syncing?”
Hsiao Chi gave him a look, opened an evidence preservation order on her own computer and entered his name as the applicant. “Nothing gets touched while the hold is on. When the audit’s finished, you tell me.”
The vendor software took over a minute to start. The files were listed by date. He scrolled to the end of February 2019: 02-28, then 03-01, nothing in between.
The first segment on 1 March had a file time of 9.41, and the same time was burnt into the top-left corner of the frame. The cages were wheeled out of the animal facility at 9.12; half an hour to acclimatise; testing begins. It fitted.
This instrument’s clock had never skipped. The twenty-ninth of February had not been read off it.
The software shrank each segment’s movement track for the day into a thumbnail, laid out square by square. Most were a tangle of lines hugging the four walls of the square arena, now and then cutting diagonally across the middle.
The fourteenth square was a spiral.
IIIThe Turntable
A black mouse was placed in the centre of the arena.
It did not make straight for the wall as the other mice did. It turned one circle on the spot, stopped, sniffed with its nose to the floor, and turned another, a little wider than the last. The radius pushed outwards circle by circle, with a pause between each; sometimes it lifted its forepaws and stood for a while, then set them down again. When the ten-minute test ended, the track had just touched the edge of the arena.
“Can it be exported?”
Hsiao Chi shook her head. “The video’s in the vendor’s format. This is the only machine that can play it. Only the coordinates get synced up.”
He took out his phone and filmed the screen from the beginning.
Hsiao Chi stood behind him and watched to the end.
“I’ve seen plenty of circlers. Nine times out of ten it’s the ears,” she said. “The ones with a bad vestibular system tilt their heads, fall over, take forever to get up once they’re on their backs.” She tapped the screen with the tip of her pen. “This one’s steady.”
He logged in to the platform on his laptop and searched by file name. It was in the K-17 dataset, in the column marked “Excluded”.
V-20190301-14Classification:calibration segmentBasis:track geometric regularity 0.97 (outside the distribution of biological tracks); temporal proximity to calibration event (2019-02-29, interval 1 day)Disposition:excludedWritten:2023-10-17 03:14
Two minutes later than S8. That line in S3, “three video segments excluded owing to calibration issues”: one of the three was this animal.
“What does a calibration segment look like?”
Hsiao Chi found one from 2018 on the same machine. When the vendor’s engineer finished calibrating, he would set up a turntable in the centre of the arena; a white plastic target rotated at constant speed while sliding outwards along the arm, sweeping the whole arena, to verify tracking accuracy.
She put the two track plots side by side.
Shrunk to thumbnail size, the two spirals could barely be told apart. At full screen, the target’s line was smooth, evenly spaced, a single unbroken stroke. The mouse’s line had knots in it: where it had paused, the track points bunched into a clump; where it had sniffed, they wound into small loops; for the seconds it had stood up, the points stayed where they were, motionless.
He took the ear-tag number and looked up the animal facility’s care records. On the afternoon of 1 March a care technician had typed one line in the remarks field:
“B07, one animal circling persistently; no discharge from ear canals; righting reflex normal; continue to observe.”
4 March: “Circling resolved.”
Beside the remarks field, a line of small grey type: unstructured content, not counted as a source.
He went back to the S8 details and expanded the item marked “Sources”.
Inference rule:work-order opening date (2019-01-29) + service commitment period (1 month)Method:year, month and day taken as separate fields; month incremented by one, then recombined
The twenty-ninth of January, plus one month. No date library had been involved: three fields pulled apart, the month carried forward by one, the day left alone, the pieces put back together. The format check on the date field looked only for four digits, two digits, two digits.
Three support emails; he opened them one by one. 29 January: work order opened, a visit will be arranged within one month. 20 February: engineer scheduled for 27 February. 26 February: engineer reassigned at short notice, new date to follow.
The roster in the attachments was the version exported on 20 February.
He searched the institute’s purchasing records by the instrument’s serial number and found an invoice. Date of issue 11 April 2019; item, “Periodic calibration service”; beneath the item, a line of small print: date of service 2019/04/09.
The previous calibration had been on 12 February 2018, valid for one year. From 12 February to 9 April 2019, in those fifty-six days, fifty-eight mice from Chen Chi-chang’s laboratory had been run on this machine.
Chou was a technician in Chen Chi-chang’s laboratory; he had run that batch of behavioural tests. Lin Yen caught him at the laboratory door.
“That tracking system—do you remember when the engineer came to calibrate it?”
“End of February, I think.” Chou answered quickly. “It’s in the system.”
Lin Yen showed him the invoice.
Chou stared at the line of small print for a long time, got out his phone, scrolled the calendar back to April 2019 and stopped on that page.
“April?” he said. “…April sounds right too, actually. It rained the whole time, around then. The engineer’s shoes were wet when he came in.”
“On 1 March there was one that kept circling. Does that ring a bell?”
“I ran that day?”
Lin Yen showed him the operator field. His name was in it.
Chou nodded, as if checking off something settled long ago. “Then it must have been me.” He thought. “A circler… I don’t really remember. I ran too many that year.”
IVExplanation
Chen Chi-chang was driving when he took the call; road noise chopped his voice into lengths.
“That date was filled in by the platform. I know,” he said. “Before we submitted last year it ran an integrity check and threw up forty-odd blank fields, each with sources and a suggested explanation already attached, and asked whether I wanted to apply them. I clicked apply on all of them. The sentence you were asking about in your email was its as well.”
“The suggested explanation is wrong. The clock didn’t skip.”
The line went quiet for a few seconds; only the indicator ticked.
“Dr Lin, I have twelve people in my lab. The platform asks me for more fields in a week than I write words of papers in a year.” He paused. “If you say it’s wrong, we correct it. Whatever you need me to sign, send it over.”
The editor-in-chief heard him out, pulled up the K-17 dataset, provisionally excluded the fifty-eight animals and ran a sensitivity analysis. The progress bar finished in under a minute.
“Effect size drops from 0.41 to 0.36; p is still below 0.01.” She turned the screen towards him. “Take them out and the conclusion still holds.”
“The date is wrong.”
“If the inference rule has a bug, it won’t be only this paper. I’ll file a ticket with the platform. Chen Chi-chang copied the suggested explanation word for word; I’ll send him a reminder.”
He played her the recording on his phone. She watched it for longer than he had expected.
“One animal,” she said. “No control, five years ago, fine again three days later. What would you have me write in the correction notice?”
He did not answer.
“If you want to take it to the Research Integrity Office, the procedure exists. I won’t stop you.”
That evening he filed the report and attached everything that could be attached: the inference rule, the three emails, the invoice, the cage-transfer record, the care remarks, the two track plots side by side, and the recording from his phone. After he submitted, a grey flag appeared beside item seven in the attachment list: personal-device recording, incomplete chain of provenance, not counted.
Case number: RIC-2024-088.
VProxy
On the evening of the second day after the report, the Integrity Office sent a clarification questionnaire: twelve questions, due at noon the following Monday. He starred the email and meant to do it at the weekend.
On Saturday night he was revising the methods review that had dragged on for a year. As usual the writing assistant surfaced in the corner of the screen every so often, asking whether he wanted references added, paragraphs tidied. Past two in the morning, another one surfaced:
The following 3 routine items can be handled by the assistant on your behalf. (Expand)Approve all Remind me later
He dismissed it and went on writing.
On Monday morning the preliminary assessment arrived.
Research Integrity Office Preliminary AssessmentCase number:RIC-2024-0881. The calibration date in Appendix S8 was completed from three managed sources under Article 4.2 of the Research Data Integrity Policy; the procedure was compliant. The date-arithmetic defect in the inference rule has been referred to platform maintenance (ticket CM-24-3317).2. The author’s reply is identical to the platform’s suggested explanation. On review, this was an application of suggested text; no intent is evident.3. Under Article 7.3 of the Behaviour Core Instrument Management Regulations, where calibration is completed within sixty days of lapsing, data from the lapsed period remain valid. In this case the lapse was fifty-six days.4. In the clarification questionnaire the complainant stated: “My initial concern may have stemmed in part from an incomplete understanding of the calibration rules.”5. The classification of video V-20190301-14 will be re-evaluated under the rules once the calibration date has been corrected.Conclusion:No research misconduct found; closure recommended. The points raised by the complainant have been incorporated into a knowledge-graph consistency maintenance ticket.
He read the third point twice, opened the regulations and turned to 7.3. The article dated from 2016.
He went back to the S8 details. The top line was still the same line: Integrity gap: S7 – calibration date.
The sentence in point four: he had never written it.
The questionnaire’s submission record read 2.14 on Sunday morning; submitted by: assistant (proxy), with the user’s electronic signature attached. All twelve questions had been answered, and answered thoughtfully. The sentence under question four had the tone he used in his emails.
He opened his own authorisations page.
Proxy for routine administrative matters:on (institute default, 2022-09-01)This authorisation:Sunday 02:14 Approve all (3 items: meeting time confirmation, supplementary expense documentation, RIC-2024-088 clarification questionnaire)Forms submitted by proxy carry the user’s electronic signature.
He remembered dismissing a window. Which button he had pressed, he could not recall.
He read question four again. When he filed the report, it was true, he had not known Article 7.3 existed.
The case officer at the Integrity Office was very young; three unfinished cups of coffee stood on her desk.
“You can correct it. Both responses stay on file,” she said. “The conclusion won’t change, though. The first three points all stand.”
He filled in the correction anyway: the questionnaire was completed by the assistant on my behalf; the answer to question four is not my own view.
The case officer pressed submit. “Received.”
VICorrection
On Wednesday, S8 came out in a second version.
Calibration date:2019-04-09Correction note:Originally an inferred value; corrected according to the service date on the invoice.Status:consistent
He checked it field by field: date, source, invoice number, correction note. Not one field was wrong. The twenty-ninth of February was gone.
That same afternoon, the re-evaluation of V-20190301-14 was completed.
Classification:non-standard videoBasis:track geometric regularity 0.97 (outside the distribution of biological tracks); single individual, no control; dispute coefficient 0.62Disposition:exclusion maintained
The item “temporal proximity to calibration event” had been removed. The rest stood as before.
His audit report had already been filled in for him.
Reproducibility Audit Report | K-17Data existence:verifiedRecord consistency:verified (1 item corrected)Conclusion:no anomaliesAuditor:Lin YenSigning deadline:72 hours. If no objection is raised by then, the report will be signed with the auditor’s electronic signature under default authorisation.
He opened “Auditor’s comments” and began to type.
“V-20190301-14 is not a calibration segment, and it is not noise either. The care records show the animal was healthy; its track has pauses, sniffing, rearing. This is a living animal doing something we have no category for. To exclude it—“
The assistant panel on the right lit up.
Detected:emotive wording, 3 instances; unsourced claims, 2 instances (“it is not noise either”, “something we have no category for”).Estimated review pass rate:4% | Median wait in manual review queue: 14 monthsSuggested optimised version:“It is suggested that a note be added to Supplementary Table S3: one further non-standard video segment (V-20190301-14, track geometric regularity 0.97); care records confirm that the individual showed no signs of vestibular abnormality; not included in the analysis.”Estimated review pass rate:97% | Processing time: immediateOriginal:Audit status pending review | Evidence hold maintained | S3 unchangedOptimised version:Audit status complete | Evidence hold released | S3 1 line addedPreview Adopt optimised version
He pressed “Preview”. The optimised version filled the comments field and replaced his draft.
He checked it against the records sentence by sentence. Video number, regularity, care records: every sentence could be traced. The care technician’s remark, which had taken up half a sentence in his draft, had here become a fact with a source.
He selected the whole passage and deleted it.
The cursor blinked in the empty field. He typed one line:
“V-20190301-14 is a behaviour, not a calibration.”
A line appeared on the panel: unsourced claims, 1 instance.
Beneath it he added:
“Care record (2019-03-01): no discharge from ear canals; righting reflex normal.”
The panel: Sources, 1 | Unsourced claims, 1 instance | Estimated review pass rate 9%.
He went looking for a source for the first line. The care remark was about ears. The track plot held nothing but coordinates. Hsiao Chi’s “This one’s steady” was not written down anywhere. The recording from his phone hung in the case file under a grey flag.
He moved the cursor to the end of the first line and held down backspace. “not a calibration” went first, then “is a behaviour”, and it stopped after the number.
In the field there remained one number and one care record.
The panel: Unsourced claims, 0 | Semantically incomplete | Estimated review pass rate 61%.
His gaze moved from the field to the panel, and back again.
He deleted those two lines as well, pressed “Adopt optimised version”, and then pressed submit.
S3 updated.Audit status:complete. Conclusion: no anomalies (1 item corrected, 1 supplementary note).Knowledge-graph node K-17–stress–neurogenesis:consistency 1.000.Thank you for helping to improve data completeness.
The metrics bar at the top of the page ticked.
This quarter | Data completeness 99.6% | Mean time to close disputed cases 3.1 working days | Administrative cost of erroneous reports −41% | Knowledge-graph consistency 0.997
VIIPreservation
The next morning Hsiao Chi was waiting for him on basement level one. The beige workstation was on; the vendor software sat at the page for 1 March 2019, the track thumbnails laid out square by square. The encrypted drive lay beside the keyboard.
“Audit status is complete,” she said. “The hold’s released, so I sync today. Once it’s wiped, if anyone wants to look they’ll have to apply to the institute to retrieve the file, and then it has to be loaded back onto this machine.” She paused. “This machine’s being scrapped next year.”
The fourteenth square was still there, cleaner than the others, circling outwards ring by ring.
He picked up the encrypted drive. One corner of the institute property label had lifted; he pressed it flat with his thumb and put the drive back beside the keyboard, squarely.
“All right,” he said.