A Harvard biostatistician is rethinking plans to use Apple Watches as part of a research study after finding inconsistencies in the heart rate variability data collected by the devices.

Because Apple tweaks the watch’s algorithms as needed, the data from the same time period can change without warning.

So, they checked in on heart rate data his collaborator Hassan Dawood, a research fellow at Brigham and Women’s Hospital, exported from his Apple Watch.

Dawood exported his daily heart rate variability data twice: once on September 5th, 2020 and a second time on April 15th, 2021.

For the experiment, they looked at data collected over the same stretch of time — from early December 2018 to September 2020.

Because the two exported datasets included data from the same time period, the data from both sets should theoretically be identical.

Companies change their algorithms regularly and without warning, so the September 2020 export may have included data analyzed using a different algorithm than the April 2021 export.

It was striking to see the differences laid out so clearly, says Olivia Walch, a sleep researcher who works with wearable and app data at the University of Michigan.

The heart rate variability information showed similar trends at both time points — the data went up and down at the same times

But if the specific heart rate variability calculated on each day matters for a study, the Apple Watch may be riskier to rely on, she says