January 2003
Patient Noncompliance with Paper Diaries
Medical treatment and research frequently rely on patients' reports of their experience of symptoms, medication-taking, adverse events, etc. Because retrospective recall is subject to error and bias, diaries are often used to collect data about real-world experience: e.g., diaries are used in 25% of all pharmaceutical clinical trials. However, a recent study showed that compliance with paper diaries is poor, thus negating the advantage of diaries and invalidating the paper diary data.
Compliance with paper diaries has been hard to assess, because the time of diary completion is not objectively documented. A study, sponsored by the National Cancer Institute (NCI) with in-kind support by invivodata, inc., was designed to document actual patient compliance with paper diaries. The investigators assessed compliance with paper diaries using a newly-developed instrumented paper diary that electronically tracked diary use unobtrusively. Compliance rates were compared with those achieved using an electronic diary implementing unique measurement principles based on the science of Ecological Momentary Assessment (EMA). EMA encompasses methods for collecting valid and reliable patient self-report data in real time and in the patients' natural environment.
Eighty chronic pain patients were randomized to use an instrumented paper diary (IPD) or an electronic patient experience diary (PED) to record pain over three weeks. They were to complete a pain assessment on three occasions each day (10 am, 4 pm, and 8 pm). The team analyzed compliance with timely completion of scheduled assessments: both reported compliance (based on the time and date written on completed diary cards) and actual compliance (based on a corresponding electronic record within ±45 minutes of the appointed time). The PED allowed entries to be made only during a window of ±15 minutes around the scheduled assessment times, and incorporated features to enhance compliance, such as sounding an alarm at the time of assessment. The diary data were collected at weekly clinic visits.
Reported compliance in the IPD group was very high: patients submitted diary cards corresponding to an average of 94% of assessment occasions. However, actual compliance was much lower, averaging 20% (95% CI: 14%-25%). Thus, 74% of paper diary cards were falsified. On 32% of study days, the IPD had not been opened at all, but patients submitted almost all the expected diary cards for those days. The IPD records indicated that patients back-filled diary cards, and that many patients completed diary cards in advance of the target time. In the patients randomized to use the ED, actual compliance was 94% (95% CI: 92%-96%).
The study demonstrated very poor compliance with paper diaries. Further, it demonstrated that compliance is often faked by patients using paper diaries, giving the false impression of good compliance.
In contrast, the patient experience diary system delivered very high rates of compliance, demonstrating that the poor compliance seen in the paper diary group was not a function of the study population and that good compliance can be achieved through a combination of proven scientific principles and solid technology. The findings raise serious doubts about the validity of data collected by paper diaries in clinical practice and research.
British Medical Journal, 324, 1193-1194.
Arthur A. Stone, Ph.D., Saul Shiffman, Ph.D., Joseph E. Schwartz, Ph.D., Joan E. Broderick, Ph.D., and Michael R. Hufford, Ph.D. (2002).
The ACDM would like to thank Invivodata for their kind permission to reproduce the above abstract.