Improving Inter-Rater Reliability

Challenge

When multiple raters will be used to assess the condition of a subject, it is important to improve inter-rater reliability, particularly if the raters are transglobal. The complexity of language barriers, nationality custom bias, and global locations requires that inter-rater reliability be monitored during the data collection period of the study. Identifying these deviations and retraining staff as quickly as possible would improve the quality and consistency of data collection.

In addition to addressing the challenges of raters being located globally, the data review monitors may also be located globally, and the study project manager and supervising monitor may also be in another global location. The client was seeking to identify an Electronic Data Capture (EDC) system that could not only manage the data collection, but could monitor multiple raters for inter-rater reliability. One way to do this would be to compare the score assigned to the observation to photos of the observed area, which would require the ability to upload and download photos, maintain metadata which ties the photos to a specific subject and visit date, allow feedback/queries and their responses, and provide a full audit trail of all data in the system. Ideally, the system would also notify the monitors that rating and photo data is entered and is ready for follow up if any discrepancies or potential inter-rater reliability issues are identified. Similarly, the system would also notify the rater when a feedback/query was entered and needed the rater’s response. Finally, the monitor would be notified when feedback/queries were resolved.

Proposal

Prelude Dynamics proposed the client use Prelude EDC, an electronic data capture system, which already provides the ability to customize the electronic Case Report Forms to collect the data necessary for the study. Prelude EDC could be customized to collect basic subject information, physical exams, patient medical history, visit history, study scales, and capture photos of the observation areas. Prelude EDC’s notification system could be programmed to notify monitors of observation and photo entries as well as resolved queries and notify the rater of queries needing to be resolved.

Additionally, Prelude EDC’s Randomization Module could provide just-in-time randomization globally across the study to ensure it would be difficult to predict the subject’s treatment group or it could randomize at the site level depending on the study design. The randomization module could also allow the stratification of subjects based on specific criteria, depending on the study’s needs.

The Inventory Management Module could be used to track medications and supplies throughout the study. The inventory module could even automatically place orders to ensure the minimum level of supplies and medications would be on hand. Ad-hoc reports and summaries would assist the monitors in keeping track of the study progress as well as monitor inter-rater reliability.

Background

When multiple clinical sites are used to collect data, it is necessary to train multiple raters, which requires taking the extra step to ensure inter-rater reliability. Assuming that the scale is sensitive enough and appropriate to use in a multiple-rater study, then any identified inter-rater reliability issues would require retraining of the rater.

It is often difficult to identify inter-rater reliability issues, particularly when a rater needs to be retrained. When the rater and the monitor are located remotely from each other, prompt monitoring is difficult because the monitor may only be able to review the data when they make a site visit, or the monitor may have access to the data, but not the related photos that are being used to monitor the observation. The necessity to retrain staff can incur costs to the study, and there are often multiple deviations of data as scales are rescored or removed from the study altogether, which is also costly.

Solution

Prelude Dynamics trained the staff to utilize Prelude EDC to capture all study data, upload photos, and take advantage of the system’s custom notification capability. Raters use tablets to record rating observations directly into the system. As the raters work through various data collection and observation screens, diagrams or illustrations will remind them of areas to be observed and the details of the rating scale, and a rating section will be provided. When scales are completed, photos can be uploaded into Prelude EDC to represent the area observed. In order to maintain study and data integrity, once submitted, the rater cannot change the scales. In fact, the scales will no longer be accessible via a tablet, and will only be visible to the rater via a read-only summary table which has captured the scale input. The Monitor can view the summary table and uploaded photos, and can compare them to the rater’s input and query the information as necessary. The solution also included:

• Standard notifications to notify the Monitor and Project Manager of the following events:
• New patient enrollment (i.e., patient successfully randomized)
• Patient terminated from study
• Changes to form state from in-work to in-review, in-review to reviewed, reviewed to finalized
• Query resolved
• Serious adverse events occurred
• Observation scale and/or photos uploaded
• Custom notifications would be created to notify of the following events
• Query entered by Monitor (notify the site personnel)

Prelude EDC’s Inventory Management Module will be used to keep track of both medication and supplies for this study. The module will also be set up to automatically order medication and supplies when they meet the minimum threshold to ensure the study progress is not compromised.

Prelude EDC additionally provides mid-study monitoring capacity through pre-set summaries, search/filter, and ad-hoc reporting. One of the standardized summary reports contains a listing of all queries both resolved and open. This will assist the monitor in keeping track of the progress of the study. For additional mid-study monitoring requiring a statistical package, export is available 24/7/365 for those roles with the appropriate permissions.

Results & Benefits

As a result of using Prelude EDC to capture study data, the Monitors are able to better track the study progress and better able to ensure inter-rater reliability. The monitors receive emails notifying them that a new scale and photos have been uploaded. They are able to manage their time better and provide prompt feedback. Designated site personnel receive an automatic email notifying them if a Monitor enters a query that needs resolution. Queries are promptly resolved when information is fresh in the researcher’s mind. As a result, the quality and reliability of the study data is improved.

The overall or primary Monitor is able to communicate directly and without delay with globally located site/country Monitors about any concerns over the scale ratings. The primary monitor has access to all scales from all sites, including photos and scale results, and can communicate to resolve any concerns or discrepancies identified. This helps ensure overall inter-rater reliability.

The Monitors and site report that data and query notifications through the system saved time, ensured quick responses, and kept them connected to the data and personnel through the study. This is especially important because all to the participating sites are active busy clinics that have their own patients and not just study participants.

Additionally, the site found it very valuable to have Prelude EDC keep track of the inventory, citing the efficiency of a system which automatically updates as patients are provided with medication and supplies, rather than keeping separate track of inventory. The automatic reordering feature reduced the stress associated with ensuring they always had enough supplies on hand to continue study progress. The Project Manager also found this feature reduced stress and logistics that otherwise would have posed a headache to handle. Imagine ordering supplies for different sites in different countries and keeping track of that! It would have been a full-time job just to do that.

The Project Manager and primary monitor tracked the study using the summaries and ad-hoc reports, staying up-to-date with the enrollment and progress at each site. They were also able to review the mix of subjects to ensure there were going to be sufficient subjects of each age and gender, and specifically noted how easy it was to copy and paste a report from Prelude EDC into a spreadsheet so they could provide monthly updates to the Sponsor.

Overall, everyone participating in the study found that Prelude EDC made the study run more efficiently and reduced their stress over participation in the study. Edit checks ensured the data was collected appropriately, and immediate feedback for missing or out-of-range data could be immediately collected  or verified as correct. They also found that it greatly improved inter-rater reliability which gave them more confidence in the data they collected throughout the study.

About Prelude Dynamics & Prelude EDC

Prelude Dynamics is a global provider of customized web-based software systems for data collection, analysis and management of clinical trials, studies and registries. We streamline eClinical operations through our unique and innovative Prelude EDC software system, which allows us to rapidly configure data collection solutions for pharmaceutical, CRO, medical device, animal health, and university organizations.