Planning a Registry

Richard E Gliklich; Nancy A Dreyer; Michelle B Leavy

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Gliklich RE, Dreyer NA, Leavy MB, editors. Registries for Evaluating Patient Outcomes: A User's Guide [Internet]. 3rd edition. Rockville (MD): Agency for Healthcare Research and Quality (US); 2014 Apr.

Cover of Registries for Evaluating Patient Outcomes

Registries for Evaluating Patient Outcomes: A User's Guide [Internet]. 3rd edition.

Show details

< Prev Next >

2Planning a Registry

1. Introduction

There is tremendous variability in size, scope, and resource requirements for registries. Registries may be large or small in terms of numbers of patients or participating sites. They may target rare or common conditions and exposures. They may require the collection of limited or extensive amounts of data, operate for short or long periods of time, and be funded generously or operate with limited financial support. In addition, the scope and focus of a registry may be adapted over time to reach broader or different populations, assimilate additional data, focus on or expand to different geographical regions, or address new research questions. While this degree of flexibility confers enormous potential, registries require good planning in order to be successful.

When planning a registry, it is desirable to follow these initial steps: (1) articulate the purpose of the registry; (2) determine if a registry is an appropriate means to achieve the purpose; (3) identify key stakeholders; and (4) assess the feasibility of a registry.

Once a decision is made to proceed, the next considerations in planning are to (5) build a registry team; (6) establish a governance and oversight plan; (7) define the scope and rigor needed; (8) define the data set, patient outcomes, and target population; (9) develop a study plan or protocol; and (10) develop a project plan. Of course, the planning for a registry is often not a linear process. Many of the steps described in this chapter occur in parallel.

Registry planners should also recognize the importance of periodic critical evaluations of the registry by key stakeholders to ensure that the objectives are being met. This is particularly important for patient registries that collect data over many years. When registry objectives are no longer being met or when clinical or other changes affect the registry (e.g., changes in treatment practices, the introduction of a new therapy), the registry may need to be adapted, or the registry may stop collecting new data. Registries may undergo a transition or cease collecting new data for many reasons. These considerations are fully discussed in Chapter 14.

The Guidelines for Good Pharmacoepidemiology Practice from the International Society of Pharmacoepidemiology is a useful resource for registry planners.¹ The Updated Guidelines for Evaluating Public Health Surveillance Systems may also be useful, especially the appendixes, which provide various checklists.² A Guide to the Project Management Body of Knowledge (PMBOK^® Guide) and the GRACE principles for comparative effectiveness (www.graceprinciples.org) may also be useful resources to registry planners.³^,⁴

2. Steps in Planning a Registry

2.1. Articulate the Registry's Purpose

One of the first steps in planning a registry is articulating its purpose. Having a clearly defined goal and/or purpose and supporting rationale makes it easier to evaluate whether a registry is the right approach for capturing the information of interest.⁵^,⁶ In addition, a clearly defined purpose helps clarify the need for certain data. Conversely, having a clear sense of how the data may be used will help refine the stated purpose. Attempts to be all inclusive may add cost but not value, resulting in overly burdensome data collection that can reduce quality and erode compliance.

A registry may have a singular purpose or several purposes.⁷ In either case, the overall purpose should be translated into specific objectives or questions to be addressed through the registry. This process needs to take into account the interests of those collaborating in the registry and the key audiences to be reached.⁸ Clear objectives are essential to define the structure and process of data collection and to ensure that the registry effectively addresses the important questions through the appropriate outcomes analyses. Specific objectives also help the registry to avoid collecting large amounts of data of limited value. The time and resources needed to collect and process data from a registry can be substantial.⁹ The identification of a core data set is essential. The benefits of any data element included in the registry must outweigh the costs of including it.

Registry planners should establish specific objectives by considering what key questions the registry needs to answer. Critical consideration should be given to defining the key questions in order to evaluate how best to proceed, as these questions will help to establish the type of registry (e.g., whether single focus or comparative), the data elements to be captured, and the types of analysis to be undertaken. Examples of key or driving questions are listed below:

What is the natural course of a disease, and how does geographic location affect the course?
Does a treatment lead to long-term benefits or harm, including delayed complications?
How is disease progression affected by available therapies?
What are significant predictors of poor outcomes?
What is the safety profile of a specific therapy?
Is a specific product or therapy teratogenic?
How do clinical practices vary, and what are the best predictors of treatment practices?
Are there disparities in the delivery and/or outcomes of care?
What characteristics or practices enhance compliance and adherence?
Do quality improvement programs affect patient outcomes, and, if so, how?
What process and outcomes metrics should be incorporated to track quality of patient care?
Should a particular procedure or product be a covered benefit in a particular population?
Was an intervention program or risk-management activity successful?
What are the resources used/economic parameters of actual use in typical patients?

2.2. Determine if a Registry Is an Appropriate Means To Achieve the Purpose

Two key questions to consider are whether a registry (or other study) is needed to address the purpose and, if the answer is yes, whether prospective data collection through a registry is an appropriate means of accomplishing the scientific objectives. Every registry developer should consider the following questions early in the planning process:

Do these data already exist?
If so, are they of sufficient quality to answer the research question?
Are they accessible, or does an entirely new data collection effort need to be initiated?

For example, could the necessary data be extracted from electronic medical records or administrative health insurance claims data? In such cases, registries might avoid re-collecting data that have already been collected elsewhere and are accessible. Thought should be given to adapting the registry (based on extant data) and/or linking to other relevant data sources (including “piggybacking” onto other registries). The Registry of Patient Registries (RoPR), developed by the Agency for Healthcare Research and Quality, is a resource for finding patient registries.¹⁰ When the required data have not been sufficiently collected or are not accessible for the desired purpose, it is appropriate to consider creating a new registry.

The next step is to consider whether the purpose would be well served by a registry. When making this decision, it is important to fully define the specific research question(s) of interest and to consider the state of current knowledge and gaps in evidence. Other factors that may influence this decision include the breadth of the target population of interest, the complexity of the current treatment patterns, the length of an observational period needed to achieve the objective, the scope and variety of treatments used, the approximate amount of funding available to address these objectives, and the urgency of decisions that will be made based on the resulting evidence. Registries may be the most appropriate choice for some research questions. For example, registries are particularly useful in situations where a comprehensive, flexible research design is needed,¹¹^,¹² or when the purpose is to discover how a product works in a wide variety of subgroups. (See Chapter 3, Section 2 for a discussion of research questions appropriate for registries.)

Other research questions, such as ones that might be used to petition a regulatory agency for a new indication, will require different approaches, such as traditional randomized controlled trials. In some cases, a hybrid approach, such as a registry that incorporates data collected retrospectively as well as prospectively, will be required. A research strategy, as opposed to a single study, may be necessary to address some research questions. For example, some research questions may require an interventional approach to address concerns about efficacy combined with an observational approach to examine long-term outcomes and quality of life in a broad patient population. When making a decision about study design, it is important to select the approach or combination of approaches best able to answer the specific research questions, from both scientific and practical standpoints. A careful evaluation of the possibilities for data collection and registry design, the degree of certainty required, and the timeframe in which this certainty is expected can help in selecting an appropriate study design.

Historically, there has been a lack of consensus standards for conducting and reporting methods and results for registries. Therefore, registries have been more variable in implementation and more difficult to assess for quality than randomized controlled trials. In recent years, advances in epidemiological and biostatistical methods have broadened the scope of questions that can be addressed through observational studies such as registries. Stratification, propensity score matching, and risk adjustment are increasingly useful approaches for addressing confounding issues and for creating comparably homogeneous subgroups for analysis within registry data sets, and advances in bias analysis are being used to help interpret results from observational studies such as registries.¹³^-¹⁵ (See Chapters 3, 13, and 18.) These techniques may allow registries to be used to support investigations of comparative safety and effectiveness. Following good registry practices, as described in this user's guide, can strengthen scientific rigor. (See Chapter 25.)

2.3. Identify Key Stakeholders

As a means of identifying potential stakeholders, it is important to consider to whom the research questions matter. It is useful to identify these stakeholders at an early stage of the registry planning process, as they may have important input into the type and scope of data to be collected, they may ultimately be users of the data, and/or they may have a key role in disseminating the results of the registry.

One or more parties could be considered stakeholders of the registry. These parties could be as specific as a regulatory agency that will be monitoring postmarketing studies or as broad as the general population, or simply those patients with the conditions of interest. Often, a stakeholder's input directly influences whether development of a registry can proceed, and it can have a strong influence on how a registry is conducted. A regulatory agency looking for management of a therapeutic product with a known toxicity profile may require a different registry design than a manufacturer with general questions about how a product is being used.

Typically, there are primary and secondary stakeholders for any registry. A primary stakeholder is usually responsible for creating and funding the registry. The party that requires the data, such as a regulatory authority, may also be considered a primary stakeholder. A secondary stakeholder is a party that would benefit from knowledge of the data or that would be impacted by the results but is not critical to establishing the registry. Treating clinicians and their patients could be considered secondary stakeholders. A partial list of possible stakeholders, both primary and secondary, follows:

Public health or regulatory authorities
Product manufacturers
Health care service providers
Payer or commissioning authorities
Patients and/or advocacy groups
Treating clinician groups
Academic institutions or consortia
Professional societies

Although interactions with potential stakeholders will vary, the registry will be best supported by defined interactions and communications with these parties. Defining these interactions during the planning stage will ensure that adequate dialog occurs and appropriate input is received to support the overall value of the registry. Interactions throughout the entire duration of the registry can also assure stakeholders that the registry is aligned with the purposes and goals that were set out during the planning stages and that the registry complies with all required guidances, rules, and/or regulations.

2.4. Assess Feasibility

A key element in determining the feasibility of developing a new registry relates to funding. Registries that meet the attributes described in this user's guide will most likely require significant funding. The degree of expense incurred will be determined by the scope of the registry, the rigor of data collection, and any audits that may be required. The larger the number of sites, the number of patients, and the scope of data collected, and the greater the need for representation of a wide variety of patient characteristics, the greater the expense will be. In addition, the method of data collection will contribute to expense. Historically, electronic data collection has been more expensive to implement, but generally less expensive to maintain, than forms that are faxed and scanned or mailed;¹⁶ however, the cost difference for startup has been lessening. Funding will be affected by whether other relevant data sources and/or infrastructures exist that capture some of the information of interest; whether the registry adapts to new issues over time; and whether multiple funding sources participate. Funding needs should also be examined in terms of the projected life of the registry and/or its long-term sustainability.

There are many potential funding sources for registries. Funding sources are likely to want to share in planning and to provide input for the many choices that need to be made in the implementation plans. Funding sources may negotiate to receive access to deidentified data as a condition for their participation. Funding models for registries may vary significantly, and there is no preferred approach. Rather, the funding model for a registry should be dictated by the needs of the registry. Potential sources of funding include:

Foundations: Nonprofit disease foundations may be interested in a registry to track the natural history of the disease of interest as well as the impact of therapeutic interventions. Registries may be used to track practice patterns and outcomes for quality improvement initiatives. Ongoing registries can sometimes serve the additional purpose of assisting in recruitment for clinical trials.¹⁷
Government: Federal agencies, such as the National Institutes of Health (NIH), Centers for Disease Control and Prevention (CDC), Centers for Medicare & Medicaid Services (CMS), Agency for Healthcare Research and Quality (AHRQ), U.S. Food and Drug Administration (FDA), and State agencies, may be interested in a registry to determine long-term outcomes of agents, devices, groups of drugs, or procedures. While the pharmaceutical industry or device manufacturers collect most long-term data on drug and device safety, many research questions arise that could potentially be suitable for government funding, ranging from clinical or comparative effectiveness to natural history of disease to the performance of health care providers based on accepted measures of quality of care. To determine if an agency might be interested in funding a registry, look for Requests for Proposals (RFPs) on its Web site. An RFP posting or direct communication with the appropriate agency staff may provide a great deal of specific information as to how a submission will be judged and what criteria would be needed in order for a proposal to be favorably ranked. Even if an RFP is not posted, contacting the appropriate agency staff may uncover potential interest in a registry to fill an unmet need.
Health plan providers: Under certain circumstances, health plan providers may be interested in funding a registry, since practical clinical research is increasingly viewed as a useful tool for providing evidence for health coverage and health care decisions.¹⁸
Patient groups: Patients may be able to contribute funding to focus on rare diseases or patient subgroups of interest for more common conditions. They may also contribute value in-kind.
Private funding: Private philanthropic individuals or charitable foundations and trusts may have an interest in furthering research to better understand the effects of a particular intervention or sets of interventions on a disease process.
Product manufacturers: Product manufacturers may be interested in studying the natural history of the disease for which they have (or are developing) a product; demonstrating the effectiveness and/or safety of existing products in real-world use through Risk Evaluation and Mitigation Strategy (REMS) programs as part of postmarketing commitments or requirements, or through studies; or assisting providers in evaluating or improving quality of care.
Professional societies: Health care professional associations are increasingly participating in developing or partnering with registries for scientific and quality measurement or improvement purposes.
Professional society/pharmaceutical industry “hybrids”: Situations may exist in which a product manufacturer funds a registry designed and implemented by a professional society to gain insight into a set of research questions.
Multiple sponsors: Registries may meet the goals of multiple stakeholders, and such stakeholders may have an interest in sharing the funding. Registries for isotretinoin and antiretrovirals in pregnancy are examples, as is INTERMACS™, a registry for patients who are receiving mechanical circulatory support device therapy to treat advanced heart failure.¹⁹ While multiple sponsorship can decrease the costs for each funding source, their varied interests and needs almost always increase the complexity and overall cost of the registry.

A public-private partnership is a service or business venture that is funded and operated through a partnership (contractual agreement) between a public agency (Federal, State, or local) and a private-sector entity or entities.²⁰ While some true public-private partnerships for registries currently exist (e.g., State-level immunization registries, bioterrorism surveillance efforts),²¹^-²³ there is great potential for growth in this approach. Both government and private sources have shown increasing interest in registries for improved safety monitoring, for comparative effectiveness goals, and for streamlining the costs of the drug development process.²⁴^-²⁹ Several legislative actions have stated or suggested the role of public-private partnerships for activities such as registry development.³⁰ There are many good reasons for multiple stakeholders, including government agencies, providers, and industry, to work together for certain purposes. Thus, it is anticipated that shared funding mechanisms are likely to become more common. Chapter 24 provides more detail on the use of public-private partnerships to support registries.

2.5. Build a Registry Team

Several different kinds of knowledge, expertise, and skills are needed to plan and implement a registry. In a small registry run by a single individual, consultants may be able to provide the critical levels of expertise needed to plan all components of the registry. In a large registry, a variety of individuals may work together as a team to contribute the necessary expertise. Depending on the size, scope, and purpose of the registry, few, some, or all of the individuals representing the components of expertise described below may be included at the time of the planning process. Whatever number of individuals is eventually assembled, it is important to build a group that can work together as a collegial team to accomplish the goals of the registry. Additionally, the team participants must understand the data sources. By understanding the goals and data sources, the registry team will enable the data to be used in the most appropriate context for the most appropriate interpretation. The different kinds of expertise and experience that are useful include the following:

Project management: Project management will be needed to coordinate the components of the registry; to manage timelines, milestones, deliverables, and budgets; and to ensure communication with sites, stakeholders, oversight committees, and funding sources. Ongoing oversight of the entire process will require a team approach. (See Section 2.6, “Establish a Governance and Oversight Plan.”)
Subject matter: A registry must be designed so that it contains the appropriate data to meet its goals as well as the needs of its stakeholders. For example, experts in the treatment of the clinical disease to be studied who are also familiar with the potential toxicities of the treatment(s) to be studied are critical to the success of the registry. Clinical experts must be able to apply all of the latest published clinical, toxicity, and outcome data to components of the registry and determine which elements are necessary, desirable, or superfluous. Depending on the outcomes and registry purpose, it is often useful to have patient representatives or advocates.
Registry science: Epidemiology and biostatistics expertise specific to the subtleties of patient registries and observational research is very important in the design, implementation, and analysis of registry data. Epidemiologists can provide the study design and can work in collaboration with biostatisticians to develop a mutual understanding of the research objectives and data needed. Health outcomes researchers and economics researchers can also lend valuable expertise to the registry team. These scientists should work with the subject matter experts to ensure that appropriate analytic methods are being used to address the clinical issues relevant to achieving the goals of the registry.
Data collection and database management: The decision to include various data elements can be made in consultation with experts in this field to place “critical fields” in a prominent and logical position on the data form for both paper-based and electronic data collection tools. (A final determination of what is usable and workable for data collection tools should be approved by all members of the team.) These experts may also need to write specific programs so that the data received from the registry are grouped, stored, and identified. They may generate reports for individuals who track registry participation, and they may provide data downloads periodically to registry analysts. This team will also be responsible for implementing and maintaining firewalls to protect the data according to accepted levels of security for similar collections of sensitive data.
Legal issues/patient privacy: It is critical that either information that identifies individual patients be excluded or applicable legal requirements for the inclusion of patient identifiable information be met (e.g., obtaining informed consent or Health Insurance Portability and Accountability Act [HIPAA] authorization, where required). The complexities of this topic are dealt with in detail in Chapters 7, 8, and 9. Legal and privacy expertise is needed to protect the patients and the owners of the database by ensuring that the registry complies with all Federal and State laws applicable to patient information.
Quality assurance: As discussed in Chapter 11, Section 3, quality assurance of procedures and data is another important component of registry success. Expertise in quality assurance will help in planning a good registry. The goals for quality assurance should be established for each registry, and the efforts made and the results achieved should be described.

2.6. Establish a Governance and Oversight Plan

Governance refers to guidance and high-level decisionmaking, including purpose, funding, execution, and dissemination of information. A goal of proper governance and oversight should be transparency to stakeholders in operations, decisionmaking, and reporting of results.

The composition and relative mix of stakeholders and experts relate largely to the purpose of the registry. For example, if the purpose of the registry is to determine a comparative effectiveness or reimbursement policy, those impacted by the policy should not solely govern the registry. Broad stakeholder involvement in governance boards is most desirable when there are many stakeholders. Depending on the size of the registry, governance may be assumed by various oversight committees made up of interested individuals who are part of the design team (internal governance) or who remain external to the day-to-day operations of the registry (external governance). Differences in the nature of the study questions, the overall resources being consumed by the registry, the soundness of the underlying data sources, and many other factors will influence the degree of involvement and role of oversight groups. In other words, the purpose of the committee functions described below is to lay out the roles that need to be assumed by the governance structure of many registries, but these should be individualized for a particular registry. It is also possible, if methods are clear and transparent, that oversight requirements may be minimal.

Registries fulfill governance roles in a variety of ways. Many of the roles, for example, could be assumed by a single committee (e.g., a steering committee) in some registries. Whatever model is adopted, it must accommodate all of the working constituencies and provide a mechanism for these individuals to work together to achieve the goals of the registry.

All aspects of governance should be codified in a written format that can be reviewed, shared, and refined over time. In addition, governance is a dynamic process, subject to change in policy as evidence emerges that is likely to lead to improvements in the process.

Governance and oversight functions that may be considered include:

Executive or steering: This function assumes responsibility for the major financial, administrative, legal/ethical, and scientific decisions that determine the direction of the registry. These decisions are made with appropriate input from legal, scientific, and administrative experts. Depending on their capabilities and the size and resources of the registry, the group serving the steering function may also assume some of the functions described below.
Scientific: This function may include experts in areas ranging from database content, to general clinical research, to epidemiology and biostatistics. This function may determine the overall direction of database inquiries and recommend specific analyses to the executive or steering group. It is strongly desirable that the reports that emerge from a registry be scientifically based analyses that are independent and transparent.³¹ To enhance credibility and in the interest of full disclosure, the role of all stakeholders in the publication process should be specified and any potential conflicts of interest identified.
Liaison: In large registries, a function may be specified to focus on maintaining relationships with the funding source, health care providers, and patients who need access to registry information. The group serving this function may develop monitoring and satisfaction tools to ensure that the day-to-day operations of the registry remain healthy.
Adjudication: Adjudication is used to review and confirm cases (outcomes) that may be difficult to classify. Individuals performing this function are generally blinded to the exposure (product or process) under study so that the confirmation of outcomes is made without knowledge of exposure.
External review: External review committees and/or advisory boards can be useful for providing independent oversight throughout the course of the registry. The majority of registries will not require a data safety monitoring board (DSMB), since a DSMB is commonly used in situations where data are randomized and treatment status is blinded. However, there may be situations in which the registry is responsible for the primary accumulation of safety data on a particular intervention; in such situations, an external committee or DSMB would be useful for conducting periodic reviews (e.g., annually).
Data access, use, and publications: This function should address the process by which registry investigators access and perform analyses of registry data for the purpose of submitting abstracts to scientific meetings and developing manuscripts for peer-reviewed journal submission. Authorship (including that of registry sponsors) in scientific publications should satisfy the conditions of the Uniform Requirements for Manuscripts Submitted to Biomedical Journals.³² The rules governing authorship may be affected by the funding source, as in the case of NIH or foundation funding, or by the biomedical journal. (See Case Examples 1 and 22.) Other investigators may request permission to access the data. For example, a Ph.D. candidate at an institution might seek registry-wide aggregate data for the purpose of evaluating a new scientific question. A process for reviewing and responding to such requests from other investigators or entities should be considered in some registries that may generate broad external interest if the registry stakeholders and participants are agreeable to such use.

2.7. Consider the Scope and Rigor Needed

2.7.1. Scope of Data

The scope of a registry may be viewed in terms of size, setting, duration, geography, and financing. The purpose and objectives of the registry should frame the scope, but other factors (aside from feasibility) may ultimately shape it. For example, the scope may be affected by:

Regulatory requirements, such as those imposed by the FDA as a condition of product marketing.
Reimbursement decisions, such as national coverage decisions by CMS or “Prior Authorization” requirements used by health insurers in some situations.
National research interests, such as those driven by NIH.
Public health policy, such as CDC policy and immunization policy.

The scope is also affected by the degree of uncertainty that is acceptable to the primary stakeholders, with that uncertainty being principally driven by the quantity, quality, and detail of the data collection balanced against its considered importance and value. Therefore, it is critical to understand the potential questions that may or may not be answerable because of the quantity and quality of the data. It should also be noted that the broader the audience of stakeholders, the broader the list of questions that may need to be included. This increased breadth can result in an increase in the number of patients who need to be enrolled and/or data points that need to be collected in order to meet the objective of the registry with an acceptable level of precision.

Some of the specific variables that can characterize the scope of a registry include:

Size: This refers to the number and complexity of data points, the frequency of data collection, and the enrollment of investigators and patients. A registry with a large number of complex data points may allow for detailed and thoughtful analyses but may be so burdensome as to discourage investigator and patient enrollments. In turn, a small registry with few patients and data points may be easier to execute, but the data could lack depth and be less meaningful.³³ Size also determines the precision with which measures of risk or risk difference can be calculated.
Duration: The planning of a registry must reflect the length of time that the registry is expected to collect the data in order to achieve its purpose and provide analysis of the data collected. Some registries are limited by commercial interests, such as when the product under study is approaching the end of its patent life.
Setting: This refers to the specific setting through which the registry will recruit investigators and patients as well as collect data (e.g., hospital, doctor's office, pharmacy, home).
Geography: A locally run registry is very different in scope from a global registry, in terms of setup, management, and analysis. A global registry poses challenges (e.g., language, cultural, time zone, regulatory) that must be taken into consideration in the planning process.
Cost: The scope of a registry will determine the cost of creating, managing, and analyzing the registry. Budgetary constraints must be carefully considered before moving from conception to reality. Additionally, the value of the information is a factor in the financial decisions. Certain choices in planning, such as building on existing infrastructure and/or linking to data sources relevant to the purposes of the registry, may increase the net return.
Richness of clinical data needed: In some situations, the outcome may be relatively simple to characterize (e.g., death). In other cases, the focus of interest may be a complex set of symptoms and measurements (e.g., for Churg-Strauss Syndrome) or may require specialized diagnostic testing or tissue sampling (e.g., sentinel node in melanoma). Some outcomes may require assessment by an independent third party. Depending on the objectives of the registry, collection and storage of biological samples may be considered. (See Section 2.7.3 below.) The collection of biosamples itself is a rapidly evolving field, and registry developers should consult both technical and legal sources regarding how to include biosamples in a registry.

2.7.2. When Data Need To Be Available for Analysis

Meaningful data on disease progression or other long-term patient outcomes may not be available through a registry for many years, whereas safety data could be examined periodically over time. Therefore, the type of data on patient outcomes and when they will be available for analysis should be addressed from the perspective of the intended uses of the data in both the short term and long term. For industry-sponsored registries, if planning begins at an early stage, it may be possible to consider whether to align registry questions with those from the clinical trial (where appropriate) so that some data can carry over for more comprehensive longitudinal analyses.

2.7.3. Scientific Rigor

The content of the data to be collected should be driven by the scientific analyses that are planned for the registry, which, in turn, are determined by the specific objectives of the registry. A registry designed primarily for monitoring safety will contain different data elements from one designed primarily for monitoring effectiveness. Similarly, the extent to which data need to be validated will depend on the purpose of the registry and the complexity of the clinical information being sought. For some outcomes, clinical diagnosis may be sufficient; for others, supporting documents from hospitalizations, referrals, or biopsies may be needed; and for others, formal adjudication by a committee may be required. Generally, registries that are undertaken for regulatory decisionmaking will require increased attention toward diagnostic confirmation (i.e., enhanced scientific rigor).

2.8. Define the Core Data Set, Patient Outcomes, and Target Population

2.8.1. Core Data Set

Elements of data to be included must have potential value in the context of the current scientific and clinical climate and must be chosen by a team of experts, preferably with input from experts in biostatistics and epidemiology. Each data element should relate to the purpose and specific objectives of the registry. Ideally, each data element should address the central questions for which the registry was designed. It is useful to consider the generalizability of the information collected, as appropriate. For example, when seeking information on cost-effectiveness, it may be preferable to collect data on resource utilization rather than actual costs of this utilization, since the broader descriptor can be more easily generalized to other settings and cost structures. While a certain number of “speculative” fields may be desired to generate and explore hypotheses, these must be balanced against the risk of overburdening sites with capturing superfluous data. A plan for quality assurance should be considered in tandem with developing the core data set.

The core data set variables (“need to know”) define the information set needed to address the critical questions for which the registry was created. At a minimum, when calculating the resource needs and overall design of the registry, registry planners must account for these fields. If additional noncore variables (“nice to know”) are included, such as more descriptive or exploratory variables, it is important that such data elements align with the goals of the registry and take into account the burden of data collection and entry at the site level. A parsimonious use of “nice to know” variables is important for several reasons.

First, when data elements change, there is a cascade effect on all dependent components of the registry process and outputs. For example, the addition of new data elements may require changes to the data collection system, retraining of site personnel on data definitions and collection practices, adjustments to the registry protocol, and amendment submissions to institutional review boards. Such changes often require additional financial resources. Ideally, the registry would both limit the total number of data elements and include, at the outset, data elements that might change from “nice to know” to “need to know” during the course of the registry. In practice, this is a difficult balance to achieve, so most registries should plan adequate resources to be used for change management.

Second, a registry should avoid attempting to accomplish too many goals, or its burden will outweigh its usefulness to the clinical sites and researchers. Examples exist, however, of registries that serve multiple purposes successfully without overburdening clinicians. (See Case Example 1.)

Third, even “need-to-know” variables can sometimes be difficult to collect reliably (e.g., use of illegal substances) or without substantial burden (e.g., unusual laboratory tests). Even with a limited core data set, feasibility must still be considered. (See Chapter 4)

Fourth, it is useful to consider what data are already available and/or collected and what additional data need to be collected. When determining additional data elements, it is imperative to consider whether the information desired is consistent with general practice or whether it might be more intensive or exceeding usual practice. For some purposes, collecting specific laboratory results or additional visits may be necessary, but could change how the registry is perceived by institutional review boards or ethics committees. The distinction between “interventional” and “observational” is straightforward in terms of random assignment to treatment, but some registries with requirements that exceed a threshold of usual practice—in Europe, for example—may be subject to additional requirements more typical of “interventional” research. In Chapter 1.7.1 of Volume 9A of the Rules Governing Medicinal Products in the European Union,³⁴ it has been clarified that registries may “collect a battery of information using standardized questionnaires in a prospective fashion” and “questionnaires, by themselves, are not considered interventional.” These rules also state that

“[T]he assignment of a patient to a particular strategy is not decided in advance by a [trial] protocol but falls within the current practice…”
“[N]o additional diagnostic or monitoring procedures shall be applied to patients.”

This last requirement can be challenging to interpret since registries sometimes perform diagnostic tests that are consistent with general practice but that may be performed more frequently than would be the case in general practice. The determination that a registry should be considered “interventional” from a regulatory perspective can add significant burden and cost to the registry program, and, therefore, the tradeoffs must be carefully considered in planning schedules for registry visits and the collection of data and/or specimens.

Finally, it is important to consider patient privacy, national and international rules concerning ethics, and regulatory requirements to assure that the registry data requirements do not jeopardize patient privacy or put institutional/ethics reviews and approvals at risk.

2.8.2. Patient Outcomes

The outcomes of greatest importance should be identified early in the concept phase of the registry. Delineating these outcomes (e.g., primary or secondary endpoints) will force registry designers to establish priorities. Prioritization of interests in the planning phase will help focus the work of the registry and will guide study size requirements. (See Chapter 3.) Identifying the patient outcomes of the greatest importance will also help to guide the selection of the data set. Avoiding the temptation to collect “nice to know” data that are likely of marginal value is of paramount importance, yet some registries do, in fact, need to collect large amounts of data to accomplish their purposes. Possessing adequate data in order to properly address potential confounders during analyses is one reason that extensive data collection is sometimes required.³⁵

Methods to ascertain the principal outcomes should be clearly established. The diagnostic requirements, level of data detail, and level of data validation and/or adjudication should also be addressed. As noted below in the context of identifying a target population, relying on established guidelines and standards to aid in defining outcomes of interest has many benefits and should be considered.

The issues of ascertainment noted here are important to consider because they will have a bearing on some attributes by which registries may be evaluated.³⁶ These attributes include sensitivity (the extent to which the methods identify all outcomes of interest) and external validity (generalizability to similar populations), among others.

2.8.3. Target Population

The target population is the population to which the findings of the registry are meant to apply. It must be defined for two basic reasons. First, the target population serves as the foundation for planning the registry. Second, it also represents a major constituency that will be impacted by the results of the registry.

One of the goals for registry data may be to enable generalization of conclusions from clinical research on narrowly defined populations to broader ones, and therefore the inclusion criteria for most (although not all) registries are relatively broad. As an example, screening criteria for a registry may allow inclusion of elderly patients, patients with multiple comorbidities, patients on multiple therapies, patients who switch treatments during the period of observation, or patients who are using products “off label.” The definition of the target population will depend on many factors (e.g., scope and cost), but ultimately will be driven by the purpose of the registry.

As with defining patient outcomes, target population criteria and/or definitions should be consistent with established guidelines and standards within the therapeutic area. Achieving this goal increases the potential utility of the registry by leveraging other data sources (historical or concurrent) with different information on the same target population and enhancing statistical power if similar information is collected on the target population.

In establishing target population criteria, consideration should be given to the feasibility of access to that population. One should try to distinguish the ideal from the real. Some questions to consider in this regard are:

How common is the exposure or disease of interest?
Can eligible people be readily identified?
Are other sources competing for data on the same patients?
Is care centralized or dispersed (e.g., in a referral or tertiary care facility)?
How mobile is the target population?

Ultimately, methods to ascertain members of the target population should be carefully considered (e.g., use of screening logs that identify all potential patients and indicate whether they participate and, if not, why not), as should the use of sources outside the registry (e.g., patient groups). Greater accessibility to the target population will reap benefits in terms of enhanced representativeness and statistical power.

Lastly, thought should be given to comparison (control) groups either internal or external to the registry. Again, much of this consideration will be driven by the purpose and specific objectives of the registry. For example, natural history registries do not need controls, but controls are especially desirable for registries created to evaluate comparative effectiveness or safety.

2.9. Develop a Study Plan or Protocol

The study plan documents the objectives of the registry and describes how those objectives will be achieved. At a minimum, the study plan should include the registry objectives, the eligibility criteria for participants, and the data collection procedures. Ideally, a full study protocol will be developed to document the objectives, design, participant inclusion/exclusion criteria, outcomes of interest, data to be collected, data collection procedures, governance procedures, and plans for complying with ethical obligations and protecting patient privacy.

In addition to a study plan or protocol, registries may have statistical analysis plans. Chapters 13 and 25 discuss the importance of analysis plans.

2.10. Develop a Project Plan

Developing an overall project plan is critically important so that the registry team has a roadmap to guide their collective efforts. Depending on the complexity of the registry project, the project plan may include some or all of the following elements:

Scope management plan to control the scope of the project. It should provide the approach to making changes to the scope through a clearly defined change-control system.
Detailed timeline and schedule management plan to ensure that the project and its deliverables are completed on time.
Cost management plan for keeping project costs within the budget. The cost management plan may provide estimates on cost of labor, purchases and acquisitions, compliance with regulatory requirements, et cetera. This plan should be aligned with the change-control system so that all changes to the scope will be reflected in the cost component of the registry project.
Quality management plan to describe the procedures to be used to test project concepts, ideas, and decisions in the process of building a registry. Having a quality management plan in place can help in detecting design errors early, formulating necessary changes to the scope, and ensuring that the final product meets stakeholders' expectations.
Staffing management plan to determine what skills will be needed and when to meet the project goals. (See Chapter 2, Section 2.5).
Communication plan that includes who is responsible for communicating information and to whom it should be communicated. Considerations include different categories of information, frequency of communications, and methods of communication. The plan should also provide steps to escalate issues that cannot be resolved on a lower staff level.
Procurement plan for external components or equipment and/or outsourced software development for the planned registry, if pertinent. Such a plan should describe how the procurement process would be managed within the organization. Decisions to procure products or services may have a direct impact on other components of the project plan, including the staffing plan and timeline.
Risk management plan to identify and mitigate risks. Many project risks are predictable events, and therefore they can and should be assessed in the very early stages of registry planning. It is important to prioritize project risks by their potential impact on the specific objectives and to develop an adequate risk response plan for the most significant risks. Some predictable risks include—
–
Disagreement between stakeholders over the scope of specific tasks.
–
Inaccurate cost estimates.
–
Delays in the timeline.

3. Summary

In summary, planning a patient registry involves several key steps, including articulating its purpose, determining whether it is an appropriate means of addressing the research question, identifying stakeholders, defining the scope and target population, assessing feasibility, and securing funding. A registry team and advisors must be assembled to develop, coordinate, and guide the registry; these individuals should be selected based on their expertise and experience. Governance and oversight for the registry should also be addressed during the planning phase. While registries differ tremendously in size, scope, and resource requirements, the basic elements of planning described here are relevant for most, if not all registries, and can help to support the launch and operation of a successful registry.

Case Example for Chapter 2

Case Example 1Creating a registry to fulfill multiple purposes and using a publications committee to review data requests

Description	The National Registry of Myocardial Infarction (NRMI) collected, analyzed, and disseminated data on patients experiencing acute myocardial infarction. Its goal was improvement of patient care at individual hospitals through the hospital team's evaluation of data and assessment of care delivery systems.
Sponsor	Genentech, Inc.
Year Started	1990
Year Ended	2006
No. of Sites	451 hospitals in the final phase of NRMI (NRMI 5). Over 2,150 hospitals participated in NRMI over 16 years.
No. of Patients	2,515,106

Challenge

Over the past 20 years, there have been significant changes in the treatment of acute myocardial infarction (AMI) patients. Evidence from large clinical trials has led to the introduction of new guidelines and therapies for treating AMI patients, including fibrinolytic therapy and percutaneous coronary intervention. While these treatments can improve both morbidity and mortality for AMI patients, they are time sensitive and must be administered very soon after hospital arrival in order to be most effective.

After the release of its first fibrinolytic therapy product in 1987, the sponsor's field representatives learned from their discussions with emergency department physicians, cardiologists, and hospital staff that most clinicians believed they were treating patients quickly, although there was no documentation or benchmarking to confirm this assumption or to identify and correct delays. At that time, many emergency departments did not have readily available diagnostic tools (such as angiography labs), and hospitals with AMI-specific decision pathways and treatment protocols were the exception rather than the rule.

In addition, since fibrinolytic therapy was being widely used for the first time, the sponsor wanted to gather safety information related to its use in real-world situations and in a broader range of patients than those treated in the controlled environment of a clinical trial.

Proposed Solution

The sponsor decided to create the registry to fulfill the multiple purposes of identifying treatment patterns, promoting time-to-treatment and other quality improvements, and gathering real-world safety data. The scope of the data collection necessary to meet these needs could have made such a registry impracticable, so the project team faced the sizable challenge of balancing the data needs with the feasibility of the registry.

The sponsor formed a scientific advisory board with members representing the various clinical stakeholders (emergency department, cardiology, nursing, research, etc.). The scientific advisory board developed the data set for the registry, keeping a few guiding principles in mind. These principles emphasized maintaining balance between the clinical research and the feasibility of the registry. The first principle was to determine whether the proposed data element was necessary by asking several key questions: How will the data element be used in generating hospital feedback reports or research analyses? Is the data element already collected? If not, should it be collected? If it should be collected, is it feasible to collect those data? The second principle focused on using existing data standards whenever possible. If a data standard did not exist, the team tried to collect the data in the simplest possible way. The third principle emphasized data consistency and making the registry user-friendly by continually refining data element definitions until they were as clear as possible.

In 1990, the sponsor launched the registry. During the 16 years that the registry was conducted, it demonstrated that the advisory board's efforts to create a feasible multipurpose registry were successful. The registry collected data on the clinical presentation, treatment, and outcomes of over 2.5 million patients with AMI from more than 2,150 participating sites.

The success of the registry presented a new challenge for the registry team. The sponsor received a large volume of requests to analyze the registry data, often for research topics that fell outside of the standardized reports developed for the registry. As a guiding principle, the registry team was committed to making the data available for research projects, but it had limited resources. To support these requests, the team developed a process that would allow outside researchers to access the registry data without overburdening the registry team.

The registry team created a publication process to determine when another group could use the data for research. The team set high-level criteria for all data requests: the analysis had to be feasible given the data in the registry, and the request could not represent a duplication of another research effort.

The registry team involved its scientific advisory board, made up of cardiologists, emergency department physicians, nurses, research scientists, pharmacists, and reviewers with specialties in biostatistics and statistical programming, in creating a publication review committee. The review committee evaluated all research proposals to determine originality, interest to peers, feasibility, appropriateness, and priority. The review committee limited its review of research proposals to a set number of reviews per year, and scheduled the reviews and deadlines around the abstract deadlines for the major cardiology conferences. Research analyses had to be intended to result in peer-reviewed presentations and publications. Researchers were asked to submit proposals that included well defined questions and an analysis plan. If the proposal was accepted, the researchers discussed any further details with the biostatisticians and statistical programmers who performed the analyses (and who were employed at an independent clinical research organization). The results were sent directly to the researchers.

The scientific advisory board and review committee remained involved in the process after a data request had been granted. All authors submitted their abstracts to the review committee before sending them to conferences. The review committee offered constructive criticism to help the authors improve their abstracts. The review committee also reviewed manuscripts before journal submission to help identify any issues or concerns that the authors should address.

Results

This publication process enabled the wealth of data collected in this registry to be used in over 150 scientific abstracts and 100 peer-reviewed articles, addressing each of the purposes of the registry as well as other research topics. By involving the scientific advisory board and providing independent biostatistical support, the registry team developed an infrastructure that enhanced the credibility of the research uses of this observational database.

Key Point

Registries can be developed to fulfill more than one purpose, but this added complexity requires careful planning to ensure that the final registry data collection burden and procedures are feasible. Making sure that the advisory board includes representatives with clinical and operational perspectives can help the board to maintain its focus on feasibility. As a registry database gains large amounts of data, the registry team will likely receive research proposals from groups interested in using the data. The registry team may want to set up a publication process during the registry design phase.

For More Information

Califf RM. The benefits of moving quality to a national level. Am Heart J. 2008;156(6):1019–22. [PMC free article: PMC3709600] [PubMed: 19032994].

Rogers WJ, Frederick PD, Stoehr E, et al. NRMI Investigators. Trends in presenting characteristics and hospital mortality among patients with ST elevation and non-ST elevation myocardial infarction in the NRMI from 1990 to 2006. Am Heart J. 2008;156(6):1026–34. [PubMed: 19032996].

Gibson CM, Pride YB, Frederick PD, et al. NRMI Investigators. Trends in reperfusion strategies, door-to-needle and door-to-balloon times, and in-hospital mortality among patients with ST-segment elevation myocardial infarction enrolled in the NRMI from 1990 to 2006. Am Heart J. 2008;156(6):1035–44. [PubMed: 19032997].

Peterson ED, Shah BR, Parsons L, et al. NRMI Investigators. Trends in quality of care for patients with acute myocardial infarction in the NRMI from 1990 to 2006. Am Heart J. 2008;156(6):1045–55. [PubMed: 19032998].

References for Chapter 2

1.: ISPE. Guidelines for good pharmacoepidemiology practices (GPP). Pharmacoepidemiol Drug Saf. 2008 Feb;17(2):200–8. [PubMed: 17868186]
2.: German RR, Lee LM, Horan JM, et al. Updated guidelines for evaluating public health surveillance systems: recommendations from the Guidelines Working Group. MMWR Recomm Rep. 2001 Jul 27;50(RR-13):1–35. quiz CE1-7. [PubMed: 18634202]
3.: Project Management Institute. A Guide to the Project Management Body of Knowledge (PMBOK Guide). 4th ed. Newtown Square, PA: Project Management Institute; 2008.
4.: Dreyer NA, Schneeweiss S, McNeil BJ, et al. GRACE principles: recognizing high-quality observational studies of comparative effectiveness. Am J Manag Care. 2010 Jun;16(6):467–71. [PubMed: 20560690]
5.: Dreyer NA, Garner S. Registries for robust evidence. JAMA. 2009 Aug 19;302(7):790–1. [PubMed: 19690313]
6.: Solomon DJ, Henry RC, Hogan JG, et al. Evaluation and implementation of public health registries. Public Health Rep. 1991 Mar-Apr;106(2):142–50. [PMC free article: PMC1580226] [PubMed: 1902306]
7.: Glaser SL, Clarke CA, Gomez SL, et al. Cancer surveillance research: a vital subdiscipline of cancer epidemiology. Cancer Causes Control. 2005 Nov;16(9):1009–19. [PubMed: 16184466]
8.: Kennedy L, Craig AM. Global registries for measuring pharmacoeconomic and quality-of-life outcomes: focus on design and data collection, analysis and interpretation. Pharmacoeconomics. 2004;22(9):551–68. [PubMed: 15209525]
9.: Bookman MA. Using tumor registry resources in analyzing concordance with guidelines and outcomes. Oncology (Williston Park). 2000 Nov;14(11A):104–7. [PubMed: 11195403]
10.: Agency for Healthcare Research and Quality. Developing a Registry of Patient Registries (RoPR). [August 17, 2012]. http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productid=690.
11.: Avorn J. In defense of pharmacoepidemiology--embracing the yin and yang of drug research. N Engl J Med. 2007 Nov 29;357(22):2219–21. [PubMed: 18046025]
12.: Vandenbroucke JP. Observational research, randomised trials, and two views of medical science. PLoS Med. 2008 Mar 11;5(3):e67. [PMC free article: PMC2265762] [PubMed: 18336067]
13.: Alter DA, Venkatesh V, Chong A, et al. Evaluating the performance of the Global Registry of Acute Coronary Events risk-adjustment index across socioeconomic strata among patients discharged from the hospital after acute myocardial infarction. Am Heart J. 2006 Feb;151(2):323–31. [PubMed: 16442894]
14.: Hernan MA, Hernandez-Diaz S, Werler MM, et al. Causal knowledge as a prerequisite for confounding evaluation: an application to birth defects epidemiology. Am J Epidemiol. 2002 Jan 15;155(2):176–84. [PubMed: 11790682]
15.: Lash TL, Fox MP, Fink AK. Applying quantitative bias analysis to epidemiologic data. New York, NY: Springer; 2009.
16.: Retchin SM, Wenzel RP. Electronic medical record systems at academic health centers: advantages and implementation issues. Acad Med. 1999 May;74(5):493–8. [PubMed: 10353279]
17.: Andersen MR, Schroeder T, Gaul M, et al. Using a population-based cancer registry for recruitment of newly diagnosed patients with ovarian cancer. Am J Clin Oncol. 2005 Feb;28(1):17–20. [PubMed: 15685029]
18.: Tunis SR. A clinical research strategy to support shared decision making. Health Aff (Millwood). 2005 Jan-Feb;24(1):180–4. [PubMed: 15647229]
19.: INTERMACS. [August 14, 2012]. http://www.uab.edu/intermacs/
20.: Wikipedia. Public-Private Partnerships. [August 14, 2012]. http://en.wikipedia.org/wiki/Public-private_partnership.
21.: Riverside County Department of Public Health. Vaxtrack Immunization Registry. [August 14, 2012]. http://www.naccho.org/topics/modelpractices/database/practice.cfm?practiceID=139.
22.: California Immunization Registry (CAIR). [August 14, 2012]. http://www.ca-siis.org.
23.: North Carolina Department of Health and Human Services. N.C Public-Private Partnership Creates Statewide Bioterrorism Surveillance System. [August 14, 2012]. http://www.infectioncontroltoday.com/news/2004/07/public-private-partnership-creates-statewide-biot.aspx.
24.: Ray WA, Stein CM. Reform of drug regulation--beyond an independent drug-safety board. N Engl J Med. 2006 Jan 12;354(2):194–201. [PubMed: 16407517]
25.: Strom BL. How the US drug safety system should be changed. JAMA. 2006 May 3;295(17):2072–5. [PubMed: 16670415]
26.: Okie S. Safety in numbers--monitoring risk in approved drugs. N Engl J Med. 2005 Mar 24;352(12):1173–6. [PubMed: 15788493]
27.: Rawlins M. De Testimonio: on the evidence for decisions about the use of therapeutic interventions. Clin Med. 2008 Dec;8(6):579–88. [PMC free article: PMC4954394] [PubMed: 19149278]
28.: Lyratzopoulos G, Patrick H, Campbell B. Registers needed for new interventional procedures. Lancet. 2008 May 24;371(9626):1734–6. [PubMed: 18502284]
29.: American Recovery and Reinvestment Act of 2009, H.R.1. [August 14, 2012]. http://www.gpo.gov/fdsys/pkg/BILLS-111hr1enr/pdf/BILLS-111hr1enr.pdf.
30.: Food and Drug Administration Amendments Act of 2007., Pub. L. No. 110-85 (2007), Title VI. Sect. 603. Critical Path Public-Private Partnerships. [August 14, 2012]. http://www.gpo.gov/fdsys/pkg/PLAW-110publ85/html/PLAW-110publ85.htm.
31.: Davidoff F, DeAngelis CD, Drazen JM, et al. Sponsorship, authorship, and accountability. Ann Intern Med. 2001 Sep 18;135(6):463–6. [PubMed: 11560460]
32.: International Committee of Medical Journal Editors, editor. Uniform Requirements for Manuscripts Submitted to Biomedical Journals. Writing and Editing for Biomedical Publication; [August 14, 2012]. http://www.icmje.org. [PubMed: 15020262]
33.: Woodward M. Sample size determination Epidemiology: study design and data analysis. Boca Raton (FL): Chapman & Hall/CRC Press; 2005. chapter 8.
34.: Guidelines on Pharmacovigilance of Medical Products for Human Use, Volume 9A of the Rules Governing Medicinal Products in the European Union. Sep, 2008. [August 14, 2012]. http://ec.europa.eu/health/documents/eudralex/vol-9/index_en.htm.
35.: Mangano DT, Tudor IC, Dietzel C, et al. The risk associated with aprotinin in cardiac surgery. N Engl J Med. 2006 Jan 26;354(4):353–65. [PubMed: 16436767]
36.: Newton J, Garner S. Disease registers in England Report commissioned by the Department of Health Policy Research Programme Institute of Health Sciences. University of Oxford; Feb, 2002. [August 27, 2012]. http://www.sepho.org.uk/Download/Public/5445/1/disease_registers_in_england.pdf.

Bookshelf ID: NBK208631

Contents