
Dr. Sarah Connelly, FDA clinical reviewer and Medical Officer, Division of Antiviral Products, presented some interesting and useful survey results at the DIA Annual Meeting on June 16th. The survey gathered information on FDA reviewers’ perception of submission quality in a number of specific areas. It was a collaboration between FDA and the DIA Medical Writers SIAC.
The survey is hot off the presses, having been completed in early June. It was designed to assess:
Responses were tabulated specifically in the CMC/Quality, Nonclinical and Clinical areas. About 53 reviewers responded.
CMC
In CMC, overall writing quality was rated fairly highly: at about 12% poor or fair, 42% average, and 46% good or excellent. However, 17% judged organization or information as poor or fair, and 31% considered completeness of information poor or fair. The worst ratings were given to adequacy of cross-references (with 38% rating fair or poor) and adequacy of bookmarks and bookmark names (32% rating fair or poor). .
In the area of M2 summaries, 19% rated quality as fair or poor, 52% as average, and 29% as good or excellent.
In comments, reviewers highlighted the following issues:
Nonclinical
In the nonclinical area, overall writing quality was rated at about 28% poor or fair, 34% average, and 37% good or excellent. Somewhat higher ratings were given to organization and completeness of information. The worst ratings again were given to adequacy of cross-references (with 46% rating fair or poor) and adequacy of bookmarks and bookmark names (52% rating fair or poor). This is not surprising considering the quantity of scanned documents presented by sponsors in nonclinical, and the reluctance of sponsors to spend time enhancing the navigation aids in these documents.
In the area of M2 summaries, 29% rated quality as fair or poor, 40% as average, and 31% as good or excellent.
In comments, reviewers highlighted the following issues:
Clinical
In the clinical area, ratings were generally higher – not surprising again as most sponsors have a larger budget and an in-house or contracted staff of professional medical writers producing clinical documents. Quality of clinical studies in general was reasonably good, with 22% considering reports poor or fair, 36% average, and 42% poor.
Overall writing quality was rated at about 15% poor or fair, 35% average, and 43% good or excellent. Similar ratings were given to completeness of information, and few reviewers had complaints about organization (possibly due to the adoption of E3 as a standard for clinical study reports). The worst ratings were given to adequacy of cross-references (with 27% rating fair or poor) and adequacy of bookmarks and bookmark names (35% rating fair or poor).
In the area of M3 summaries, 17% rated quality as fair or poor, 39% as average, and 42% as good or excellent. ISS and ISE were a little less satisfactory.
In comments, clinical reviewers mentioned similar issues repetition (including cutting and pasting into summaries), lack of interpretation, non-searchable documents, and inadequate narrative summaries. They also mentioned lack of hyperlinks and difficulty in finding information. On a somewhat harsher note, some reviewers believe that the people writing clinical summaries don’t understand the science.
Summary
There are certainly some significant trends in the feedback from reviewers – poor quality bookmarking and hyperlinking, repetition, and lack of summary and interpretation being the most consistent. Something that struck me was the lack of “excellent” ratings – only one area reached as high as 10% and most were well below that. I interpret that as meaning almost all sponsors have some room for improvement. This is a great opportunity to review your practices and see where you might not be meeting the reviewers’ expectations. After all, impeding the reviewers in making their assessments is a major contributor to complete response.
I’m going to deviate from my usual approach of not mentioning my company’s product s and services to mention that our soon-to-be-released GlobalSubmit 2010 release has significant enhancements in support of the quality of PDF documents, bookmarks and hyperlinks. Not only will we identify over 40 PDF-related errors at the validation stage, but our new CrossCheckTM product provides a revolutionary way to QC bookmarks and hyperlinks without having to open target documents. Our clients who have previewed CrossCheck view it as a real productivity boost. For further information or a demo, contact sales@globalsubmit.com.
Finally, Dr. Connolly’s presentation, CTD/eCTD Quality: FDA Survey Results is available on GlobalSubmit’s presentation page.
Attending the DIA EU EDM Conference in December gave me a great opportunity to catch up on eCTD-related status and activities at various European agencies. We heard from a number of presenters representing EMEA (now just European Medicines Agency), the MEB, SwissMedic, and AGES PharmMed. Since the updates are fairly lengthy, today I’ll cover EMEA, and will address the other agencies in a future posting.
Tim Buxton gave the update from the EMEA. He clarified what eCTD implementation means to this agency:
EMEA now receives over 500 eCTD sequences a month. In November, they also received 149 NeeS sequences. (A further update can be found in the recently published Update on the implementation of the EU Telematics strategy, which states that since July 1st 2008, over 2,500 eCTD submissions have been received by EMEA, and 406 centrally-authorised products are managed in eCTD format, representing more than two thirds of the total number of centrally-authorised products).
electronic Application Form (eAF)
Although this was an important initiative for the EMEA, adoption has not been good in the past because no tool was provided to create this XML document. The upcoming release of the eMF will include a Data Exchange Standard, receiving tool (initially EMEA only), authoring tool, and validation tool. Prototypes of the receiving tool and authoring tool under evaluation. Support for variations is still under development.
PIM
Likewise, for PIM, EMEA is delivering a Data Exchange Standard, PIM Review System, PIM Light Authoring Tool, and PIM Data Validation Engine. A statement of intent and migration details are still in pilot. The timetable for PIM (from the Statement of Intent) is:
There has been a change of approach for migration to PIM - EMEA had planned to migrate sponsor’s data but sponsors want to do it themselves with “hand holding”.
eSubmission Gateway
The eSubmission Gateway is in production for ICSRs and has been tested for MAAs. Tim characterized the go-live of the gateway as “around the corner” (but said he was glad that he declined to commit to a date at the DIA annual back in June).
Digital Signatures
A limited pilot was being conducted for digital signatures, but is on hold right now due to other priorities. It won’t be completed in 2010, but may be implementatd in 2011. SAFE is a not the only valid form of eSig. Rules in some countries specify some types of electronic signatures. The EC has just released a call for ideas on how to harmonize eSignatures requirements across Europe. To quote Tim - “The storm for eSig is just around the corner – ignore at your own peril.”
Other Initiatives
Other current initiatives include identification of medicinal products, and ICSRs (update of E2B standard for better ID of medicinal products causing problems).
Upcoming initiatives include eCTD Next Major Version (Regulated Product Submissions) – by the way EMEA has just added a web page for this topic, including links to last year’s meeting minutes.
New Q&A/Change Request Tracking Table
In other EMEA news, a new version (V1.21) of the EU Telematics EU eCTD Change Request/Q&A Tracking Table has been posted. This is an update following discussion of open CRs by the TIGes subgroup, and general review of status and presentation of all CRs. All closed/withdrawn/rejected/duplicated CRs have been moved to new worksheets; all CRs implemented in EU M1 v1.4 were moved to the appropriate ‘Implemented in EU M1 v1.4′ worksheet. Most importantly, all CRs for a potential EU M1 v1.4.1 (spec update only) have been identified and marked.
Well,it’s back to business after the holiday break… and as promised, here’s my update on what’s happening at Health Canada.
At the recent DIA eSubmissions conference, Health Canada’s Vianney Caron presented an update on eCTD at Health Canada.
The E-Review Program
The purpose of Health Canada’s E-Review program is to support branch business processes throughout the health product life cycle across product lines using an integrated secure electronic environment that facilitates the exchange of information and conforms to international standards.
Health Canada has made a lot of progress in E-Review recently. Important milestones achieved include:
Regulatory Activities in eCTD Format
Metrics were provided on eCTDs received:
Cumulative through September of 2009:
Totals for 2009 (as of September), received from 53 sponsors:
Health Canada is not seeing a significant rise in applications and sequences received, such as has been experienced by the FDA and EMEA – perhaps because they still require at least part of the dossier in paper. As of 2009, 8.7% of regulatory activities are received in eCTD format.
eCTD Validation
Sponsors are improving the quality of their submissions to Health Canada. In 2009, 4.5% of sequences failed validation. This is down considerably from previous years: 8.3% failed in 2008 and 11% failed in 2007.
Top 10 errors and warnings encountered during validation include:
1. Inactive bookmarks -no link destination
2. Inactive hyperlinks -no link destination
3. External links -pointing to the folder or destination that does not exist in the folder structure on the CD/DVD. The folder or destination would only exist in the sponsor’s repository.
4. Incorrect naming of the 3011 form; correct name is hc-sc-3011-en.pdf
5. Unreferenced files in the index.xml or ca.regional.xml
6. Life cycle management of the document: invalid file reference or no previous ID found
7. Subfolders created in ca regional folder
8. Missing top level folder
9. Missing attestation letter; content as per eCTD Guidance, letter needs to be dated and signed
10. Printed content of MD-5 Checksum does not match the one in the index-md5.txt file
Preliminary Experience with eCTD
Health Canada reports that they have buy-in from all stakeholders, but no legal basis to make eCTD mandatory. They continue to experience logistical and process issues. Reviewers perceive that paper & electronic in parallel is difficult and the process is excessively manual. Validation is perceived as an extra step. Other review issues including understanding lifecycle management in the eCTD and dealing with scanned documents and resolution of graphics.
Some upcoming milestones include the retirement of hybrid submissions as of January 2010 (this month). A date is yet to be announced for accepting fully electronic submissions.
Assessment of Hybrid Pilot
Health Canada surveyed reviewers participating in their hybrid pilot with some interesting results:
Next Steps at Health Canada
Health Canada defined their next steps as:
Health Canada closed by saying that they strongly recommend electronic submissions in eCTD format, and support the development of common standards. They will be expanding the existing infrastructure to support as many application types across product line as possible. Questions can be sent to them at ereview@hc-sc.gc.ca.
Announcement: See GlobalSubmit’s web site for a brand new presentations page linking to recent agency presentations!
GlobalSubmit recently attended two informative DIA conferences:
During these conferences, industry received updates from a number of regulators, including the FDA, Health Canada, the European Medicines Agency, the MEB, AGES PharmMed, and SwissMedic. In my next series of blog postings, I’ll be passing on interesting news from the regulators. Since there’s quite a bit of material, I’ll cover it in three postings: US, Canada, and Europe.
The FDA’s presentations focused on three major areas:
Status and metrics for FDA initiatives
Gary Gensinger provided updates in both San Diego and Vienna. Some of the more significant metrics included:
Gary spoke about the importance of the emerging RPS standard. He emphasized that RPS is critical to FDA’s meeting their PDUFA IV commitments – as well as supporting their goal of to conducting all their business electronically. Gary referenced a Draft Standard for Trial Use (DSTU) date of January 2010, and a target acceptance date for RPS Submissions for drugs and biologics of the 4th quarter of 2011.
Gary also provided an update on the DARRTS initiative. DARRTS is “A flexible, integrated, fully electronic workflow tracking and information management system to receive, log, track, assign, process, and manage official submissions with internal and external stakeholders. The system maintains the official submission records and will manage and track all communications and documentation concerning submission.” Release 3 of DARRTS was implemented successfully in July 2009, resulting in the retirement of 17 legacy systems. Phase 4 (CDER and CBER BLAs) is being planned.
Data Standards
Lilliam Rosario described a major FDA challenge: The FDA receives massive amounts of clinical research data in extremely disparate formats, using a variety of proprietary standards. This makes it extremely difficult, if not impossible, to do cross-study and application reviews.
FDA has been working towards a standardized approach to capture, receive, and analyze study data. Standardization of study data is vital to integrate pre-marketing study data and post-marketing safety data to improve public health and patient safety. Central to this vision is the creation of an enterprise data infrastructure within FDA to improve the management of all structured scientific data (Janus).
Data standards to support this vision are needed in three broad categories: Exchange standards, analysis standards, and terminology standards. FDA is moving towards XML exchange standards based on the HL7 Reference Information Model to submit study data to the FDA. FDA is also currently working on a proposed rule that would require the electronic submission of study data to the FDA. Study data content for creation of SDTM views will be sent to FDA as an XML files modeled using the HL7 RIM.
Study Tagging Files
Virginia Ventura of the Office of Business Process Support spoke on “Study Tagging Files: Their Vital Role In Submissions To The FDA.”
Virginia described some of the most common problems she sees with STFs:
She clarified that an STF is needed any time you are including documents in Modules 4 or 5, except 5.2 Tabular Listings, and 4.3 or 5.4 Literature references.
A case study provided by Virginia should give sponsors pause.
She continued by providing descriptions of additional problems and their correction strategies:
And wrapped up with some sound advice if you run into problems:
Next time… Health Canada news.
Rob de Haan, deputy director at MEB, spoke about the status of eCTD at MEB, and also presented a variety of EU statistics, many of them updates from the eCTD Implementation Survey Report published in December of 2008.
MEB now has a full e-working environment (although they make almost no use of their eCTD viewer, for reasons Rob didn’t go into). eSubmissions (eCTD and NeeS) for human medicines are mandatory. Case managers and assessors have secured access to electronic files in a central repository, and thus can work remotely (through Citrix) without downloading files. Veterinary submissions are still on paper. eSubmissions are via CD or DVD now as MEB does not yet have an electronic gateway.
Europe is making slow progress toward the goal of all agencies excepting eSubmissions without paper by the end of 2009. Currently 15 out of 32 agencies are ready (up from 13 six months earlier). Agencies report receiving 54% of all submissions electronically in Q3/Q4 2008, out of a total of 203,941 submissions.
When broken down by submission type, the total percentage by format was as follows:
· 6% eCTD (up from 3% in Q1/Q2)
· 58% NeeS (down from 62% in Q1/Q2)
· 31% Other (down from 33% in Q1/Q2)
[not quite sure why the totals only add up to 95% or why eCTD+NeeS at 64% is much higher than the total of 54% electronic reported in the previous paragraph, but you get the idea!]
At this point, DCP/MRP are about 54% electronic, NP about 50% electronic, and CP almost 100% (see my previous post for EMEA experience).
Rob’s observation is that NeeS will eventually disappear and no longer be accepted by the individual HAs. This won’t happen right away but validation will continue to get tougher, with HAs adding more rules and moving to zero tolerance.
Most agencies really want to move to eCTD, but obstacles remain such as pushback from the sponsors (who sometimes say that NeeS is easier than eCTD). In addition, some regulators don’t want to have to train reviewers and install review tools. Finally, of course, some countries don’t have a legal basis at this time for accepting electronic documents as official or archival copies in lieu of paper, and there is only so much the HAs can do to push legislation.
I caught up with Tim Buxton of EMEA at the DIA meeting in June. Tim’s big news is that EMEA’s Electronic gateway will be operational shortly, although he could not release a date. The gateway has already been tested with a variety of submissions.
Tim stressed that the EMEA is serious about enforcing the use of eCTD and no longer accepting NeeS as of the published 1 January 2010 deadline. He mentioned that forward-looking companies are already looking beyond eCTD and asking him about RPS.
In Tim’s presentation, he mentioned that the eAF (electronic application form, which is an XML form) will be ready by the end of the year. EMEA is finalizing their receiving tool as well as an authoring tool that will be available both on-line and stand-alone.
In the area of PIM, they are finalizing a stand-alone light authoring tool that will be downloadable from their web site. Although they are still working out kinks in the pilot project and are not ready to make PIM mandatory yet, Tim said that “it won’t be too long” – just the push that sponsors have been waiting for… in the meantime, EMEA will assist sponsors in converting.
Next post, news from the MEB and European statistics…
This year’s DIA annual meeting was more of a checkpoint rather than revealing startling new information – at least in the eSubmissions area. However, a lot of interesting information was exchanged. Today, I’ll focus on the FDA’s presentations. I’ll follow up later with information from other authorities and industry.
Gary Gensinger provided his usual update on eSubmissions at the FDA:
· In the CDER town hall meeting, Gary noted that FDA is looking at drafting legislation to require eCTD – however, this is a lengthy process and no target date is being released at this point.
· As of shortly before the meeting, FDA had received a total of 6015 applications consisting of 72154 sequences. FDA expects to receive in excess of 4500 eCTD sequences in July of 2009.
· There are some interesting trends by application type:
o 52% of NDAs received are now fully electronic.
o ANDAs have grown by 70% since November 2008 – a huge uptick. If you’re in the generics industry and still submitting on paper, you are behind the curve.
o While only 12% of INDs are submitted electronically (up by 1% since last year), 28% of IND Amendments are submitted electronically – a result of maintaining an ever larger number of INDs submitted electronically to date. (By the way, I had an FDA reviewer who stopped by my booth mention that he cringes when presented with a paper IND these days…)
o An impressive 83% of Efficacy supplements are electronic, up by 12%
o CMC supplements are holding flat at 47% electronic
· FDA is working on an agency level receipt date guidance (the current document applies only to currently only CBER/CDER)
· Gary reminded sponsors that briefing packages must still be submitted in paper even if they are also submitted as an eCTD sequence.
Mark Gray gave an overview of RPS, and stated that RPS will be in production use at FDA by September 2011. Mark and Gary mentioned that ICH will not be making any significant updates to the eCTD spec – there will be no DTD changes as ICH and the regulators look towards moving to RPS as “eCTD Next Major Version”.
The DIA Annual Meeting will be held in San Diego June 20-25. If you are attending and we haven’t met yet, I’d love to meet you! Look for me in our booth, 923 (get your badge scanned to enter a drawing for our extremely cool giveaway - I’m not going to reveal what it is). I’ll be joined in the booth by my collegues and well-known industry experts Jason Rock and Rahul Mistry.
I’ll also be in some of the eCTD-related sessions, which include:
I’ll be commenting on the presentations on this blog in late June or early July.
At the DIA Doc Management conference, the FDA presenters spoke with one voice on their intention to start rejection sequences that have what they consider significant technical flaws.
Gary Gensinger reviewed the guidance that has been issued by FDA on this topic (see my previous post) and the three types of eCTD Validation:
The most common problems encountered during ASR validation include:
The eCTD tool validation done with GlobalSubmit VALIDATE 5 enforces the criteria defined in Specifications for eCTD Validation Criteria. For those of you who have received validation reports back from FDA, those reports were generated directly from GS VALIDATE.
Virginia Ventura also discussed validation in her presentation “CDER eCTD & Validation, Issues Update”. As always, Ginny’s presentation gave helpful, practical advice. Ginny gave the following as top reasons for rejection:
In discussing rejections, she made the following points:
Separately, we have been hearing that FDA already stepped up the rejection process. Below is an example of the rejection notice issued by FDA for technically deficient submissions.

DIA held the 22nd annual Doc Management meeting in Philadelphia the week of February 9th. Attendance at the conference was good. I talked with colleagues from many different sponsor companies and vendors - both folks I already knew and new acquaintances. I was interested and encouraged to hear that most are seeing progress in eCTD adoption, despite the recent economic downturn.
On “Regulatory Day”, we heard from a variety of FDA speakers on the topic of eCTD (and RPS), including:
Some key points about eCTD that I heard from the FDA presentations included: