Emergency $8.7M Contract Awarded on Eve of Healthcare.gov Launch

    In a stunning admission, a recently released government document reveals that the original contract for Healthcare.gov contained no "specific concurrent user performance requirement".  The federal government's online marketplace was plagued from the outset, in part due to the lack of ability to handle the amount of traffic to the site.
    Less than a week before the long-anticipated launch of the Obamacare marketplaces, testing revealed that the cloud computing services needed for the operation of Healthcare.gov and the Data Services Hub could only handle 20% of the projected traffic beginning October 1.  The Centers for Medicare and Medicaid Services (CMS) executed an $8.7 million contract modification on September 30 with Terremark Federal Group in an attempt to rectify the situation.  The Justification document for the modification claims that CMS "could not have anticipated the need"  before the testing conducted on September 26, because the original "task order did not have a specific concurrent user performance requirement":

CMS could not have anticipated the need to add these additional services. Previous resource estimates gave CMS reasonable assurance that the capacity procured through this order was sufficient for the expected demands. CMS learned through performance and stress testing conducted on September 26, 2013 that the Terremark cloud would only be able to handle about 10,000 concurrent users. This was far short of the expected peak traffic of 50,000 concurrent users. The task order did not have a specific concurrent user performance requirement. OIS quickly realized that the cloud services would have to be expanded in order to be able to meet the expected user demands. 

    While the documents authorizing the contract modification were signed by those immediately responsible on September 30, months passed before signatures were obtained for those farther up the chain of command.  Some signatures were added throughout November, but the final signature, that of then-Chief Operating Officer Michelle Snyder, was not added until December 18.  Snyder retired at the end of 2013.

    The $8.7 million modification increased the value of the Terremark contract from $37.3 million to $46 million.  CMS believed that without the additional capacity, the "Exchanges would not function as designed and citizens would not be able to comparison shop for affordable health care insurance." Internal emails at CMS from the week preceding the launch revealed the major testing debacle when the system could not even handle 10,000 concurrent users, much less 50,000, but the emails do not appear to directly reference this cloud computing contract modification.  Subsequent events, however, confirmed CMS's fears, although lack of capacity was far from the only problem ultimately faced by the troubled website.


Note: A version of this post first appeared on The Weekly Standard.