Steamship Authority releases reports on Martha’s Vineyard internet opening system issues
FALMOUTH (February 18, 2020) – The Woods Hole, Martha’s Vineyard and Nantucket Steamship Authority today released several reports investigating the causes of the January 21, 2020 web server environment issues that affected the general internet opening for Martha’s Vineyard summer reservations.
The reports – one by the Authority, the other by Imarc, the Authority’s website service vendor – detail the steps taken prior to the Vineyard opening to ensure the system’s reliability; the issues that were encountered that day; and the steps taken to correct them. The reports also outline the next necessary steps to bolster the Authority’s system capacities and improve information technology practices so as to ensure improved performance moving forward.
“In reviewing both our internal report and the report from Imarc, it is clear that this regrettable incident is the result of missed opportunities from both parties. In hindsight, it is easy to find ways in which the server environment problems could have been avoided ahead of time. But there are also several points at which reasonable actions should have been taken that would have, if not stopped this from happening, lessened the burdens our customers encountered that day,” General Manager Robert B. Davis said.
The reports, along with a summary written by Mr. Davis, were distributed to the Port Council and Board this morning at their joint meeting, held at the Authority’s administrative office in Falmouth, Massachusetts. The reports are also available on the Authority’s website, www.steamshipauthority.com/about/news.
According to the reports, two main issues negatively affected the website on January 21, 2020:
- The Authority experienced an unexpected volume of connection requests to the servers for the Authority’s mobile website. Those mobile website servers were using an inefficient connection type that taxed the Authority’s web database servers and made it impossible for users to log in and make reservations.
- Following a switch of the mobile servers to a more efficient connection type, the web database servers still were unable to handle the volume of connection requests following the delayed noon opening, leading to their inability to restart correctly and two failures in the afternoon. Only by reducing the allowed number of connections to the server was the website able to function normally.
The causal factors leading to the opening day issues included:
- The mobile servers were not switched to the more efficient connection type after the same change was made to the other web servers following the 2018 opening day reservation difficulties.
- The mobile servers were not included in prior load tests.
- Key system settings were not verified prior to the internet general opening to determine that production servers were properly configured.
- Memory resources on the web servers were exhausted.
Ultimately, the number of transactions processed on the Vineyard general opening (14,853) was more than the number processed in 2019 (14,244), although “that fact is of no comfort to our customers whose day was disrupted by the website’s inability to properly function,” Mr. Davis said.
While the short-term problems with the website were addressed on the day of the Vineyard internet opening, both Imarc and the Authority have identified several mid- to long-range recommendations that will be implemented to further increase the robustness of the Authority’s web services. They include:
- Unified log and resource monitoring
- Review load testing procedures, including current tools used, test design to include mobile servers, and participation of Imarc’s technical team and other vendors in the process.
- Study of website upgrade/redesign in preparation for an RFP issuance by end of 2020 for a back-end and front-end upgrade of site.
In addition, the Authority will fully examine and, where appropriate, execute the specific solutions regarding its IT department in Section 4 of the 2018 comprehensive review of the Authority’s operations performed by HMS Consulting, Glosten Associates and Rigor Analytics. A review of those issues and prioritization of their execution will be among the first tasks for the Authority’s next director of information technologies. The search for that individual is now under way.
“Just as we share responsibility for the website infrastructure crash, Imarc and the Authority share the challenges of moving forward to make significant and lasting changes on our information technology infrastructure,” Davis said. “I look forward to our continued commitment to providing the best service possible for our customers.”