Case Study

Two growing commercial firms with limited IT resources needed to ensure their mission-critical enterprise-class servers operated reliably and securely. RedLine introduced high-availability features and eliminated single points of failure in hardware and software to maximize availability, reliability, and the customers’ return on their investments.

Challenge

Companies that depend upon the availability, performance, and serviceability of their back-office servers often purchase enterprise-class servers designed for this purpose. These systems are frequently configured for the production environment and deployed by the hardware vendor or other systems integrators. Challenges arise when the company’s most important Information Technology (IT) resource is transitioned to local IT support staff, inexperienced with the new class of hardware and software. Over time, failure to follow best practices for systems administration, such as applying regular system updates and routinely monitoring for hardware failures in redundant components, places company operations at risk with a high probability that the systems are not being consistently used to their maximum capacity and capabilities.

Solution

RedLine designed solutions to ensure sustained, secure operations for two different small-to-midsize business customers based upon their specific use requirements and budgets.

Company A

Challenge: A food processing plant’s operations, including accounts receivable, accounts payable, and shipping and receiving, were centralized on a single server. The internal staff managing the server had been making changes to the system in a non-standard way. File permissions were improperly set, operating system maintenance had not occurred since installation and, while system backups were being regularly performed, they were incomplete. Although the system had been configured with a mirrored boot drive, a failure of one of the drives had gone undetected by local staff for over two years. The second boot drive failed following a power outage and the company’s operations came to a halt. Recovery took longer than three days, during which plant operations were effectively halted. Fortunately, the majority of the operations software and data resided on a different storage array or the outage might have lasted significantly longer.

Solution: Once engaged by the customer, RedLine met with the operations team and performed a thorough systems survey and risk assessment. The product of that survey was a prioritized list of recommendations to increase reliability, availability, and serviceability. The RedLine team subsequently worked with the customer to implement those recommendations, which included:

• A backup and recovery solution that included an off-site copy
• Introduction of a second server and a shared storage array configured as a “hot standby”
• System monitoring capabilities that alert the customer and RedLine support staff 24/7
• Implementation of best practices for systems administration that included restructuring file-system permissions to fit a policy of least privilege, and defining processes and procedures that ensured changes implemented on the primary system were accurately reflected on the standby server

In addition to the changes detailed above, RedLine and the customer developed and implemented a shared services relationship that split support responsibilities. Routine systems administration tasks were performed by the customer and RedLine staff managed more complex tasks such as implementing system updates, monitoring system health and performance, scheduling and performing failover testing, and providing 24/7 Tier 2 technical support.

Results:

• Minimizing downtime and revenue losses by increasing availability and uptime with the introduction of a second server on site, better monitoring capabilities, and a disaster recovery solution off site
• Increasing security, decreasing run-time errors, and minimizing downtime by restructuring file system layouts and permissions
• Increasing performance revenue by managing complex tasks and tuning remotely as a part of shared services

Company B

Challenge: A software-as-a-service company relied upon a single server for both operations and development. Deployed as a single system image, the development workload would often adversely impact customer-facing production applications. This single server configuration was susceptible to risk from not only hardware or software failures, but also self-induced outages and degraded performance, resulting in an unsatisfactory user experience for its customer base.

Solution: After performing extensive analysis and system profiling, RedLine made a number of recommendations to reduce risk and increase availability and reliability. Similar to the solution above, a second server and shared storage array was introduced. Due to budgetary constraints, the second server was configured with fewer physical resources such as CPU and memory. The systems were configured for high availability, separating production and development environments. In the event of a production server failure, development would halt and production would operate on the smaller server, albeit with reduced performance, until the production server could be repaired.

Results: RedLine’s solution minimized downtime and revenue losses by increasing availability and uptime with the introduction of a second on-site server solution.