Holiday Load Testing: Self Service or Full Service?
In this installment of Neustar’s continuing series of advice on holiday preparations to protect your ecommerce platform, we look at two options for ensuring operational performance capacity through load testing.
The first option for executing load tests is to use a self-service model. One advantage to this model is that you can schedule and execute load tests at almost any time. This is especially helpful during the holiday season when unplanned situations arise and companies suddenly find themselves in a pinch. Another advantage is that it gives you full control over the load test: You define the duration, number of concurrent users, and any other test parameters. If the tests don’t go as planned, you can simply adjust the parameters until the tests execute to your liking.
The disadvantage to the self-service model is it requires having employees with some level of technical expertise and experience to properly set up and execute a load test. That expertise may vary, and could include an understanding of:
- Scripting the load testing scenarios
- The concept of concurrent users in load tests
- The infrastructure that the ecommerce platform is hosted on, including the number of CPUs and CPU cores
While technical expertise is not absolutely required for the self-service model, it does provide the benefit of allowing you to properly prepare and execute your load tests the first time. Otherwise, if they’re not executed properly, these tests can waste a good lot of your company’s time, resources, and money.
If your company lacks this expertise, many load test companies offer a full-service model, where the load testing is executed by a Professional Services team. Such a team may:
- Provide initial consultation of your load testing needs
- Provide various script load testing scenarios
- Plan and execute load tests from end to end, as per you request
- Generate reports (with executive summaries) of findings after every test
- Deliver scripts to you for future use (perhaps via self-service option)
While the full-service option usually requires some amount of lead time, especially for more complex engagements, it can be a worthwhile investment. In this engagement, unlike in the self-service model, the engineer assigned will be doing much of the leg work. This frees you to focus on results and modifications as opposed to technical minutiae, which can be distracting. One consideration, however, is the lack of flexibility. For example, if a load test needed to be rushed, consultation between the customer and the professional service team would be required to determine if the proposed timeline were possible.
When a professional services team is hired for load testing, a scope has to be developed and documented in paperwork. If additional work is needed, this also has to be documented and accounted for. All of this is to ensure that the team delivers exactly what you would need.
Regardless of the model of service you choose for load testing, following are steps you’ll need to take beforehand:
- Define the scope of your load test. This includes the specific sites you wish to test and the scenarios you wish to test against. Regardless of which testing model you use, the scope will be needed to determine the cost of the load tests.
- Establish the goals of your load test. Do you wish the test to push your capacity to theoretical maximum or just a specific level that you have defined?
- Establish a timeline for the load testing as early as possible. Once a timeline is established, you can explore whether the self-service or full-service model is for you. If timelines are tight, self-service may be your only option.
- Determine the volume of concurrent users needed for your tests. My prior blog post discusses the importance of concurrent users in load tests.
- Determine how many tests you wish to execute. Note that this number is not fixed and will fluctuate. But establishing an initial baseline number of tests will allow you to determine costs and adjust the number of tests accordingly.
Now is the time to prepare your website for the weight of holiday traffic—and load testing is a valuable tool for doing so. There are many options out there including Neustar’s. We offer both self-service and full-service load testing through our professional services team. To learn more, visit our page on Load Testing.