SRE Insights - 11 Lessons for Tech Resilience

1. Risk Assessment During Incidents:

- Monitor and evaluate incident severity.
- Choose mitigation paths commensurate with risk.
- Informed decisions during broken scenarios.

2. Practice Makes Perfect:

- Regularly practice recovery mechanisms.
- Verify effectiveness through testing.
- Double down on testing for improved reliability.

3. Canary Deployments for Global Changes:

- Implement global changes incrementally.
- Utilize progressive rollout strategies.
- Prevent unintended consequences with canary deployments.

4. Backup Communication Channels:

- Establish non-dependent backup channels.
- Ensure thorough testing of backup communication.
- Vital during unexpected incidents affecting primary channels.

5. Graceful Degradation for Continuous Functionality:

- Design systems for continuous minimum functionality.
- Provide a consistent user experience during degraded modes.
- Intentional and careful construction of degraded performance modes.

6. Disaster Resilience and Recovery Testing:

- Beyond unit and integration testing, prioritize resilience and recovery.
- Simulate extreme scenarios through tabletop exercises.
- Prepare for natural disasters or cyber-attacks.

7. Automated Mitigations for Faster Resolution:

- Automate mitigations for faster resolution.
- Swift responses to clear signals of failure.
- Minimize user impact through automated actions.

8. Frequent Rollouts for Safety:

- Conduct frequent rollouts with thorough testing.
- Reduce surprises through consistent testing.
- Ensure safety in complex, multi-component systems.

9. Diverse Infrastructure for Resilience:

- Maintain diverse infrastructure to prevent total outages.
- Mitigate latent bugs with diverse network backbones.
- The difference between a troublesome outage and a total one.

10. Reduce MTTR with Automated Measures:

- Automate mitigating measures during network failures.
- Clear signals trigger automated mitigation.
- Preserve root-cause analysis for user impact avoidance.

11. Timely Rollouts for Critical Functions:

- Avoid long delays between rollouts, especially in complex systems.
- Frequent rollouts with proper testing prevent critical function failures.
- Maintain diverse infrastructure to identify latent bugs.

Monitor Unix server without any monitoring tool | Linux server monitoring from command mode | Script for server monitoring in performance testing

#!/bin/bash

# Specify the time duration for which you want to collect metrics

start_time="10:00:00"

end_time="11:00:00"

# Interval in seconds

interval=30

# Output file

output_file="metrics.txt"

# Loop to collect metrics every 30 seconds

current_time=$start_time

while [[ "$current_time" < "$end_time" ]]; do

    sar -u -r -d -n DEV -s "$current_time" -e "$current_time" >> "$output_file"

    current_time=$(date -d "$current_time + $interval seconds" +%H:%M:%S)

    sleep $interval

done

Save this script to a file, make it executable (chmod +x script_name.sh), and then run it. It will collect metrics using the sar command every 30 seconds within the specified time range and append the output to the metrics.txt file.

After the script runs, you can open the metrics.txt file in Excel or another tool for analysis. Keep in mind that running this script for an hour will generate a large amount of data, so ensure you have enough disk space for the output file.

Convert Microfocus Loadrunner (VuGen) scripts to Gatling Scripts | Loadrunner to Gatling

Gatling's scripting, based on its specific DSL in Scala, may seem daunting initially. However, the process of converting LoadRunner scripts into Gatling scripts is made accessible through Gatling Recorder and the HAR mode.

Method:
Step 1: Save LoadRunner Script as HAR File: After replaying, save the LoadRunner script as a HAR file. We may see the option on the right hand side of the page as shown in below image.

Step 2: Import this HAR file into Gatling Recorder & Click on the Start button. This will convert HAR file into DSL/Scala script

Step 3:Conversion to DSL/Scala Script: Click on Start in Gatling Recorder, converting the HAR file into a DSL/Scala script.

Step 4:Import into IDE or Editor: Go to the output folder and import RecordedSimulationXXX.scala in your preferred IDE or Editor.


For those with existing scripts in HP Load Generator, the HAR mode provides a bridge to adapt and run them seamlessly in Gatling. In a landscape where cost, concurrency, and efficiency are paramount, Gatling emerges as a beacon of efficiency. Its asynchronous thread approach, coupled with a streamlined process for script conversion, makes it an attractive option for organizations seeking flexibility and cost-effectiveness in their performance testing endeavors. As the software development landscape continues to evolve, Gatling stands ready to meet the challenges head-on, ensuring applications can handle the demands of an ever-expanding user base.

Note: You can also use fiddler to to record the flow and generate .har file, from there we can generate the Gatling scripts too.

Running Multiple Iterations Simultaneously in Load Runner

Consider a scenario where the first iteration involves logging into a site, saving the session's cookies to global variables. Subsequent iterations should skip the login process, utilize saved cookies, and execute REST calls. The goal is to run multiple iterations simultaneously, each starting after a specified delay, thus simulating concurrent user interactions.

One apparent challenge is the pacing settings within testing tools, which typically dictate that the next iteration starts only after the previous one completes. This limitation can hinder efforts to introduce delays between iterations. Possible Solutions are as follows.

Multiple Scripts with Shared Session Cookies:

   - After the first iteration (login), save the session cookies to a file.

   - Create multiple independent scripts that read from the same cookie file.

   - Run these scripts simultaneously, effectively simulating concurrent users.

   - This approach ensures independence between iterations, allowing for parallel execution.

Parallel Execution Tools or Frameworks:

   - Explore parallel execution tools or testing frameworks designed to run multiple instances concurrently.

   - Configure these tools to introduce delays between iterations, achieving the desired pacing.

   - This approach provides more control over the timing of script execution.

Custom Scripting Logic:

   - Implement custom scripting logic within your testing tool.

   - Introduce scripting logic that simulates delays between iterations.

   - This method may require a deeper understanding of the testing tool's scripting capabilities.

Achieving concurrent iterations with session persistence in performance testing may require a combination of creative scripting, external tools, or frameworks. While some testing tools have limitations in directly supporting concurrent pacing, exploring alternative solutions, such as multiple scripts with shared session data or parallel execution tools, can provide the flexibility needed to simulate real-world scenarios effectively.