Uncovering Hidden Memory Usage -A Dynatrace Investigation Story

In the world of cloud computing, the challenge of optimizing costs while ensuring optimal performance is ever-present. A cost optimization investigation was conducted for a valued customer, which involved a deep dive into the complexities of memory management. Unbeknownst to the investigator, this journey would lead to the discovery of a hidden aspect of memory usage that had escaped the monitoring tools.

The scenario involved the task of reducing cloud costs for a customer by rightsizing their servers. To gain insights into resource utilization, Dynatrace, a robust monitoring tool, was utilized. However, while examining memory usage metrics, a perplexing observation was made—substantial memory consumption was occurring without any specific processes accounting for the usage.

In one instance, nearly 5GiB of memory was being utilized on a web server, but upon closer examination, the top processes accounted for only around 200MiB of memory usage. This discrepancy left the investigator puzzled as to where the rest of the memory was being utilized.

Further investigation revealed that Dynatrace's memory metrics did not provide a breakdown of the memory allocated to shared memory, concealing crucial details about actual memory usage on the system. To gain a clearer understanding, the 'free' command was run, revealing a different story.

The 'free' command output showed that a significant portion of the memory, amounting to 4.4GB in this case, was allocated to shared buffers. This shared memory allocation was attributed to the use of ramdisks, a technique for improving performance by allocating RAM as disk drives.

The scenario underscored the importance of understanding the intricacies of memory usage, particularly regarding shared memory allocations. By uncovering this hidden aspect of memory usage, valuable insights were provided to the customer, enabling informed decisions about rightsizing server instances.

In conclusion, while monitoring tools like Dynatrace are invaluable for tracking resource utilization, it is crucial to complement them with other tools and commands such as the 'free' command to gain a comprehensive understanding of system performance. This approach allows organizations to optimize costs while ensuring optimal resource utilization in their cloud environments.

This insight was inspired by a friend's post on LinkedIn, where they shared their experience with similar challenges and solutions in the realm of cloud computing.

How to monitor a linux server during Load Test | Monitor Linux server without any monitoring tool

Recently I came across a situation where my client doesn't have any monitoring tool but we need to monitor the servers during the load testify the server is windows we can directly use PERFMON default windows monitoring tool, but the Linux doesn't have any such features, so I have created a csv file with a simple command in Linux machine. i have triggered this for

$vmstat -n [delay [count]] | awk '{now=strftime("%Y-%m-%d %T "); print now $0}'>{file path}

Example:  vmstat -n 1 10 | awk '{now=strftime("%Y-%m-%d %T "); print now $0}'>/tmp/vmstat.csv

The given command line combines the output of the vmstat command with awk to create a log file that records the current timestamp along with the system statistics. Here's how it works:

• vmstat -n [delay [count]]: The vmstat command is used to monitor virtual memory statistics and system performance. The -n flag disables the header output, providing only data. The command accepts optional arguments:

  • delay: The time interval, in seconds, between data samples.
  • count: The number of data samples to collect.

• awk '{now=strftime("%Y-%m-%d %T "); print now $0}': This part of the command processes each line of the vmstat output:

  • now=strftime("%Y-%m-%d %T "): This awk code creates a timestamp in the format Year-Month-Day Hour:Minute:Second.
  • print now $0: This prints the current timestamp followed by the original line from the vmstat output.

• >{file path}: The output of the command, which is the timestamped vmstat data, is redirected to the specified file path. For example, /tmp/vmstat.txt.