Replication
Replication
The process of copying data or VM state to another system or location.
In everyday business: It can shorten recovery time after a server or site failure, but copied corruption can also replicate.
STORAGE TROUBLESHOOTING
Storage Performance Troubleshooting Playbook
Use latency, IOPS, throughput and queue behaviour to locate a storage bottleneck instead of blaming the slowest-looking disk graph.
Book 4 | Chapter 5
Chapter opening
Chapter 5 moves from the course overview into storage performance troubleshooting. The aim is not to memorise product menus or commands. The reader should understand what problem the technology solves, how it fits into the wider storage technical textbook, what normal operation looks like, and how to prove the result in the field.
The previous chapter introduced iscsi multipath and san troubleshooting playbook. This chapter continues the sequence by focusing on storage performance troubleshooting playbook, so the reader can connect the architecture to a more specific operational problem.
Learning objectives
- Explain the main purpose of Storage Performance Troubleshooting in plain language.
- Distinguish capacity, performance, resilience and backup instead of treating them as the same requirement.
- Use evidence to separate a design problem from a configuration or physical fault.
- Describe the business impact of a failure and the evidence required for handover.
Why this matters in a real company
Storage is where company files, databases and virtual machines live. The business impact is not only capacity: performance, resilience, backup and recovery determine whether data remains usable when something fails.
Course: Storage Technical Textbook
PLAIN ENGLISH
Technical terms used in this chapter
You do not need to memorise the jargon first. Understand what each term does and why somebody running a company would care about it.
Throughput
Throughput
The amount of useful data moved in a period of time.
In everyday business: It helps describe how fast large backups, files or storage transfers can complete.
Snapshot
Snapshot
A point-in-time storage or VM state used for short-term rollback depending on the platform.
In everyday business: It can quickly undo a bad change, but it is not automatically an independent backup.
Latency
Latency
The delay before an operation or packet completes.
In everyday business: Users experience high latency as slowness even when headline bandwidth looks good.
Backup
Backup
An independent recoverable copy of data or system state.
In everyday business: It is the safety net for deletion, corruption, ransomware, hardware loss or major mistakes.
iSCSI
Internet Small Computer Systems Interface
A protocol that carries block-storage commands over IP networks.
In everyday business: A server can use remote storage almost as if it were a local disk.
RAID
Redundant Array of Independent Disks
A method of combining drives for capacity, performance and/or tolerance of certain drive failures.
In everyday business: A failed disk does not always mean immediate data loss, but RAID is not a backup.
IOPS
Input/Output Operations Per Second
A count of how many storage read/write operations complete each second.
In everyday business: It helps describe workloads with many small file or database operations.
BUSINESS TRANSLATION
What this means outside the server room
This topic affects where company data lives, how fast applications respond and whether the business can continue after hardware or data problems.
RAID
What it means: RAID combines several drives so the storage system can keep operating through certain drive failures, depending on the RAID level.
In normal business language: Think of RAID as keeping the business open when one filing cabinet drawer breaks. It helps availability, but it does not protect you if somebody deletes the wrong folder, malware encrypts the files, or the whole building is lost.
Why the decision matters: Use RAID to reduce downtime from disk failure, and use backup for recovery from deletion, corruption, ransomware and larger disasters.
Virtual Machine
What it means: A virtual machine is a software-defined computer with its own operating system, memory, CPU allocation, disks and network interfaces.
In normal business language: It behaves like a separate server even though several VMs may share one physical machine.
Why the decision matters: VMs are useful for separating roles such as domain control, file sharing and applications while keeping management and recovery flexible.
Backup
What it means: A backup is an independent recoverable copy of data or system state.
In normal business language: Think of it as having a duplicate of important company records stored safely away from the live filing cabinet.
Why the decision matters: A backup only counts if it can be restored. Test recovery, not just backup completion.
Snapshot
What it means: A snapshot is a point-in-time state of a storage volume or VM on the same platform.
In normal business language: It is like a quick undo point inside the same office. Very useful for rolling back a mistake, but not enough if the whole storage system is lost.
Why the decision matters: Use snapshots for fast rollback, but keep independent backups for real disaster recovery.
Replication
What it means: Replication copies data or VM state to another system or location.
In normal business language: It is like keeping a second live copy of the filing room at another branch. It can reduce recovery time, but bad changes can also be copied.
Why the decision matters: Replication improves availability and recovery speed, but it does not replace retention and backup.
NAT
What it means: NAT changes IP addresses as traffic crosses a network boundary.
In normal business language: It is similar to a company switchboard: many internal extensions can share one public number.
Why the decision matters: NAT helps with address use and publishing services, but it is not a substitute for firewall security.
5.1 There is no universal "good latency" number
NetApp explicitly notes that it does not define one universal latency or IOPS threshold for all workloads. Use the stable normal state of the real application as a baseline and investigate changes together with user symptoms.
Measure the workload shape
- Read vs write.
- Random vs sequential.
- I/O size.
- IOPS.
- Throughput.
- Latency.
- Queue depth/concurrency.
Low IOPS can still show odd average latency
With very few operations, one slow request can skew the average strongly. NetApp documents this statistical effect. Do not declare a storage system unhealthy from one average-latency value without checking operation count and application symptoms.
Find the delay layer
- Application: is the app waiting on storage or something else?
- Host: filesystem, queue, CPU, memory pressure?
- Network: packet loss, errors, congestion, iSCSI path?
- Array: controller/cache/data layer?
- Media: SSD/HDD response and rebuild/background work?
Background operations
RAID rebuild, scrubbing, snapshots, replication, deduplication and backup can consume resources. Correlate performance changes with maintenance activity before replacing hardware.
Baseline
Keep normal busy-hour graphs for IOPS, latency and throughput. A baseline is more useful than a generic Internet threshold because it shows what the application normally needs on this exact system.
5.2 Latency breakdown
When the platform exposes component latency, separate network, protocol, controller/data and disk/media delay. This prevents a high total latency number from being blamed on drives when the delay is actually in the network or application layer.
Read/write asymmetry
A workload can read well and write poorly because parity, cache policy, replication or media characteristics affect writes differently. Always split read and write metrics when the platform supports it.
Benchmark carefully
Do not run destructive or unrealistic benchmarks on production data. A synthetic sequential test may prove interface throughput but say little about a database workload with small random writes.
Rebuild impact
Measure performance before, during and after a rebuild if the system is business critical. The client should know whether a second failure or heavy production load during rebuild changes the operational risk.
5.3 Throughput math
Approximate throughput can be related to IOPS and average I/O size: throughput is roughly IOPS multiplied by bytes per operation. This is arithmetic, not a performance guarantee. Protocol overhead, caching and mixed I/O patterns affect what the application sees.
Queue depth
A deeper queue can keep fast storage busy, but it can also increase latency because more work waits in line. Compare queue depth with application response time rather than tuning it in isolation.
Cache hit effects
Reads served from cache can be dramatically faster than reads from media. A benchmark that fits entirely in cache may not represent a large production dataset. Record test duration and working-set size.
HDD vs SSD behaviour
Rotating disks pay mechanical seek/rotation cost, so random I/O behaves differently from sequential I/O. SSDs remove mechanical seek but still have controller, flash-management and endurance characteristics. Pick media from workload and resilience requirements, not only headline throughput.
Network ceiling
A storage array capable of high throughput can still be limited by a 1 GbE client link, an oversubscribed switch uplink or a single TCP/session path. Confirm every layer's negotiated speed and utilisation before concluding the array is slow.
DEEPER TEACHING
Reading storage performance like a workload
Storage performance is often discussed with three numbers: IOPS, throughput and latency. They describe different things and must be read together.
IOPS
IOPS counts completed I/O operations per second. Ten thousand tiny random operations and ten thousand large sequential operations are very different workloads even though the IOPS number is the same.
Throughput
Throughput measures the amount of data transferred per second. A rough arithmetic relationship is:
Throughput ≈ IOPS x average I/O size
Example: 5,000 IOPS with an average I/O size of 16 KiB is roughly 80,000 KiB/s, or about 78.1 MiB/s before protocol/filesystem effects. This is arithmetic, not a performance guarantee.
Latency
Latency is the response time for an operation. NetApp exposes IOPS, throughput and latency as separate metrics because a system can move substantial data while individual requests are still slow. User-facing transactional applications often notice latency before they saturate headline throughput.
Worked diagnosis
Suppose a VM host reports:
| Metric | Normal busy hour | Problem period |
|---|---|---|
| IOPS | 4,000 | 4,500 |
| Throughput | 120 MiB/s | 125 MiB/s |
| Average latency | 2 ms | 28 ms |
| Queue depth | 4 | 42 |
Demand increased only modestly, but latency and queue depth rose sharply. That pattern points toward a service bottleneck rather than simply "the users are doing more work". Now correlate with rebuilds, backups, network drops, controller utilisation, cache status and media latency.
Do not benchmark the wrong thing
A large sequential benchmark can prove maximum transfer throughput while telling you little about a database that performs small random writes. Match the test to the workload and avoid destructive production benchmarks.
Baseline beats folklore
There is no single latency threshold that defines every workload as healthy or unhealthy. Record the real system when users are satisfied, then compare future incidents against that baseline.
WORKED BUSINESS SCENARIO
Scenario: storage is online, but the business says it is slow
A storage system has free capacity and all disks show healthy, yet accounting reports are taking much longer and VM backups are overlapping into business hours.
What the chapter teaches us: Capacity and health LEDs do not prove performance. The investigation must separate application delay, host queues, network path, controller/cache behaviour and media performance.
Think it through
- Which measurements distinguish latency from throughput?
- Could the network be the bottleneck even when the array is healthy?
- What baseline would help prove whether the problem is new?
PRACTICAL WORK
Hands-on lab
Document a lab storage path from application to virtual or physical disk. Record usable capacity, RAID layout, network path if any, and backup destination.
Troubleshooting exercise
Storage is slow. Separate application, host queue, network, controller or cache and media metrics before replacing drives.
Chapter field checklist
- I can explain the subject without relying only on acronyms.
- I can draw or describe the main traffic, storage, power or service path.
- I know what normal operation should look like.
- I know which logs, counters or test results prove the result.
- I can explain the business impact if this component fails.
- I would document the final configuration and evidence at handover.
Chapter summary
Storage Performance Troubleshooting Playbook should now be understood as a business service with a technical implementation, not simply a collection of commands or product names. The important habit is to know the purpose, understand the dependencies, measure the result and document what was proven.
END OF CHAPTER
Review questions
- Explain the subject in plain language to a business owner.
- List the main dependencies that must be healthy before this service can work.
- Describe one realistic failure and the first three pieces of evidence you would collect.
- Explain what should be documented at handover.
- Why are RAID and backup different controls?
- How do latency, IOPS and throughput describe different aspects of storage performance?
A good answer should explain the reason, not only repeat a product name or command.
Technical references
Use the current project specification and the exact product documentation for the installed equipment. These references support the technical principles used in this guide.