Yet another hiatus. Sorry, I was very busy with my job as a performance architect at Calxeda. Will try to be regular again.
I have recently been interviewing people at Calxeda, my new employer. There are a few fundamental concepts I expect every engineer/CS major to understand, regardless of what position they are applying for. One of them is the difference between a channel’s throughput and its latency. It is surprising how many candidates get it wrong. I will not only try to explain the concepts of latency and throughput using a simple analogy, but also try to hypothesize why IMO most people get them confused.
Quick background: I was first asked to differentiate between latency and throughput when I was a college sophomore. It was during an internship interview at nVidia Austin in Spring 2002, more than a decade ago. Sure enough, I was not very clear about it myself and the interviewer had to explain it to me. I wish I remembered his name to give him credit since I am using the same analogy that he used.
When you go to buy a water pipe, there are two completely independent parameters that you look at: the diameter of the pipe and its length. The diameter determines the throughput of the pipe and the length determines the latency, i.e., the time it will take for a water droplet to travel across the pipe. Key point to note is that the length and diameter are independent, thus, so are are latency and throughput of a communication channel.
More formally, Throughput is defined as the amount of water entering or leaving the pipe every second and latency is the average time required to for a droplet to travel from one end of the pipe to the other.
Let’s do some math:
For simplicity, assume that our pipe is a 4inch x 4inch square and its length is 12inches. Now assume that each water droplet is a 0.1in x 0.1in x 0.1in cube. Thus, in one cross section of the pipe, I will be able to fit 1600 water droplets. Now assume that water droplets travel at a rate of 1 inch/second.
Throughput: Each set of droplets will move into the pipe in 0.1 seconds. Thus, 10 sets will move in 1 second, i.e., 16000 droplets will enter the pipe per second. Note that this is independent of the length of the pipe.
Latency: At one inch/second, it will take 12 seconds for droplet A to get from one end of the pipe to the other regardless of pipe’s diameter. Hence the latency will be 12 seconds.
Queuing: Note that droplets may be arriving at a rate faster than 16,000/second, say 16,100/second. Since the pipe cannot let more than 16,000 droplets to enter each second, the extra 100 droplets will have to wait. Said another way, they are put in queue where they will get to enter the next second. The time a droplet waits in the queue before entering the pipe is called the queuing delay.
Possible source of confusion: Since queuing delay is related to throughput –NOT latency– many scientists/engineers confuse queuing delay with latency and conclude that latency is related to throughput — which is wrong. Latency itself is independent of throughput, its the queuing delay which isn’t.
This and many other cool concepts about queuing are explained by Queuing theory.
one cross section can fit 16,000 droplets, not 1600
(4* 4) / (0.1 * 0.1 * 0.1) = 16,000
It’s supposed the length of a cross section is equals to one water droplet’s length, I think. So one cross section has 4 x 4 x 0.1 cubic inch.
Therefore, one cross section can fit (4 x 4 x 0.1) / (0.1 x 0.1 x 0.1) = 1.600 droplets.
Hi,
In our day to day lives either latency or message size is important but not both. Basically, either you’re using the mail system or you’re talking on the phone (can you name a non-computer situation where both have to be balanced?). Given that networks are rated on their throughput it’s assumed that there’s no latency. It requires a bit of a kick to get people out of this mode of thinking; that both latency and throughput can be important for the same system.
- Andrew
Hi Andrew, call from your past! Sorry, I missed this comment entirely. Well said point. Latency is becoming vey important now IMO. With Amazon and Google and many others saying that latency is == revenue. Google said that 500ms delay costs them 25% searches. Amazon said that 100ms delay costs them 1% in revenue. I think the mindset will get fixed with time. Money is after all the only thing that matters;)
Aater
My example is usually not water pipes,
It is car engines, and I compare throughput =horsepower and latency=torque , but that assumes that the person understands car engines….
Jacob, Interesting suggestion. Will add that to my set of analogies as well, with your permission of course.
Hi,
very ease to understand the throughput and latency.
I guess that confusion stems from another source. It seems that most people imagine networks, applications and so on as if it would be single production line manufacturing cars, for example. And in this case latency is inverse of throughput. Assume, you have one car off the line each 4 hours. It is latency. You have 1/4 of car per hour or 6 cars per 24 hours. It is throughput. The difference appears when we add parallelism – second production line.