I genuinely enjoy listening to Neil Degrasse Tyson. Regardless of the subject, sometimes, I find myself losing the track of time while listening, watching, or reading his works!
Probably, being a space geek is one of the most obvious reasons for this!
The cover of Astrophysics for People in a Hurry
But the passion, the excitement, and the way he articulates complex subjects in a very approachable manner are equally as effective.
I remember once, in particular, he was talking about measurements. Basically, he was saying that there is no such thing as..! Well, What’s the fun in me telling the story? Let’s see what he actually said, instead of just mentioning the gist of it here!
The Uncertainty Principle1
“Oh, of my favorite subjects to think about is measurement, something we so take for granted every day. There are measurements all the time that we’re exposed to, that we’re handed, that we do ourselves.”
Having someone help you to measure your height may improve accuracy :D (https://bit.ly/3gYdtWo)
“What’s a typical one? Oh, how tall are you? Oh, I’m 5’ 8”, you say. Well, I say, well, how do you know that? Well, I was just at the doctor’s. I just had a checkup last week.
OK, how did they measure your height? Well, I took off my shoes. And I stood on this little thing. And there’s a little rod that comes up and bends down over the top of my head. OK, fine. And they made the reading. There it was. 5’ 8”. I might ask, was it 5’ 8” and 1/2? No, it was right on 5’ 8”.
OK, you’re content with that answer. But it was a measurement. And every measurement ever made, and ever will be made, has uncertainties built into it. You may see that term in books called measurement error. But it’s not an error in the sense that you made a mistake. That’s why I stay away from the word in that context. It’s a measurement uncertainty.”
To conclude, “all measurement in life comes down to the approximation that you’re comfortable with”. In fact, “This is true for anything you will measure. There is no precise answer. There’s only the answer that you’ll be happy with.”
I, personally, like to think of this concept this way: Every measurement you make, about anything, is as accurate as your measuring device. For instance, this one should ring a bell for all of us:
Here we’ve created a very simple measuring device, to measure the execution time of the surrounded block of code.
Even in this simple example, our measurement device can be wrong or off for a lot of reasons. Did we account for NTP resets here? How about GC pauses? What if our process get preempted while executing the code? What if compiler threads compete for too much resources? CPU migrations? False sharing? CPU caches?
These questions, among many others, are the reasons why our measurement might be off. Most of the time, we are so concerned about what we’re measuring. This is a very valid concern, to be fair. However, knowing our measuring device is equally as important as what we’re measuring.
Enough with the concepts, let’s see a more concrete example.
First, we need to implement something that we can measure its performance. To that end, let’s implement a simple gRPC service the echos back whatever it receives. Here is the Protobuf description:
And the corresponding service implementation:
To bootstrap the gRPC server, we can do something like:
Here, we’re registering our simple
EchoService as one of the service implementations. After that, we’re making sure that the server listens to port
9000 for the incoming requests.
Now that we have a working server, let’s measure its throughput.
For starters, let’s send 10 thousand requests to our gRPC server. First, the connection:
Here we’re using the work stealing pool to handle the client requests. The second step is to initialize a
CountDownLatch equal to the number of requests:
This way we can wait for all the requests to finish.
Open fire? (https://bit.ly/3p4k7gi)
Now let’s send some requests:
Here we’re sending 10000 requests. Each time we receive a response from the server, we decrement the latch by one. After sending all requests, we use the
latch.await() to wait for all responses to arrive.
Finally, let’s print our measurements:
The first time we run the benchmark, the result would be something like:
Probably, the server didn’t warm up properly, yet. Re-running the same benchmark a few more times and the result would be more stable:
It’s better but I believe we can do even much better!
Let’s implement the same benchmark using Golang. First off, we should generate the proto stubs via the
Next step, let’s connect to the server:
Then we should prepare the
WaitGroup (latch implementation in Golang):
ELF archers are ready to go!
Here we’re launching 10000 goroutines to send the benchmark requests. Each time a goroutine performs its duty, we decrement the
WaitGroup. Different syntax but the logic is the same as the Java code. Now let’s see the result:
So the same gRPC server just handled 34K RPS! Something doesn’t quite add up there.
So what can be the culprit for such a huge difference in the throughput of the same server? Maybe Golang is just much more powerful than JVM:
Goroutines and Channels for the win! (https://bit.ly/34oqbbS)
Golang might found a better way to wait for IO! But let’s compare these measuring devices for a moment to find a better explanation:
- Java is a compiled language, which compiles to an intermediate language. JVM is responsible for executing the compiled IL or bytecode. The HotSpot implementation is using a technique called Profile Guided Optimization (PGO) to optimize the code while executing them.
- Golang is a compiled language, as well. However, the code will be compiled to a native binary that is directly executable on the machine. So all the optimizations are done at compile-time. Consequently, there is no PGO there.
Two valid technical choices with dramatic effects on the runtime. The more the HotSpot JVM executes a piece of code, the more the optimizations, and possibly a better performance. On the other hand, Golang artifacts already packaged with the native instructions. Therefore, there isn’t much happening at runtime in terms of compiler optimizations.
A few moments ago, we did send a few thousands of requests to the server, just to warm up the Java server (what we were measuring!). On the contrary, we didn’t give any chance to the Java client to warm up properly!
Once I warm up, bring it on! (https://bit.ly/3nyBu8O)
To test this hypothesis, let’s perform a few rounds of warmup:
Here we’re running the same benchmark 10 times in the same JVM instance. And the result:
Look at the sudden jump in RPS between the first and second iterations, 17K RPS difference! And at the end, the warmed up client managed to record 39K requests per second. Paying some attention to the measuring device paid off!
To reiterate, every measurement we make, about anything, is as accurate as our measuring device. Therefore, knowing our measuring device is as important as knowing what we’re measuring.
This knowledge helps us to understand the measurement uncertainties much better. For instance, we saw how does the runtime characteristics affect our measurement result.
Does that mean we did consider everything in our example? Absolutely not! That was only the compiler effect. We didn’t consider the GC effect on both Golang and Java. Golang does not need that much of a warmup but the underlying machine does! We did run both the server and benchmarks on the same machine. Therefore competing resources (CPU, RAM, FDs, etc.) on the client and server-side might affect the result. This list goes on and on!
At some point, we should accept the fact that our measurement fails to take some parameters into account. Let’s call it the “approximation that we’re comfortable with”.
The bottom line is, there is no such thing as absolute and infinite precision. There’s only the precision that we’ll be happy with!
1. “In the early 1800s, after the success of scientific theories such as Sir Newton’s theory of Gravity, Laplace suggested that the universe is completely deterministic. That is if we know the state of the universe at some point, we can predict the next states.
In 1926, the famous German scientist Werner Heisenberg suggested that to predict the future position and velocity of a particle, one has to be able to measure the current position and velocity precisely. As it turned out, measuring the position and the velocity are tightly bound to each other.
The more we try to measure the position of a particle accurately, our velocity calculations will be less precise and vice versa! Heisenberg showed that the uncertainty in the position and velocity of a particle times the mass of it can never be smaller than a certain amount known as Planck’s constant.
The Heisenberg’s Uncertainty Principle is a fundamental, inescapable property of the world” – A Brief History of Time, Stephen Hawking.
2. The idea of the first heading and consequently the first footnote is inspired by a comment from Ehsan.