Tutorial

Wrapping Your Head Around Numerical Precision

April 14, 2022
Table of Contents

Has this ever happened to you?

You're working on an algorithmic problem, and all of a sudden, your little errors become _really big errors_. And it's not a cosmic bit-flip, either; this keeps happening again and again.

*Editor's Note*: Our algorithm engineer Paul Schroeder hit this, so we asked him to explain the resolution in detail. His deep dive into numerical precision is thorough and _fascinating_. Check out the Jupyter notebook below for some detailed insights into numerical precision, and what it means for programmers everywhere.

_User Experience Warning: There are links in the embedded Jupyter notebook below. If you click one of the links, the embedded notebook will not display a "back" button. You will need to use your browser's "back" button to return to where you were previously in the Jupyter notebook._

> As always, the Jupyter notebook found here is also hosted on the [Tangram Visions Blog Repository](https://gitlab.com/tangram-vision/oss/tangram-visions-blog).

<hr>

<hr>

If numerical precision keeps you up at night, (1) We're sorry and (2) us too. That being said, you'd probably love the work we're doing here at Tangram Vision. Good news: we're hiring! Check out our hiring page, and start to rest easy.

Share On: