Recently, I've been focusing a lot on the health of our test suite at Knock. Fast, reliable tests are a boost for confidence and developer happiness, so I naturally care a lot about this. I've been wrestling with two main things: slow tests, and flaky tests. My sworn enemies.

The good news here is that I believe that the "solution" to both these issues are proper, robust asynchronous tests. When tests are asynchronous, single tests can even be somewhat slow because you have ways to overcome that. You can scale horizontally with more parallel tests, more cores, and so on. You can split up test cases and parallelize slow tests even more. What about flakiness? Right. In my experience, so much of the flakiness comes from either improperly-written asynchronous tests or slow timeouts and assertions on messages. Go fix the latter, I have no silver bullet for that.

This post is about beautiful, elegant, robust asynchronous tests in Elixir. It's also a kind of ode to OTP concurrency and about parallel-forward thinking and maybe some other made up words.

Other people do this. I assume, because only Louis' post popped up in my timeline, but hey, one's enough.

I've never done a write-up on stuff I use, but I'm a sucker for reading this kind of posts, so I figured I'd join in on the fun. Here's to this becoming a yearly tradition.

By the way, an up-to-date list is always available at my /uses page.

The Elixir compiler does what most modern compilers have to do: it only recompiles the files it needs to. You change a file, and the compiler figures out all the files that somehow depend on that file. Those are the only files that get recompiled. This way, you get to avoid recompiling big projects when changing only a few files.

All good and nice, but it all relies on files not having too many dependent files. That's what was happening with Gettext, Elixir's localization and internationalization library. This post goes through the issue in detail, and how we ended up fixing it.

If you've ever had to interact with Apple APIs related to App Store purchases and whatnot (like the App Store Server Notifications), there's a chance you had to verify JWTs at some point. I had used, signed, and verified JWTs extensively before this, but Apple uses a bit of a unique way of signing their JWTs that I had never stumbled upon in the past. In this short post, I'll show some code around verifying Apple's JWT signatures in Elixir.

Elixir's ecosystem has quite a few HTTP clients at this point. But what's the best one? In this post, I want to break down a bunch of the clients we have available. I'll give an overview of the clients I personally like the most, and I'll talk about which clients are the best choice in different use cases. I'll also share some advice with library authors.

I recently published a new screencast series where I solve some network programming puzzles in Elixir. The challenges are provided by Protohackers, a website I recently found on the interwebs (it's sort of like Advent of Code, but for network challenges). You can find all the videos in this post.

I'm doing Advent of Code 2022 in Rust as a chance to get more familiar with the language. In this ongoing post, I'm documenting each day's progress, my thought process, and so on. I'm better at writing than recording screencasts or what have you. This is essentially a small journal. Ah, and I'm trying AI tools for doing this as well, so this is bound to be fun.

Database migrations are great. I love to be able to change the shape of tables and move data around in a controlled way to avoid issues and downtime. However, lately I started to view migrations more like Git commits than like active pieces of code in my applications. In this post, I want to dig a bit deeper into the topic. I'll start with some context on database migrations, I'll expand on the Git commits analogy, and I'll show you what I've been doing instead.

At Community, we run most of our infrastructure and services on AWS. We use several AWS services. Many of our own services interact with AWS directly, such as by uploading and downloading files from S3, querying Athena, and more. Lately, I've been trying to improve how we test the interaction between our services and AWS, testing error conditions and edge cases as well as running reproducible integration tests. In this post, I'll talk about Localstack, mocks, ExAws, and more.

I mostly use property-based testing to test stateless functional code. A technique I love to use is to pair property-based tests together with example-based tests (that is, "normal" tests) in order to have some tests that check real input. Let's dive deeper into this technique, some contrived blog-post-adequate examples, and links to real-world examples.

At Community, we use RabbitMQ, a lot. It's the infrastructure backbone that allows our services (over forty at this point) to communicate with each other. That mostly happens through events (since we have an event-sourced system), but in some cases what we need is a request-response interaction between two services. This is the best tool in a few use cases, like retrieving data on the fly or asking a service to do something and return a response. An industry standard for such interactions is HTTP, but we are not big fans of that. Instead, since RabbitMQ is so ubiquitous in our system, we settled on using it for request-response interactions as well in the form of Remote Procedure Calls (RPCs). In this post, I'll go over the architecture of such interactions. I'll talk about the RabbitMQ topologies we use to make them work, the benefits around reliability, the compromises around performance, and finally how this all implemented to be as fault-tolerant as possible with Elixir.

When you have a limited number of resources that you have to share for your all application, like database connections or worker processes, what you need is a pool. In this post, we're going to take a look at one possible pooling strategy that highly leverages Elixir's built-in Registry resulting in fast, reliable, and cleanly designed pools.

The system that we're building at Community.com is made of a few services (around fifteen at the time of writing) that interact with each other through a basic version of event sourcing. All events flow (published and consumed) through RabbitMQ and are serialized with Protobuf. With several services already and many more coming in the future, managing the Protobuf schemas becomes a painful part of evolving and maintaining the system. Do we copy the schemas in all services? Do we keep them somewhere and use something akin to Git submodules to keep them in sync in all of our projects? What do we do?! In this post, I'll go through the tooling that we came up with in order to sanely manage our Protobuf schemas throughout our services and technology stack.

Our applications often interact with external systems. In many cases, we need a persistent connection to one or more of these external services. For example, if your application makes continuous use of a database, you'll likely want to stay connected to such database so that you can avoid spending time and resources connecting and disconnecting each time you perform a request. With Erlang and Elixir, the natural abstraction to maintain a persistent connection is a process. In this post, we'll have a look at how we can take advantage of the gen_statem behaviour to write state machine processes that act as persistent connections to external systems.

Now the story of an Elixir library and the one author who had no choice but to take it away.

Property-based testing is a common tool to improve testing by testing properties of a piece of software over many values drawn at random from a large space of valid values. This methodology was first introduced in the paper QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs, which describes the basic idea and shows a possible implementation in Haskell. Since then, many tools to aid in property based testing appeared for many programming languages: as of the time of writing, there are libraries for Haskell, Erlang, Clojure, Python, Scala, and many others. A few days ago I released the first version of StreamData, a property testing (and data generation) library for Elixir (that is a candidate for inclusion in Elixir itself in the future). This post is not an introduction to property-based testing nor a tutorial on how to use StreamData: what I want to do is dig into the mechanics of how StreamData works, its design, and how it compares to some of the other property-based testing libraries mentioned above.

Erlang supports a way to implement functions in C and use them transparently from Erlang. These functions are called NIFs (native implemented functions). There are two scenarios where NIFs can turn out to be the perfect solution: when you need raw computing speed and when you need to interface to existing C bindings from Erlang. In this article, we're going to take a look at both use cases.

Macros are a very common way to do metaprogramming in Elixir. There are many resources that explain what macros are and how to use them (much better than I could): there's the Macro chapter from the "Getting Started" guide on Elixir's website, an awesome series of articles by Saša Jurić, and even a book (Metaprogramming Elixir) by Chris McCord. In this article, I'll assume you are familiar with macros and how they work, and I'll talk about another use case of macros that is rarely examined: doing compile-time things in macros.

Elixir is frequently used in network-aware applications because of the core design of Erlang and the Erlang VM. In this context, there's often the need to connect to external services through the network: for example, a classic web application could connect to a relational database and a key-value store, while an application that runs on embedded systems could connect to other nodes on the network.

Lexical analysis (tokenizing) and parsing are very important concepts in computer science and programming. There is a lot of theory behind these concepts, but I won't be talking about any of that here because, well, it's a lot. Also, I feel like approaching these topics in a "scientific" way makes them look a bit scary; however, using them in practice turns out to be pretty straightforward. If you want to know more about the theory, head over to Wikipedia (lexical analysis and parsing) or read the amazing dragon book (which I recommend to all programmers, it's fantastic).