At Community we run most of our infrastructure and services on AWS. We use several different AWS services. Many of our own services interact with AWS directly, such as by uploading and downloading files from S3, querying Athena, and more. Lately, I’ve been trying to improve how we test the interaction between our services and AWS, testing error conditions and edge cases as well as running reproducible integration tests. In this post, I’ll talk about Localstack, mocks, ex_aws, and more.

I mostly use property-based testing to test stateless functional code. A technique I love to use is to pair property-based tests together with example-based tests (that is, “normal” tests) in order to have some tests that check real input. Let’s dive deeper into this technique, some contrived blog-post-adequate examples, and links to real-world examples.

At Community we use RabbitMQ, a lot. It’s the infrastructure backbone that allows our services (over fourty at this point) to communicate with each other. That mostly happens through events (since we have an event-sourced system), but in some cases what we need is a request-response interaction between two services. This is the best tool in a few use cases, like retrieving data on the fly or asking a service to do something and return a response. An industry standard for such interactions is HTTP, but we are not big fans of that. Instead, since RabbitMQ is so ubiquitous in our system, we settled on using it for request-response interactions as well in the form of Remote Procedure Calls (RPCs). In this post, I’ll go over the architecture of such interactions, the RabbitMQ topologies we use to make them work, the benefits around reliability and the compromises around performance, and finally how this all implemented to be as fault-tolerant as possible with Elixir.

When you have a limited number of resources that you have to share for your all application, like database connections or worker processes, what you need is a pool. In this post, we’re going to take a look at one possible pooling strategy that highly leverages Elixir’s built-in Registry resulting in fast, reliable, and cleanly designed pools.

The system that we’re building at Community.com is made of a few services (around fifteen at the time of writing) that interact with each other through a basic version of event sourcing. All events are exchanged (published and consumed) through RabbitMQ and are serialized with Protobuf. With several services already and many more coming in the future, managing the Protobuf schemas becomes a painful part of evolving and maintaining the system. Do we copy the schemas in all services? Do we keep them somewhere and use something akin to Git submodules to keep them in sync in all of our projects? What do we do?! In this post, I’ll go through the tooling that we came up with in order to sanely manage our Protobuf schemas throughout our services and technology stack.

Our applications often interact with external systems. In many cases, we need a persistent connection to one or more of these external services. For example, if your application makes continuous use of a database, you’ll likely want to stay connected to such database so that you can avoid spending time and resources connecting and disconnecting each time you perform a request. With Erlang and Elixir, the natural abstraction to maintain a persistent connection is a process. In this post, we’ll have a look at how we can take advantage of the gen_statem behaviour to write state machine processes that act as persistent connections to external systems.

Now the story of an Elixir library and the one author who had no choice but to take it away.

Property-based testing is a common tool to improve testing by testing properties of a piece of software over many values drawn at random from a large space of valid values. This methodology was first introduced in the paper QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs, which describes the basic idea and shows a possible implementation in Haskell. Since then, many tools to aid in property based testing appeared for many programming languages: as of the time of writing, there’s libraries for Haskell, Erlang, Clojure, Python, Scala, and many others. A few days ago I released the first version of StreamData, a property testing (and data generation) library for Elixir (that is a candidate for inclusion in Elixir itself in the future). This post is not an introduction to property-based testing nor a tutorial on how to use StreamData: what I want to do is dig into the mechanics of how StreamData works, its design, and how it compares to some of the other property-based testing libraries mentioned above.

Erlang supports a way to implement functions in C and use them transparently from Erlang. These functions are called NIFs (native implemented functions). There are two scenarios where NIFs can turn out to be the perfect solution: when you need raw computing speed and when you need to interface to existing C bindings from Erlang. In this article, we’re going to take a look at both use cases.

Macros are a very common way to do metaprogramming in Elixir. There are many resources that explain what macros are and how to use them (much better than I could): there’s the Macro chapter from the “Getting Started” guide on Elixir’s website, an awesome series of articles by Saša Jurić, and even a book (Metaprogramming Elixir) by Chris McCord. In this article, I’ll assume you are familiar with macros and how they work and I’ll talk about another use case of macros that is rarely examined: doing compile-time things in macros.

Elixir is frequently used in network-aware applications because of the core design of Erlang and the Erlang VM. In this context, there’s often the need to connect to external services through the network: for example, a classic web application could connect to a relational database and a key-value store, while an application that runs on embedded systems could connect to other nodes on the network.

Lexical analysis (tokenizing) and parsing are very important concepts in computer science and programming. There is a lot of theory behind these concepts, but I won’t be talking about any of that here because, well, it’s a lot. Also, I feel like approaching these topics in a “scientific” way makes them look a bit scary; however, using them in practice turns out to be pretty straightforward. If you want to know more about the theory, head over to Wikipedia (lexical analysis and parsing) or read the amazing dragon book (which I recommend to all programmers, it’s fantastic).