Jekyll2023-07-25T07:45:50-06:00http://localhost:4000/feed.xmlAaron Renner’s BlogPersonal blog for Aaron Renner covering topics including Elixir, Software Architecture, and various tips and tricks picked up as a Senior Software Engineer.Walkthrough of Elixir’s Adapter Pattern2023-07-22T00:00:00-06:002023-07-22T00:00:00-06:00http://localhost:4000/2023/07/22/elixir-adapter-pattern<div id="toc"></div>
<p>Sometimes we need to be able to change how parts of our application behave depending on the environment we’re running in. Examples of this include:</p>
<ul>
<li>Toggling sms gateway adapters (production sends a real sms while staging redirects sms to a test phone number)</li>
<li>Testing with a stubbing library like <a href="https://hex.pm/packages/mox">Mox</a></li>
</ul>
<p>If the behavior change is large enough and needs to happen application-wide (like using a different implementation when running in prod vs non-prod), the Adapter pattern makes a lot of sense.</p>
<h2 id="the-adapter-pattern">The Adapter pattern</h2>
<p>The overall structure of the Adapter pattern looks like this:</p>
<p><img src="/assets/img/adapter-pattern/Adapter_Pattern_-_Overview.png" alt="Adapter Pattern - Overview" /></p>
<p>Application code calls the API module, which delegates to the current adapter. In order to keep the API module and adapters in sync, we define a behaviour that each module must implement.</p>
<p>The file tree for this pattern looks like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>lib/
my_app/
sms_gateway.ex - The top-level module that all application code calls
sms_gateway/
adapter.ex - This is the behavior module that defines the callbacks needed for an adapter
default_adapter.ex - An adapter implementation
interceptor_adapter.ex - Adapter implementation
local_adapter.ex - Another adapter implmentation
</code></pre></div></div>
<p>The beauty of this directory structure is when exploring <code class="language-plaintext highlighter-rouge">lib/my_app</code> we can see that there’s a <code class="language-plaintext highlighter-rouge">SMSGateway</code> that we can call, but it just looks like any other module in our system. If the developer is curious how the <code class="language-plaintext highlighter-rouge">SMSGateway</code> works, they can open the <code class="language-plaintext highlighter-rouge">sms_gateway/</code> directory and discover there are multiple adapters. In most cases these adapters are tucked away so they don’t add noise to the codebase.</p>
<h3 id="the-api-module">The API module</h3>
<p><img src="/assets/img/adapter-pattern/Adapter_Pattern_-_API_Module.png" alt="Adapter Pattern - API Module" /></p>
<p>The API module is the top-level module that the rest of the application calls. Application code should be completely unaware of the adapter it’s calling — it’s just calling an interface that fulfills a contract.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span> <span class="k">do</span>
<span class="nv">@moduledoc</span> <span class="sd">"""
Main API for sending SMS
"""</span>
<span class="n">alias</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span><span class="o">.</span><span class="no">DeliveryError</span>
<span class="nv">@type</span> <span class="n">phone_number</span> <span class="p">::</span> <span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">()</span>
<span class="nv">@type</span> <span class="n">message</span> <span class="p">::</span> <span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">()</span>
<span class="nv">@type</span> <span class="n">message_id</span> <span class="p">::</span> <span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">()</span>
<span class="nv">@behaviour</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span><span class="o">.</span><span class="no">Adapter</span>
<span class="nv">@doc</span> <span class="sd">"""
Send a SMS
"""</span>
<span class="nv">@impl</span> <span class="no">true</span>
<span class="nv">@spec</span> <span class="n">send_sms</span><span class="p">(</span><span class="n">phone_number</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span> <span class="p">::</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">message_id</span><span class="p">}</span> <span class="o">|</span> <span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="no">DeliveryError</span><span class="o">.</span><span class="n">t</span><span class="p">()}</span>
<span class="k">def</span> <span class="n">send_sms</span><span class="p">(</span><span class="n">phone_number</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span> <span class="k">do</span>
<span class="n">adapter</span><span class="p">()</span><span class="o">.</span><span class="n">send_sms</span><span class="p">(</span><span class="n">phone_number</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">adapter</span> <span class="k">do</span>
<span class="no">Application</span><span class="o">.</span><span class="n">get_env</span><span class="p">(</span><span class="ss">:my_app</span><span class="p">,</span> <span class="ss">:sms_adapter</span><span class="p">,</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span><span class="o">.</span><span class="no">DefaultAdapter</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>This module is nothing more than a passthrough to the current adapter and should contain very little logic to keep things easy to understand. Its contract should be consistent regardless of the adapter its delegating to. The adapters should wrap error responses in a custom error type to prevent implementation details from leaking through.</p>
<h3 id="implementation-modules">Implementation modules</h3>
<p><img src="/assets/img/adapter-pattern/Adapter_Pattern_-_Implementations.png" alt="Adapter Pattern - Implementations" /></p>
<p>The implementation modules are where the actual logic is written.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span><span class="o">.</span><span class="no">DefaultAdapter</span> <span class="k">do</span>
<span class="nv">@moduledoc</span> <span class="no">false</span>
<span class="n">alias</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span><span class="o">.</span><span class="no">DeliveryError</span>
<span class="nv">@behaviour</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span><span class="o">.</span><span class="no">Adapter</span>
<span class="nv">@impl</span> <span class="no">true</span>
<span class="k">def</span> <span class="n">send_sms</span><span class="p">(</span><span class="n">phone_number</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span> <span class="k">do</span>
<span class="k">case</span> <span class="no">TwilioClient</span><span class="o">.</span><span class="n">send_message</span><span class="p">(</span><span class="n">api_key</span><span class="p">(),</span> <span class="n">phone_number</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="p">%{</span><span class="ss">message_id:</span> <span class="n">message_id</span><span class="p">}</span> <span class="o">-></span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">message_id</span><span class="p">}</span>
<span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="n">reason</span><span class="p">}</span> <span class="o">-></span>
<span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="no">DeliveryError</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="ss">reason:</span> <span class="n">reason</span><span class="p">)}</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Our LocalAdapter may look completely different, which is fine as long as it conforms to the same contract.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span><span class="o">.</span><span class="no">LocalAdapter</span> <span class="k">do</span>
<span class="nv">@moduledoc</span> <span class="no">false</span>
<span class="n">alias</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span><span class="o">.</span><span class="no">DeliveryError</span>
<span class="nv">@behaviour</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span><span class="o">.</span><span class="no">Adapter</span>
<span class="kn">require</span> <span class="no">Logger</span>
<span class="nv">@impl</span> <span class="no">true</span>
<span class="k">def</span> <span class="n">send_sms</span><span class="p">(</span><span class="n">phone_number</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span> <span class="k">do</span>
<span class="no">Logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="sd">"""
SMS would be sent to #{phone_number}.
#{message}
"""</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">random_id</span><span class="p">()}</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">random_id</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="no">Base</span><span class="o">.</span><span class="n">encode64</span><span class="p">(</span><span class="ss">:crypto</span><span class="o">.</span><span class="n">strong_rand_bytes</span><span class="p">(</span><span class="mi">20</span><span class="p">))</span>
<span class="k">end</span>
</code></pre></div></div>
<p>The key is that the API module says it will return <code class="language-plaintext highlighter-rouge">{:ok, message_id}</code> so we need to ensure we fulfill this contract. Since the LocalAdapter doesn’t have a real message id from an SMS provider, we can just make one up to fulfill the contract.</p>
<h3 id="the-behaviour-module">The Behaviour module</h3>
<p>With a top-level API module and multiple implementation modules, it can be difficult to keep the functions, arguments and return types in sync. This is where the Behaviour module comes in.</p>
<p><img src="/assets/img/adapter-pattern/Adapter_Pattern_-_Behaviour.png" alt="Adapter Pattern - Behaviour" /></p>
<p>To write the behaviour, we define the functions each adapter must implement using <code class="language-plaintext highlighter-rouge">@callback</code> .</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span><span class="o">.</span><span class="no">Adapter</span> <span class="k">do</span>
<span class="nv">@moduledoc</span> <span class="no">false</span>
<span class="n">alias</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span>
<span class="n">alias</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span><span class="o">.</span><span class="no">DeliveryError</span>
<span class="nv">@callback</span> <span class="n">send_sms</span><span class="p">(</span><span class="no">SMSGateway</span><span class="o">.</span><span class="n">phone_number</span><span class="p">(),</span> <span class="no">SMSGateway</span><span class="o">.</span><span class="n">message</span><span class="p">())</span> <span class="p">::</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="no">SMSGateway</span><span class="o">.</span><span class="n">message_id</span><span class="p">()</span> <span class="o">|</span> <span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="no">DeliveryError</span><span class="o">.</span><span class="n">t</span><span class="p">()}</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Again, it’s important to create common success and error return values so each implementation can return common values and not leak implementation details (hence the custom <code class="language-plaintext highlighter-rouge">DeliveryError</code> type).</p>
<p>Then in the API module and each one of the adapter modules, use the behaviour by defining the <code class="language-plaintext highlighter-rouge">@behaviour</code> module attribute and setting <code class="language-plaintext highlighter-rouge">@impl true</code> on each of the behaviour’s functions.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span> <span class="k">do</span>
<span class="nv">@moduledoc</span> <span class="sd">"""
Main API for sending SMS
"""</span>
<span class="nv">@behaviour</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span><span class="o">.</span><span class="no">Adapter</span>
<span class="nv">@impl</span> <span class="no">true</span>
<span class="k">def</span> <span class="n">send_sms</span><span class="p">(</span><span class="n">phone_number</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span> <span class="k">do</span>
<span class="c1">#...</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span><span class="o">.</span><span class="no">LocalAdapter</span> <span class="k">do</span>
<span class="nv">@moduledoc</span> <span class="no">false</span>
<span class="nv">@behaviour</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span><span class="o">.</span><span class="no">Adapter</span>
<span class="nv">@impl</span> <span class="no">true</span>
<span class="k">def</span> <span class="n">send_sms</span><span class="p">(</span><span class="n">phone_number</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span> <span class="k">do</span>
<span class="c1"># ...</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Now if a module doesn’t implement the correct functions or arities, the compiler will emit a warning:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="ss">warning:</span> <span class="n">function</span> <span class="n">send_sms</span><span class="o">/</span><span class="mi">2</span> <span class="n">required</span> <span class="n">by</span> <span class="n">behaviour</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span><span class="o">.</span><span class="no">Adapter</span> <span class="n">is</span> <span class="ow">not</span> <span class="n">implemented</span> <span class="p">(</span><span class="ow">in</span> <span class="n">module</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span><span class="o">.</span><span class="no">LocalAdapter</span><span class="p">)</span>
<span class="n">lib</span><span class="o">/</span><span class="n">my_app</span><span class="o">/</span><span class="n">sms_gateway</span><span class="o">/</span><span class="n">local_adapter</span><span class="o">.</span><span class="ss">ex:</span><span class="mi">1</span><span class="p">:</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span><span class="o">.</span><span class="no">LocalAdapter</span> <span class="p">(</span><span class="n">module</span><span class="p">)</span>
</code></pre></div></div>
<h2 id="benefits-of-this-pattern">Benefits of this pattern</h2>
<p>While there is some boilerplate code with this pattern, I’d argue that the following benefits outweigh the drawbacks of other patterns.</p>
<h3 id="adapter-switching-is-isolated-to-a-single-place">Adapter switching is isolated to a single place</h3>
<p>Sometimes developers will read the adapter from the application enviroment throughout the codebase.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">sms_gateway</span> <span class="o">=</span> <span class="no">Application</span><span class="o">.</span><span class="n">fetch_env!</span><span class="p">(</span><span class="ss">:my_app</span><span class="p">,</span> <span class="ss">:sms_gateway</span><span class="p">)</span>
<span class="n">sms_gateway</span><span class="o">.</span><span class="n">send_sms</span><span class="p">(</span><span class="s2">"+15551239876"</span><span class="p">,</span> <span class="s2">"Hello world"</span><span class="p">)</span>
</code></pre></div></div>
<p>One problem with this is now the rest of the codebase has to be aware that there are multiple adapters. Compare this to a simple function call that determines the adapter behind the scenes.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">SMSGateway</span><span class="o">.</span><span class="n">send_sms</span><span class="p">(</span><span class="s2">"+15551239876"</span><span class="p">,</span> <span class="s2">"Hello World"</span><span class="p">)</span>
</code></pre></div></div>
<p>Allowing our application code to make simple function calls on modules keeps our code cleaner, easier to refactor and even simpler to remove the adapter pattern in the future.</p>
<h3 id="maintains-compatibility-with-dialyzer">Maintains compatibility with dialyzer</h3>
<p>When calling a function on a variable, Dialyzer doesn’t know what module will be returned and therefore can’t do type-checking.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">sms_gateway</span> <span class="o">=</span> <span class="no">Application</span><span class="o">.</span><span class="n">fetch_env!</span><span class="p">(</span><span class="ss">:my_app</span><span class="p">,</span> <span class="ss">:sms_gateway</span><span class="p">)</span>
<span class="n">sms_gateway</span><span class="o">.</span><span class="n">send_sms</span><span class="p">(</span><span class="s2">"+15551239876"</span><span class="p">,</span> <span class="s2">"Hello world"</span><span class="p">)</span>
</code></pre></div></div>
<p>When calling a function on a module, Dialyzer can now give us parameter and return type checking like it would anywhere else in the application.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">SMSGateway</span><span class="o">.</span><span class="n">send_sms</span><span class="p">(</span><span class="s2">"+15551239876"</span><span class="p">,</span> <span class="s2">"Hello World"</span><span class="p">)</span>
</code></pre></div></div>
<p>Furthermore, because we’re using a Behaviour, each of our implementation modules will also be checked to ensure they meet the desired typespecs.</p>
<h2 id="toggling-for-the-right-reasons">Toggling for the right reasons</h2>
<p>Generally, we want to minimize our usage of the adapter pattern because it creates more layers of indirection.</p>
<p>DO:</p>
<ul>
<li>Create an adapter if we need to be able to run our application with substantially different functionality depending on the environment. An example of this would be sending real SMS in production vs logging SMS messages locally.</li>
<li>Create an adapter that allows us to change global configuration with <a href="https://hex.pm/packages/mox">Mox</a> so our tests can be run with <code class="language-plaintext highlighter-rouge">async: true</code>. More information is available in the testing section below.</li>
</ul>
<p>DON’T:</p>
<ul>
<li>Use the adapter pattern because we’re unsure how to test code that accesses an external service. Instead, use something like <a href="https://hexdocs.pm/bypass/Bypass.html">bypass</a> that creates a mock version of the external service so our adapter code is fully exercised in test.</li>
</ul>
<p>Chances are we’ll need to use the adapter pattern significantly less than we’d think, but it’s a good tool to have when we need it.</p>
<h2 id="testing-multiple-adapters">Testing multiple adapters</h2>
<p>One place where I like to use the adapter pattern is the central module where I access application config.</p>
<p>Instead of having calls to <code class="language-plaintext highlighter-rouge">Application.get_env/3</code> throughout my app, I’ll centralize them to a single module like <code class="language-plaintext highlighter-rouge">MyApp.Config</code>:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Config</span> <span class="k">do</span>
<span class="nv">@moduledoc</span> <span class="sd">"""
API for accessing the app's configuration
"""</span>
<span class="nv">@spec</span> <span class="n">sms_gateway</span> <span class="p">::</span> <span class="n">module</span>
<span class="k">def</span> <span class="n">sms_gateway</span> <span class="k">do</span>
<span class="no">Application</span><span class="o">.</span><span class="n">get_env</span><span class="p">(</span><span class="ss">:my_app</span><span class="p">,</span> <span class="ss">:sms_adapter</span><span class="p">,</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span><span class="o">.</span><span class="no">DefaultAdapter</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Then I update my API module to call the config module to determine the current adapter.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span> <span class="k">do</span>
<span class="k">def</span> <span class="n">send_sms</span><span class="p">(</span><span class="n">phone_number</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span> <span class="k">do</span>
<span class="n">adapter</span><span class="p">()</span><span class="o">.</span><span class="n">send_sms</span><span class="p">(</span><span class="n">phone_number</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">adapter</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Config</span><span class="o">.</span><span class="n">sms_gateway</span><span class="p">()</span>
<span class="k">end</span>
</code></pre></div></div>
<p>By having these calls centralized to a single module, I can use the adapter pattern for <code class="language-plaintext highlighter-rouge">MyApp.Config</code> to swap in a Mox-based config adapter and allow me to change the adapter used in each test in a way that’s compatible with <code class="language-plaintext highlighter-rouge">async: true</code>. Without it, setting the current adapter using <code class="language-plaintext highlighter-rouge">Application.put_env/3</code> would affect all tests running concurrently in the BEAM because it’s a global operation.</p>
<h3 id="setting-up-the-adapter-pattern-for-the-config-module">Setting up the adapter pattern for the config module</h3>
<p>First, ensure the adapter being tested is controlled by the central config module.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Config</span> <span class="k">do</span>
<span class="nv">@moduledoc</span> <span class="sd">"""
API for accessing the app's configuration
"""</span>
<span class="nv">@spec</span> <span class="n">sms_gateway</span> <span class="p">::</span> <span class="n">module</span>
<span class="k">def</span> <span class="n">sms_gateway</span> <span class="k">do</span>
<span class="no">Application</span><span class="o">.</span><span class="n">get_env</span><span class="p">(</span><span class="ss">:my_app</span><span class="p">,</span> <span class="ss">:sms_adapter</span><span class="p">,</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span><span class="o">.</span><span class="no">DefaultAdapter</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Now we can inject the adapter pattern behind <code class="language-plaintext highlighter-rouge">MyApp.Config</code>. First, copy the function we want to mock to <code class="language-plaintext highlighter-rouge">MyApp.Config.DefaultAdapter</code> .</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Config</span><span class="o">.</span><span class="no">DefaultAdapter</span> <span class="k">do</span>
<span class="nv">@moduledoc</span> <span class="no">false</span>
<span class="nv">@behaviour</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Config</span><span class="o">.</span><span class="no">Adapter</span>
<span class="nv">@impl</span> <span class="no">true</span>
<span class="k">def</span> <span class="n">sms_gateway</span> <span class="k">do</span>
<span class="no">Application</span><span class="o">.</span><span class="n">get_env</span><span class="p">(</span><span class="ss">:my_app</span><span class="p">,</span> <span class="ss">:sms_adapter</span><span class="p">,</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span><span class="o">.</span><span class="no">DefaultAdapter</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Update <code class="language-plaintext highlighter-rouge">MyApp.Config</code> to delegate to its current adapter. The adapter will default to the <code class="language-plaintext highlighter-rouge">DefaultAdapter</code> if nothing is configured.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Config</span> <span class="k">do</span>
<span class="nv">@moduledoc</span> <span class="sd">"""
API for accessing the app's configuration
"""</span>
<span class="nv">@behaviour</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Config</span><span class="o">.</span><span class="no">Adapter</span>
<span class="nv">@impl</span> <span class="no">true</span>
<span class="k">def</span> <span class="n">sms_gateway</span> <span class="k">do</span>
<span class="n">adapter</span><span class="o">.</span><span class="n">sms_gateway</span><span class="p">()</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">adapter</span> <span class="k">do</span>
<span class="no">Application</span><span class="o">.</span><span class="n">get_env</span><span class="p">(</span><span class="ss">:my_app</span><span class="p">,</span> <span class="ss">:config_adapter</span><span class="p">,</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Config</span><span class="o">.</span><span class="no">DefaultAdapter</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Define the behaviour module to keep the API and implementation in sync. Then update these modules to use the behaviour.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Config</span><span class="o">.</span><span class="no">Adapter</span> <span class="k">do</span>
<span class="nv">@moduledoc</span> <span class="no">false</span>
<span class="nv">@callback</span> <span class="n">sms_gateway</span> <span class="p">::</span> <span class="n">module</span>
<span class="k">end</span>
</code></pre></div></div>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span><span class="o">.</span><span class="no">LocalAdapter</span> <span class="k">do</span>
<span class="nv">@moduledoc</span> <span class="no">false</span>
<span class="nv">@behaviour</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span><span class="o">.</span><span class="no">Adapter</span>
<span class="c1">#...</span>
<span class="k">end</span>
</code></pre></div></div>
<h3 id="testing-multiple-adapters-with-mox">Testing multiple adapters with Mox</h3>
<p>Now that there’s a behaviour for <code class="language-plaintext highlighter-rouge">MyApp.Config</code>, we can define a mock adapter and use it to stub config values for each test.</p>
<p>First, add Mox to our test dependencies in <code class="language-plaintext highlighter-rouge">mix.exs</code></p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">defp</span> <span class="n">deps</span> <span class="k">do</span>
<span class="p">[{</span><span class="ss">:mox</span><span class="p">,</span> <span class="s2">"~> 1.0"</span><span class="p">,</span> <span class="ss">only:</span> <span class="ss">:test</span><span class="p">}]</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Then in <code class="language-plaintext highlighter-rouge">test/test_helper.exs</code> define the mock and configure it via the application environment:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Mox</span><span class="o">.</span><span class="n">defmock</span><span class="p">(</span><span class="no">MyApp</span><span class="o">.</span><span class="no">ConfigMock</span><span class="p">,</span> <span class="ss">for:</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Config</span><span class="o">.</span><span class="no">Adapter</span><span class="p">)</span>
<span class="no">Application</span><span class="o">.</span><span class="n">put_env</span><span class="p">(</span><span class="ss">:my_app</span><span class="p">,</span> <span class="ss">:config_adapter</span><span class="p">,</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">ConfigMock</span><span class="p">)</span>
</code></pre></div></div>
<p>Now in our tests we can use <code class="language-plaintext highlighter-rouge">Mox.stub/3</code> to set the SMS gateway adapter.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGatewayTest</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">Case</span><span class="p">,</span> <span class="ss">async:</span> <span class="no">true</span>
<span class="kn">import</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">CaptureLog</span>
<span class="n">alias</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span>
<span class="n">describe</span> <span class="s2">"with local adapter"</span> <span class="k">do</span>
<span class="n">setup</span> <span class="k">do</span>
<span class="no">Mox</span><span class="o">.</span><span class="n">stub</span><span class="p">(</span><span class="no">MyApp</span><span class="o">.</span><span class="no">ConfigMock</span><span class="p">,</span> <span class="ss">:sms_gateway</span><span class="p">,</span> <span class="k">fn</span> <span class="o">-></span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span><span class="o">.</span><span class="no">LocalAdapter</span> <span class="k">end</span><span class="p">)</span>
<span class="ss">:ok</span>
<span class="k">end</span>
<span class="n">test</span> <span class="s2">"successfully sending a message"</span> <span class="k">do</span>
<span class="p">{</span><span class="n">result</span><span class="p">,</span> <span class="n">log</span><span class="p">}</span> <span class="o">=</span>
<span class="n">with_log</span><span class="p">(</span><span class="k">fn</span> <span class="o">-></span>
<span class="no">SMSGateway</span><span class="o">.</span><span class="n">send_sms</span><span class="p">(</span><span class="s2">"+15551239876"</span><span class="p">,</span> <span class="s2">"Hello world"</span><span class="p">)</span>
<span class="k">end</span><span class="p">)</span>
<span class="n">assert</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">_message_id</span><span class="p">}</span> <span class="o">=</span> <span class="n">result</span>
<span class="n">assert</span> <span class="n">log</span> <span class="o">=~</span> <span class="s2">"Hello world"</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">describe</span> <span class="s2">"with default adapter"</span> <span class="k">do</span>
<span class="n">setup</span> <span class="k">do</span>
<span class="no">Mox</span><span class="o">.</span><span class="n">stub</span><span class="p">(</span><span class="no">MyApp</span><span class="o">.</span><span class="no">ConfigMock</span><span class="p">,</span> <span class="ss">:sms_gateway</span><span class="p">,</span> <span class="k">fn</span> <span class="o">-></span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SMSGateway</span><span class="o">.</span><span class="no">DefaultAdapter</span> <span class="k">end</span><span class="p">)</span>
<span class="ss">:ok</span>
<span class="k">end</span>
<span class="n">test</span> <span class="s2">"successfully sending a message"</span> <span class="c1">#...</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Now anywhere we want to change the SMS gateway adapter (or any other config setting) in a way that’s compatible with <code class="language-plaintext highlighter-rouge">async: true</code>, we can use the following pattern.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Mox</span><span class="o">.</span><span class="n">stub</span><span class="p">(</span><span class="no">MyApp</span><span class="o">.</span><span class="no">ConfigMock</span><span class="p">,</span> <span class="ss">:<</span><span class="n">function_name</span><span class="o">></span><span class="p">,</span> <span class="k">fn</span> <span class="o">-></span> <span class="s2">"<value_here>"</span> <span class="k">end</span><span class="p">)</span>
</code></pre></div></div>
<h3 id="falling-back-to-myappconfigdefaultadapter">Falling back to <code class="language-plaintext highlighter-rouge">MyApp.Config.DefaultAdapter</code></h3>
<p>The more we use the adapter pattern on the centralized config module, we may find ourselves wanting to fall back to the DefaultAdapter’s behavior for most cases unless we have a reason to override it.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Config</span> <span class="k">do</span>
<span class="nv">@moduledoc</span> <span class="sd">"""
API for accessing the app's configuration
"""</span>
<span class="nv">@behaviour</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Config</span><span class="o">.</span><span class="no">Adapter</span>
<span class="nv">@impl</span> <span class="no">true</span>
<span class="k">def</span> <span class="n">sms_gateway</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="n">adapter</span><span class="o">.</span><span class="n">sms_gateway</span><span class="p">()</span>
<span class="nv">@impl</span> <span class="no">true</span>
<span class="k">def</span> <span class="n">twilio_base_url</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="n">adapter</span><span class="o">.</span><span class="n">twilio_base_url</span><span class="p">()</span>
<span class="nv">@impl</span> <span class="no">true</span>
<span class="k">def</span> <span class="n">twilio_api_token</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="n">adapter</span><span class="o">.</span><span class="n">twilio_api_token</span><span class="p">()</span>
<span class="nv">@impl</span> <span class="no">true</span>
<span class="k">def</span> <span class="n">postmark_base_url</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="n">adapter</span><span class="o">.</span><span class="n">postmark_base_url</span><span class="p">()</span>
<span class="nv">@impl</span> <span class="no">true</span>
<span class="k">def</span> <span class="n">postmark_api_key</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="n">adapter</span><span class="o">.</span><span class="n">postmark_api_key</span><span class="p">()</span>
<span class="k">defp</span> <span class="n">adapter</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="c1">#...</span>
<span class="k">end</span>
</code></pre></div></div>
<p>In that case, we can use the following line of code to have any function that hasn’t been stubbed on <code class="language-plaintext highlighter-rouge">MyApp.ConfigMock</code> fall back to <code class="language-plaintext highlighter-rouge">MyApp.Config.DefaultAdapter</code>.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Mox</span><span class="o">.</span><span class="n">stub_with</span><span class="p">(</span><span class="no">MyApp</span><span class="o">.</span><span class="no">ConfigMock</span><span class="p">,</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Config</span><span class="o">.</span><span class="no">DefaultAdapter</span><span class="p">)</span>
</code></pre></div></div>
<p>This can be added to a setup helper in a custom <code class="language-plaintext highlighter-rouge">ExUnit.CaseTemplate</code> so it can be run automatically, or just called whenever we need it.</p>
<p>With this fallback set, we can now override only the values we want:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">setup</span> <span class="k">do</span>
<span class="no">MyApp</span><span class="o">.</span><span class="no">ConfigMock</span>
<span class="o">|></span> <span class="no">Mox</span><span class="o">.</span><span class="n">stub</span><span class="p">(</span><span class="ss">:twilio_base_url</span><span class="p">,</span> <span class="k">fn</span> <span class="o">-></span> <span class="s2">"http://localhost:2345"</span> <span class="k">end</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Mox</span><span class="o">.</span><span class="n">stub</span><span class="p">(</span><span class="ss">:twilio_api_token</span><span class="p">,</span> <span class="k">fn</span> <span class="o">-></span> <span class="s2">"supersecret"</span> <span class="k">end</span><span class="p">)</span>
<span class="ss">:ok</span>
<span class="k">end</span>
</code></pre></div></div>
<h2 id="conclusion">Conclusion</h2>
<p>The adapter pattern is a powerful pattern that allows us to swap application behavior when running in different environments. We need to take care not to use it in the wrong scenarios, but when used in the proper situations it can be a critical tool in our toolbox.</p>
<p>The code presented in this article is also available to be reviewed commit-by-commit: <a href="https://github.com/aaronrenner/adapter-pattern-example">https://github.com/aaronrenner/adapter-pattern-example</a>.</p>Application Layering - A Pattern for Extensible Elixir Application Design2019-09-18T00:00:00-06:002019-09-18T00:00:00-06:00http://localhost:4000/2019/09/18/application-layering-a-pattern-for-extensible-elixir-application-design<!-- <div id="toc"></div> -->
<h1 id="introduction">Introduction</h1>
<p>When designing any application, the question is inevitably raised: “where should this code go?” Many times this answer isn’t readily available, resulting in code that ends up in a project’s junk drawer (like utils or models). If this happens enough, the codebase becomes tangled and the team’s ability to maintain the software over time is significantly decreased. This is not a sign of developer inexperience or “naming is hard” — instead it’s most likely a symptom of an application’s lack of structure.</p>
<p>The goal of this paper is to help Elixir developers learn how to structure large codebases so they can be maintainable, adaptable and extensible, without getting bogged down in a tangled web of interdependencies and technical cruft.</p>
<h1 id="background">Background</h1>
<p>Phoenix contexts are a good first step toward app organization and can work well for small scale apps. However as the app continues to grow, there is a tendency to keep all contexts as siblings in a single layer without a way to indicate the different levels of abstraction within the codebase. Things increase in complexity when business logic requires the assembly of data from multiple contexts and there’s no clear place where this logic should go.</p>
<p><img src="/assets/img/application-layering/multiple-levels-of-abstraction-in-phoenix-context.png" alt="Image of multiple levels of abstraction in phoenix contexts"" /></p>
<p>The real problem is there are many levels of abstraction which are all grouped into a single context. When levels of abstraction are mixed in the same module, or there is no place to put over-arching business logic, the codebase becomes muddy and difficult to reason about.</p>
<h2 id="an-alternative-approach">An alternative approach</h2>
<p>Sometimes to come up with an alternative approach, one has to take several steps back and examine the problem from a different point of view.</p>
<p>Most libraries are a layer of abstraction over the functionality they provide. Developers write API client libraries, so they don’t have to think about HTTP requests, parsing responses, serializing data, and everything else that goes into an API client. The same idea applies with database adapters, web servers, hardware drivers, and many other libraries that provide a clean API to wrap potentially complex operations. They all hide layers of complexity so the developer can focus on writing their application without having to worry about low level concerns like opening a socket to a web server.</p>
<p>Even the libraries we use every day delegate complexity to other libraries that focus on lower levels of abstraction. As an example, let’s look at ecto_sql’s dependency tree.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ecto_sql ~> 3.0 (Hex package)
├── db_connection ~> 2.0 (Hex package)
│ └── connection ~> 1.0.2 (Hex package)
├── ecto ~> 3.1.0 (Hex package)
│ └── decimal ~> 1.6 (Hex package)
├── postgrex ~> 0.14.0 or ~> 0.15.0 (Hex package)
└── telemetry ~> 0.4.0 (Hex package)
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">ecto_sql</code> provides an API for applications to interact with a database. The code inside <code class="language-plaintext highlighter-rouge">ecto_sql</code> focuses on writing application data into the database and reading it back out. However, when it comes to lower level concerns like maintaining database connections, connection pooling, metrics, and database specific communications, it calls out to other libraries that focus on those lower levels of abstraction.</p>
<p><img src="/assets/img/application-layering/ecto-sql-dependency-tree-levels-of-abstraction.png" alt="ecto_sql's dependencies levels of abstraction" /></p>
<p>The idea of having libraries that focus on certain levels of abstraction is practiced fruitfully on a daily basis in the Elixir ecosystem. Library authors try to provide easy to use APIs that other applications can build on top of without getting bogged down in the levels of abstraction their library is meant to wrap. This sort of modularity and separation of concerns is a key factor in allowing developers to build better, more maintainable applications because they can focus on their business logic and delegate lower-level concerns to other libraries within the Elixir ecosystem.</p>
<p>Since the pattern of layering modular dependencies to isolate levels of abstraction works so well, why not extend this pattern into our application codebases so they have a tree of modular components that isolate logic to the appropriate levels of abstraction?</p>
<h1 id="the-pattern">The Pattern</h1>
<p>The pattern of Application Layering consists of two parts:</p>
<ol>
<li>Breaking an app into a tree of layers based on the app’s various levels of abstraction</li>
<li>Allowing each layer’s implementation to be easily swapped with an alternative implementation to improve testability and increase adaptability to changing business requirements</li>
</ol>
<p>As we go through this pattern, there is also an example repo that has been built using the techniques discussed: <a href="https://github.com/aaronrenner/zone-meal-tracker">https://github.com/aaronrenner/zone-meal-tracker</a>. This app has been refactored using some of the techniques discussed in later sections of this article, so you will have to go back in the commit history to see some of the earlier steps.</p>
<h2 id="break-the-app-into-multiple-layers">Break the app into multiple layers</h2>
<p>It is common practice for developers in the Elixir/Phoenix community to split their projects into separate web and business logic apps through the use of Phoenix contexts. This was originally inspired by Lance Halvorsen’s talk <a href="https://www.youtube.com/watch?v=lDKCSheBc-8">Phoenix Is Not Your Application</a> and further championed by the Phoenix framework’s code generators. This separation greatly simplifies the web app by allowing it to only focus on web concerns and be a thin interface to the underlying business application. This also allows developers to focus on the core logic of their project separately — an Elixir application that solves a business need.</p>
<p>Although we see huge gains by separating a web layer from the rest of the application, many projects stop with just these two layers. This can be fine for a small codebase, but in larger codebases this can lead to an application that becomes tangled and complicated.</p>
<p>The main idea behind Application Layering is to further decompose an app into multiple layers that isolate the application’s various levels of abstraction. This concept was originally introduced as the Layers pattern in <a href="https://www.amazon.com/Pattern-Oriented-Software-Architecture-System-Patterns/dp/0471958697">Pattern-Oriented Software Architecture - Volume 1</a> in 1996. Some of this pattern’s benefits include:</p>
<ul>
<li>Understandability - Because layers are focused on a single level of abstraction, the code becomes easier to follow. For example, a business logic layer can focus on the process of registering a user (creating the user in the database, sending a welcome email, etc), without being cluttered with the lower-level details of SQL commands or email delivery.</li>
<li>Maintainability - Code that is more understandable is simpler for a developer to update. When the application is separated into multiple layers, it’s much easier to understand where new logic should be written. Updates to how data is serialized when it’s being sent to an external API make sense to go in a low-level layer like an API client. Likewise, logic supporting new business processes should go in a higher-level business logic layer.</li>
<li>Adaptability - Since parent layers communicate with child layers over clearly defined APIs, the implementation of a child layer can be replaced with an improved implementation that conforms to the same API. This allows for replacing the implementation of an entire layer without changes rippling up to the parent.</li>
</ul>
<p>In Application Layering, we take the strict-variant of the Layers pattern (a layer can only be coupled to its direct child layers) and instead of creating application wide layers, we create a tree of child layers focused on the particular levels of abstraction that go into solving the task at hand.</p>
<p><img src="/assets/img/application-layering/tree-of-layers.png" alt="Tree of layers" /></p>
<p>The great part of this “tree of layers” approach is each implementation can be decomposed into logical child modules, and each child module can be further decomposed into more child modules, if necessary. This decomposition naturally creates layers that move toward lower levels of abstraction. Business logic layers are able to stay simple, readable and flexible, even though they may be coordinating several complex lower-level layers. It also makes it easy for low level layers like API clients to be extracted into their own standalone libraries because they are decoupled from higher level layers.</p>
<h3 id="building-a-top-level-api">Building a top-level API</h3>
<p>Now that we’ve discussed the high level structure of Application layering, let’s look at how to implement it.</p>
<p>In order to make it clear what functionality our application exposes, we’ll treat our application’s top-level module as its Public API and the only place the outside world should interact with our code. <a href="https://hexdocs.pm/elixir/writing-documentation.html#hiding-internal-modules-and-functions">Elixir’s Writing Documentation guide</a> indicates that only our Public API should show up in the documentation and any internal functions/modules should be hidden with <code class="language-plaintext highlighter-rouge">@moduledoc false</code>. This allows a developer to get a clear view of the application’s contract with the outside world without getting confused by other internal modules that are actually implementation details.</p>
<p><img src="/assets/img/application-layering/top-level-api-docs.png" alt="ExDoc for Public API" /></p>
<p>It is very important to only have a single module (and its related structs) as the application’s public API for a couple of reasons:</p>
<ol>
<li>If additional modules are made public, it becomes very difficult to understand if a child module is part of the Public API or a lower-level implementation detail that’s only meant to be called internally.</li>
<li>When adding a new business process that spans multiple components (like user registration which creates and account and sends a welcome notification), the top-level Public API module serves as a place to tie these components together into a single business process. Without a single, agreed upon place for the top level business logic, it becomes unclear where this logic should go.</li>
</ol>
<h3 id="leveraging-namespaces-to-indicate-layers">Leveraging namespaces to indicate layers</h3>
<p>If our top level module is the Public API, then child modules should only be one of two things:</p>
<ol>
<li>Struct-only modules referenced by the public API.</li>
<li>Internal helper modules that are implementation details of the public API and not meant to be called publicly.</li>
</ol>
<p>As mentioned previously, if we only document modules that are part of the public API, it’s easy to tell which modules and functions are public vs. internal.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">ZoneMealTracker</span> <span class="k">do</span>
<span class="nv">@moduledoc</span> <span class="sd">"""
Public API for ZoneMealTracker application
"""</span>
<span class="n">alias</span> <span class="no">ZoneMealTracker</span><span class="o">.</span><span class="no">User</span>
<span class="nv">@doc</span> <span class="sd">"""
Registers a new user with email and password.
"""</span>
<span class="nv">@spec</span> <span class="n">register_user</span><span class="p">(</span><span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">(),</span> <span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">())</span> <span class="p">::</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="no">User</span><span class="o">.</span><span class="n">t</span><span class="p">()}</span> <span class="o">|</span> <span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="ss">:email_already_registered</span><span class="p">}</span>
<span class="k">def</span> <span class="n">register_user</span><span class="p">(</span><span class="n">email</span><span class="p">,</span> <span class="n">password</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="c1">#...</span>
<span class="k">end</span>
</code></pre></div></div>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">ZoneMealTracker</span><span class="o">.</span><span class="no">User</span> <span class="k">do</span>
<span class="nv">@moduledoc</span> <span class="sd">"""
User struct
"""</span>
<span class="c1"># This is a struct module used by the public API</span>
<span class="k">defstruct</span> <span class="p">[</span><span class="ss">:id</span><span class="p">,</span> <span class="ss">:email</span><span class="p">]</span>
<span class="nv">@type</span> <span class="n">id</span> <span class="p">::</span> <span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">()</span>
<span class="nv">@type</span> <span class="n">t</span> <span class="p">::</span> <span class="p">%</span><span class="bp">__MODULE__</span><span class="p">{</span>
<span class="ss">id:</span> <span class="n">id</span><span class="p">,</span>
<span class="ss">email:</span> <span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">()</span>
<span class="p">}</span>
<span class="k">end</span>
</code></pre></div></div>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">ZoneMealTracker</span><span class="o">.</span><span class="no">Notifications</span> <span class="k">do</span>
<span class="nv">@moduledoc</span> <span class="no">false</span>
<span class="c1"># This module is just an implementation detail of ZoneMealTracker</span>
<span class="c1"># and not exposed on the public API. You can tell this by `@moduledoc false`</span>
<span class="c1"># and it doesn't define a struct.</span>
<span class="c1">#</span>
<span class="c1"># No modules other than ZoneMealTracker should call this because it's</span>
<span class="c1"># not part of the public API.</span>
<span class="nv">@spec</span> <span class="n">send_welcome_message</span><span class="p">(</span><span class="no">User</span><span class="o">.</span><span class="n">t</span><span class="p">)</span> <span class="k">do</span>
<span class="k">end</span>
</code></pre></div></div>
<p>The great thing about this approach is we’re exposing our functionality to the world through the Public API layer (module), and any child helper modules are part of a lower implementation layer that isn’t meant to be called by the public. This gives us the flexibility to organize and reorganize child modules in the implementation layer and have the security to know that they shouldn’t be called by any other module than their parent, the Public API.</p>
<p>In an effort to maintain simplicity, it’s important to keep functions with side-effects out of our struct-only modules. If we added <code class="language-plaintext highlighter-rouge">ZoneMealTracker.User.fetch/1</code> to retrieve a user from the database, we are effectively splitting our public API across multiple modules, which leads to several problems including confusion around what our public API is, and not having a clear place for over-arching business logic. Instead, we can write <code class="language-plaintext highlighter-rouge">ZoneMealTracker.fetch_user/1</code> which either has the lookup logic directly in the function, or delegates to a function on an internal data store module like <code class="language-plaintext highlighter-rouge">ZoneMealTracker.UserStore.fetch/1</code>.</p>
<h3 id="leveraging-namespaces-all-the-way-down">Leveraging namespaces all the way down</h3>
<p>Previously we used namespaces to indicate:</p>
<ol>
<li>The top level module is the Public API</li>
<li>The child modules are either:
<ol>
<li>Struct-only modules referenced by the public API.</li>
<li>Internal helper modules that are implementation details of the public API and not meant to be called publicly.</li>
</ol>
</li>
</ol>
<p>The great thing about this pattern is we can repeat it many levels deep in our application and it still provides the same structure and guarantees.</p>
<p><img src="/assets/img/application-layering/tree-of-namespaces.png" alt="Tree of namespaces" /></p>
<p>Really, the pattern we’re setting forth is:</p>
<ol>
<li>The current module is the API</li>
<li>Child modules are either
<ol>
<li>Structs referenced by the current module’s API</li>
<li>Internal modules that are used by the API’s implementation</li>
</ol>
</li>
<li>Modules should only access their direct children. No accessing siblings, grandchildren, great grandchildren, etc.
<ol>
<li>If you need to access siblings, then the logic should go in the next higher namespace</li>
<li>If you need to access grandchildren, then the child module needs to provide an API for that functionality.</li>
</ol>
</li>
</ol>
<h3 id="naturally-created-layers">Naturally created layers</h3>
<p>The great thing about using namespaces this way is it naturally creates layers. For example, <code class="language-plaintext highlighter-rouge">ZoneMealTracker.Notifications.NotificationPreferenceStore</code> is only focused on storing user preferences for the notifications system, where <code class="language-plaintext highlighter-rouge">ZoneMealTracker</code> is focused around the overall business logic of the application. As a developer, this layered structure provides a couple benefits:</p>
<ol>
<li><strong><em>No need to look at the children of a module unless you care how the module’s logic is implemented.</em></strong> For example, if you’re calling the <code class="language-plaintext highlighter-rouge">ZoneMealTracker.register_user/2</code> function, you should be able to trust that the user is registered properly. The only reason to look at its code or child modules is if you need to know <strong><em>how</em></strong> it registers the user.</li>
<li><strong><em>Keeps the codebase easy to understand.</em></strong> If your boss says “now we need to send off metrics when a user is registered,” <code class="language-plaintext highlighter-rouge">ZoneMealTracker.register_user/2</code> is a nice, logical place to integrate this new functionality. If this top-level public API wasn’t available, user registration may be placed on a traditional context like <code class="language-plaintext highlighter-rouge">ZoneMealTracker.Accounts.register_user/1</code>. However, this makes the <code class="language-plaintext highlighter-rouge">ZoneMealTracker.Accounts</code> module more difficult to understand because some of its functions operate at the business logic layer, and others operate at the persistence layer. Now the developer has to keep in mind which functions are high level (business logic) and which functions are lower level (persistence logic) and call the functions that are appropriate to the level in which they are currently working. It’s much simpler if a module’s API only operates at a single level of abstraction.</li>
</ol>
<p><img alt="Multiple levels of abstraction in same module" src="/assets/img/application-layering/multiple-levels-of-abstraction-in-same-module.png" style="max-width: 400px;" /></p>
<h2 id="make-implementations-swappable">Make implementations swappable</h2>
<p>Now that the application has been layered through the use of namespaces and we have well-defined APIs on each layer, we can take this one step further and allow the code behind these APIs to be swapped out with different implementations.</p>
<p>Although the Layers pattern hints at being able to swap implementations when they mention “Exchangeability”, this concept is covered in more detail in the “Hexagonal Architecture” paper by Alistair Cockburn. Hexagonal architecture (aka ports and adapters) says in order to keep the application flexible, business logic should communicate to things in the outside world (databases, HTTP APIs, etc) over a well defined interface. This gives the developer a huge amount of flexibility, because once the well-defined interface is created, the current implementation can be replaced with any other implementation that also conforms to that well defined interface. In our case, each layer of the application communicates to the layers below it through these interfaces, and the implementations of these lower-level layers can be easily swapped without affecting any code in higher-level layers.</p>
<p>In hexagonal architecture diagrams, instead of talking about higher-level and lower-level layers they turn the diagram 90 degrees to the left and say:</p>
<ul>
<li>higher-level layers call the logic inside the hexagon through “left side ports” and</li>
<li>the logic inside the hexagon calls lower-level layers through “right side ports”</li>
</ul>
<p>The hexagon shape isn’t significant; it’s just an easy to draw shape where each flat side represents a port.</p>
<figure>
<img src="/assets/img/application-layering/hexagonal-architecture-original-adapters.png" alt="Hexagonal architecture diagram of ZoneMealTracker with original adapters" />
<figcaption>Original ZoneMealTracker app</figcaption>
</figure>
<figure>
<img src="/assets/img/application-layering/hexagonal-architecture-new-adapters.png" alt="Hexagonal architecture diagram of ZoneMealTracker with new adapters" />
<figcaption>ZoneMealTracker with new implementations swapped in</figcaption>
</figure>
<p>The power to swap implementations on the right side ports, allows us to:</p>
<ul>
<li>Easily unit test business logic without calling out to external services</li>
<li>Build implementations of child layers that return fake data so higher layers of an app can be developed while other teams are still building out the lower layers</li>
<li>Solidify the functions of the lower layer’s API without actually writing any lower layer implementation code. This allows us to defer technical decisions like which data store to use, how to structure database tables, etc. until we have a clearer understanding of the interface we need to provide.</li>
<li>Easily adapt to changing business requirements. For example, when the business wants us to migrate to a new API provider, all we’d need to do is swap in a new implementation of that child layer. As long as the new implementation can provide the same code-level interface as the previous implementation, the new implementation can be swapped in without affecting anything upstream.</li>
</ul>
<h3 id="mechanics-of-swapping-out-child-layers-in-elixir">Mechanics of swapping out child layers in Elixir</h3>
<p>Although having the ability to swap in different implementations sounds great, actually implementing it can be difficult. Here are a few ways I’ve tried:</p>
<ul>
<li>Injecting collaborating functions as optional parameters</li>
<li>Looking up the current implementation before calling a function</li>
<li>Calling an API module that delegates to the current implementation behind the scenes</li>
</ul>
<p>For reference, here is an example <code class="language-plaintext highlighter-rouge">register_user/2</code> function that we’ll modify using the different methods of swapping implementations.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">@spec</span> <span class="n">register_user</span><span class="p">(</span><span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">(),</span> <span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">())</span> <span class="p">::</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="no">User</span><span class="o">.</span><span class="n">t</span><span class="p">()}</span> <span class="o">|</span> <span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="ss">:email_already_registered</span><span class="p">}</span>
<span class="k">def</span> <span class="n">register_user</span><span class="p">(</span><span class="n">email</span><span class="p">,</span> <span class="n">password</span><span class="p">)</span> <span class="k">do</span>
<span class="k">case</span> <span class="no">AccountStore</span><span class="o">.</span><span class="n">create_user</span><span class="p">(</span><span class="n">email</span><span class="p">,</span> <span class="n">password</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="p">%</span><span class="no">User</span><span class="p">{</span><span class="ss">id:</span> <span class="n">user_id</span><span class="p">}</span> <span class="o">=</span> <span class="n">user</span><span class="p">}</span> <span class="o">-></span>
<span class="ss">:ok</span> <span class="o">=</span> <span class="no">Notifications</span><span class="o">.</span><span class="n">set_user_email</span><span class="p">(</span><span class="n">user_id</span><span class="p">,</span> <span class="n">email</span><span class="p">)</span>
<span class="ss">:ok</span> <span class="o">=</span> <span class="no">Notifications</span><span class="o">.</span><span class="n">send_welcome_message</span><span class="p">(</span><span class="n">user_id</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">user</span><span class="p">}</span>
<span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="ss">:email_not_unique</span><span class="p">}</span> <span class="o">-></span>
<span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="ss">:email_already_registered</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<h3 id="swapping-approach-injecting-collaborating-functions-as-optional-parameters">Swapping approach: Injecting collaborating functions as optional parameters</h3>
<p>This approach involves modifying a function to accept new parameters so the developer can swap implementations during test.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">@type</span> <span class="n">register_user_opt</span> <span class="p">::</span>
<span class="p">{</span><span class="ss">:create_user_fn</span><span class="p">,</span> <span class="p">(</span><span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">(),</span> <span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">()</span> <span class="o">-></span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="no">User</span><span class="o">.</span><span class="n">t</span><span class="p">()}</span> <span class="o">|</span> <span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="no">Changeset</span><span class="o">.</span><span class="n">t</span><span class="p">})</span> <span class="o">|</span>
<span class="p">{</span><span class="ss">:set_primary_notification_email_fn</span><span class="p">,</span> <span class="p">(</span><span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">(),</span> <span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">()</span> <span class="o">-></span> <span class="ss">:ok</span><span class="p">)}</span> <span class="o">|</span>
<span class="p">{</span><span class="ss">:send_welcome_message_fn</span><span class="p">,</span> <span class="p">(</span><span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">()</span> <span class="o">-></span> <span class="ss">:ok</span><span class="p">)}</span>
<span class="nv">@spec</span> <span class="n">register_user</span><span class="p">(</span><span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">(),</span> <span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">())</span> <span class="p">::</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="no">User</span><span class="o">.</span><span class="n">t</span><span class="p">()}</span> <span class="o">|</span> <span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="ss">:email_already_registered</span><span class="p">}</span>
<span class="k">def</span> <span class="n">register_user</span><span class="p">(</span><span class="n">email</span><span class="p">,</span> <span class="n">password</span><span class="p">,</span> <span class="n">opts</span> <span class="p">\\</span> <span class="p">[])</span> <span class="k">do</span>
<span class="n">create_user_fn</span> <span class="o">=</span> <span class="no">Keyword</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">opts</span><span class="p">,</span> <span class="ss">:create_user_fn</span><span class="p">,</span> <span class="o">&</span><span class="no">AccountStore</span><span class="o">.</span><span class="n">create_user</span><span class="o">/</span><span class="mi">2</span><span class="p">)</span>
<span class="n">set_primary_notification_email_fn</span> <span class="o">=</span>
<span class="no">Keyword</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">opts</span><span class="p">,</span> <span class="ss">:set_primary_notification_email</span><span class="p">,</span> <span class="o">&</span><span class="no">Notifications</span><span class="o">.</span><span class="n">set_user_email</span><span class="o">/</span><span class="mi">2</span><span class="p">)</span>
<span class="n">send_welcome_message_fn</span> <span class="o">=</span>
<span class="no">Keyword</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">opts</span><span class="p">,</span> <span class="ss">:send_welcome_message</span><span class="p">,</span> <span class="o">&</span><span class="no">Notifications</span><span class="o">.</span><span class="n">send_welcome_message</span><span class="o">/</span><span class="mi">1</span><span class="p">)</span>
<span class="k">case</span> <span class="n">create_user_fn</span><span class="o">.</span><span class="p">(</span><span class="n">email</span><span class="p">,</span> <span class="n">password</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="p">%</span><span class="no">User</span><span class="p">{</span><span class="ss">id:</span> <span class="n">user_id</span><span class="p">}</span> <span class="n">user</span><span class="p">}</span> <span class="o">-></span>
<span class="ss">:ok</span> <span class="o">=</span> <span class="n">set_primary_notification_email_fn</span><span class="o">.</span><span class="p">(</span><span class="n">user_id</span><span class="p">,</span> <span class="n">email</span><span class="p">)</span>
<span class="ss">:ok</span> <span class="o">=</span> <span class="n">send_welcome_message_fn</span><span class="o">.</span><span class="p">(</span><span class="n">user_id</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">user</span><span class="p">}</span>
<span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="ss">:email_not_unique</span><span class="p">}</span> <span class="o">-></span>
<span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="ss">:email_already_registered</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p><strong>Benefits</strong></p>
<ul>
<li>Low barrier to entry</li>
<li>Allows collaborating functions to be swapped with any other implementation. Doesn’t require additional modules to be defined.</li>
<li>Alternative implementations can be swapped in per function call. No global configuration in the application environment.</li>
</ul>
<p><strong>Drawbacks</strong></p>
<ul>
<li>Quickly makes the function more complicated and harder to reason about</li>
<li>Requires significant code changes</li>
<li>Easy for mocks and alternative implementations to drift from what’s expected</li>
<li>No dialyzer support</li>
</ul>
<h3 id="swapping-approach-looking-up-the-current-implementation-before-calling-a-function">Swapping approach: Looking up the current implementation before calling a function</h3>
<p>This approach involves looking up the module that contains the current implementation from the application environment. This happens in the module that’s calling the function.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">@spec</span> <span class="n">register_user</span><span class="p">(</span><span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">(),</span> <span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">())</span> <span class="p">::</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="no">User</span><span class="o">.</span><span class="n">t</span><span class="p">()}</span> <span class="o">|</span> <span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="ss">:email_already_registered</span><span class="p">}</span>
<span class="k">def</span> <span class="n">register_user</span><span class="p">(</span><span class="n">email</span><span class="p">,</span> <span class="n">password</span><span class="p">)</span> <span class="k">do</span>
<span class="k">case</span> <span class="n">account_store</span><span class="p">()</span><span class="o">.</span><span class="n">create_user</span><span class="p">(</span><span class="n">email</span><span class="p">,</span> <span class="n">password</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="p">%</span><span class="no">User</span><span class="p">{</span><span class="ss">id:</span> <span class="n">user_id</span><span class="p">}</span> <span class="o">=</span> <span class="n">user</span><span class="p">}</span> <span class="o">-></span>
<span class="ss">:ok</span> <span class="o">=</span> <span class="n">notifications</span><span class="p">()</span><span class="o">.</span><span class="n">set_user_email</span><span class="p">(</span><span class="n">user_id</span><span class="p">,</span> <span class="n">email</span><span class="p">)</span>
<span class="ss">:ok</span> <span class="o">=</span> <span class="n">notifications</span><span class="p">()</span><span class="o">.</span><span class="n">send_welcome_message</span><span class="p">(</span><span class="n">user_id</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">user</span><span class="p">}</span>
<span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="ss">:email_not_unique</span><span class="p">}</span> <span class="o">-></span>
<span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="ss">:email_already_registered</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">account_store</span> <span class="k">do</span>
<span class="no">Application</span><span class="o">.</span><span class="n">get_env</span><span class="p">(</span><span class="ss">:zone_meal_tracker</span><span class="p">,</span> <span class="ss">:account_store</span><span class="p">,</span> <span class="no">AccountStore</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">notifications</span> <span class="k">do</span>
<span class="no">Application</span><span class="o">.</span><span class="n">get_env</span><span class="p">(</span><span class="ss">:zone_meal_tracker</span><span class="p">,</span> <span class="ss">:notifications</span><span class="p">,</span> <span class="no">Notifications</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div></div>
<p><strong>Benefits</strong></p>
<ul>
<li>Compatible with Mox</li>
</ul>
<p><strong>Drawbacks</strong></p>
<ul>
<li>Requires code changes when calling functions</li>
<li>Doesn’t work with dialyzer because the module name is dynamically resolved</li>
<li>Every caller module has to look up the current implementation themselves. This becomes a significant burden.</li>
<li>The current implementation is controlled globally via the application environment. This makes it difficult for multiple implementations to be run in parallel</li>
</ul>
<h3 id="swapping-approach-calling-an-api-module-that-delegates-to-the-current-implementation-behind-the-scenes">Swapping approach: Calling an API module that delegates to the current implementation behind the scenes</h3>
<p>This approach lets client code continue to call the original module with no change. Instead, the original module is modified to delegate function calls to the current implementation. This approach basically splits the module’s public API from its implementation.</p>
<p><img src="/assets/img/application-layering/hexagonal-architecture-separated-apis-and-implementations.png" alt="Separating API from implementation" /></p>
<p>In this approach the client code remains the same.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">@spec</span> <span class="n">register_user</span><span class="p">(</span><span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">(),</span> <span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">())</span> <span class="p">::</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="no">User</span><span class="o">.</span><span class="n">t</span><span class="p">()}</span> <span class="o">|</span> <span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="ss">:email_already_registered</span><span class="p">}</span>
<span class="k">def</span> <span class="n">register_user</span><span class="p">(</span><span class="n">email</span><span class="p">,</span> <span class="n">password</span><span class="p">)</span> <span class="k">do</span>
<span class="k">case</span> <span class="no">AccountStore</span><span class="o">.</span><span class="n">create_user</span><span class="p">(</span><span class="n">email</span><span class="p">,</span> <span class="n">password</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="p">%</span><span class="no">User</span><span class="p">{</span><span class="ss">id:</span> <span class="n">user_id</span><span class="p">}</span> <span class="o">=</span> <span class="n">user</span><span class="p">}</span> <span class="o">-></span>
<span class="ss">:ok</span> <span class="o">=</span> <span class="no">Notifications</span><span class="o">.</span><span class="n">set_user_email</span><span class="p">(</span><span class="n">user_id</span><span class="p">,</span> <span class="n">email</span><span class="p">)</span>
<span class="ss">:ok</span> <span class="o">=</span> <span class="no">Notifications</span><span class="o">.</span><span class="n">send_welcome_message</span><span class="p">(</span><span class="n">user_id</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">user</span><span class="p">}</span>
<span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="ss">:email_not_unique</span><span class="p">}</span> <span class="o">-></span>
<span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="ss">:email_already_registered</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>The collaborating module is where the adjustment is made.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">ZoneMealTracker</span><span class="o">.</span><span class="no">AccountStore</span> <span class="k">do</span>
<span class="nv">@moduledoc</span> <span class="no">false</span>
<span class="n">alias</span> <span class="no">ZoneMealTracker</span><span class="o">.</span><span class="no">AccountStore</span><span class="o">.</span><span class="no">User</span>
<span class="nv">@behaviour</span> <span class="no">ZoneMealTracker</span><span class="o">.</span><span class="no">AccountStore</span><span class="o">.</span><span class="no">Impl</span>
<span class="nv">@impl</span> <span class="no">true</span>
<span class="nv">@spec</span> <span class="n">create_user</span><span class="p">(</span><span class="no">User</span><span class="o">.</span><span class="n">email</span><span class="p">(),</span> <span class="no">User</span><span class="o">.</span><span class="n">password</span><span class="p">())</span> <span class="p">::</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="no">User</span><span class="o">.</span><span class="n">t</span><span class="p">()}</span> <span class="o">|</span> <span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="ss">:email_not_unique</span><span class="p">}</span>
<span class="k">def</span> <span class="n">create_user</span><span class="p">(</span><span class="n">email</span><span class="p">,</span> <span class="n">password</span><span class="p">)</span> <span class="k">do</span>
<span class="n">impl</span><span class="p">()</span><span class="o">.</span><span class="n">create_user</span><span class="p">(</span><span class="n">email</span><span class="p">,</span> <span class="n">password</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">impl</span> <span class="k">do</span>
<span class="no">Application</span><span class="o">.</span><span class="n">get_env</span><span class="p">(</span><span class="ss">:zone_meal_tracker</span><span class="p">,</span> <span class="ss">:account_store</span><span class="p">,</span> <span class="bp">__MODULE__</span><span class="o">.</span><span class="no">PostgresImpl</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Then the actual implementation is moved into its own module.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">ZoneMealTracker</span><span class="o">.</span><span class="no">AccountStore</span><span class="o">.</span><span class="no">PostgresImpl</span> <span class="k">do</span>
<span class="nv">@moduledoc</span> <span class="no">false</span>
<span class="n">alias</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Changeset</span>
<span class="n">alias</span> <span class="no">ZoneMealTracker</span><span class="o">.</span><span class="no">AccountStore</span><span class="o">.</span><span class="no">PostgresImpl</span><span class="o">.</span><span class="no">InvalidDataError</span>
<span class="n">alias</span> <span class="no">ZoneMealTracker</span><span class="o">.</span><span class="no">AccountStore</span><span class="o">.</span><span class="no">PostgresImpl</span><span class="o">.</span><span class="no">Repo</span>
<span class="n">alias</span> <span class="no">ZoneMealTracker</span><span class="o">.</span><span class="no">AccountStore</span><span class="o">.</span><span class="no">User</span>
<span class="nv">@behaviour</span> <span class="no">ZoneMealTracker</span><span class="o">.</span><span class="no">AccountStore</span><span class="o">.</span><span class="no">Impl</span>
<span class="nv">@impl</span> <span class="no">true</span>
<span class="nv">@spec</span> <span class="n">create_user</span><span class="p">(</span><span class="no">User</span><span class="o">.</span><span class="n">email</span><span class="p">(),</span> <span class="no">User</span><span class="o">.</span><span class="n">password</span><span class="p">())</span> <span class="p">::</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="no">User</span><span class="o">.</span><span class="n">t</span><span class="p">()}</span> <span class="o">|</span> <span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="ss">:email_not_unique</span><span class="p">}</span>
<span class="k">def</span> <span class="n">create_user</span><span class="p">(</span><span class="n">email</span><span class="p">,</span> <span class="n">password</span><span class="p">)</span> <span class="ow">when</span> <span class="n">is_email</span><span class="p">(</span><span class="n">email</span><span class="p">)</span> <span class="ow">and</span> <span class="n">is_password</span><span class="p">(</span><span class="n">password</span><span class="p">)</span> <span class="k">do</span>
<span class="p">%</span><span class="no">User</span><span class="p">{}</span>
<span class="o">|></span> <span class="no">User</span><span class="o">.</span><span class="n">changeset</span><span class="p">(%{</span><span class="ss">email:</span> <span class="n">email</span><span class="p">,</span> <span class="ss">password:</span> <span class="n">password</span><span class="p">})</span>
<span class="o">|></span> <span class="no">Repo</span><span class="o">.</span><span class="n">insert</span><span class="p">()</span>
<span class="o">|></span> <span class="k">case</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="p">%</span><span class="no">User</span><span class="p">{}</span> <span class="o">=</span> <span class="n">user</span><span class="p">}</span> <span class="o">-></span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">user</span><span class="p">}</span>
<span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="p">%</span><span class="no">Changeset</span><span class="p">{</span><span class="ss">errors:</span> <span class="n">errors</span><span class="p">}}</span> <span class="o">-></span>
<span class="k">if</span> <span class="no">Enum</span><span class="o">.</span><span class="n">any?</span><span class="p">(</span><span class="n">errors</span><span class="p">,</span> <span class="o">&</span><span class="n">match?</span><span class="p">({</span><span class="ss">:email</span><span class="p">,</span> <span class="p">{</span><span class="s2">"is_already_registered"</span><span class="p">,</span> <span class="n">_</span><span class="p">}},</span> <span class="nv">&1</span><span class="p">))</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="ss">:email_not_unique</span><span class="p">}</span>
<span class="k">else</span>
<span class="k">raise</span> <span class="no">InvalidDataError</span><span class="p">,</span> <span class="ss">errors:</span> <span class="n">errors</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Finally the API and implementations are kept in sync with a behaviour.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">ZoneMealTracker</span><span class="o">.</span><span class="no">AccountStore</span><span class="o">.</span><span class="no">Impl</span> <span class="k">do</span>
<span class="nv">@moduledoc</span> <span class="no">false</span>
<span class="n">alias</span> <span class="no">ZoneMealTracker</span><span class="o">.</span><span class="no">AccountStore</span><span class="o">.</span><span class="no">User</span>
<span class="nv">@callback</span> <span class="n">create_user</span><span class="p">(</span><span class="no">User</span><span class="o">.</span><span class="n">email</span><span class="p">(),</span> <span class="no">User</span><span class="o">.</span><span class="n">password</span><span class="p">())</span> <span class="p">::</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="no">User</span><span class="o">.</span><span class="n">t</span><span class="p">()}</span> <span class="o">|</span> <span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="ss">:email_not_unique</span><span class="p">}</span>
<span class="k">end</span>
</code></pre></div></div>
<p><img src="/assets/img/application-layering/behaviour-keeping-api-and-implementation-in-sync.png" alt="Diagram of Behaviour keeping API and Implementation in sync" /></p>
<p><strong>Benefits</strong></p>
<ul>
<li>Doesn’t require changing client code</li>
<li>Dialyzer works</li>
<li>Single place to swap implementations</li>
<li>Compatible with Mox</li>
<li>The extra namespace makes it easy to keep multiple implementations separate. Each implementation will have its own namespace, and deleting an implementation is as easy as deleting its namespace.</li>
</ul>
<p><strong>Drawbacks</strong></p>
<ul>
<li>Requires more scaffolding</li>
<li>Adds an extra level to the namespace hierarchy</li>
<li>The current implementation is controlled globally via the application environment. This makes it difficult for multiple implementations to be run in parallel.</li>
</ul>
<p>The “Calling an API module that delegates to the current implementation behind the scenes” approach is the one that I’ve found to work best and is the approach I’ll be using for the rest of this article.</p>
<h2 id="unit-testing">Unit Testing</h2>
<p>With the swapping mechanism in place, now we can easily test <code class="language-plaintext highlighter-rouge">ZoneMealTracker</code> in isolation without having to set up (or even have the implementation completed for) <code class="language-plaintext highlighter-rouge">ZoneMealTracker.AccountStore</code> or <code class="language-plaintext highlighter-rouge">ZoneMealTracker.Notifications</code>.</p>
<p>Below is the <code class="language-plaintext highlighter-rouge">ZoneMealTracker</code> module, along with its tests.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># lib/zone_meal_tracker.ex</span>
<span class="k">defmodule</span> <span class="no">ZoneMealTracker</span> <span class="k">do</span>
<span class="nv">@moduledoc</span> <span class="sd">"""
Public API for ZoneMealTracker
"""</span>
<span class="n">alias</span> <span class="no">ZoneMealTracker</span><span class="o">.</span><span class="no">AccountStore</span>
<span class="n">alias</span> <span class="no">ZoneMealTracker</span><span class="o">.</span><span class="no">Notifications</span>
<span class="nv">@spec</span> <span class="n">register_user</span><span class="p">(</span><span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">(),</span> <span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">())</span> <span class="p">::</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="no">User</span><span class="o">.</span><span class="n">t</span><span class="p">()}</span> <span class="o">|</span> <span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="ss">:email_already_registered</span><span class="p">}</span>
<span class="k">def</span> <span class="n">register_user</span><span class="p">(</span><span class="n">email</span><span class="p">,</span> <span class="n">password</span><span class="p">)</span> <span class="k">do</span>
<span class="k">case</span> <span class="no">AccountStore</span><span class="o">.</span><span class="n">create_user</span><span class="p">(</span><span class="n">email</span><span class="p">,</span> <span class="n">password</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="p">%</span><span class="no">User</span><span class="p">{</span><span class="ss">id:</span> <span class="n">user_id</span><span class="p">}</span> <span class="o">=</span> <span class="n">user</span><span class="p">}</span> <span class="o">-></span>
<span class="ss">:ok</span> <span class="o">=</span> <span class="no">Notifications</span><span class="o">.</span><span class="n">set_user_email</span><span class="p">(</span><span class="n">user_id</span><span class="p">,</span> <span class="n">email</span><span class="p">)</span>
<span class="ss">:ok</span> <span class="o">=</span> <span class="no">Notifications</span><span class="o">.</span><span class="n">send_welcome_message</span><span class="p">(</span><span class="n">user_id</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">user</span><span class="p">}</span>
<span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="ss">:email_not_unique</span><span class="p">}</span> <span class="o">-></span>
<span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="ss">:email_already_registered</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># test/test_helper.exs</span>
<span class="no">Mox</span><span class="o">.</span><span class="n">defmock</span><span class="p">(</span><span class="no">ZoneMealTracker</span><span class="o">.</span><span class="no">MockAccountStore</span><span class="p">,</span>
<span class="ss">for:</span> <span class="no">ZoneMealTracker</span><span class="o">.</span><span class="no">AccountStore</span><span class="o">.</span><span class="no">Impl</span>
<span class="p">)</span>
<span class="no">Application</span><span class="o">.</span><span class="n">put_env</span><span class="p">(</span>
<span class="ss">:zone_meal_tracker</span><span class="p">,</span>
<span class="ss">:account_store</span><span class="p">,</span>
<span class="no">ZoneMealTracker</span><span class="o">.</span><span class="no">MockAccountStore</span>
<span class="p">)</span>
<span class="no">Application</span><span class="o">.</span><span class="n">put_env</span><span class="p">(</span>
<span class="ss">:zone_meal_tracker</span><span class="p">,</span>
<span class="ss">:notifications_impl</span><span class="p">,</span>
<span class="no">ZoneMealTracker</span><span class="o">.</span><span class="no">MockNotifications</span>
<span class="p">)</span>
<span class="no">Mox</span><span class="o">.</span><span class="n">defmock</span><span class="p">(</span><span class="no">ZoneMealTracker</span><span class="o">.</span><span class="no">MockNotifications</span><span class="p">,</span>
<span class="ss">for:</span> <span class="no">ZoneMealTracker</span><span class="o">.</span><span class="no">Notifications</span><span class="o">.</span><span class="no">Impl</span>
<span class="p">)</span>
</code></pre></div></div>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># test/zone_meal_tracker.exs</span>
<span class="k">defmodule</span> <span class="no">ZoneMealTrackerTest</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">Case</span><span class="p">,</span> <span class="ss">async:</span> <span class="no">true</span>
<span class="kn">import</span> <span class="no">Mox</span>
<span class="n">alias</span> <span class="no">ZoneMealTracker</span>
<span class="n">alias</span> <span class="no">ZoneMealTracker</span><span class="o">.</span><span class="no">MockAccountStore</span>
<span class="n">alias</span> <span class="no">ZoneMealTracker</span><span class="o">.</span><span class="no">MockNotifications</span>
<span class="n">alias</span> <span class="no">ZoneMealTracker</span><span class="o">.</span><span class="no">User</span>
<span class="n">setup</span> <span class="p">[</span><span class="ss">:set_mox_from_context</span><span class="p">,</span> <span class="ss">:verify_on_exit!</span><span class="p">]</span>
<span class="n">test</span> <span class="s2">"register_user/2 when email is unique"</span> <span class="k">do</span>
<span class="n">email</span> <span class="o">=</span> <span class="s2">"foo@bar.com"</span>
<span class="n">password</span> <span class="o">=</span> <span class="s2">"password"</span>
<span class="n">user_id</span> <span class="o">=</span> <span class="s2">"123"</span>
<span class="n">user</span> <span class="o">=</span> <span class="p">%</span><span class="no">User</span><span class="p">{</span><span class="ss">id:</span> <span class="n">user_id</span><span class="p">,</span> <span class="ss">email:</span> <span class="n">email</span><span class="p">}</span>
<span class="n">expect</span><span class="p">(</span><span class="no">MockAccountStore</span><span class="p">,</span> <span class="ss">:create_user</span><span class="p">,</span> <span class="k">fn</span> <span class="o">^</span><span class="n">email</span><span class="p">,</span> <span class="o">^</span><span class="n">password</span> <span class="o">-></span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">user</span><span class="p">}</span>
<span class="k">end</span><span class="p">)</span>
<span class="no">MockNotifications</span>
<span class="o">|></span> <span class="n">expect</span><span class="p">(</span><span class="ss">:set_user_email</span><span class="p">,</span> <span class="k">fn</span> <span class="o">^</span><span class="n">user_id</span><span class="p">,</span> <span class="o">^</span><span class="n">email</span> <span class="o">-></span> <span class="ss">:ok</span> <span class="k">end</span><span class="p">)</span>
<span class="o">|></span> <span class="n">expect</span><span class="p">(</span><span class="ss">:send_welcome_message</span><span class="p">,</span> <span class="k">fn</span> <span class="o">^</span><span class="n">user_id</span> <span class="o">-></span> <span class="ss">:ok</span> <span class="k">end</span><span class="p">)</span>
<span class="n">assert</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="o">^</span><span class="n">user</span><span class="p">}</span> <span class="o">=</span> <span class="no">ZoneMealTracker</span><span class="o">.</span><span class="n">register_user</span><span class="p">(</span><span class="n">email</span><span class="p">,</span> <span class="n">password</span><span class="p">)</span>
<span class="k">end</span>
<span class="n">test</span> <span class="s2">"register_user/2 when email is already taken"</span> <span class="k">do</span>
<span class="n">email</span> <span class="o">=</span> <span class="s2">"foo@bar.com"</span>
<span class="n">password</span> <span class="o">=</span> <span class="s2">"password"</span>
<span class="n">expect</span><span class="p">(</span><span class="no">MockAccountStore</span><span class="p">,</span> <span class="ss">:create_user</span><span class="p">,</span> <span class="k">fn</span> <span class="o">^</span><span class="n">email</span><span class="p">,</span> <span class="o">^</span><span class="n">password</span> <span class="o">-></span>
<span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="ss">:email_not_unique</span><span class="p">}</span>
<span class="k">end</span><span class="p">)</span>
<span class="n">assert</span> <span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="ss">:email_already_registered</span><span class="p">}</span> <span class="o">=</span> <span class="no">ZoneMealTracker</span><span class="o">.</span><span class="n">register_user</span><span class="p">(</span><span class="n">email</span><span class="p">,</span> <span class="n">password</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Since we have clear APIs for each of our dependencies and an easy way to
swap in mock implementations, it’s very easy to unit test our business logic
at the same level of abstraction the code was written at. With mock implementations
swapped in, we can easily test that no notification is sent when
<code class="language-plaintext highlighter-rouge">AccountStore.create_user/2</code> returns <code class="language-plaintext highlighter-rouge">{:error, :email_not_unique}</code>. The mock saves
us from having to first create a user in the database with the same email, so we
could ensure the second user registration failed and no notifications were sent on failure.</p>
<p>Instead, this approach allows us to unit test our layers in isolation and relies on our
contracts between modules (APIs) — enforced by dialyzer, the <code class="language-plaintext highlighter-rouge">AccountStore.Impl</code> behaviour
and double-checked by integration tests — to ensure the modules work together as expected.
This approach allows higher layers to focus on testing known responses from lower layers
without being coupled to details of the underlying implementation. Furthermore, if the
account store is migrated from a local postgres-based implementation, to a high-availability
riak-based implementation, the business logic tests in <code class="language-plaintext highlighter-rouge">ZoneMealTracker</code> don’t have to be
modified to be compatible with the new <code class="language-plaintext highlighter-rouge">ZoneMealTracker.AccountStore</code> implementation.</p>
<h2 id="swappable-layers-all-the-way-down">Swappable layers all the way down</h2>
<p><img src="/assets/img/application-layering/swappable-layers-all-the-way-down.png" alt="Swappable layers all the way down" /></p>
<p>Although the scope of Hexagonal Architecture is limited to saying that business logic inside a hexagon should communicate to each outside system over a well-defined API, this pattern also lends itself to being repeated at multiple layers throughout the application. If we allow for swappable implementations at each layer, we end up with a tree of neatly layered dependencies, all having implementations that can be easily replaced at any point. Not only does this significantly help in testing, but it lets the tests for each layer focus on its own level of abstraction without being coupled to the underlying implementation. For example, the tests for <code class="language-plaintext highlighter-rouge">ZoneMealTracker.Notifications.send_welcome_message/1</code> ensure if an email address is registered for the given user id, a welcome email is sent to that address. Because we can swap in mocks for the <code class="language-plaintext highlighter-rouge">NotificationPreferenceStore</code> and <code class="language-plaintext highlighter-rouge">Emails</code> modules, our tests won’t be coupled to how notification preferences are stored or how emails are delivered. Instead, they will be able to focus on the business logic of an email being sent to the appropriate user when the function is called.</p>
<h3 id="file-structure-mirrors-application-structure">File Structure mirrors application structure</h3>
<p>Another great thing about this pattern is it makes the separation between public API modules and their underlying implementations even more clear.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>lib/zone_meal_tracker/
account_store.ex <- Top level API module
account_store/
impl.ex <- Behaviour to be implemented by top level module and impls
in_memory_impl.ex <- An in-memory implementation
in_memory_impl/ <- modules used only by this specific implementation
state.ex
login.ex <- struct returned by public API
postgres_impl.ex <- A postgres-backed implementation
postgres_impl/ <- modules used only by this specific implementation
domain_translator.ex
exceptions.ex
login.ex
repo.ex
supervisor.ex
user.ex
supervisor.ex <- Helper module used across all implementations (@moduledoc false)
user.ex <- struct returned by public API
</code></pre></div></div>
<p>When looking through a module’s folder, there are a few things that should stand out:</p>
<ul>
<li>Anything ending in <code class="language-plaintext highlighter-rouge">_impl</code> is an implementation. Anything inside the <code class="language-plaintext highlighter-rouge"><impl_name>_impl/</code> folder is a helper module used only by that implementation</li>
<li><code class="language-plaintext highlighter-rouge">impl.ex</code> is the behaviour that keeps the top level module and implementations in sync.</li>
<li>Most other modules in the folder (with the exception of well known modules like <code class="language-plaintext highlighter-rouge">application.ex</code> or <code class="language-plaintext highlighter-rouge">supervisor.ex</code>) will be struct-only modules that are referenced by the API.</li>
</ul>
<p>This structure and uniformity makes it easy to understand a layer just by glancing at the file tree. Furthermore, we may have started out with an <code class="language-plaintext highlighter-rouge">InMemoryImpl</code> and expanded to a <code class="language-plaintext highlighter-rouge">PostgresImpl</code> as we improved our understanding of the API we needed to provide. This structure allows the <code class="language-plaintext highlighter-rouge">PostgresImpl</code> to be developed in isolation while the <code class="language-plaintext highlighter-rouge">InMemoryImpl</code> is still being used. When it’s time to switch from the <code class="language-plaintext highlighter-rouge">InMemoryImpl</code> to the <code class="language-plaintext highlighter-rouge">PostgresImpl</code>, it can be done with a single line change in the top level module that sets the default implementation to <code class="language-plaintext highlighter-rouge">PostgresImpl</code> (it could also be changed via the application environment to allow for easy rollback). Once the migration is complete and we’re ready to remove the old implementation, the only thing that needs to be done is to delete <code class="language-plaintext highlighter-rouge">in_memory_impl.ex</code> and <code class="language-plaintext highlighter-rouge">in_memory_impl/</code>. Since the entire implementation is contained in that file and folder, removing it is extremely simple. This idea was originally inspired by <a href="https://www.youtube.com/watch?v=yk_-t_Pw3XA">Johnny Winn’s talk: Just Delete It</a>.</p>
<h2 id="integration-testing">Integration testing</h2>
<p>Although each layer has been fully unit tested and dialyzer is running type checking between the layers, it is still important to do some end-to-end tests to ensure all of the default implementations integrate as expected. This could be a simple as writing an automated test that goes through the user registration flow to ensure everything works as expected.</p>
<p>Since our tests manipulate the application environment to swap in mock implementations at various layers, it’s generally best to have an integration tester outside of the umbrella app that starts the app in a clean environment.</p>
<blockquote>
<p>There is an example integration tester in the <code class="language-plaintext highlighter-rouge">integration_tester</code> folder of the ZoneMealTracker app</p>
</blockquote>
<p>If you need to do browser-based integration testing, <a href="https://github.com/keathley/wallaby">Wallaby</a> and <a href="https://github.com/HashNuke/hound/">Hound</a> are great tools that allow you to drive your app via a real web browser. If you’re writing an app with an API, it’s worth considering building an API client to make integration testing easier (it’s also very helpful when it’s time to call your API from another elixir application).</p>
<h2 id="tips-and-tricks">Tips and tricks</h2>
<p>Although this pattern works very well and gives you flexibility when maintaining a long lived application, there are a few items to keep in mind when working in an application with this sort of structure.</p>
<h3 id="only-add-swapping-mechanisms-when-needed">Only add swapping mechanisms when needed</h3>
<p>When first starting out, it’s tempting to add swapping mechanisms between every module. Unfortunately, too many swapping mechanisms will only bring needless pain and frustration. Only insert swapping between:</p>
<ol>
<li>A business logic layer and a layer that reaches out to an external service (API, database, etc)</li>
<li>Distinct business logic layers. For example, a developer could insert a swapping mechanism for <code class="language-plaintext highlighter-rouge">ZoneMealTracker.Notifications</code> so its parent layer, <code class="language-plaintext highlighter-rouge">ZoneMealTracker</code>, could be tested in isolation. When testing <code class="language-plaintext highlighter-rouge">ZoneMealTracker</code>, we don’t care how notifications are sent, just that the appropriate function is called on <code class="language-plaintext highlighter-rouge">ZoneMealTracker.Notifications</code>.</li>
</ol>
<p>Many times, there are other helper modules like <code class="language-plaintext highlighter-rouge">ZoneMealTracker.Notifications.Logger.Formatter</code>that contain only pure functions that have been extracted from their parent (<code class="language-plaintext highlighter-rouge">ZoneMealTracker.Notifications.Logger</code>). These modules don’t interact with any external services or parts of the system and therefore don’t need swapping logic. Adding swapping logic to these modules would only complicate testing and the overall application structure.</p>
<h3 id="keep-implementation-details-out-of-apis">Keep implementation details out of APIs</h3>
<p>It’s also important to not let implementation details leak into the app’s APIs. If a developer isn’t disciplined and returns Ecto schemas and changesets in their API, they may not be able to swap out their ecto-based local accounts store, with a new implementation that reaches out over HTTP to an accounts microservice. The key is to ensure structs that are accepted or returned via the public API don’t have any coupling to the underlying implementation. This means no Ecto schemas or changesets should leak out into the public API because they would prevent us from switching to a non-ecto data store.</p>
<h3 id="working-with-long-namespaces">Working with long namespaces</h3>
<p>After we’ve started layering our application into the appropriate levels of abstraction (complete with swapping logic), there are times that our namespaces can get very deep. When nesting 3 or more swappable layers, module names can get very long and cumbersome.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">ZoneMealTracker</span><span class="o">.</span><span class="no">DefaultImpl</span><span class="o">.</span><span class="no">Notifications</span><span class="o">.</span><span class="no">DefaultImpl</span>
<span class="o">.</span><span class="no">NotificationPreferenceStore</span><span class="o">.</span><span class="no">PostgresImpl</span><span class="o">.</span><span class="no">PrimaryEmail</span> <span class="k">do</span>
<span class="k">end</span>
</code></pre></div></div>
<p>When that point comes, it works well to pick a logical, stand-alone layer like <code class="language-plaintext highlighter-rouge">ZoneMealTracker.DefaultImpl.Notifications</code> and extract it into its own application under the umbrella. To indicate it’s an internal app, I prefix the application with the project’s initials like <code class="language-plaintext highlighter-rouge">ZMTNotifications</code>. Not only does this extraction shorten the length of the namespaces, but it also brings several other benefits:</p>
<ul>
<li>Each internal app has its own mix.exs, which makes it easy to tell which layer introduces a dependency</li>
<li>Standalone libraries like API clients can easily be extracted out of the umbrella into their own projects and published to Hex.</li>
<li>Even on an internal application, its top level module exposes an API that the rest of the system can build on. Since this module has enough functionality that made it worth extracting, I also add documentation for this top-level internal API. Now when developers generate the ExDoc for the project, they’ll not only see the main Public API (<code class="language-plaintext highlighter-rouge">ZoneMealTracker</code>), but any internal APIs they can reach for as well (like <code class="language-plaintext highlighter-rouge">ZMTNotifications</code>).</li>
</ul>
<p><img src="/assets/img/application-layering/api-docs-for-internal-apis.png" alt="ExDoc for internal APIs" /></p>
<h3 id="no-logic-in-api-modules">No logic in API modules</h3>
<p>When building API modules, their functions should only be passthroughs to the current implementation. If a developer adds logic to an API module, they are forced to also keep this logic in mind when they want to return specific data from a function. Here is an example:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># API Module that contains logic</span>
<span class="k">defmodule</span> <span class="no">ZMTNotifications</span> <span class="k">do</span>
<span class="nv">@spec</span> <span class="n">fetch_registered_email</span><span class="p">(</span><span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">())</span> <span class="o">|</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">()}</span> <span class="o">|</span> <span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="ss">:not_found</span><span class="p">}</span>
<span class="k">def</span> <span class="n">fetch_registered_email</span><span class="p">(</span><span class="n">email</span><span class="p">)</span> <span class="k">do</span>
<span class="n">current_impl</span><span class="p">()</span><span class="o">.</span><span class="n">fetch_registered_email</span><span class="p">(</span>
<span class="k">end</span>
<span class="nv">@spec</span> <span class="n">email_registered?</span><span class="p">(</span><span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">)</span> <span class="p">::</span> <span class="n">boolean</span>
<span class="k">def</span> <span class="n">email_registered?</span><span class="p">(</span><span class="n">email</span><span class="p">)</span> <span class="k">do</span>
<span class="c1"># This contains logic instead of a passthrough</span>
<span class="n">match?</span><span class="p">({</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">_</span><span class="p">},</span> <span class="n">fetch_email_registered</span><span class="p">(</span><span class="n">email</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="c1"># Module being tested</span>
<span class="k">defmodule</span> <span class="no">ZoneMealTracker</span> <span class="k">do</span>
<span class="nv">@spec</span> <span class="n">email_registered?</span><span class="p">(</span><span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">)</span> <span class="p">::</span> <span class="n">boolean</span>
<span class="k">def</span> <span class="n">email_registered?</span><span class="p">(</span><span class="n">email</span><span class="p">)</span> <span class="k">do</span>
<span class="c1"># Code under test</span>
<span class="no">ZMTNotifications</span><span class="o">.</span><span class="n">email_registered?</span><span class="p">(</span><span class="n">email</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="c1"># Test Case</span>
<span class="k">defmodule</span> <span class="no">ZoneMealTrackerTest</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">Case</span>
<span class="kn">import</span> <span class="no">Mox</span>
<span class="n">alias</span> <span class="no">ZoneMealTracker</span><span class="o">.</span><span class="no">MockZMTNotifications</span>
<span class="n">test</span> <span class="s2">"email_registered?/1 returns true when email is registered"</span> <span class="k">do</span>
<span class="c1"># Although the code under test is calling</span>
<span class="c1"># `ZMTNotifications.email_registered?/1`, we have to know to mock</span>
<span class="c1"># `fetch_registered_email/1` because there is logic in</span>
<span class="c1"># `ZMTNotifications.email_registered?/1` instead of being just a straight</span>
<span class="c1"># passthrough.</span>
<span class="n">expect</span><span class="p">(</span><span class="no">MockZMTNotifications</span><span class="p">,</span> <span class="ss">:fetch_registered_email</span><span class="p">,</span> <span class="k">fn</span> <span class="n">email</span> <span class="o">-></span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">email</span><span class="p">})</span>
<span class="n">assert</span> <span class="no">ZoneMealTracker</span><span class="o">.</span><span class="n">email_registered?</span><span class="p">(</span><span class="s2">"foo@bar.com"</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>From the example above, instead of being able to simply expect that <code class="language-plaintext highlighter-rouge">MockZMTNotifications.email_registered?/1</code> returned the value we wanted, we need to know that the underlying implementation calls <code class="language-plaintext highlighter-rouge">MockZMTNotifications.fetch_registered_email/1</code>. In order to make <code class="language-plaintext highlighter-rouge">MockZMTNotifications.email_registered?/1</code> return <code class="language-plaintext highlighter-rouge">true</code>, we actually have to make <code class="language-plaintext highlighter-rouge">MockZMTNotifications.fetch_registered_email/1</code> return <code class="language-plaintext highlighter-rouge">{:ok, email}</code>. This is a lot of unnecessary coupling and complication to test a seemingly simple function.</p>
<p>The only bit of logic I would put in an API module would be guard clauses to ensure the appropriate data types are passed to the underlying implementation. This makes error messages nicer when calling functions on an API module with invalid data and keeps us from forgetting guard clauses on our actual implementations. Other than that, functions on an API module should be a complete pass-through to the current implementation.</p>
<h1 id="relationship-to-prior-works">Relationship to prior works</h1>
<p>In software design, most ideas are applying somebody’s earlier work in a slightly different context. Greg Young’s “<a href="https://vimeo.com/108441214">The art of destroying software</a>” is an excellent talk where he discusses composing large systems out of a collection of tiny programs (components) — each of which can be deleted and rewritten in about a week. This helps keep parts of larger systems, small, understandable and easily adaptable to changing business requirements.</p>
<p>The idea of Application Layering meshes extremely well with this philosophy. With Application Layering, we build the application as a tree of components that can be swapped out at any time. For example, if we want to change the <code class="language-plaintext highlighter-rouge">ZMTNotifications.DefaultImpl.NotificationPreferenceStore</code> to write to a riak database instead of a postgres database, we can write a new <code class="language-plaintext highlighter-rouge">ZMTNotifications.DefaultImpl.NotificationPreferenceStore.RiakImpl</code> module and swap it in when it’s ready. When the transition from postgres to riak is complete, we can delete <code class="language-plaintext highlighter-rouge">ZMTNotifications.DefaultImpl.NotificationPreferenceStore.PostgresImpl</code> and all of its child modules. By having our database logic isolated into a small component, we can easily follow Greg’s idea of being able to delete and rewrite a component in about a week.</p>
<p>To take this one step further, when business requirements change, we may find that we need to completely change the design of the notification system. With our existing swapping infrastructure in place, we can define a new <code class="language-plaintext highlighter-rouge">ZMTNotifications.EnhancedImpl</code> module and start work on the new, improved notifications system. This implementation may need separate data stores and services than the <code class="language-plaintext highlighter-rouge">ZMTNotifications.DefaultImpl</code> implementation, but they can all be tucked under the <code class="language-plaintext highlighter-rouge">ZMTNotifications.EnhancedImpl</code> namespace. As long as it can fulfill the contract defined by the <code class="language-plaintext highlighter-rouge">ZMTNotifications.Impl</code> behaviour, we’re free to rewrite this tree of components without affecting any upstream layers. Also, because this component is isolated from the rest of the system, in many cases it can also be rewritten in about a week.</p>
<p>In Greg’s talk he mentions the benefits of being able to rewrite any of the tiny programs (components) within an application in a week. In Application Layering, each swappable implementation is analogous to one of those tiny programs. Greg mentions “the difference between great code and sucky code is the size of the programs (layers).” If each layer is focused only on a single level of abstraction (business logic, persistence, etc), each layer stays small and can be easily replaced as requirements change. This sort of structure unshackles developers from their large, complex codebases and gives them freedom to rewrite small parts of their system as changes are needed. From my experience, this makes for an extremely enjoyable application to work in.</p>
<h1 id="final-advice">Final advice</h1>
<p>Building software is a process — the proper software design will be come apparent as you go. Many times I’ve tried to skip steps and create layers before I actually need them, only to realize I’ve chosen wrong and need to rewrite my code. This is why I try to follow the following process:</p>
<ol>
<li>Public functions should be made easier to read by extracting private functions</li>
<li>If several related private functions have been extracted or I need to write unit tests for these private functions, I’ll extract them into a child module.</li>
<li>If tests for my business logic are difficult to write because they require calls to an external service or separate section of business logic, I’ll insert a swappable layer at the module that serves as a boundary between my business logic and the external service/separate domain.</li>
</ol>
<p>It also helps when writing a module to ask myself “what is this module’s Public API?” This helps me to think about the contract I need to fulfill to support the rest of the system, while giving me the flexibility to experiment with the implementation. When working in higher layers, I’ll call functions that I wish a child layer had, and then go implement the lower-level logic once I’ve solidified that child layer’s API. This helps me make sure my child layer APIs make sense.</p>
<p>Although it takes thought and discipline to design a system this way, it leads to an amazingly flexible codebase that is easy to adapt and maintain. I’d encourage everyone with a non-trivial application to try this pattern at some edge of your application. This could either be at a boundary to an external service (like an API client), or at the top-most boundary of your application (between the web layer and business logic). With time, you’ll figure out the ins and outs of what works and how to best structure your system. Ultimately, if done correctly, you should end up with a flexible, adaptable codebase. That said, don’t forget to add some full stack integration tests to ensure your application continues to function as you restructure your codebase.</p>
<p>Happy coding!</p>
<h1 id="resources">Resources</h1>
<ul>
<li><a href="https://aaronrenner.io/2018/09/30/elixir-conf-growing-applications-and-taming-complexity.html">Growing Applications and Taming Complexity</a> - Aaron Renner - ElixirConf 2018</li>
<li><a href="https://github.com/aaronrenner/zone-meal-tracker">ZoneMealTracker repo</a></li>
<li><a href="https://www.youtube.com/watch?v=th4AgBcrEHA&list=PLGl1Jc8ErU1w27y8-7Gdcloy1tHO7NriL">Alistair in the “Hexagone”</a></li>
</ul>Conference Talk: Growing Applications and Taming Complexity2018-09-30T00:00:00-06:002018-09-30T00:00:00-06:00http://localhost:4000/2018/09/30/elixir-conf-growing-applications-and-taming-complexity<p>Earlier this month, I finally got to give my <a href="https://elixirconf.com/">ElixirConf</a> talk
“Growing Applications and Taming Complexity”. This was my first
full-length conference talk and it was an amazing experience.
I didn’t realize the momumental amount of work that goes into giving
a presentation, but my family stuck by me and I’m super grateful.</p>
<p>This talk includes design patterns I’ve learned over the past 10
years of how to build a system for the long term while keeping it understandable,
maintainable and adaptable to change. It covers concepts including
designing code-level APIs, layering applications, and
<a href="http://alistair.cockburn.us/Hexagonal+architecture">hexagonal architecture</a>.</p>
<style>
.embed-container { position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%; } .embed-container iframe, .embed-container object, .embed-container embed { position: absolute; top: 0; left: 0; width: 100%; height: 100%; }
</style>
<div class="embed-container">
<iframe width="560" height="315" src="https://www.youtube.com/embed/Ue--hvFzr0o" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
</div>
<p>I’d like to give a huge thanks to:</p>
<ul>
<li><a href="https://keathley.io/">Chris Keathley</a> for helping me write an
abstract that got my talk accepted,</li>
<li><a href="https://github.com/koudelka">Michael Shapiro</a> for listening to my talk
multiple times and giving me lots of great feedback, and</li>
<li><a href="https://activeprospect.com">ActiveProspect</a> for sponsoring me to go to
this conference, allowing me to work on Elixir-based systems
full time, and being an overall great place to work.</li>
</ul>
<p>Slides are available upon request, but they only make sense if you can open
them in Keynote due to all of the animations.</p>Earlier this month, I finally got to give my ElixirConf talk “Growing Applications and Taming Complexity”. This was my first full-length conference talk and it was an amazing experience. I didn’t realize the momumental amount of work that goes into giving a presentation, but my family stuck by me and I’m super grateful.Lightning Talk: Elixir Deployment Tools2017-09-30T00:00:00-06:002017-09-30T00:00:00-06:00http://localhost:4000/2017/09/30/lightning-talk-elixir-deployment-tools<p>A couple of weeks ago, I had the opportunity to give a lightning talk
at <a href="https://elixirconf.com/2017">ElixirConf 2017</a> about my experiences in the last few months
working in Elixir’s deployment ecosystem. In this talk, I cover using
<a href="https://github.com/bitwalker/distillery">distillery</a> to build OTP releases, <a href="https://github.com/edeliver/edeliver">edeliver</a> for
building and deploying to remote machines, and <a href="https://github.com/bitwalker/conform">conform</a> for
configuring application settings at runtime.</p>
<p>A big thanks to <a href="https://activeprospect.com/">ActiveProspect</a> for sending me to this
conference and being a great place to work on Elixir full time.</p>
<style>
.embed-container { position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%; } .embed-container iframe, .embed-container object, .embed-container embed { position: absolute; top: 0; left: 0; width: 100%; height: 100%; }
</style>
<div class="embed-container">
<iframe width="560" height="315" src="https://www.youtube.com/embed/2R5QO0O8cCE" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
</div>A couple of weeks ago, I had the opportunity to give a lightning talk at ElixirConf 2017 about my experiences in the last few months working in Elixir’s deployment ecosystem. In this talk, I cover using distillery to build OTP releases, edeliver for building and deploying to remote machines, and conform for configuring application settings at runtime.Lightning Talk: Mocks, Adapters and Micro-services2017-02-16T00:00:00-07:002017-02-16T00:00:00-07:00http://localhost:4000/2017/02/16/lightning-talk-mocks-adapters-and-microservices<p>Last fall, <a href="https://telnyx.com">my employer</a> generously sent me to ElixirConf 2016 in Orlando, Florida. While I was there, thought I’d put together a lightning talk about our experiences testing our Elixir API adapters in our microservices environment. Thanks to <a href="http://confreaks.tv/">ConFreaks</a>, I’m able to post the video of my talk here for posterity.</p>
<style>
.embed-container { position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%; } .embed-container iframe, .embed-container object, .embed-container embed { position: absolute; top: 0; left: 0; width: 100%; height: 100%; }
</style>
<div class="embed-container">
<iframe width="560" height="315" src="https://www.youtube.com/embed/zuzDQKWrr6Q?start=2916" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
</div>
<p>And here are my slides: <a href="http://slides.com/aaronrenner/deck#/">http://slides.com/aaronrenner/deck#/</a></p>Last fall, my employer generously sent me to ElixirConf 2016 in Orlando, Florida. While I was there, thought I’d put together a lightning talk about our experiences testing our Elixir API adapters in our microservices environment. Thanks to ConFreaks, I’m able to post the video of my talk here for posterity.Navigating Vim projects like a Boss2017-01-31T00:00:00-07:002017-01-31T00:00:00-07:00http://localhost:4000/2017/01/31/navigating-vim-projects-like-a-boss<p><img src="/assets/img/1*4FAL9COB9lNq-ReX3hBGbA.jpeg" alt="" /></p>
<p>I’ll admit it, I’m a Vim addict. I started using Vim while programming
Ruby on Rails and quickly got hooked on
<a href="https://github.com/tpope/vim-rails">vim-rails</a>’ project navigation shortcuts.
Commands like <code class="language-plaintext highlighter-rouge">:A</code> (meaning “alternate ”) to jump between a Ruby file and its
spec were huge timesavers. There were also other cool commands to jump to a
specific type of file.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>:Econtroller - Jump to controller
:Emodel - Jump to model
:Emigration - Jump to migration
:Einitializer - Jump to initializer
</code></pre></div></div>
<p>In the past year, my day job has helped me venture beyond Ruby into Elixir and Angular. It’s fun getting to work in these new languages and frameworks, but I sorely missed my shortcuts to jump around projects. Fortunately, <a href="https://github.com/tpope">Tim Pope</a> has written <a href="https://github.com/tpope/vim-projectionist">vim-projectionist</a> to tackle just this problem.</p>
<h3 id="quick-install">Quick install</h3>
<p>I use <a href="https://github.com/junegunn/vim-plug">vim-plug</a> to manage <a href="https://github.com/aaronrenner/dotfiles">my dotfiles</a>, but any Vim plugin manager will do. If you happen to be using vim-plug, all you have to do is add</p>
<div class="language-vim highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Plug <span class="s1">'tpope/vim-projectionist'</span>
</code></pre></div></div>
<p>to your .vimrc, then open Vim and run <code class="language-plaintext highlighter-rouge">:BundleInstall</code>.</p>
<h3 id="your-firstprojectionsjson">Your first .projections.json</h3>
<p>To try out vim-projectionist let’s clone the
<a href="https://github.com/angular/angular.js">angular.js</a> repo and open it up
in vim. As you look around, notice that for almost every file in the
<code class="language-plaintext highlighter-rouge">src/</code> folder, there’s a matching spec file in the <code class="language-plaintext highlighter-rouge">test/</code> directory.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>src/ngMock/angular-mocks.js
test/ngMock/angular-mocksSpec.js
</code></pre></div></div>
<p>Since we know this is the pattern of how files are named in this project,
let’s create a .projections.json file in the root of the project that allows
use :A to toggle between the src file and its corresponding test.</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"src/*.js"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"alternate"</span><span class="p">:</span><span class="w"> </span><span class="s2">"test/{}Spec.js"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"test/*Spec.js"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"alternate"</span><span class="p">:</span><span class="w"> </span><span class="s2">"src/{}.js"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>Now open up <code class="language-plaintext highlighter-rouge">src/ngMock/angular-mocks.js</code> and type <code class="language-plaintext highlighter-rouge">:A</code> to watch it switch to <code class="language-plaintext highlighter-rouge">test/ngMock/angular-mocksSpec.js</code>. Type <code class="language-plaintext highlighter-rouge">:A</code> again and you should be switched back to the original file.</p>
<p>Here’s how it works.</p>
<p>1.<code class="language-plaintext highlighter-rouge">"src/*.js"</code> uses the * to capture the string between <code class="language-plaintext highlighter-rouge">src/</code> and
<code class="language-plaintext highlighter-rouge">.js</code>. In the <code class="language-plaintext highlighter-rouge">src/ngMock/angular-mocks.js</code> example, the <code class="language-plaintext highlighter-rouge">*</code> captures
<code class="language-plaintext highlighter-rouge">ngMock/angular-mocks</code>.</p>
<ol>
<li><code class="language-plaintext highlighter-rouge">"alternate": "test/{}Spec.js"</code> specifies the file to switch to
when using <code class="language-plaintext highlighter-rouge">:A</code>. <code class="language-plaintext highlighter-rouge">{}</code> is replaced with the value captured in Step 1.
If the captured value was <code class="language-plaintext highlighter-rouge">ngMock/angular-mocks</code> then the alternate file
would be <code class="language-plaintext highlighter-rouge">test/ngMock/angular-mocksSpec.js</code>.</li>
<li>To be able to switch back from the test file to the src file, we have to
define the reverse association with a <code class="language-plaintext highlighter-rouge">"test/*Spec.js"</code> section.</li>
</ol>
<h3 id="more-advancedexample">More advanced example</h3>
<p>Now that you have made your first .projections.json file, let’s try some
more advanced features. For this example, clone the
<a href="https://github.com/hexpm/hex_web">hex-web</a> elixir project and open it
up in vim. Take a look around and notice for every file in the web folder,
there is usually a matching file in the test folder.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>web/controllers/dashboard_controller.ex
test/controllers/dashboard_controller_test.exs
</code></pre></div></div>
<p>We can use this pattern to build a basic .projections.json file.</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"web/*.ex"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"alternate"</span><span class="p">:</span><span class="w"> </span><span class="s2">"test/{}_test.exs"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"test/*_test.exs"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"alternate"</span><span class="p">:</span><span class="w"> </span><span class="s2">"web/{}.ex"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>Now open up <code class="language-plaintext highlighter-rouge">web/controllers/api/docs_controller.ex</code> and type <code class="language-plaintext highlighter-rouge">:A</code> to watch
it switch to <code class="language-plaintext highlighter-rouge">test/controllers/api_docs_controller_test.exs</code>. Type <code class="language-plaintext highlighter-rouge">:A</code> again
and you should be switched back to the original controller file.</p>
<p>Now, let’s make it so we can jump to the controllers using an <code class="language-plaintext highlighter-rouge">:Econtroller</code>
shortcut like we could in vim-rails. Reopen the .projections.json, and add
the following section:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"web/controllers/*\_controller.ex"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"controller"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"web/*.ex"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"alternate"</span><span class="p">:</span><span class="w"> </span><span class="s2">"test/{}</span><span class="se">\_</span><span class="s2">test.exs"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"test/*\_test.exs"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"alternate"</span><span class="p">:</span><span class="w"> </span><span class="s2">"web/{}.ex"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>Now type the <code class="language-plaintext highlighter-rouge">:Econtroller api/</code> command and hit Tab to see your options.
Projectionist will show you all of the possible controllers that match
what you’ve typed so far.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>:Econtroller api/
api/docs api/key api/package api/retirement
api/index api/owner api/release api/user
:Econtroller api/
</code></pre></div></div>
<p>You can tab through all of the available options and hit enter to open. Once
it’s open you can use <code class="language-plaintext highlighter-rouge">:A</code> to switch to the test file and <code class="language-plaintext highlighter-rouge">:A</code> again to switch
back. You can also use the following variations of the <code class="language-plaintext highlighter-rouge">:Econtroller</code> command
to control how the file is opened.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>:Econtroller - Open the controller in the current window
:Tcontroller - Open the controller in a new tab
:Vcontroller - Open the controller in a vertical split
:Scontroller - Open the controller in a horizontal split
</code></pre></div></div>
<p>You can add as many of these “type” definitions to your .projections.json as
you like. For example, the following section would add the <code class="language-plaintext highlighter-rouge">:Emodel</code>,
<code class="language-plaintext highlighter-rouge">:Tmodel</code>, <code class="language-plaintext highlighter-rouge">:Vmodel</code>, and <code class="language-plaintext highlighter-rouge">:Smodel</code> commands.</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"web/models/*.ex"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"model"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="err">/*</span><span class="w"> </span><span class="err">Rest</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">.projections.json</span><span class="w"> </span><span class="err">file</span><span class="w"> </span><span class="err">*/</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<h3 id="takeways">Takeways</h3>
<p>These are just a few of the features offered by vim-projectionist. Other
features include using templates for new files and transforming the path
captured by <code class="language-plaintext highlighter-rouge">*</code>. Run <code class="language-plaintext highlighter-rouge">:help projectionist</code> to see the full documentation.</p>
<p>Provided a project has a standard structure, you can easily build a
.projections.json file that allows you to quickly navigate a new repo. I’m
definitely indebted to <a href="https://github.com/tpope">Tim Pope</a> for putting this
plugin together, and can’t fathom the amount of time it’s saved me.</p>
<p>Thanks so much for reading this post and please follow my blog or share this
article if you found it helpful.</p>Phoenix acceptance tests on Semaphore CI2016-01-22T00:00:00-07:002016-01-22T00:00:00-07:00http://localhost:4000/2016/01/22/phoenix-acceptance-tests-on-semaphore-ci<p>Plataformatec just published a great <a href="http://blog.plataformatec.com.br/2016/01/writing-acceptance-tests-in-phoenix/">introduction to testing Phoenix apps with Hound</a>.
<a href="https://github.com/HashNuke/hound">Hound</a> is really cool because it gives you a
nice API to build browser-based acceptance tests that exercise your complete
application — from your Ember/React/Angular frontend to your
<a href="http://www.phoenixframework.org/">Phoenix</a> backend.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">Myapp</span><span class="o">.</span><span class="no">Integration</span><span class="o">.</span><span class="no">AuthenticationTest</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Myapp</span><span class="o">.</span><span class="no">IntegrationCase</span>
<span class="n">test</span> <span class="s2">"log in and log out"</span> <span class="k">do</span>
<span class="n">insert_user</span> <span class="ss">email:</span> <span class="s2">"user@example.com"</span><span class="p">,</span> <span class="ss">password:</span> <span class="s2">"secret"</span>
<span class="n">navigate_to</span> <span class="s2">"/"</span>
<span class="n">click</span><span class="p">({</span><span class="ss">:link_text</span><span class="p">,</span> <span class="s2">"Sign In"</span><span class="p">})</span>
<span class="n">sign_in_with</span><span class="p">(</span><span class="s2">"user@example.com"</span><span class="p">,</span> <span class="s2">"secret"</span><span class="p">)</span>
<span class="n">assert</span> <span class="n">logged_in?</span>
<span class="n">click</span><span class="p">({</span><span class="ss">:link_text</span><span class="p">,</span> <span class="s2">"Sign Out"</span><span class="p">})</span>
<span class="n">refute</span> <span class="n">logged_in?</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">logged_in?</span> <span class="k">do</span>
<span class="n">page_source</span> <span class="o">=~</span> <span class="s2">"Sign Out"</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">sign_in_with</span><span class="p">(</span><span class="n">email</span><span class="p">,</span> <span class="n">password</span><span class="p">)</span> <span class="k">do</span>
<span class="n">fill_field</span><span class="p">({</span><span class="ss">:name</span><span class="p">,</span> <span class="s2">"session[email]"</span><span class="p">},</span> <span class="n">email</span><span class="p">)</span>
<span class="n">fill_field</span><span class="p">({</span><span class="ss">:name</span><span class="p">,</span> <span class="s2">"session[password]"</span><span class="p">},</span> <span class="n">password</span><span class="p">)</span>
<span class="n">click</span><span class="p">({</span><span class="ss">:css</span><span class="p">,</span> <span class="s2">"[type=submit]"</span><span class="p">})</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<h2 id="running-tests-in-ci">Running tests in CI</h2>
<p>Once you start writing acceptance tests, the next step is to get them running
in continuous integration. <a href="https://semaphoreci.com/">Semaphore CI</a> has a
<a href="https://semaphoreci.com/blog/2015/10/22/semaphore-continuous-deployment-with-phoenix-1-0.html">great blog post</a>
about setting up CI for a basic Phoenix app. However if you want to test with
<a href="https://github.com/HashNuke/hound">Hound</a>, there are a couple more steps that
you need to do.</p>
<h2 id="modifications-to-the-app">Modifications to the app</h2>
<p>The only code modification you should need to make is to allow the database
username and password to be set via environment variables in <code class="language-plaintext highlighter-rouge">config/test.exs</code>.
On <a href="https://semaphoreci.com/">Semaphore</a>, these variables are
<code class="language-plaintext highlighter-rouge">DATABASE_POSTGRESQL_USERNAME</code> and <code class="language-plaintext highlighter-rouge">DATABASE_POSTGRESQL_PASSWORD</code>.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Configure your database</span>
<span class="n">config</span> <span class="ss">:myapp</span><span class="p">,</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Repo</span><span class="p">,</span>
<span class="ss">adapter:</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Adapters</span><span class="o">.</span><span class="no">Postgres</span><span class="p">,</span>
<span class="ss">username:</span> <span class="no">System</span><span class="o">.</span><span class="n">get_env</span><span class="p">(</span><span class="s2">"DATABASE_POSTGRESQL_USERNAME"</span><span class="p">)</span> <span class="o">||</span> <span class="s2">"postgres"</span><span class="p">,</span>
<span class="ss">password:</span> <span class="no">System</span><span class="o">.</span><span class="n">get_env</span><span class="p">(</span><span class="s2">"DATABASE_POSTGRESQL_PASSWORD"</span><span class="p">)</span> <span class="o">||</span> <span class="s2">"postgres"</span><span class="p">,</span>
<span class="ss">database:</span> <span class="s2">"myapp_test"</span><span class="p">,</span>
<span class="ss">hostname:</span> <span class="s2">"localhost"</span><span class="p">,</span>
<span class="ss">pool:</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Adapters</span><span class="o">.</span><span class="no">SQL</span><span class="o">.</span><span class="no">Sandbox</span>
</code></pre></div></div>
<h2 id="setting-up-yourproject">Setting up your project</h2>
<p>To get started, head over to <a href="https://semaphoreci.com/users/sign_up">Semaphore CI</a>
and create an account. Once your account is set up, click “Build a new project”.</p>
<p><img src="/assets/img/0*yYeKSc1VRQ19XfxR.png" alt="" /></p>
<p>Next, choose where your repo is hosted…</p>
<p><img src="/assets/img/0*b5YZc-25SZmAeJeQ.png" alt="" /></p>
<p>pick the project…</p>
<p><img src="/assets/img/0*EcnVaeGwW6eHcO48.png" alt="" /></p>
<p>and the branch you want to build.</p>
<p><img src="/assets/img/0*kfFp2fzaV5yppe0n.png" alt="" /></p>
<p>Now Semaphore will analyze the project and give you some sensible defaults for your first build.</p>
<p><img src="/assets/img/0*s13tZ21mel0MRex-.png" alt="" /></p>
<h3 id="customizing-the-build-plan">Customizing the build plan</h3>
<p>After Semaphore is done analyzing, it proposes a build plan like this:</p>
<p><img src="/assets/img/0*9_R7YMFbxBxeqHcf.png" alt="" />
Choose the Elixir version and review the commands in the Setup section. Semaphore’s default Elixir build plan includes two Setup steps:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mix local.hex --force
mix deps.get --only test
</code></pre></div></div>
<p>This is great for a basic Elixir project, but there are a few more steps if you’re testing with <a href="https://github.com/HashNuke/hound">Hound</a>. In the Setup section, click “Edit Thread” and paste in these commands.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mix local.hex --force
mix deps.get --only test
nvm use 5.3.0
npm install
bower install
node_modules/.bin/brunch build
MIX_ENV=test mix do deps.compile, compile
phantomjs --wd --webdriver-loglevel=ERROR &
</code></pre></div></div>
<p><img src="/assets/img/0*XXeybvMMGsGAmMPi.png" alt="" /></p>
<h4 id="what-did-i-just-paste-in">What did I just paste in?</h4>
<p>Let’s take these additional commands line by line:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nvm use 5.3.0
</code></pre></div></div>
<p>First, we set the node version to 5.3.0. My project is using the latest version of <a href="http://brunch.io/">Brunch</a> to build my assets, and Brunch requires a recent version of Node.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>npm install
</code></pre></div></div>
<p>Next, we install all of our NPM dependences. This also installs <a href="http://brunch.io/">Brunch</a>, Phoenix’s default tool for building assets.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bower install
</code></pre></div></div>
<p>My project has a few bower dependencies, so I run this to ensure everything is downloaded before the build.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>node_modules/.bin/brunch build
</code></pre></div></div>
<p>Next, we generate the assets needed for our acceptance tests. Running mix test doesn’t automatically build our assets, so this step must be run in order for Hound to find our javascript.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>MIX_ENV=test mix do deps.compile, compile
</code></pre></div></div>
<p>After that, we compile our app and its dependencies in the test environment. By running command this as part of the setup phase, we get a lot less noise in our test output later.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>phantomjs --wd --webdriver-loglevel=ERROR &
</code></pre></div></div>
<p>Finally, we start our PhantomJS webdriver server for Hound. I send
it to the background with <code class="language-plaintext highlighter-rouge">&</code> so it keeps running for the rest of the
build. If you were using Chromedriver instead, you can run <code class="language-plaintext highlighter-rouge">chromedriver &</code>.</p>
<h3 id="your-first-build">Your First Build.</h3>
<p>Once you’ve updated your build plan, scroll to the bottom and click “Build With These Settings.” If all goes well, you should have your first green build in a minute or two.</p>
<p><img src="/assets/img/0*B908NlSS7eBb20uI.png" alt="" /></p>
<p>If you do run into a problem, you can click the “Launch SSH” button to connect to a build box and run your commands manually.</p>
<h3 id="wrapping-up">Wrapping up</h3>
<p>If you’re going to spend the time writing acceptance tests, it’s a great idea to make sure they’re run in CI. <a href="https://semaphoreci.com/">Semaphore</a> already has great support for <a href="http://elixir-lang.org/">Elixir</a>, and with a few extra commands you can run full acceptance tests with <a href="https://github.com/HashNuke/hound">Hound</a>.</p>
<p>I also want to make sure to give credit where credit is due. Semaphore’s
“<a href="https://semaphoreci.com/blog/2015/10/22/semaphore-continuous-deployment-with-phoenix-1-0.html">Continuous Deployment with Phoenix</a>” post was great for
getting started. I hope this post has been helpful in showing how to put
<a href="https://github.com/HashNuke/hound">Hound</a> into your testing mix.</p>
<p>Happy Testing!</p>
<p><em>Originally published at <a href="http://blog.animascodelabs.com/2016/01/22/running-hound-on-semaphoreci/">blog.animascodelabs.com</a> on January 22, 2016.</em></p>Plataformatec just published a great introduction to testing Phoenix apps with Hound. Hound is really cool because it gives you a nice API to build browser-based acceptance tests that exercise your complete application — from your Ember/React/Angular frontend to your Phoenix backend.Running PhantomJS Ghost Driver as an OS X service2015-12-31T00:00:00-07:002015-12-31T00:00:00-07:00http://localhost:4000/2015/12/31/running-phantomjs-ghost-driver-as-an-os-x-service<p><img src="/assets/img/1*6T5xUXWJ9DwF4V64nEF0Qg.png" alt="" /></p>
<p>Recently, I have been experimenting with the
<a href="http://www.phoenixframework.org/">Phoenix Framework</a> and testing tool
called <a href="https://github.com/HashNuke/hound">Hound</a>. Hound allows you to write
integration tests that drive your app with a headless
<a href="http://phantomjs.org/">PhantomJS</a> browser. However, in order to use PhantomJS
with Hound you must have a <a href="https://github.com/detro/ghostdriver">Ghost Driver</a>
session running on your Mac.</p>
<h2 id="manually-running-ghostdriver">Manually Running Ghost Driver</h2>
<p>Ghost Driver comes bundled with PhantomJS, and can be installed on OS X
using <a href="http://brew.sh/">Homebrew</a>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ brew install phantomjs
</code></pre></div></div>
<p>After that, running a Ghost Driver session is just a single command.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ phantomjs --wd
</code></pre></div></div>
<p>Your tests will pass now, but you have to remember to run this command and
keep it running while your working on your app.</p>
<h2 id="letting-launchd-take-care-ofit">Letting launchd take care of it</h2>
<p>I figured there had to be an easier way to automatically start Ghost Driver.
Then I remembered all those <code class="language-plaintext highlighter-rouge">.plist</code> files that <a href="http://brew.sh/">Homebrew</a>
tells you to link to <code class="language-plaintext highlighter-rouge">~/Library/LaunchAgents</code> if you want services to start
automatically.</p>
<p>After a little experimentation, I created my own LaunchAgent file and stored
it in <code class="language-plaintext highlighter-rouge">~/Library/LaunchAgents/org.phantomjs.ghostdriver.plist</code>.</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?xml version="1.0" encoding="UTF-8"?></span>
<span class="cp"><!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"></span>
<span class="nt"><plist</span> <span class="na">version=</span><span class="s">"1.0"</span><span class="nt">></span>
<span class="nt"><dict></span>
<span class="nt"><key></span>Label<span class="nt"></key></span>
<span class="nt"><string></span>org.phantomjs.ghostdriver<span class="nt"></string></span>
<span class="nt"><key></span>RunAtLoad<span class="nt"></key></span>
<span class="nt"><true/></span>
<span class="nt"><key></span>KeepAlive<span class="nt"></key></span>
<span class="nt"><false/></span>
<span class="nt"><key></span>ProgramArguments<span class="nt"></key></span>
<span class="nt"><array></span>
<span class="nt"><string></span>/usr/local/opt/phantomjs/bin/phantomjs<span class="nt"></string></span>
<span class="nt"><string></span>--wd<span class="nt"></string></span>
<span class="nt"></array></span>
<span class="nt"><key></span>StandardErrorPath<span class="nt"></key></span>
<span class="nt"><string></span>/usr/local/var/log/phantomjs-ghostdriver-error.log<span class="nt"></string></span>
<span class="nt"><key></span>StandardOutPath<span class="nt"></key></span>
<span class="nt"><string></span>/usr/local/var/log/phantomjs-ghostdriver-output.log<span class="nt"></string></span>
<span class="nt"></dict></span>
<span class="nt"></plist></span>
</code></pre></div></div>
<p>Now I can manually start Ghost Driver with my new script:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ launchctl load ~/Library/LaunchAgents/org.phantomjs.ghostdriver.plist
</code></pre></div></div>
<p>If I need to stop it, I can run the following:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ launchctl unload ~/Library/LaunchAgents/org.phantomjs.ghostdriver.plist
</code></pre></div></div>
<p>The great part about using <code class="language-plaintext highlighter-rouge">launchctl</code> is it will launch this agent every time
I start my computer. It a “set it and forget it” script, that will make
sure my tests won’t fail just because Ghost Driver isn’t started.</p>
<h2 id="other-notes">Other notes</h2>
<p>When working with <code class="language-plaintext highlighter-rouge">launchctl</code>, make sure you are not running in a tmux
session. Homebrew warns you about this as part of many installation scripts
and it still applies here.</p>
<p>If you have any other comments or suggestions, feel free to hit me up on <a href="https://twitter.com/bayfieldcoder">Twitter</a>.</p>
<p><em>Originally published at <a href="http://blog.animascodelabs.com/2015/12/31/running-phantomjs-ghostdriver-as-an-osx-service/">blog.animascodelabs.com</a> on December 31, 2015.</em></p>Building Workflows with Sidekiq Batches2015-12-28T00:00:00-07:002015-12-28T00:00:00-07:00http://localhost:4000/2015/12/28/building-workflows-with-sidekiq-batches<p>A few weeks ago, <a href="http://sidekiq.org/">Sidekiq</a>’s creator, Mike Perham, posted a great article about how to handle
<a href="https://github.com/mperham/sidekiq/wiki/Really-Complex-Workflows-with-Batches">Really Complex Workflows with Batches</a>.
His article shows how you can wait for a batch of jobs to complete before moving on to the next job/batch in the workflow.</p>
<p><img src="/assets/img/0*zK60QwnR7JRY-nwI.png" alt="" /></p>
<p>I first learned about this technique during Sidekiq’s <a href="http://sidekiq.org/support">weekly happy hour</a> back in March,
and have been using it on <a href="http://recognized.io/">Recognized.io</a> for the last 6 months. Although the example in the
documentation shows the basics of dependent batches, I wanted to share how we created standalone Workflow classes to
manage this process.</p>
<p>Here is an example of a workflow class using Sidekiq’s batch callbacks.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">GenerateComplexReportWorkflow</span>
<span class="kp">include</span> <span class="no">Sidekiq</span><span class="o">::</span><span class="no">Worker</span>
<span class="c1"># The main entry point into the workflow, just</span>
<span class="c1"># like a normal Sidekiq job</span>
<span class="k">def</span> <span class="nf">perform</span><span class="p">(</span><span class="n">report_id</span><span class="p">)</span>
<span class="n">step_1</span><span class="p">(</span><span class="n">report_id</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">step_1</span><span class="p">(</span><span class="n">report_id</span><span class="p">)</span>
<span class="n">generate_calculations_batch</span> <span class="o">=</span> <span class="no">Sidekiq</span><span class="o">::</span><span class="no">Batch</span><span class="p">.</span><span class="nf">new</span>
<span class="n">generate_calculations_batch</span><span class="p">.</span><span class="nf">on</span><span class="p">(</span>
<span class="s2">"success"</span><span class="p">,</span> <span class="c1">#{self.class}#step2",</span>
<span class="s2">"report_id"</span> <span class="o">=></span> <span class="n">report_id</span>
<span class="p">)</span>
<span class="n">generate_calculations_batch</span><span class="p">.</span><span class="nf">jobs</span> <span class="k">do</span>
<span class="no">Customer</span><span class="p">.</span><span class="nf">find_each</span> <span class="k">do</span> <span class="o">|</span><span class="n">customer</span><span class="o">|</span>
<span class="no">ExpensiveCalculationForCustomer</span><span class="p">.</span><span class="nf">perform_async</span><span class="p">(</span>
<span class="n">report_id</span><span class="p">,</span> <span class="n">customer</span><span class="p">.</span><span class="nf">id</span>
<span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">step_2</span><span class="p">(</span><span class="n">status</span><span class="p">,</span> <span class="n">options</span><span class="p">)</span>
<span class="n">report_id</span> <span class="o">=</span> <span class="n">options</span><span class="p">[</span><span class="s2">"report_id"</span><span class="p">]</span>
<span class="n">generate_summaries_batch</span> <span class="o">=</span> <span class="no">Sidekiq</span><span class="o">::</span><span class="no">Batch</span><span class="p">.</span><span class="nf">new</span>
<span class="n">generate_summaries_batch</span><span class="p">.</span><span class="nf">on</span><span class="p">(</span>
<span class="s2">"success"</span><span class="p">,</span> <span class="c1">#{self.class}#step3",</span>
<span class="s2">"report_id"</span> <span class="o">=></span> <span class="n">report_id</span>
<span class="p">)</span>
<span class="n">generate_summaries_batch</span><span class="p">.</span><span class="nf">jobs</span> <span class="k">do</span>
<span class="no">Region</span><span class="p">.</span><span class="nf">find_each</span> <span class="k">do</span> <span class="o">|</span><span class="n">region</span><span class="o">|</span>
<span class="no">ExpensiveRegionalSummary</span><span class="p">.</span><span class="nf">perform_async</span><span class="p">(</span>
<span class="n">report_id</span><span class="p">,</span> <span class="n">region</span><span class="p">.</span><span class="nf">id</span>
<span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">step_3</span><span class="p">(</span><span class="n">status</span><span class="p">,</span> <span class="n">options</span><span class="p">)</span>
<span class="n">report</span> <span class="o">=</span> <span class="no">Report</span><span class="p">.</span><span class="nf">find</span><span class="p">(</span><span class="n">options</span><span class="p">[</span><span class="s2">"report_id"</span><span class="p">])</span>
<span class="n">report</span><span class="p">.</span><span class="nf">mark_as_finished!</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>The key thing to notice is we set a success callback on the batch that triggers the next step in the
workflow. This callback also passes along <code class="language-plaintext highlighter-rouge">report_id</code>, which is needed in subsequent steps:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">generate_calculations_batch</span><span class="p">.</span><span class="nf">on</span><span class="p">(</span>
<span class="s2">"success"</span><span class="p">,</span> <span class="s2">"</span><span class="si">#{</span><span class="nb">self</span><span class="p">.</span><span class="nf">class</span><span class="si">}</span><span class="s2">#step2"</span><span class="p">,</span>
<span class="s2">"report_id"</span> <span class="o">=></span> <span class="n">report_id</span>
<span class="p">)</span>
</code></pre></div></div>
<h2 id="gotchas">Gotchas</h2>
<p>When we start using <code class="language-plaintext highlighter-rouge">#each</code> to dynamically create jobs in a workflow, there are a few edge cases that make things tricky.</p>
<h3 id="no-jobs-in-abatch">No Jobs in a Batch</h3>
<p>In the code below, if there aren’t any customers, it won’t generate any jobs for the batch.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">generate_calculations_batch</span><span class="p">.</span><span class="nf">jobs</span> <span class="k">do</span>
<span class="no">Customer</span><span class="p">.</span><span class="nf">find_each</span> <span class="k">do</span> <span class="o">|</span><span class="n">customer</span><span class="o">|</span>
<span class="no">ExpensiveCalculationForCustomer</span><span class="p">.</span><span class="nf">perform_async</span><span class="p">(</span>
<span class="n">report_id</span><span class="p">,</span> <span class="n">customer</span><span class="p">.</span><span class="nf">id</span>
<span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>I found out the hard way if there are no jobs in a batch, the callbacks won’t run and trigger
next step in the workflow. To fix this, we add a <code class="language-plaintext highlighter-rouge">NullWorker</code> job to the batch, which ensures
we always have at least one job and the callbacks will be run.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">NullWorker</span>
<span class="kp">include</span> <span class="no">Sidekiq</span><span class="o">::</span><span class="no">Worker</span> <span class="k">def</span> <span class="nf">perform</span>
<span class="c1">#NOOP</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">class</span> <span class="nc">GenerateComplexReportWorkflow</span>
<span class="k">def</span> <span class="nf">step_1</span><span class="p">(</span><span class="n">report_id</span><span class="p">)</span>
<span class="c1"># omitted...</span>
<span class="n">generate_calculations_batch</span><span class="p">.</span><span class="nf">jobs</span> <span class="k">do</span>
<span class="no">Customer</span><span class="p">.</span><span class="nf">find_each</span> <span class="k">do</span> <span class="o">|</span><span class="n">customer</span><span class="o">|</span>
<span class="no">ExpensiveCalculationForCustomer</span><span class="p">.</span><span class="nf">perform_async</span><span class="p">(</span>
<span class="n">report_id</span><span class="p">,</span> <span class="n">customer</span><span class="p">.</span><span class="nf">id</span>
<span class="p">)</span>
<span class="k">end</span>
<span class="c1"># Ensure batch has at least 1 job so callbacks</span>
<span class="c1"># are run.</span>
<span class="no">NullWorker</span><span class="p">.</span><span class="nf">perform_async</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="c1"># other steps omitted...</span>
<span class="k">end</span>
</code></pre></div></div>
<h3 id="adding-a-large-number-of-jobs-to-abatch">Adding a large number of jobs to a batch</h3>
<p>Sometimes you need to create a large number of jobs (500,000+) in a single batch. Since all
jobs defined in a <code class="language-plaintext highlighter-rouge">jobs</code> block are created atomically, it can take a few minutes to persist
your batch in Redis. With enough jobs, this can even lead to a Redis time out error.</p>
<p>Fortunately, you can add jobs to your batch in groups. Here’s an example:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">GenerateComplexReportWorkflow</span>
<span class="k">def</span> <span class="nf">step_1</span><span class="p">(</span><span class="n">report_id</span><span class="p">)</span>
<span class="c1"># omitted...</span>
<span class="no">Customer</span><span class="p">.</span><span class="nf">all</span><span class="p">.</span><span class="nf">pluck</span><span class="p">(</span><span class="ss">:id</span><span class="p">).</span><span class="nf">each_slice</span><span class="p">(</span><span class="mi">1000</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">customer_ids</span><span class="o">|</span>
<span class="n">generate_calculations_batch</span><span class="p">.</span><span class="nf">jobs</span> <span class="k">do</span>
<span class="n">customer_ids</span><span class="p">.</span><span class="nf">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">customer_id</span><span class="o">|</span>
<span class="no">ExpensiveCalculationForCustomer</span><span class="p">.</span><span class="nf">perform_async</span><span class="p">(</span>
<span class="n">report_id</span><span class="p">,</span> <span class="n">customer_id</span>
<span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="c1"># Add one last job to make sure the callbacks are run</span>
<span class="n">generated_calculations_batch</span><span class="p">.</span><span class="nf">jobs</span> <span class="k">do</span>
<span class="no">NullWorker</span><span class="p">.</span><span class="nf">perform_async</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="c1"># other steps omitted...</span>
<span class="k">end</span>
</code></pre></div></div>
<p>There are a couple of things to note in this example. First, we use <code class="language-plaintext highlighter-rouge">Customer.all.pluck(:id)</code> to only load
the relevant customer ids, instead of the entire customer object. This saves a ton of time and memory.</p>
<p>Second, we slice the customer ids into groups, so jobs can be added to the batch in smaller groups and not
stress out Redis. This also allows your workers to get started on processing a batch while the rest of the
jobs are still being created. It’s important to note if all of your jobs complete before the next set of
jobs have been created, your batches’ callbacks will be triggered prematurely. However, if you are using
background processing to handle heavy computation, the jobs will almost always take longer to complete
than the time it takes to add another set of jobs to the batch.</p>
<p>Finally, we add our NullWorker job to the batch to make sure the callbacks are run, even if there is no data
in the system.</p>
<p>I have been using this approach on <a href="http://recognized.io/">Recognized.io</a> for the past several months and
it’s allowed me to create some very large batches without issue.</p>
<h2 id="wrapping-up">Wrapping Up</h2>
<p><a href="http://sidekiq.org/products/pro">Sidekiq Pro</a>’s batch support enables developers to achieve maximum
parallelization, while sending jobs through complex workflows. Although it’s a paid addon, it’s highly
worth the money when you consider the significant performance improvements you’ll be able to make on your app.</p>
<p><em>Originally published at <a href="http://blog.animascodelabs.com/2015/12/28/workflows-with-sidekiq-batches/">blog.animascodelabs.com</a> on December 28, 2015.</em></p>A few weeks ago, Sidekiq’s creator, Mike Perham, posted a great article about how to handle Really Complex Workflows with Batches. His article shows how you can wait for a batch of jobs to complete before moving on to the next job/batch in the workflow.Eager-loading Polymorphic Relationships in Rails2014-10-30T00:00:00-06:002014-10-30T00:00:00-06:00http://localhost:4000/2014/10/30/eager-loading-polymorphic-relationships-in-rails<p><em>Update: The internal Preloader class has changed between Rails 3 and Rails 4. The code examples have been updated to reflect that.</em></p>
<p><img src="/assets/img/1*hrNtNZLOi1k2hjMAzsNk9Q.jpeg" alt="" /></p>
<p>ActiveRecord has some great utilities for eager loading associations to solve the
<a href="http://guides.rubyonrails.org/active_record_querying.html#eager-loading-associations">N+1 query problem</a>. Take,
for example, the following code to print a list of sellers’ email addresses in an online marketplace:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">listings</span> <span class="o">=</span> <span class="no">Listing</span><span class="p">.</span><span class="nf">limit</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> <span class="n">listings</span><span class="p">.</span><span class="nf">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">listing</span><span class="o">|</span>
<span class="nb">puts</span> <span class="n">listing</span><span class="p">.</span><span class="nf">seller</span><span class="p">.</span><span class="nf">email</span>
<span class="k">end</span>
</code></pre></div></div>
<p>This code will run the following sql queries.</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">listings</span> <span class="k">LIMIT</span> <span class="mi">10</span>
<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">sellers</span> <span class="k">WHERE</span> <span class="n">sellers</span><span class="p">.</span><span class="n">id</span> <span class="o">=</span> <span class="mi">1</span>
<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">sellers</span> <span class="k">WHERE</span> <span class="n">sellers</span><span class="p">.</span><span class="n">id</span> <span class="o">=</span> <span class="mi">2</span>
<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">sellers</span> <span class="k">WHERE</span> <span class="n">sellers</span><span class="p">.</span><span class="n">id</span> <span class="o">=</span> <span class="mi">3</span>
<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">sellers</span> <span class="k">WHERE</span> <span class="n">sellers</span><span class="p">.</span><span class="n">id</span> <span class="o">=</span> <span class="mi">4</span>
<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">sellers</span> <span class="k">WHERE</span> <span class="n">sellers</span><span class="p">.</span><span class="n">id</span> <span class="o">=</span> <span class="mi">5</span>
<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">sellers</span> <span class="k">WHERE</span> <span class="n">sellers</span><span class="p">.</span><span class="n">id</span> <span class="o">=</span> <span class="mi">6</span>
<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">sellers</span> <span class="k">WHERE</span> <span class="n">sellers</span><span class="p">.</span><span class="n">id</span> <span class="o">=</span> <span class="mi">7</span>
<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">sellers</span> <span class="k">WHERE</span> <span class="n">sellers</span><span class="p">.</span><span class="n">id</span> <span class="o">=</span> <span class="mi">8</span>
<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">sellers</span> <span class="k">WHERE</span> <span class="n">sellers</span><span class="p">.</span><span class="n">id</span> <span class="o">=</span> <span class="mi">9</span>
<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">sellers</span> <span class="k">WHERE</span> <span class="n">sellers</span><span class="p">.</span><span class="n">id</span> <span class="o">=</span> <span class="mi">10</span>
</code></pre></div></div>
<p>However, using ActiveRecord’s <code class="language-plaintext highlighter-rouge">#includes</code> method</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">listings</span> <span class="o">=</span> <span class="no">Listing</span><span class="p">.</span><span class="nf">includes</span><span class="p">(</span><span class="ss">:seller</span><span class="p">).</span><span class="nf">limit</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
<span class="n">listings</span><span class="p">.</span><span class="nf">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">listing</span><span class="o">|</span>
<span class="nb">puts</span> <span class="n">listing</span><span class="p">.</span><span class="nf">seller</span><span class="p">.</span><span class="nf">email</span>
<span class="k">end</span>
</code></pre></div></div>
<p>allows you to shorten it to just two queries.</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="nv">"listings"</span><span class="p">.</span><span class="o">*</span> <span class="k">FROM</span> <span class="nv">"listings"</span> <span class="k">LIMIT</span> <span class="mi">10</span>
<span class="k">SELECT</span> <span class="nv">"sellers"</span><span class="p">.</span><span class="o">*</span> <span class="k">FROM</span> <span class="nv">"sellers"</span> <span class="k">WHERE</span> <span class="nv">"sellers"</span><span class="p">.</span><span class="nv">"id"</span> <span class="k">IN</span> <span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">7</span><span class="p">,</span> <span class="mi">8</span><span class="p">,</span> <span class="mi">9</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="mi">11</span><span class="p">,</span> <span class="mi">12</span><span class="p">)</span>
</code></pre></div></div>
<h3 id="preloading-polymorphic-associations">Preloading polymorphic associations</h3>
<p>Now, let’s say that we have two types of sellers, Users and Organizations. To get the email for a
listing sold by a User we still call <code class="language-plaintext highlighter-rouge">listing.seller.email</code>, but if the listing is sold by an
Organization we need to call <code class="language-plaintext highlighter-rouge">listing.seller.primary_contact.email</code>. This means we need to preload
seller on listings sold by users and <code class="language-plaintext highlighter-rouge">seller.primary_contact</code> on listings sold by Organizations. Here’s
how to do it.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">listings_with_associations_preloaded</span>
<span class="n">listings</span> <span class="o">=</span> <span class="no">Listing</span><span class="p">.</span><span class="nf">includes</span><span class="p">(</span><span class="ss">:seller</span><span class="p">)</span>
<span class="n">listings_by_seller_type</span> <span class="o">=</span> <span class="n">listings</span><span class="p">.</span><span class="nf">group_by</span><span class="p">(</span><span class="o">&</span><span class="ss">:seller_type</span><span class="p">)</span>
<span class="n">organization_listings</span>
<span class="n">listings_by_seller_type</span><span class="p">.</span><span class="nf">fetch</span><span class="p">(</span><span class="s2">"Organization"</span><span class="p">,[])</span>
<span class="n">preload_record_array</span><span class="p">(</span>
<span class="n">organization_listings</span><span class="p">,</span>
<span class="ss">seller: </span><span class="p">[</span><span class="ss">:primary_contact</span><span class="p">]</span>
<span class="p">)</span>
<span class="n">listings</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">preload_record_array</span><span class="p">(</span><span class="n">record_array</span><span class="p">,</span> <span class="n">preload_hash</span><span class="p">)</span>
<span class="no">ActiveRecord</span><span class="o">::</span><span class="no">Associations</span><span class="o">::</span><span class="no">Preloader</span><span class="p">.</span><span class="nf">new</span>
<span class="p">.</span><span class="nf">preload</span><span class="p">(</span><span class="n">record_array</span><span class="p">,</span> <span class="n">preload_hash</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Let’s take this example line by line. First, we preload all of the related seller objects since we will need them for every record.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">listings</span> <span class="o">=</span> <span class="no">Listing</span><span class="p">.</span><span class="nf">includes</span><span class="p">(</span><span class="ss">:seller</span><span class="p">)</span>
</code></pre></div></div>
<p>Then, we group the listings by <code class="language-plaintext highlighter-rouge">#seller_type</code>.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">listings_by_seller_type</span> <span class="o">=</span> <span class="n">listings</span><span class="p">.</span><span class="nf">group_by</span><span class="p">(</span><span class="o">&</span><span class="ss">:seller_type</span><span class="p">)</span>
<span class="c1">#=> {"User" => [...], "Organization" => [...]}</span>
</code></pre></div></div>
<p>Next, we need to get the array of listings with an organization as the seller. We use Ruby’s <code class="language-plaintext highlighter-rouge">Hash#fetch</code> to return an
empty array if no organizations are found.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">organization_listings</span> <span class="o">=</span>
<span class="n">listings_by_seller_type</span><span class="p">.</span><span class="nf">fetch</span><span class="p">(</span><span class="s2">"Organization"</span><span class="p">,[])</span>
</code></pre></div></div>
<p>Now it’s time to do the actual preloading. Since <code class="language-plaintext highlighter-rouge">organization_listings</code> is an <code class="language-plaintext highlighter-rouge">Array</code> instead of an <code class="language-plaintext highlighter-rouge">ActiveRecord::Relation</code>
we can’t use the <code class="language-plaintext highlighter-rouge">#includes</code> method that we used previously. Instead we need build a lower-level
<a href="http://www.rubydoc.info/docs/rails/ActiveRecord/Associations/Preloader"><code class="language-plaintext highlighter-rouge">ActiveRecord::Associations::Preloader</code></a> object and
call <code class="language-plaintext highlighter-rouge">#preload</code> on it.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">ActiveRecord</span><span class="o">::</span><span class="no">Associations</span><span class="o">::</span><span class="no">Preloader</span><span class="p">.</span><span class="nf">new</span>
<span class="p">.</span><span class="nf">preload</span><span class="p">(</span><span class="n">record_array</span><span class="p">,</span> <span class="n">preload_hash</span><span class="p">)</span>
</code></pre></div></div>
<p>The syntax of this constructor includes the records you’d like loaded as the first argument and the associations to load as
the second argument. The associations to load argument uses the same syntax as the <code class="language-plaintext highlighter-rouge">#includes</code> method that is called on an
<code class="language-plaintext highlighter-rouge">ActiveRecord::Relation</code>.</p>
<h3 id="in-rails-3">In Rails 3</h3>
<p>This API changed between Rails 3 and Rails 4. If you are using Rails 3, the preloader should be called like this.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">ActiveRecord</span><span class="o">::</span><span class="no">Associations</span><span class="o">::</span><span class="no">Preloader</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span>
<span class="n">record_array</span><span class="p">,</span> <span class="n">preload_hash</span>
<span class="p">).</span><span class="nf">run</span><span class="p">()</span>
</code></pre></div></div>
<h3 id="wrapping-up">Wrapping Up</h3>
<p>The models in this example may be a bit contrived, but it illustrates how to preload different types of objects in a polymorphic
relationships, as well as preload objects that are in an <code class="language-plaintext highlighter-rouge">Array</code> instead of an <code class="language-plaintext highlighter-rouge">ActiveRecord::Relation</code>. There is an open
<a href="https://github.com/rails/rails/issues/11719">Rails issue</a> because this class is marked as <code class="language-plaintext highlighter-rouge">:nodoc:</code> and is for internal use only.
However, this class has proven to so useful and I think it’s completely worth blogging about.</p>
<p>Finally, to give credit where credit is due, I wouldn’t have stumbled upon this solution without
<a href="http://stackoverflow.com/questions/12561688/how-to-eager-load-association-for-an-array-of-model-records">this Stack Overflow question and answer</a>.</p>
<p>If you have any feedback, please contact me on Twitter at <a href="https://twitter.com/bayfieldcoder">@bayfieldcoder</a>.</p>
<p><em>Originally published at <a href="http://blog.animascodelabs.com/2014/10/30/preloading-polymorphic-relationships-in-rails/">blog.animascodelabs.com</a> on October 30, 2014.</em></p>Update: The internal Preloader class has changed between Rails 3 and Rails 4. The code examples have been updated to reflect that.