Jekyll2022-02-21T18:48:10-06:00https://milmazz.uno/atom.xmlmilmazzTo teach is to learn twice. - Joseph JoubertMilton Mazzarrime@milmazz.unoOban: Testing your Workers and Configuration2022-02-21T00:00:00-06:002022-02-21T00:00:00-06:00https://milmazz.uno/article/2022/02/21/oban-testing-your-workers-and-configuration<p>In this article, I will continue talking about Oban, but I’ll focus on how to
test your workers and, also, your configuration.</p>
<!--more-->
<p>This article is the second one of a series about Oban, an Elixir library for
background job processing:</p>
<ul>
<li><a href="/article/2022/02/11/oban-job-processing-package-for-elixir/">Oban: job processing library for Elixir</a></li>
<li><strong>Oban: Testing your Workers and Configuration (you are here)</strong></li>
<li>Oban in production</li>
</ul>
<h2 id="testing-the-implementation-of-the-obanworker-behaviour">Testing the implementation of the Oban.Worker behaviour</h2>
<p>Before implementing an Oban Worker, I start writing the unit test that will
insert and execute the job; these tests are usually short because your workers
should be as lean as possible, I tend to treat <code class="language-plaintext highlighter-rouge">Workers</code> as <code class="language-plaintext highlighter-rouge">Resolvers</code> or
<code class="language-plaintext highlighter-rouge">Controllers</code> that orchestrate a series of actions that, most of the time,
execute functions that are already being tested and are part of the business
logic layer. It would also help validate that the job is running the side
effects in these unit tests.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Migrations</span><span class="o">.</span><span class="no">MyDataMigrationWorkerTest</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">Case</span><span class="p">,</span> <span class="ss">async:</span> <span class="no">true</span>
<span class="kn">use</span> <span class="no">Oban</span><span class="o">.</span><span class="no">Testing</span><span class="p">,</span> <span class="ss">repo:</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Repo</span>
<span class="n">alias</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Migrations</span><span class="o">.</span><span class="no">MyDataMigrationWorker</span>
<span class="n">describe</span> <span class="s2">"perform/1"</span> <span class="k">do</span>
<span class="n">test</span> <span class="s2">"performs data migration"</span> <span class="k">do</span>
<span class="c1"># prepare your test</span>
<span class="n">assert</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">result</span><span class="p">}</span> <span class="o">=</span> <span class="n">perform_job</span><span class="p">(</span><span class="no">MyDataMigrationWorker</span><span class="p">,</span> <span class="p">%{</span><span class="s2">"my_arg"</span> <span class="o">=></span> <span class="mi">1</span><span class="p">})</span>
<span class="c1"># assert the side-effects of the Oban Worker</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>In the previous test, we used the
<a href="https://hexdocs.pm/oban/Oban.Testing.html#perform_job/3"><code class="language-plaintext highlighter-rouge">perform_job/2</code></a>
helper from the <code class="language-plaintext highlighter-rouge">Oban.Testing</code> module, this helper constructs a job and execute
it with the given worker module. Apart from reducing the ceremony when
constructing jobs for unit test, one of the neat things about <code class="language-plaintext highlighter-rouge">perform_job/2</code>
is that it does some assertions for us.</p>
<p>From the docs, we have the following:</p>
<blockquote>
<ul>
<li>That the worker implements the <code class="language-plaintext highlighter-rouge">Oban.Worker</code> behaviour</li>
<li>That the options provided build a valid job</li>
<li>That the return is valid, e.g. <code class="language-plaintext highlighter-rouge">:ok</code>, <code class="language-plaintext highlighter-rouge">{:ok, value}</code>, <code class="language-plaintext highlighter-rouge">{:error, value}</code>, etc.</li>
</ul>
<p>If all of the assertions pass, the function returns the result of <code class="language-plaintext highlighter-rouge">perform/1</code>
for you to make additional assertions.</p>
</blockquote>
<p>Apart from <code class="language-plaintext highlighter-rouge">perform_job/2,3</code> there are other testing helpers such as:
<code class="language-plaintext highlighter-rouge">assert_enqueued/2,3</code>, <code class="language-plaintext highlighter-rouge">all_enqueued/2</code>, <code class="language-plaintext highlighter-rouge">refute_enqueued/2,3</code>. For more
information please check the <a href="https://hexdocs.pm/oban/Oban.Testing.html">Oban.Testing</a> documentation.</p>
<p>It is important to note that if you want to test a worker included in Oban Pro
like the <a href="https://hexdocs.pm/oban/batch.html">Batch</a>, <a href="https://hexdocs.pm/oban/chunk.html">Chunk</a>, or <a href="https://hexdocs.pm/oban/workflow.html">Workflow</a>, you should <code class="language-plaintext highlighter-rouge">import</code> the module
<code class="language-plaintext highlighter-rouge">Oban.Pro.Testing</code> and use the <code class="language-plaintext highlighter-rouge">process_job/2,3</code> helper instead:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="no">Oban</span><span class="o">.</span><span class="no">Pro</span><span class="o">.</span><span class="no">Testing</span>
<span class="n">alias</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">MyBatchWorker</span>
<span class="n">describe</span> <span class="s2">"process/1"</span> <span class="k">do</span>
<span class="n">test</span> <span class="s2">"archives the given thing"</span> <span class="k">do</span>
<span class="c1"># prepare your test</span>
<span class="n">assert</span> <span class="ss">:ok</span> <span class="o">==</span> <span class="n">process_job</span><span class="p">(</span><span class="no">MyBatchWorkerBatch</span><span class="p">,</span> <span class="p">%{</span><span class="ss">thing_id:</span> <span class="n">thing</span><span class="o">.</span><span class="n">id</span><span class="p">})</span>
<span class="c1"># assert the side-effects of the Oban Batch Worker</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>One important thing about testing, don’t forget to test your Oban
configuration. Let’s talk about it next.</p>
<h2 id="testing-your-oban-configuration">Testing your Oban Configuration</h2>
<p>Do not wait until you deploy your application into your <em>staging</em> environment
to catch errors in your Oban configuration, try to catch those errors as soon
as possible; you can start by doing:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">ObanConfigTest</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">Case</span><span class="p">,</span> <span class="ss">async:</span> <span class="no">true</span>
<span class="n">test</span> <span class="s2">"check oban configuration"</span> <span class="k">do</span>
<span class="p">[</span><span class="n">config</span><span class="p">,</span> <span class="n">prod</span><span class="p">]</span> <span class="o">=</span>
<span class="no">Enum</span><span class="o">.</span><span class="n">map</span><span class="p">([</span><span class="s2">"config/config.exs"</span><span class="p">,</span> <span class="s2">"config/prod.exs"</span><span class="p">],</span> <span class="k">fn</span> <span class="n">path</span> <span class="o">-></span>
<span class="n">path</span>
<span class="o">|></span> <span class="no">Config</span><span class="o">.</span><span class="no">Reader</span><span class="o">.</span><span class="n">read!</span><span class="p">(</span><span class="ss">imports:</span> <span class="p">[],</span> <span class="ss">env:</span> <span class="ss">:prod</span><span class="p">,</span> <span class="ss">target:</span> <span class="ss">:host</span><span class="p">)</span>
<span class="o">|></span> <span class="n">get_in</span><span class="p">([</span><span class="ss">:my_app</span><span class="p">,</span> <span class="no">Oban</span><span class="p">])</span>
<span class="k">end</span><span class="p">)</span>
<span class="n">assert</span> <span class="p">%</span><span class="no">Oban</span><span class="o">.</span><span class="no">Config</span><span class="p">{</span><span class="ss">plugins:</span> <span class="n">_plugins</span><span class="p">}</span> <span class="o">=</span>
<span class="n">config</span>
<span class="o">|></span> <span class="no">Keyword</span><span class="o">.</span><span class="n">merge</span><span class="p">(</span><span class="n">prod</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Oban</span><span class="o">.</span><span class="no">Config</span><span class="o">.</span><span class="n">new</span><span class="p">()</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p><em>NOTE: In the previous test, I merged the Oban config that comes from
<code class="language-plaintext highlighter-rouge">config/config.exs</code> and <code class="language-plaintext highlighter-rouge">config/prod.exs</code>; you should adapt this test based on
your scenario.</em></p>
<p>But, for the previous unit test to make sense, you need to know a bit about the
internal implementation of Oban. Well, the gist is this, when you try to start
the <em>supervision tree</em> for Oban, it first tries to create an <code class="language-plaintext highlighter-rouge">Oban.Config</code>
struct via
<a href="https://github.com/sorentwo/oban/blob/fec80cdbafe59b0fba13c0f67ed1739c1f86e703/lib/oban/config.ex#L44-L63"><code class="language-plaintext highlighter-rouge">Oban.Config.new/1</code></a>,
and if you supply the wrong configuration, it will blow up.</p>
<p>For example, let’s assume you made a typo in the <code class="language-plaintext highlighter-rouge">repo</code> option:</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>mix <span class="nb">test test</span>/my_app/oban_config_test.exs
<span class="go">
1) test check oban configuration (MyApp.ObanConfigTest)
test/my_app/oban_config_test.exs:4
** (ArgumentError) expected :repo to be an Ecto.Repo, got: MyApp.Rep0
</span><span class="gp"> code: |></span><span class="w"> </span>Oban.Config.new<span class="o">()</span>
<span class="go"> stacktrace:
</span><span class="c"> ...
</span></code></pre></div></div>
<p>Once you fix the typo, you get:</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>mix <span class="nb">test test</span>/my_app/oban_config_test.exs
<span class="c">.
</span><span class="go">Finished in 0.02 seconds (0.02s async, 0.00s sync)
1 test, 0 failures
</span></code></pre></div></div>
<p>It is important to note that the <code class="language-plaintext highlighter-rouge">Oban.Config.new/1</code> function doesn’t validate
the <em>internal configuration</em> for the plugins; it only validates that the given
plugin is an <code class="language-plaintext highlighter-rouge">atom</code>, is loaded, the plugin in question exports an <code class="language-plaintext highlighter-rouge">init/1</code>
function, and also the given <code class="language-plaintext highlighter-rouge">opts</code> is a keyword list. We will explore a
workaround for this limitation in the following sub-section.</p>
<h3 id="testing-your-plugins-configuration">Testing your plugins configuration</h3>
<p>Previously I mentioned that you should test your Oban configuration. Still,
there were limitations in the previous approach; mainly, <code class="language-plaintext highlighter-rouge">Oban.config.new/1</code>
doesn’t validate the <em>internal configuration</em> that you pass to each plugin
listed under the <code class="language-plaintext highlighter-rouge">plugins</code> option.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">test</span> <span class="s2">"check oban configuration"</span> <span class="k">do</span>
<span class="c1"># ...</span>
<span class="n">assert</span> <span class="p">%</span><span class="no">Oban</span><span class="o">.</span><span class="no">Config</span><span class="p">{</span><span class="ss">plugins:</span> <span class="n">_plugins</span><span class="p">}</span> <span class="o">=</span>
<span class="n">config</span>
<span class="o">|></span> <span class="no">Keyword</span><span class="o">.</span><span class="n">merge</span><span class="p">(</span><span class="n">prod</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Oban</span><span class="o">.</span><span class="no">Config</span><span class="o">.</span><span class="n">new</span><span class="p">()</span>
<span class="k">end</span>
</code></pre></div></div>
<p>So, if you want to go further and validate the given options to the
<code class="language-plaintext highlighter-rouge">Oban.Plugins.Cron</code> plugin, for example, you need to know a bit about the
internal implementation and use <code class="language-plaintext highlighter-rouge">Oban.Plugins.Cron.validate!/1</code> and assert that
the returned value is <code class="language-plaintext highlighter-rouge">:ok</code>.</p>
<p>I like to validate the <code class="language-plaintext highlighter-rouge">cron</code> plugin configuration because it constantly
evolves, and it’s easy to catch typos in the cron worker module names also
validating the cron expressions by doing the following:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">assert</span> <span class="p">%</span><span class="no">Oban</span><span class="o">.</span><span class="no">Config</span><span class="p">{</span><span class="ss">plugins:</span> <span class="n">plugins</span><span class="p">}</span> <span class="o">=</span>
<span class="n">config</span>
<span class="o">|></span> <span class="no">Keyword</span><span class="o">.</span><span class="n">merge</span><span class="p">(</span><span class="n">prod</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Oban</span><span class="o">.</span><span class="no">Config</span><span class="o">.</span><span class="n">new</span><span class="p">()</span>
<span class="p">{</span><span class="n">_</span><span class="p">,</span> <span class="n">cron_opts</span><span class="p">}</span> <span class="o">=</span> <span class="no">Enum</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="n">plugins</span><span class="p">,</span> <span class="o">&</span><span class="n">match?</span><span class="p">({</span><span class="no">Oban</span><span class="o">.</span><span class="no">Plugins</span><span class="o">.</span><span class="no">Cron</span><span class="p">,</span> <span class="n">_</span><span class="p">},</span> <span class="nv">&1</span><span class="p">))</span>
<span class="n">assert</span> <span class="ss">:ok</span> <span class="o">=</span> <span class="no">Oban</span><span class="o">.</span><span class="no">Plugins</span><span class="o">.</span><span class="no">Cron</span><span class="o">.</span><span class="n">validate!</span><span class="p">(</span><span class="n">cron_opts</span><span class="p">)</span>
</code></pre></div></div>
<p>A word of caution about the previous unit test segment, regardless of that
<code class="language-plaintext highlighter-rouge">Oban.Plugins.Cron.validate!/1</code> is a public function; it has a <code class="language-plaintext highlighter-rouge">@doc false</code>,
which usually means that function is for internal use. There could be
unannounced breaking changes in the future, but you gain the following checks:</p>
<ul>
<li>that the value for the <code class="language-plaintext highlighter-rouge">crontab</code> option is a list.</li>
<li>if you use the <code class="language-plaintext highlighter-rouge">timezone</code> option, it checks that the given value it’s a known timezone</li>
</ul>
<p>For example, let’s assume that you’re using an invalid timezone:</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>mix <span class="nb">test test</span>/my_app/oban_config_test.exs
<span class="go">
1) test check oban configuration (MyApp.ObanConfigTest)
test/my_app/oban_config_test.exs:4
** (ArgumentError) expected :timezone to be a known timezone
code: assert :ok = Oban.Plugins.Cron.validate!(cron_opts)
stacktrace:
</span><span class="c"> ...
</span></code></pre></div></div>
<p>In the specific case of the cron plugin, the <code class="language-plaintext highlighter-rouge">validate!/1</code> function also parses
the crontab expressions; if a term is invalid, the validation will fail.
But on the other hand, it also validates that the given module is loaded and
implements the <code class="language-plaintext highlighter-rouge">perform/1</code> callback, so this will catch possible typos.</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>mix <span class="nb">test test</span>/my_app/oban_config_test.exs
<span class="go">
1) test check oban configuration (MyApp.ObanConfigTest)
test/my_app/oban_config_test.exs:4
** (ArgumentError) MyApp.DailWorker not found or can't be loaded
code: assert :ok = Oban.Plugins.Cron.validate!(cron_opts)
</span></code></pre></div></div>
<p>In the previous test, you can see that we tried to load a <code class="language-plaintext highlighter-rouge">MyApp.DialWorker</code>,
but in this case, the correct name for that module is <code class="language-plaintext highlighter-rouge">MyApp.DailyWorker</code>.</p>
<p>I think it is worth taking the risk of using a <em>non-documented public function</em>
in this case; you gain more in your daily routine because it will allow you to
catch errors in your Oban Configuration as soon as possible.</p>
<p>After talking with Parker about the previous challenges, he suggested opening
the following issue in the Oban repository: <a href="https://github.com/sorentwo/oban/issues/632">Improve the testing experience for
Oban and its plugins</a></p>
<h2 id="testing-workers-included-in-oban-pro">Testing workers included in Oban Pro</h2>
<p>In a previous section, I mentioned that when you’re testing a worker included
in Oban Pro, like the <a href="https://hexdocs.pm/oban/batch.html">Batch</a>, <a href="https://hexdocs.pm/oban/chunk.html">Chunk</a>, or <a href="https://hexdocs.pm/oban/workflow.html">Workflow</a>, you should
<code class="language-plaintext highlighter-rouge">import</code> the module <code class="language-plaintext highlighter-rouge">Oban.Pro.Testing</code> and use the <code class="language-plaintext highlighter-rouge">process_job/2,3</code> helper
instead:</p>
<p>Possibly, the most challenging worker to test is the BatchWorker, especially if
you want to try the whole cycle, including their <em>handle callbacks</em>, but let’s
start with some dummy Batch Worker. Then we can start talking about the details
of its tests.</p>
<p>So, let’s imagine the following worker:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">MyBatchWorker</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Oban</span><span class="o">.</span><span class="no">Pro</span><span class="o">.</span><span class="no">Workers</span><span class="o">.</span><span class="no">Batch</span><span class="p">,</span>
<span class="ss">queue:</span> <span class="ss">:default</span>
<span class="kn">require</span> <span class="no">Logger</span>
<span class="nv">@impl</span> <span class="no">Batch</span>
<span class="k">def</span> <span class="n">process</span><span class="p">(</span><span class="n">_</span><span class="p">)</span> <span class="k">do</span>
<span class="ss">:ok</span>
<span class="k">end</span>
<span class="nv">@impl</span> <span class="no">Batch</span>
<span class="k">def</span> <span class="n">handle_exhausted</span><span class="p">(%</span><span class="no">Job</span><span class="p">{}</span> <span class="o">=</span> <span class="n">_job</span><span class="p">)</span> <span class="k">do</span>
<span class="no">Logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"exhausted callback"</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>The body of the module <code class="language-plaintext highlighter-rouge">MyApp.ByBatchWorker</code> is very similar to the usual
<code class="language-plaintext highlighter-rouge">Oban.Worker</code>. The exception is that you need to handle your work under
<code class="language-plaintext highlighter-rouge">process/1</code> instead of the <code class="language-plaintext highlighter-rouge">perform/1</code>; the latter is used internally. One of
the excellent additions of the <code class="language-plaintext highlighter-rouge">Oban.Pro.Workers.Batch</code> behaviour allows you to
define a few callbacks that are perfectly explained in the <a href="https://hexdocs.pm/oban/batch.html">behaviour
documentation</a>. In this example, I’m using the <code class="language-plaintext highlighter-rouge">handle_exhausted</code>
callback, called after <em>all jobs</em> in the batch have either a <code class="language-plaintext highlighter-rouge">completed</code> or
<code class="language-plaintext highlighter-rouge">discarded</code> state. You also see that I’m just logging that the callback was
called in this case to keep the example as simple as possible, but you can do
whatever you want here.</p>
<p>Okay, now the exciting part, let’s see how to test this <em>worker</em>.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">MyBatchWorkerTest</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">Case</span>
<span class="kn">import</span> <span class="no">Oban</span><span class="o">.</span><span class="no">Pro</span><span class="o">.</span><span class="no">Testing</span>
<span class="n">alias</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">MyBatchWorker</span>
<span class="n">describe</span> <span class="s2">"process/1"</span> <span class="k">do</span>
<span class="n">test</span> <span class="s2">"processes the background job"</span> <span class="k">do</span>
<span class="n">assert</span> <span class="ss">:ok</span> <span class="o">=</span> <span class="n">process_job</span><span class="p">(</span><span class="no">MyBatchWorker</span><span class="p">,</span> <span class="p">%{})</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Here we’re testing the “bulk” of the worker. Now, let’s check one of the
handler callbacks as a unit:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="kn">use</span> <span class="no">Oban</span><span class="o">.</span><span class="no">Testing</span><span class="p">,</span> <span class="ss">repo:</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Repo</span>
<span class="n">describe</span> <span class="s2">"handle_exhausted/1"</span> <span class="k">do</span>
<span class="n">test</span> <span class="s2">"handle exhausted behaves correctly"</span> <span class="k">do</span>
<span class="n">assert</span> <span class="ss">:ok</span> <span class="o">=</span> <span class="n">perform_job</span><span class="p">(</span><span class="no">MyBatchWorker</span><span class="p">,</span> <span class="p">%{},</span> <span class="ss">meta:</span> <span class="p">%{</span><span class="s2">"callback"</span> <span class="o">=></span> <span class="s2">"exhausted"</span><span class="p">})</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>As far as I know, the test helper <code class="language-plaintext highlighter-rouge">process_job/2,3</code> will not work in the
previous test because it ends calling our <code class="language-plaintext highlighter-rouge">MyApp.MyBatchWorker.process/1</code>, and
we actually want to test our <code class="language-plaintext highlighter-rouge">MyApp.MyBatchWorker.perform/1</code>, which coordinates
the execution of the callbacks. Again, here you need to know how Oban works
internally. That’s why I ended up using: <code class="language-plaintext highlighter-rouge">perform_job/3</code>, passing the <code class="language-plaintext highlighter-rouge">meta</code>
option. But please, be aware that <code class="language-plaintext highlighter-rouge">perform_job/3</code> will not complain if your
handle callback last expression returns things like <code class="language-plaintext highlighter-rouge">{:ok, value}</code>, <code class="language-plaintext highlighter-rouge">{:error,
value}</code>, etc. Please remember that the handle callbacks contract specifies that
you must return <code class="language-plaintext highlighter-rouge">:ok</code>. To fix this issue, a possible solution that is being
considered is to include a <code class="language-plaintext highlighter-rouge">perform_callback/3</code> helper under the
<code class="language-plaintext highlighter-rouge">Oban.Pro.Testing</code> module.</p>
<p>Now, let’s assume that we want to go further in our test and check the whole
cycle. In that case, we do:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">MyBatchWorkerTest</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">Case</span>
<span class="kn">import</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">CaptureLog</span>
<span class="n">alias</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">MyBatchWorker</span>
<span class="kn">require</span> <span class="no">Logger</span>
<span class="c1"># ...</span>
<span class="n">setup</span> <span class="k">do</span>
<span class="n">oban_name</span> <span class="o">=</span> <span class="n">start_supervised_oban!</span><span class="p">(</span><span class="ss">queues:</span> <span class="p">[</span><span class="ss">default:</span> <span class="mi">3</span><span class="p">])</span>
<span class="no">Logger</span><span class="o">.</span><span class="n">put_module_level</span><span class="p">(</span><span class="no">MyBatchWorker</span><span class="p">,</span> <span class="ss">:all</span><span class="p">)</span>
<span class="n">on_exit</span><span class="p">(</span><span class="k">fn</span> <span class="o">-></span>
<span class="no">Logger</span><span class="o">.</span><span class="n">delete_module_level</span><span class="p">(</span><span class="no">MyBatchWorker</span><span class="p">)</span>
<span class="k">end</span><span class="p">)</span>
<span class="p">%{</span><span class="ss">oban_name:</span> <span class="n">oban_name</span><span class="p">}</span>
<span class="k">end</span>
<span class="n">test</span> <span class="s2">"testing our batch worker"</span><span class="p">,</span> <span class="p">%{</span><span class="ss">oban_name:</span> <span class="n">oban_name</span><span class="p">}</span> <span class="k">do</span>
<span class="n">batch</span> <span class="o">=</span>
<span class="p">[</span><span class="s2">"foo@example.com"</span><span class="p">,</span> <span class="s2">"bar@example.com"</span><span class="p">]</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="o">&</span><span class="p">%{</span><span class="ss">email:</span> <span class="nv">&1</span><span class="p">})</span>
<span class="o">|></span> <span class="no">MyBatchWorker</span><span class="o">.</span><span class="n">new_batch</span><span class="p">()</span>
<span class="n">log</span> <span class="o">=</span>
<span class="n">capture_log</span><span class="p">(</span><span class="k">fn</span> <span class="o">-></span>
<span class="no">Oban</span><span class="o">.</span><span class="n">insert_all</span><span class="p">(</span><span class="n">oban_name</span><span class="p">,</span> <span class="n">batch</span><span class="p">)</span>
<span class="no">Oban</span><span class="o">.</span><span class="n">drain_queue</span><span class="p">(</span><span class="ss">queue:</span> <span class="ss">:default</span><span class="p">)</span>
<span class="k">end</span><span class="p">)</span>
<span class="n">assert</span> <span class="n">log</span> <span class="o">=~</span> <span class="s2">"exhausted callback"</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">start_supervised_oban!</span><span class="p">(</span><span class="n">opts</span><span class="p">)</span> <span class="k">do</span>
<span class="n">default_opts</span> <span class="o">=</span> <span class="p">[</span>
<span class="ss">name:</span> <span class="n">make_ref</span><span class="p">(),</span>
<span class="ss">repo:</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Repo</span><span class="p">,</span>
<span class="ss">plugins:</span> <span class="p">[</span>
<span class="p">{</span><span class="no">Oban</span><span class="o">.</span><span class="no">Pro</span><span class="o">.</span><span class="no">Plugins</span><span class="o">.</span><span class="no">BatchManager</span><span class="p">,</span> <span class="ss">debounce_interval:</span> <span class="mi">5</span><span class="p">},</span>
<span class="p">{</span><span class="no">Oban</span><span class="o">.</span><span class="no">Plugins</span><span class="o">.</span><span class="no">Repeater</span><span class="p">,</span> <span class="ss">interval:</span> <span class="mi">25</span><span class="p">}</span>
<span class="p">],</span>
<span class="ss">poll_interval:</span> <span class="ss">:timer</span><span class="o">.</span><span class="n">minutes</span><span class="p">(</span><span class="mi">10</span><span class="p">),</span>
<span class="ss">shutdown_grace_period:</span> <span class="mi">25</span>
<span class="p">]</span>
<span class="n">opts</span> <span class="o">=</span> <span class="no">Keyword</span><span class="o">.</span><span class="n">merge</span><span class="p">(</span><span class="n">default_opts</span><span class="p">,</span> <span class="n">opts</span><span class="p">)</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">opts</span><span class="p">[</span><span class="ss">:name</span><span class="p">]</span>
<span class="n">start_supervised!</span><span class="p">({</span><span class="no">Oban</span><span class="p">,</span> <span class="n">opts</span><span class="p">})</span>
<span class="n">name</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Wow, I know this is a lot to digest at once, so let’s split the previous
snipped.</p>
<p>Suppose you saw the talk <a href="https://www.youtube.com/watch?v=PZ48omi0NKU">Testing Oban Jobs From the Inside Out</a>
from Parker. In that case, you remember that he calls this kind of testing
“<em>inline testing</em>”, and this should be used when you absolutely must normally
run jobs during your test (e.g., LiveView, Browser Testing), or in my case, I
wanted to test the whole path in my <em>Batch Workers</em>. So, let’s start analyzing
the <code class="language-plaintext highlighter-rouge">start_supervised_oban!/1</code> helper:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">defp</span> <span class="n">start_supervised_oban!</span><span class="p">(</span><span class="n">opts</span><span class="p">)</span> <span class="k">do</span>
<span class="n">default_opts</span> <span class="o">=</span> <span class="p">[</span>
<span class="ss">name:</span> <span class="n">make_ref</span><span class="p">(),</span>
<span class="ss">repo:</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Repo</span><span class="p">,</span>
<span class="ss">plugins:</span> <span class="p">[</span>
<span class="p">{</span><span class="no">Oban</span><span class="o">.</span><span class="no">Pro</span><span class="o">.</span><span class="no">Plugins</span><span class="o">.</span><span class="no">BatchManager</span><span class="p">,</span> <span class="ss">debounce_interval:</span> <span class="mi">5</span><span class="p">},</span>
<span class="p">{</span><span class="no">Oban</span><span class="o">.</span><span class="no">Plugins</span><span class="o">.</span><span class="no">Repeater</span><span class="p">,</span> <span class="ss">interval:</span> <span class="mi">25</span><span class="p">}</span>
<span class="p">],</span>
<span class="ss">poll_interval:</span> <span class="ss">:timer</span><span class="o">.</span><span class="n">minutes</span><span class="p">(</span><span class="mi">10</span><span class="p">),</span>
<span class="ss">shutdown_grace_period:</span> <span class="mi">25</span>
<span class="p">]</span>
<span class="n">opts</span> <span class="o">=</span> <span class="no">Keyword</span><span class="o">.</span><span class="n">merge</span><span class="p">(</span><span class="n">default_opts</span><span class="p">,</span> <span class="n">opts</span><span class="p">)</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">opts</span><span class="p">[</span><span class="ss">:name</span><span class="p">]</span>
<span class="n">start_supervised!</span><span class="p">({</span><span class="no">Oban</span><span class="p">,</span> <span class="n">opts</span><span class="p">})</span>
<span class="n">name</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Here, with the help of <a href="https://hexdocs.pm/ex_unit/ExUnit.Callbacks.html#start_supervised!/2o"><code class="language-plaintext highlighter-rouge">start_supervised!/2</code></a>, which comes
with <a href="https://hexdocs.pm/ex_unit">ExUnit</a>, we’re setting up an Oban supervisor with some default options;
the essential thing in the previous snippet is the plugins. So first,
<code class="language-plaintext highlighter-rouge">Oban.Pro.Plugins.BatchManager</code> is needed because that’s the plugin that will
track the execution of the Oban jobs within a batch and enqueue
the callback jobs. So, yes, the callbacks are also Oban Jobs.</p>
<p>Then, we have the <a href="https://hexdocs.pm/oban/Oban.Plugins.Repeater.html"><code class="language-plaintext highlighter-rouge">Oban.Plugins.Repeater</code></a> plugin, which will poll
every 25 milliseconds to look for new jobs. This plugin is essential in our
case because inside the unit tests, PostgreSQL notifications don’t work. We’re
using the Ecto Sandbox, which means that every unit test runs inside a
transaction. Also, note that I’m using <code class="language-plaintext highlighter-rouge">make_ref</code> to create a unique reference
that we will use as the name for the Oban supervisor; you cannot have more than
one supervisor with the same <code class="language-plaintext highlighter-rouge">id</code>.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">setup</span> <span class="k">do</span>
<span class="n">oban_name</span> <span class="o">=</span> <span class="n">start_supervised_oban!</span><span class="p">(</span><span class="ss">queues:</span> <span class="p">[</span><span class="ss">default:</span> <span class="mi">3</span><span class="p">])</span>
<span class="no">Logger</span><span class="o">.</span><span class="n">put_module_level</span><span class="p">(</span><span class="no">MyBatchWorker</span><span class="p">,</span> <span class="ss">:all</span><span class="p">)</span>
<span class="n">on_exit</span><span class="p">(</span><span class="k">fn</span> <span class="o">-></span>
<span class="no">Logger</span><span class="o">.</span><span class="n">delete_module_level</span><span class="p">(</span><span class="no">MyBatchWorker</span><span class="p">)</span>
<span class="k">end</span><span class="p">)</span>
<span class="p">%{</span><span class="ss">oban_name:</span> <span class="n">oban_name</span><span class="p">}</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Then, in our <code class="language-plaintext highlighter-rouge">setup</code> function, we use the previous helper
<code class="language-plaintext highlighter-rouge">start_supervised_oban!/1</code> to start a new Oban supervisor that will only handle
jobs for the <code class="language-plaintext highlighter-rouge">default</code> queue. Then we say that we want to include all the
logging levels available in the <code class="language-plaintext highlighter-rouge">MyApp.MyBatchWorker</code> module, the <code class="language-plaintext highlighter-rouge">on_exit/1</code>
callback will clean up this setting at the end of our test. Finally, we put the
Oban supervisor’s name in the test context.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">test</span> <span class="s2">"testing our batch worker"</span><span class="p">,</span> <span class="p">%{</span><span class="ss">oban_name:</span> <span class="n">oban_name</span><span class="p">}</span> <span class="k">do</span>
<span class="n">batch</span> <span class="o">=</span>
<span class="p">[</span><span class="s2">"foo@example.com"</span><span class="p">,</span> <span class="s2">"bar@example.com"</span><span class="p">]</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="o">&</span><span class="p">%{</span><span class="ss">email:</span> <span class="nv">&1</span><span class="p">})</span>
<span class="o">|></span> <span class="no">MyBatchWorker</span><span class="o">.</span><span class="n">new_batch</span><span class="p">()</span>
<span class="n">assert</span> <span class="n">capture_log</span><span class="p">(</span><span class="k">fn</span> <span class="o">-></span>
<span class="no">Oban</span><span class="o">.</span><span class="n">insert_all</span><span class="p">(</span><span class="n">oban_name</span><span class="p">,</span> <span class="n">batch</span><span class="p">)</span>
<span class="no">Oban</span><span class="o">.</span><span class="n">drain_queue</span><span class="p">(</span><span class="n">oban_name</span><span class="p">,</span> <span class="ss">queue:</span> <span class="ss">:default</span><span class="p">)</span>
<span class="k">end</span><span class="p">)</span> <span class="o">=~</span> <span class="s2">"exhausted callback"</span>
<span class="k">end</span>
</code></pre></div></div>
<p>And here is where we test our batch worker. First, we create a new batch with
<code class="language-plaintext highlighter-rouge">MyApp.MyBatchWorker.new_batch/1</code> and immediately insert with
<code class="language-plaintext highlighter-rouge">Oban.insert_all/2</code>. It is essential to pass the <code class="language-plaintext highlighter-rouge">oban_name</code> as the first
argument to that function. We drain the <code class="language-plaintext highlighter-rouge">default</code> queue right after using
<code class="language-plaintext highlighter-rouge">Oban.drain_queue/2</code>. The way I’m testing here that the <code class="language-plaintext highlighter-rouge">handle_exhausted/1</code> is
called is by capturing the log or side-effect produced for that callback, but
as I said before, you can do here whatever you want; you need to check the
side-effects produced by your callback.</p>
<p>But wait, something is missing in our previous <code class="language-plaintext highlighter-rouge">start_supervised_oban!/1</code>
helper. Do you remember that I mentioned before that the <em>plugins</em> implement
the <a href="https://hexdocs.pm/elixir/GenServer.html">GenServer</a> behaviour? Also, in
the <code class="language-plaintext highlighter-rouge">test</code> environment, we’re using <a href="https://hexdocs.pm/ecto_sql/Ecto.Adapters.SQL.Sandbox.html#module-collaborating-processes"><code class="language-plaintext highlighter-rouge">Ecto.Adapters.SQL.Sandbox</code></a>
to allow concurrent transactional tests. So we need to enable collaboration
from Oban processes to run these tests successfully for these conditions. All
these processes should use the same connection, so they all belong to the same
transaction.</p>
<p>From the <code class="language-plaintext highlighter-rouge">Ecto.Adapters.SQL.Sandbox</code> documentation, we have the following:</p>
<blockquote>
<p>The test above will fail with an error similar to:</p>
<p><code class="language-plaintext highlighter-rouge">** (DBConnection.OwnershipError) cannot find ownership process for #PID<0.35.0></code></p>
<p>That’s because the <code class="language-plaintext highlighter-rouge">setup</code> block is checking out the connection only for the
test process. Once the worker attempts to perform a query, there is no
connection assigned to it and it will fail.</p>
<p>The sandbox module provides two ways of doing so, via allowances or by
running in shared mode.</p>
</blockquote>
<p>And also, we have:</p>
<blockquote>
<p>The idea behind allowances is that you can explicitly tell a process which
checked out connection it should use, allowing multiple processes to
collaborate over the same connection.</p>
</blockquote>
<p>So, we need to include this allowance in our <code class="language-plaintext highlighter-rouge">start_supervised_oban!/1</code> helper.
Thankfully, Parker recently shared in the <code class="language-plaintext highlighter-rouge">#oban</code> channel the following gist to
<a href="https://gist.github.com/sorentwo/1286196276d1adecbebfa082320a991b">demonstrate how to integration test batch
callbacks</a>,
there you can see this clever piece to allow the Oban Producers, Plugins, and
other modules to use the checked out connection:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="c1"># For Oban < v2.11 you can remove the `{:_, Oban.Peer}` part</span>
<span class="n">for</span> <span class="n">key</span> <span class="o"><-</span> <span class="p">[{</span><span class="ss">:_</span><span class="p">,</span> <span class="no">Oban</span><span class="o">.</span><span class="no">Peer</span><span class="p">},</span> <span class="p">{</span><span class="ss">:_</span><span class="p">,</span> <span class="p">{</span><span class="ss">:producer</span><span class="p">,</span> <span class="ss">:_</span><span class="p">}},</span> <span class="p">{</span><span class="ss">:_</span><span class="p">,</span> <span class="p">{</span><span class="ss">:plugin</span><span class="p">,</span> <span class="ss">:_</span><span class="p">}}],</span>
<span class="n">pid</span> <span class="o"><-</span> <span class="no">Registry</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="no">Oban</span><span class="o">.</span><span class="no">Registry</span><span class="p">,</span> <span class="p">[{{</span><span class="n">key</span><span class="p">,</span> <span class="ss">:"$2"</span><span class="p">,</span> <span class="ss">:_</span><span class="p">},</span> <span class="p">[],</span> <span class="p">[</span><span class="ss">:"$2"</span><span class="p">]}])</span> <span class="k">do</span>
<span class="no">Ecto</span><span class="o">.</span><span class="no">Adapters</span><span class="o">.</span><span class="no">SQL</span><span class="o">.</span><span class="no">Sandbox</span><span class="o">.</span><span class="n">allow</span><span class="p">(</span><span class="no">MyApp</span><span class="o">.</span><span class="no">Repo</span><span class="p">,</span> <span class="n">self</span><span class="p">(),</span> <span class="n">pid</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div></div>
<p>But again, this piece of code assumes you know some internals, so I will try to
explain a bit the previous code.</p>
<p>In this <a href="https://github.com/sorentwo/oban/pull/331">excellent PR</a>, Saša Jurić
introduced <a href="https://hexdocs.pm/elixir/Registry.html">Registry</a> into Oban to
replace all the locally registered names, and also to hold the configuration
for those processes as part of the Registry metadata.</p>
<p>So, if you get everything that’s in the Oban Registry you see something like:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="n">iex</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="o">></span> <span class="no">Registry</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="no">Oban</span><span class="o">.</span><span class="no">Registry</span><span class="p">,</span> <span class="p">[{{</span><span class="ss">:"$1"</span><span class="p">,</span> <span class="ss">:"$2"</span><span class="p">,</span> <span class="ss">:"$3"</span><span class="p">},</span> <span class="p">[],</span> <span class="p">[{{</span><span class="ss">:"$1"</span><span class="p">,</span> <span class="ss">:"$2"</span><span class="p">,</span> <span class="ss">:"$3"</span><span class="p">}}]}])</span>
<span class="p">[</span>
<span class="p">{{</span><span class="no">Oban</span><span class="p">,</span> <span class="p">{</span><span class="ss">:watchman</span><span class="p">,</span> <span class="s2">"default"</span><span class="p">}},</span> <span class="c1">#PID<0.576.0>, nil},</span>
<span class="p">{{</span><span class="no">Oban</span><span class="p">,</span> <span class="no">Oban</span><span class="o">.</span><span class="no">Notifier</span><span class="p">},</span> <span class="c1">#PID<0.568.0>, nil},</span>
<span class="p">{{</span><span class="no">Oban</span><span class="p">,</span> <span class="p">{</span><span class="ss">:plugin</span><span class="p">,</span> <span class="no">Oban</span><span class="o">.</span><span class="no">Plugins</span><span class="o">.</span><span class="no">Stager</span><span class="p">}},</span> <span class="c1">#PID<0.571.0>, nil},</span>
<span class="p">{{</span><span class="no">Oban</span><span class="p">,</span> <span class="p">{</span><span class="ss">:foreman</span><span class="p">,</span> <span class="s2">"default"</span><span class="p">}},</span> <span class="c1">#PID<0.574.0>, nil},</span>
<span class="p">{{</span><span class="no">Oban</span><span class="p">,</span> <span class="p">{</span><span class="ss">:plugin</span><span class="p">,</span> <span class="no">Oban</span><span class="o">.</span><span class="no">Plugins</span><span class="o">.</span><span class="no">Pruner</span><span class="p">}},</span> <span class="c1">#PID<0.570.0>, nil},</span>
<span class="p">{{</span><span class="no">Oban</span><span class="p">,</span> <span class="p">{</span><span class="ss">:supervisor</span><span class="p">,</span> <span class="s2">"default"</span><span class="p">}},</span> <span class="c1">#PID<0.573.0>, nil},</span>
<span class="p">{{</span><span class="no">Oban</span><span class="p">,</span> <span class="p">{</span><span class="ss">:producer</span><span class="p">,</span> <span class="s2">"default"</span><span class="p">}},</span> <span class="c1">#PID<0.575.0>, nil},</span>
<span class="p">{</span><span class="no">Oban</span><span class="p">,</span> <span class="c1">#PID<0.567.0>, %Oban.Config{...}},</span>
<span class="p">{{</span><span class="no">Oban</span><span class="p">,</span> <span class="no">Oban</span><span class="o">.</span><span class="no">Midwife</span><span class="p">},</span> <span class="c1">#PID<0.569.0>, nil}</span>
<span class="p">]</span>
</code></pre></div></div>
<p>Then, based on the previous <code class="language-plaintext highlighter-rouge">Registry</code> output, the <code class="language-plaintext highlighter-rouge">for</code> comprehension filters
the results based on the <code class="language-plaintext highlighter-rouge">key</code> pattern, and we return the <code class="language-plaintext highlighter-rouge">pid</code> (a.k.a.
<code class="language-plaintext highlighter-rouge">:"$2"</code>) for each of those coincidences. Once we get the <code class="language-plaintext highlighter-rouge">pid</code>, we explicitly
assign the test process’s connection to each of these Oban processes.</p>
<p>And with the previous pattern, you can execute e2e tests, integration testing,
and so forth.</p>
<p>Finally, if you want to know more about how to test Oban jobs, I highly
recommend watching the talk <a href="https://www.youtube.com/watch?v=PZ48omi0NKU">Testing Oban Jobs From the Inside Out</a>
from Parker Selbert.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Oban offers some good helpers to run your Unit and Integration tests; you
should pay special attention to the <a href="https://hexdocs.pm/oban/Oban.Testing.html">Oban.Testing</a> documentation, and also
keep in mind that you also have helpers to test workers included in Oban Pro
such as <a href="https://hexdocs.pm/oban/batch.html">Batch</a>, <a href="https://hexdocs.pm/oban/chunk.html">Chunk</a>, or <a href="https://hexdocs.pm/oban/workflow.html">Workflow</a>.</p>
<p>But, indeed, there is always room for improvement. Especially when you want to
test your Oban configuration as soon as possible, it would be awesome to have
helpers validate all the settings, including plugins, at once. Also, in my
opinion, the plugins should have a uniform way to test their configuration. As
I mentioned in this <a href="https://github.com/sorentwo/oban/issues/632">issue</a>, a
possible solution could be:</p>
<blockquote>
<p>A public test helper to validate the Oban configuration, including plugins. I
think it’s also worth normalizing the <em>plugins</em> under a contract or behavior;
Oban now enforces defining an <code class="language-plaintext highlighter-rouge">init/1</code> function for plugins, but I think the
behaviour should also include a <code class="language-plaintext highlighter-rouge">validate!/1</code> function. At least the
<code class="language-plaintext highlighter-rouge">validate!/1</code> should help with the <em>testing experience</em>.</p>
<p>There are also plugins in the Pro package that could include complex
configurations, like the <code class="language-plaintext highlighter-rouge">Oban.Pro.Plugins.DynamicPruner</code>. So, I think it’s
worth it to offer a unified way to validate these configs.</p>
</blockquote>
<p>I hope you find this article helpful, and you have a better idea of how to test
your Oban Workers and your Oban Configuration.</p>
<p>If you want to reach me, you can do it at <a href="https://twitter.com/milmazz"><code class="language-plaintext highlighter-rouge">@milmazz</code></a> on Twitter, or
you can find me in the <code class="language-plaintext highlighter-rouge">#oban</code> channel on the <a href="https://elixir-slackin.herokuapp.com/">Elixir Slack</a>.</p>
<p>That’s all, folks! Thanks for reading.</p>
<h2 id="acknowledgments">Acknowledgments</h2>
<p>Thank you, <a href="https://sorentwo.com">Parker Selbert</a>, and <a href="https://andrewek.github.io">Andrew
Ek</a> for reviewing drafts of this post.</p>Milton Mazzarrime@milmazz.unoIn this article, I will continue talking about Oban, but I’ll focus on how to test your workers and, also, your configuration.Oban: job processing library for Elixir2022-02-11T12:27:31-06:002022-02-11T12:27:31-06:00https://milmazz.uno/article/2022/02/11/oban-job-processing-package-for-elixir<p>After working for years on different organizations, one common theme
is scheduling background jobs. In this article, I’ll share my experience with <a href="https://getoban.pro">Oban</a>, an open-source job processing
package for <a href="https://elixir-lang.org">Elixir</a>. I’ll also cover some features, like real-time monitoring
with Oban Web and complex workflow management with Oban Pro.</p>
<!--more-->
<p>This article is the first of a series; later we will explore other important topics such as:</p>
<ul>
<li><a href="/article/2022/02/21/oban-testing-your-workers-and-configuration">Oban: Testing your Workers and Configuration</a></li>
<li>Oban in production</li>
</ul>
<p>Stay tuned!</p>
<p>Let’s begin with an overview of Oban.</p>
<h2 id="oban-overview">Oban Overview</h2>
<p>In this context, when we talk about Oban, I’m not referring to the <a href="https://en.wikipedia.org/wiki/Oban">town in Scotland</a>. Instead, I will be talking about an Elixir library that offers a
background job system built on top of <a href="https://www.postgresql.org">PostgreSQL</a> with the primary goals of
reliability, consistency, and observability. One of the cool features of Oban,
given that it’s built on top of PostgreSQL is that you can enqueue jobs and other
database changes, ensuring that everything is committed or rolled back
atomically.</p>
<p>I will also talk about <em>Oban Pro</em>, which, according to the <a href="https://getoban.pro">official site</a>:</p>
<blockquote>
<p>Oban Pro is a collection of plugins, workers and extensions that improve Oban’s reliability and make difficult workflows possible.</p>
</blockquote>
<p>You can see a comparison between the open-source and the paid version <a href="https://getoban.pro/#feature-comparison">here</a>.</p>
<p>Let’s start by examining the following configuration:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">config</span> <span class="ss">:my_app</span><span class="p">,</span> <span class="no">Oban</span><span class="p">,</span>
<span class="ss">repo:</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Repo</span><span class="p">,</span>
<span class="ss">queues:</span> <span class="p">[</span><span class="ss">default:</span> <span class="mi">5</span><span class="p">,</span> <span class="ss">uno:</span> <span class="mi">3</span><span class="p">],</span>
<span class="ss">engine:</span> <span class="no">Oban</span><span class="o">.</span><span class="no">Pro</span><span class="o">.</span><span class="no">Queue</span><span class="o">.</span><span class="no">SmartEngine</span><span class="p">,</span>
<span class="ss">plugins:</span> <span class="o">...</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">repo</code> option specifies the <a href="https://hexdocs.pm/ecto/Ecto.html">Ecto</a> repository used to insert and retrieve jobs. The <code class="language-plaintext highlighter-rouge">queues</code> option is always, except when you use <code class="language-plaintext highlighter-rouge">false</code> as a value, a keyword list where the keys are the queue names, and its value specifies the concurrency limit. For our specific configuration, we define two queues, <code class="language-plaintext highlighter-rouge">default</code> with a local concurrency limit of 5 and the <code class="language-plaintext highlighter-rouge">uno</code> queue, which I will use later to explain how to schedule <em>one-off</em> jobs, the local concurrency limit for this queue is 3.</p>
<p>We can extend Oban functionality via plugins and callback modules as I already mentioned. Oban Pro takes advantage of this feature providing a collection of plugins, workers, and extensions.</p>
<p>One extension offered by Oban Pro is the
<code class="language-plaintext highlighter-rouge">Oban.Pro.Queue.SmartEngine</code>. It is an alternate queue engine that enables true global concurrency and global rate limiting. The open-source package offers a limited basic engine, <code class="language-plaintext highlighter-rouge">Oban.Queue.BasicEngine</code>.</p>
<p>The following list of plugins is for an <em>advanced</em> configuration, and it uses plugins from both Oban and Oban Pro. Please adjust the following based on your scenarios:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="ss">plugins:</span> <span class="p">[</span>
<span class="no">Oban</span><span class="o">.</span><span class="no">Plugins</span><span class="o">.</span><span class="no">Gossip</span><span class="p">,</span>
<span class="no">Oban</span><span class="o">.</span><span class="no">Plugins</span><span class="o">.</span><span class="no">Stager</span><span class="p">,</span>
<span class="no">Oban</span><span class="o">.</span><span class="no">Pro</span><span class="o">.</span><span class="no">Plugins</span><span class="o">.</span><span class="no">BatchManager</span><span class="p">,</span>
<span class="no">Oban</span><span class="o">.</span><span class="no">Pro</span><span class="o">.</span><span class="no">Plugins</span><span class="o">.</span><span class="no">Lifeline</span><span class="p">,</span>
<span class="no">Oban</span><span class="o">.</span><span class="no">Pro</span><span class="o">.</span><span class="no">Plugins</span><span class="o">.</span><span class="no">Reprioritizer</span><span class="p">,</span>
<span class="p">{</span>
<span class="no">Oban</span><span class="o">.</span><span class="no">Pro</span><span class="o">.</span><span class="no">Plugins</span><span class="o">.</span><span class="no">DynamicPruner</span><span class="p">,</span>
<span class="ss">mode:</span> <span class="p">{</span><span class="ss">:max_age</span><span class="p">,</span> <span class="p">{</span><span class="mi">1</span><span class="p">,</span> <span class="ss">:day</span><span class="p">}},</span>
<span class="ss">limit:</span> <span class="mi">25_000</span><span class="p">,</span>
<span class="ss">queue_overrides:</span> <span class="p">[</span>
<span class="ss">uno:</span> <span class="p">{</span><span class="ss">:max_age</span><span class="p">,</span> <span class="ss">:infinity</span><span class="p">}</span>
<span class="p">],</span>
<span class="ss">state_overrides:</span> <span class="p">[</span>
<span class="ss">cancelled:</span> <span class="p">{</span><span class="ss">:max_age</span><span class="p">,</span> <span class="p">{</span><span class="mi">5</span><span class="p">,</span> <span class="ss">:days</span><span class="p">}},</span>
<span class="ss">discarded:</span> <span class="p">{</span><span class="ss">:max_age</span><span class="p">,</span> <span class="p">{</span><span class="mi">5</span><span class="p">,</span> <span class="ss">:days</span><span class="p">}}</span>
<span class="p">]</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="no">Oban</span><span class="o">.</span><span class="no">Pro</span><span class="o">.</span><span class="no">Plugins</span><span class="o">.</span><span class="no">DynamicCron</span><span class="p">,</span>
<span class="ss">crontab:</span> <span class="p">[</span>
<span class="p">{</span><span class="s2">"30 7 * * *"</span><span class="p">,</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">MyWorker</span><span class="p">},</span>
<span class="p">{</span><span class="s2">"@reboot"</span><span class="p">,</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Migrations</span><span class="o">.</span><span class="no">MyDataMigrationWorker</span><span class="p">}</span>
<span class="p">]</span>
<span class="p">}</span>
<span class="p">]</span>
</code></pre></div></div>
<p>You can see that we’re using some plugins, like <code class="language-plaintext highlighter-rouge">Oban.Pro.Plugins.Lifeline</code>,
which offers a way to rescue orphaned jobs. Or the
<code class="language-plaintext highlighter-rouge">Oban.Pro.Plugins.Reprioritizer</code>, which prevents queue starvation by
automatically adjusting priorities to ensure all jobs are eventually processed; this plugin is handy when you’re using different priorities in your <code class="language-plaintext highlighter-rouge">Oban.Worker</code>, a classic example is given in the plugin documentation:</p>
<blockquote>
<p>For example, a queue that processes jobs from various customers may
prioritize customers that are in a higher tier or plan. All high priority (<code class="language-plaintext highlighter-rouge">0</code>)
jobs are guaranteed to run before any with lower priority (<code class="language-plaintext highlighter-rouge">1..3</code>), which is
wonderful for the higher tier customers but can lead to resource starvation.
When there is a constant flow of high priority jobs the lower priority jobs
will never get the chance to run.</p>
<p>The <code class="language-plaintext highlighter-rouge">Reprioritizer</code> plugin automatically adjusts lower job’s priorities so that
all jobs are eventually processed.</p>
</blockquote>
<p>In another section of this article, I will be giving more details about the
<code class="language-plaintext highlighter-rouge">Oban.Pro.Plugins.DynamicPruner</code> plugin, so let’s skip that for now.</p>
<p>At the end of our plugin list, you can see that we’re using
<code class="language-plaintext highlighter-rouge">Oban.Pro.Plugins.DynamicCron</code>, which is an advanced version of the
<code class="language-plaintext highlighter-rouge">Oban.Plugins.Cron</code> plugin for cron scheduling. The pro version allows changing
its configuration globally across your entire cluster at runtime.</p>
<p>For more details about Oban, visit the official site at <a href="https://getoban.pro">getoban.pro</a> or
on HexDocs at <a href="https://hexdocs.pm/oban">hexdocs.pm/oban</a></p>
<p>Now let’s review some of the features from Oban Web.</p>
<h2 id="oban-web">Oban Web</h2>
<p>The best place to check what Oban Web has to offer is the <a href="https://getoban.pro/oban">live Web dashboard
demo</a>, according to the authors:</p>
<blockquote>
<p>[The Oban Web Dashboard Demo is] a playful combination of randomly generated
workers using fake data and random failures makes the demo a chaotic
simulation of a production workload.
…
The demo is a beautiful canary because it uses the latest OSS, Web, and Pro
releases, utilizing all the plugins and most available features. With error
monitoring, we receive notifications that help us diagnose and fix issues
from a constantly running production instance, often (but not <em>always</em>) before
any customers report a problem! It’s crowdsourcing <em>and</em> dogfooding rolled into
one,</p>
</blockquote>
<p>Let’s start reviewing a few parts from this live demo.</p>
<h3 id="dashboard">Dashboard</h3>
<p><img src="/images/2022-02-11-oban-job-processing-library-for-elixir/oban-dashboard.png" alt="oban-dashboard" /></p>
<p>In this image, you can see the list of jobs that Oban is executing. On the
sidebar, you see different sections, such as nodes, states, and queues.</p>
<p>Currently, they have two nodes acting as workers to run Oban Jobs in this demo.</p>
<p>Each node has six queues, one of them is <code class="language-plaintext highlighter-rouge">analysis</code>, and for this queue on each node, we have a local limit of 20, giving us a total of 40, which is what you
see in the <code class="language-plaintext highlighter-rouge">limit</code> column. You can see that the <code class="language-plaintext highlighter-rouge">mailers</code> and the <code class="language-plaintext highlighter-rouge">media</code>
queues have some symbols in the <code class="language-plaintext highlighter-rouge">mode</code> column. The <em>chart line down</em> character in
the queues <code class="language-plaintext highlighter-rouge">media</code> and <code class="language-plaintext highlighter-rouge">mailers</code> means that those queues are rate limited,
which is possible given that this demo uses the <code class="language-plaintext highlighter-rouge">SmartEngine</code>. Still, you can see another icon for the <code class="language-plaintext highlighter-rouge">media</code> queue, the <em>globe</em> icon, meaning that the <code class="language-plaintext highlighter-rouge">media</code> queue has a global limit in the cluster.</p>
<p>You can also see that the demo has already completed more than 359k jobs, and
there is just one job available. One thing to notice here is that regardless of
the number of jobs available, Oban will not process more jobs than is allowed
on each queue at a given time, imposing a back-pressure mechanism, which is essential to avoid overloading our system.</p>
<h3 id="job-details">Job details</h3>
<p><img src="/images/2022-02-11-oban-job-processing-library-for-elixir/oban-job-executing.png" alt="oban-job-executing" /></p>
<p>In this image, you can see the job details in real-time; for example, in this
capture, you can see the current state, the specific arguments for this job,
which node is executing this job, schedule time, and so on. Note that you have a button on the upper right side that could allow you to cancel the
job that’s being executed.</p>
<p><img src="/images/2022-02-11-oban-job-processing-library-for-elixir/oban-job-completed.png" alt="oban-job-completed" /></p>
<p>In this image, you can see that a specific job is completed without errors.</p>
<p><img src="/images/2022-02-11-oban-job-processing-library-for-elixir/oban-job-discarded.png" alt="oban-job-discarded" /></p>
<p>But in this image, you can appreciate that this job was discarded, meaning that
we reached the maximum number of attempts, but on each try, we got errors,
this traceback view increases the observability and could help you find an issue on your code, or, if you’re dealing with a flaky third-party service, you can hit the <em>Retry</em> button, for example.</p>
<h3 id="queues">Queues</h3>
<p><img src="/images/2022-02-11-oban-job-processing-library-for-elixir/oban-queues.png" alt="oban-queues" /></p>
<p>If you press the <em>Queues</em> tab, you can see more details per queue, including
nodes, how many jobs per queue are available, their local and global limit, if
they are rate-limited, and so on. If you have the proper permissions, you can even stop or resume each queue on-demand from here. For example, in the previous screenshot, you
can see that I stopped the <code class="language-plaintext highlighter-rouge">analysis</code> queue in one of the available nodes.</p>
<h3 id="smart-engine-extension">Smart Engine extension</h3>
<p><img src="/images/2022-02-11-oban-job-processing-library-for-elixir/oban-smart-engine.png" alt="oban-smart-engine" /></p>
<p>Under the <em>Queues</em> tab, you will see an image
similar to the previous one if you click on a queue name. Using the <code class="language-plaintext highlighter-rouge">SmartEngine</code>, you can set limits by rate, global, or both per queue.</p>
<p>One neat feature of the rate limit section is that you can go a bit further
and apply <em>partitioned rate-limiting</em> by <code class="language-plaintext highlighter-rouge">worker</code>, <code class="language-plaintext highlighter-rouge">args</code>, or both within a queue.
For example, in the <code class="language-plaintext highlighter-rouge">media</code> queue, you can see that there is a <em>local limit</em> of
10, a <em>global limit</em> of 25, but only 20 jobs are allowed per worker (<em>Partition
Field</em>), every 60 seconds, across every instance of the <code class="language-plaintext highlighter-rouge">media</code> queue in this
cluster.</p>
<p>Here you can see that the <code class="language-plaintext highlighter-rouge">mailers</code> queue is rate limited. We can
define Oban Jobs or Workers that interact with external services, and we just
set the rate limit at the queue level. There is no need to worry about rate limit implementation at the <em>worker level</em>.</p>
<p>Okay, I covered enough Oban Pro and Oban Web features with this section. But, if you want to know more details about the Oban Architecture, I recommend checking these slides <a href="https://speakerdeck.com/sorentwo/the-architecture-of-oban">The Architecture of Oban</a> from
Parker Selbert.</p>
<p>Now, let’s review some conventions that I tend to follow.</p>
<h2 id="conventions">Conventions</h2>
<p>I will briefly examine some conventions I like to follow when using Oban in the following sub-sections.</p>
<h3 id="naming-and-filedirectory-organization">Naming and file/directory organization</h3>
<p>I tend to follow this code organization for Oban workers; adding subdirectories under the <code class="language-plaintext highlighter-rouge">workers</code> directory is valid.</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="go">my_app/
├── README.md
├── lib
│ ├── my_app
│ │ └── workers
│ │ └── archive_account.ex
│ └── my_app.ex
├── mix.exs
└── test
├── my_app
│ └── workers
│ └── archive_account_test.ex
├── my_app_test.exs
└── test_helper.exs
</span></code></pre></div></div>
<p>And the module naming is as follows:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">MyApp.Workers.ArchiveAccount</code> for the worker implementation</li>
<li><code class="language-plaintext highlighter-rouge">MyApp.Workers.ArchiveAccountTest</code> for the unit tests associated with the previous worker implementation.</li>
</ul>
<p>I’ve worked on some projects that follow the <a href="https://www.phoenixframework.org">Phoenix</a> style, meaning that
the directory structure is very similar to the previous one, but the file names are a bit different:</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="go">my_app/
├── README.md
├── lib
│ ├── my_app
│ │ └── workers
│ │ └── archive_account_worker.ex
│ └── my_app.ex
├── mix.exs
└── test
├── my_app
│ └── workers
│ └── archive_account_worker_test.ex
├── my_app_test.exs
└── test_helper.exs
</span></code></pre></div></div>
<p>Also, the module names are a bit different:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">MyApp.ArchiveAccountWorker</code> for the worker implementation</li>
<li><code class="language-plaintext highlighter-rouge">MyApp.ArchiveAccountWorkerTest</code> for the unit tests associated with the previous worker implementation.</li>
</ul>
<p>I’ve worked fine with both approaches; once your team has made the decision, you should stick with it; that’s the most important thing for me to
be honest.</p>
<p>So, before starting coding, take some time and define what code
organization and naming convention you and your team want to follow.</p>
<p>I will follow the “Phoenix” way of doing things in the following code samples.</p>
<h3 id="keep-calls-to-obaninsert-or-obaninsert_all-contained-in-your-worker">Keep calls to <code class="language-plaintext highlighter-rouge">Oban.insert</code> or <code class="language-plaintext highlighter-rouge">Oban.insert_all</code> contained in your worker</h3>
<p>I highly recommend keeping the knowledge about how to enqueue an Oban job in
their <code class="language-plaintext highlighter-rouge">Oban.Worker</code> implementation. Following this approach, you also avoid
polluting your controllers, resolvers, or contexts with a sequence of calls
like the following:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">my_job_args</span>
<span class="o">|></span> <span class="no">MyApp</span><span class="o">.</span><span class="no">MyWorker</span><span class="o">.</span><span class="n">new</span><span class="p">()</span>
<span class="o">|></span> <span class="no">Oban</span><span class="o">.</span><span class="n">insert</span><span class="p">()</span>
</code></pre></div></div>
<p>Instead, you can create a <code class="language-plaintext highlighter-rouge">enqueue/1</code> function like this:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">MyWorker</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Oban</span><span class="o">.</span><span class="no">Worker</span><span class="p">,</span>
<span class="ss">queue:</span> <span class="ss">:things</span><span class="p">,</span>
<span class="ss">max_attempts:</span> <span class="mi">5</span><span class="p">,</span>
<span class="ss">unique:</span> <span class="p">[</span><span class="ss">period:</span> <span class="n">_period_in_seconds</span> <span class="o">=</span> <span class="n">round</span><span class="p">(</span><span class="ss">:timer</span><span class="o">.</span><span class="n">hours</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="o">/</span> <span class="mi">1000</span><span class="p">)]</span>
<span class="n">alias</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Thing</span>
<span class="nv">@doc</span> <span class="sd">"""
Enqueues an Oban job to do something with the given thing
"""</span>
<span class="nv">@spec</span> <span class="n">enqueue</span><span class="p">(</span><span class="no">Thing</span><span class="o">.</span><span class="n">t</span><span class="p">())</span> <span class="p">::</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="no">Job</span><span class="o">.</span><span class="n">t</span><span class="p">()}</span> <span class="o">|</span> <span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="no">Job</span><span class="o">.</span><span class="n">changeset</span><span class="p">()}</span> <span class="o">|</span> <span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="n">term</span><span class="p">()}</span>
<span class="k">def</span> <span class="n">enqueue</span><span class="p">(%</span><span class="no">Thing</span><span class="p">{</span><span class="ss">id:</span> <span class="n">thing_id</span><span class="p">})</span> <span class="k">do</span>
<span class="p">%{</span><span class="ss">thing_id:</span> <span class="n">thing_id</span><span class="p">}</span>
<span class="o">|></span> <span class="n">new</span><span class="p">()</span>
<span class="o">|></span> <span class="no">Oban</span><span class="o">.</span><span class="n">insert</span><span class="p">()</span>
<span class="k">end</span>
<span class="nv">@impl</span> <span class="no">Oban</span><span class="o">.</span><span class="no">Worker</span>
<span class="k">def</span> <span class="n">perform</span><span class="p">(%</span><span class="no">Job</span><span class="p">{</span><span class="ss">args:</span> <span class="p">%{</span><span class="s2">"thing_id"</span> <span class="o">=></span> <span class="n">_thing_id</span><span class="p">}}</span> <span class="o">=</span> <span class="n">_job</span><span class="p">)</span> <span class="k">do</span>
<span class="ss">:ok</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Remember that your <code class="language-plaintext highlighter-rouge">enqueue</code> function doesn’t need to have an arity of one;
adjust the number of arguments depending on what your worker expects.</p>
<p>Even for more <em>complex workers</em>, you can apply the same convention, for example:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">TranscodeWorker</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Oban</span><span class="o">.</span><span class="no">Pro</span><span class="o">.</span><span class="no">Workers</span><span class="o">.</span><span class="no">Workflow</span>
<span class="n">alias</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">IndexingWorker</span>
<span class="n">alias</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">NotifyWorker</span>
<span class="n">alias</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">RecognizeWorker</span>
<span class="n">alias</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">SentimentWorker</span>
<span class="n">alias</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">TopicsWorker</span>
<span class="n">alias</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">TranscribeWorker</span>
<span class="k">def</span> <span class="n">process_video</span><span class="p">(</span><span class="n">video_id</span><span class="p">)</span> <span class="k">do</span>
<span class="n">args</span> <span class="o">=</span> <span class="p">%{</span><span class="ss">id:</span> <span class="n">video_id</span><span class="p">}</span>
<span class="n">new_workflow</span><span class="p">()</span>
<span class="o">|></span> <span class="n">add</span><span class="p">(</span><span class="ss">:transcode</span><span class="p">,</span> <span class="n">new</span><span class="p">(</span><span class="n">args</span><span class="p">))</span>
<span class="o">|></span> <span class="n">add</span><span class="p">(</span><span class="ss">:transcribe</span><span class="p">,</span> <span class="no">TranscribeWorker</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="n">args</span><span class="p">),</span> <span class="ss">deps:</span> <span class="p">[</span><span class="ss">:transcode</span><span class="p">])</span>
<span class="o">|></span> <span class="n">add</span><span class="p">(</span><span class="ss">:indexing</span><span class="p">,</span> <span class="no">IndexingWorker</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="n">args</span><span class="p">),</span> <span class="ss">deps:</span> <span class="p">[</span><span class="ss">:transcode</span><span class="p">])</span>
<span class="o">|></span> <span class="n">add</span><span class="p">(</span><span class="ss">:recognize</span><span class="p">,</span> <span class="no">RecognizeWorker</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="n">args</span><span class="p">),</span> <span class="ss">deps:</span> <span class="p">[</span><span class="ss">:transcode</span><span class="p">])</span>
<span class="o">|></span> <span class="n">add</span><span class="p">(</span><span class="ss">:sentiment</span><span class="p">,</span> <span class="no">SentimentWorker</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="n">args</span><span class="p">),</span> <span class="ss">deps:</span> <span class="p">[</span><span class="ss">:transcribe</span><span class="p">])</span>
<span class="o">|></span> <span class="n">add</span><span class="p">(</span><span class="ss">:topics</span><span class="p">,</span> <span class="no">TopicsWorker</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="n">args</span><span class="p">),</span> <span class="ss">deps:</span> <span class="p">[</span><span class="ss">:transcribe</span><span class="p">])</span>
<span class="o">|></span> <span class="n">add</span><span class="p">(</span><span class="ss">:notify</span><span class="p">,</span> <span class="no">NotifyWorker</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="n">args</span><span class="p">),</span> <span class="ss">deps:</span> <span class="p">[</span><span class="ss">:indexing</span><span class="p">,</span> <span class="ss">:recognize</span><span class="p">,</span> <span class="ss">:sentiment</span><span class="p">])</span>
<span class="o">|></span> <span class="no">Oban</span><span class="o">.</span><span class="n">insert_all</span><span class="p">()</span>
<span class="k">end</span>
<span class="c1"># ...</span>
<span class="k">end</span>
</code></pre></div></div>
<p><em>NOTE: The previous example was borrowed and slightly modified from the _Workflow
Example_ available in the <a href="https://sorentwo.com/2021/05/06/composing-jobs-with-oban-pro.html">Composing Jobs With Oban Pro</a> post by <a href="https://sorentwo.com">Shannon and Parker</a>.</em></p>
<h2 id="one-off-jobs">One-off jobs</h2>
<p>The easiest way to run one-off functions is via <code class="language-plaintext highlighter-rouge">bin/RELEASE_NAME remote</code> (or <code class="language-plaintext highlighter-rouge">remote_console</code> if you use <code class="language-plaintext highlighter-rouge">distillery</code> to create your release) on
production nodes, but that’s not always available. In these cases, you can use Oban to run
your one-off jobs.</p>
<p>If you plan to do a <em>data migration</em>, for example, consider wrapping this
process into an <a href="https://getoban.pro">Oban</a> job doing the following.</p>
<p>Add a new file under <code class="language-plaintext highlighter-rouge">lib/my_app/workers/migrations/</code>, leaving the suffix
<code class="language-plaintext highlighter-rouge">_worker.ex</code>, for example:
<code class="language-plaintext highlighter-rouge">lib/my_app/workers/migrations/my_data_migration_worker.ex</code>, your new
worker module should be something similar to the following:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Migrations</span><span class="o">.</span><span class="no">MyDataMigrationWorker</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Oban</span><span class="o">.</span><span class="no">Worker</span><span class="p">,</span>
<span class="ss">queue:</span> <span class="ss">:uno</span><span class="p">,</span>
<span class="ss">max_attempts:</span> <span class="mi">5</span><span class="p">,</span>
<span class="ss">unique:</span> <span class="p">[</span><span class="ss">period:</span> <span class="ss">:infinity</span><span class="p">,</span> <span class="ss">states:</span> <span class="no">Oban</span><span class="o">.</span><span class="no">Job</span><span class="o">.</span><span class="n">states</span><span class="p">()]</span>
<span class="nv">@impl</span> <span class="no">Oban</span><span class="o">.</span><span class="no">Worker</span>
<span class="k">def</span> <span class="n">perform</span><span class="p">(%</span><span class="no">Job</span><span class="p">{}</span> <span class="o">=</span> <span class="n">job</span><span class="p">)</span> <span class="k">do</span>
<span class="c1"># TODO: data migration</span>
<span class="c1"># should return a valid value</span>
<span class="c1"># See: https://hexdocs.pm/oban/Oban.Worker.html#module-defining-workers</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>From the previous code snippet, notice that you must use the <code class="language-plaintext highlighter-rouge">uno</code> queue and
define a <code class="language-plaintext highlighter-rouge">unique: [period: :infinity, states: Oban.Job.states()]</code> to indicate that any attempt to enqueue a subsequent job will be considered a duplicate <em>as long as jobs are retained</em>
in the database, and for the specific case of the <code class="language-plaintext highlighter-rouge">uno</code> queue, we keep those jobs indefinitely. You can adjust the number of <code class="language-plaintext highlighter-rouge">max_attempts</code> based on your
scenario.</p>
<p>You can test your data migration adding a new file under
<code class="language-plaintext highlighter-rouge">test/my_app/workers/migrations/my_data_migration_worker_test.exs</code>.</p>
<p>Once you have unit tested your worker following the suggestions that I will offer in the
<em>Testing your Workers and Configuration</em> article, proceed to add an entry in your configuration file:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">config</span> <span class="ss">:my_app</span><span class="p">,</span> <span class="no">Oban</span><span class="p">,</span>
<span class="c1"># ..</span>
<span class="ss">plugins:</span> <span class="p">[</span>
<span class="c1"># ...</span>
<span class="p">{</span>
<span class="no">Oban</span><span class="o">.</span><span class="no">Pro</span><span class="o">.</span><span class="no">Plugins</span><span class="o">.</span><span class="no">DynamicCron</span><span class="p">,</span>
<span class="ss">crontab:</span> <span class="p">[</span>
<span class="c1"># ...</span>
<span class="p">{</span><span class="s2">"@reboot"</span><span class="p">,</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Migrations</span><span class="o">.</span><span class="no">MyDataMigrationWorker</span><span class="p">},</span>
<span class="c1"># ...</span>
<span class="p">]</span>
<span class="p">}</span>
<span class="p">]</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">@reboot</code> string is a “non-standard syntax” that allows executing the given
job at boot time in one single node in the cluster.</p>
<h2 id="challenges">Challenges</h2>
<p>Now, I think it’s time to examine a few challenges that you could find while working with Oban.</p>
<h3 id="inserting-oban-jobs-in-bulk">Inserting Oban jobs in bulk</h3>
<p>Sometimes you need to enqueue many Oban Jobs at once or in bulk.</p>
<p>Please, don’t do this:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">my_data_stream</span>
<span class="o">|></span> <span class="no">Stream</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="o">&</span><span class="no">MyApp</span><span class="o">.</span><span class="no">MyWorker</span><span class="o">.</span><span class="n">new</span><span class="p">(%{</span><span class="ss">asset_id:</span> <span class="nv">&1</span><span class="o">.</span><span class="n">id</span><span class="p">})</span> <span class="c1"># <- use the arguments you really need for your worker</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="o">&</span><span class="no">Oban</span><span class="o">.</span><span class="n">insert</span><span class="o">/</span><span class="mi">1</span><span class="p">)</span>
</code></pre></div></div>
<p>This will produce a lot of roundtrips to the database, instead, you should use <code class="language-plaintext highlighter-rouge">Oban.insert_all/4</code>:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">my_data_stream</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="o">&</span><span class="no">MyApp</span><span class="o">.</span><span class="no">MyWorker</span><span class="o">.</span><span class="n">new</span><span class="p">(%{</span><span class="ss">asset_id:</span> <span class="nv">&1</span><span class="o">.</span><span class="n">id</span><span class="p">}))</span>
<span class="o">|></span> <span class="no">Oban</span><span class="o">.</span><span class="n">insert_all</span><span class="p">()</span>
</code></pre></div></div>
<p>While the previous approach avoids doing many roundtrips to the database, you can have problems depending on the number of jobs you’re trying to insert
at once. Keep in mind that PostgreSQL’s binary protocol has a limit of 65,535
parameters that you may send in a single call. That presents an upper limit on
the number of rows you may insert at one time and, therefore, the number of
jobs you may insert in all at once.</p>
<p>So, it’s safer to split the previous stream into chunks:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">timeouts</span> <span class="o">=</span>
<span class="n">my_data_stream</span>
<span class="o">|></span> <span class="no">Stream</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="o">&</span><span class="no">MyApp</span><span class="o">.</span><span class="no">MyWorker</span><span class="o">.</span><span class="n">new</span><span class="p">(%{</span><span class="ss">asset_id:</span> <span class="nv">&1</span><span class="o">.</span><span class="n">id</span><span class="p">})</span>
<span class="o">|></span> <span class="no">Stream</span><span class="o">.</span><span class="n">chunk_every</span><span class="p">(</span><span class="n">chunk_size</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Task</span><span class="o">.</span><span class="n">async_stream</span><span class="p">(</span><span class="o">&</span><span class="no">Oban</span><span class="o">.</span><span class="n">insert_all</span><span class="o">/</span><span class="mi">1</span><span class="p">,</span> <span class="ss">ordered:</span> <span class="no">false</span><span class="p">,</span> <span class="ss">timeout:</span> <span class="n">timeout_ms</span><span class="p">,</span> <span class="ss">on_timeout:</span> <span class="ss">:kill_task</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Stream</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="o">&</span> <span class="nv">&1</span> <span class="o">==</span> <span class="p">{</span><span class="ss">:exit</span><span class="p">,</span> <span class="ss">:timeout</span><span class="p">})</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">count</span><span class="p">()</span>
<span class="c1"># TODO: handle timeouts </span>
</code></pre></div></div>
<p>Here we’re using our <em>beloved</em> <a href="https://hexdocs.pm/elixir/Task.html#async_stream/3">Task.async_stream/3</a>, that will return a stream that runs the given function, <code class="language-plaintext highlighter-rouge">Oban.insert_all/1</code>, <em>concurrently</em> over each <em>chunk</em> in the enumerable.</p>
<p>Adjust the <code class="language-plaintext highlighter-rouge">chunk_size</code> accordingly to handle your specific scenarios.</p>
<p>One important thing to keep in mind when you use <code class="language-plaintext highlighter-rouge">Oban.insert_all/2,4</code> is that
you can insert duplicate jobs, even if your worker defines <code class="language-plaintext highlighter-rouge">unique</code> options
like:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">MyWorker</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Oban</span><span class="o">.</span><span class="no">Worker</span><span class="p">,</span>
<span class="ss">unique:</span> <span class="p">[</span><span class="ss">period:</span> <span class="n">_period_in_seconds</span> <span class="o">=</span> <span class="n">round</span><span class="p">(</span><span class="ss">:timer</span><span class="o">.</span><span class="n">hours</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="o">/</span> <span class="mi">1000</span><span class="p">)]</span>
<span class="c1"># ...</span>
<span class="k">end</span>
</code></pre></div></div>
<p>As noted in the documentation:</p>
<blockquote>
<p>[Oban.insert_all/2] insertion respects <code class="language-plaintext highlighter-rouge">prefix</code> and <code class="language-plaintext highlighter-rouge">log</code> settings, but it <em>does not use</em> per-job unique configuration. You must use <code class="language-plaintext highlighter-rouge">insert/2,4</code> or <code class="language-plaintext highlighter-rouge">insert!/2</code> for per-job unique support.</p>
</blockquote>
<p>But, sometimes, using <code class="language-plaintext highlighter-rouge">Oban.insert/2,4</code> is too costly. You might want to insert
hundreds or thousands of unique jobs as fast as possible; in these cases, you
have two possibilities, at least that I’m aware of so far.</p>
<p>The first one is to guarantee the uniqueness in the stream pipeline, but, this
could be risky because you are discarding the possibility of introducing a duplicate job that’s already in the <code class="language-plaintext highlighter-rouge">oban_jobs</code> table. In these cases, it’s
safer to set up a <a href="https://www.postgresql.org/docs/current/indexes-partial.html">partial unique index</a> in PostgreSQL, you can
set up a migration to create your <em>partial unique index</em> as follows:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="no">Migrations</span><span class="o">.</span><span class="no">CreateUniqueIndexForMyWorker</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Migration</span>
<span class="nv">@disable_ddl_transaction</span> <span class="no">true</span>
<span class="nv">@disable_migration_lock</span> <span class="no">true</span>
<span class="nv">@index_name</span> <span class="s2">"oban_jobs_unique_my_worker_index"</span>
<span class="nv">@worker_name</span> <span class="s2">"MyApp.MyWorker"</span>
<span class="k">def</span> <span class="n">up</span> <span class="k">do</span>
<span class="n">execute</span><span class="p">(</span><span class="sd">"""
CREATE UNIQUE INDEX CONCURRENTLY #{@index_name} ON oban_jobs (worker, args) WHERE worker = '#{@worker_name}'
"""</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">down</span> <span class="k">do</span>
<span class="n">execute</span><span class="p">(</span><span class="s2">"DROP INDEX IF EXISTS </span><span class="si">#{</span><span class="nv">@index_name</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>You can include more conditions to your <em>partial unique index</em>, adjust this
settings based on your specific case.</p>
<p>But you may be wondering why this would work at all. It happens that
<code class="language-plaintext highlighter-rouge">Oban.insert_all/2,4</code> it’s a wrapper around <code class="language-plaintext highlighter-rouge">Repo.insert_all/4</code> and it sets the
<code class="language-plaintext highlighter-rouge">on_conflict</code> option to <code class="language-plaintext highlighter-rouge">:nothing</code>. Better, you can test this behavior, once you
have run the previous migration yourself by creating s unit test similar to
this one:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">refute</span> <span class="n">payload</span>
<span class="o">|></span> <span class="no">MyBatchWorker</span><span class="o">.</span><span class="n">new_batch</span><span class="p">(</span><span class="ss">batch_id:</span> <span class="n">project</span><span class="o">.</span><span class="n">id</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Oban</span><span class="o">.</span><span class="n">insert_all</span><span class="p">()</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">any?</span><span class="p">(</span><span class="o">&</span> <span class="nv">&1</span><span class="o">.</span><span class="n">conflict?</span><span class="p">)</span>
<span class="c1"># second time all the jobs must create a conflict, but not an insertion</span>
<span class="n">assert</span> <span class="n">payload</span>
<span class="o">|></span> <span class="no">MyBatchWorker</span><span class="o">.</span><span class="n">new_batch</span><span class="p">(</span><span class="ss">batch_id:</span> <span class="n">project</span><span class="o">.</span><span class="n">id</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Oban</span><span class="o">.</span><span class="n">insert_all</span><span class="p">()</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">all?</span><span class="p">(</span><span class="o">&</span> <span class="nv">&1</span><span class="o">.</span><span class="n">conflict?</span><span class="p">)</span>
</code></pre></div></div>
<p>In this unit test, we use the key <code class="language-plaintext highlighter-rouge">:conflict?</code> to detect the job uniqueness. From the Oban README, we have:</p>
<blockquote>
<p>When unique settings match an existing job, the return value of <code class="language-plaintext highlighter-rouge">Oban.insert/2</code>
is still <code class="language-plaintext highlighter-rouge">{:ok, job}</code>. However, you can detect a unique conflict by checking the
jobs’ <code class="language-plaintext highlighter-rouge">:conflict?</code> field. If there was an existing job, the field is <code class="language-plaintext highlighter-rouge">true</code>;
otherwise it is <code class="language-plaintext highlighter-rouge">false</code>.</p>
</blockquote>
<p>So, in the first pass, we check that the Oban Jobs doesn’t have <em>any</em> entry like: <code class="language-plaintext highlighter-rouge">%Oban.Job{conflict?: true}</code>. We check that <em>all</em> the entries have <code class="language-plaintext highlighter-rouge">%Oban.Job{conflict?: true}</code> in the second pass.</p>
<p>Nice, huh?</p>
<h2 id="complex-workers">Complex workers</h2>
<p>I won’t explain in detail the Workers offered by <code class="language-plaintext highlighter-rouge">Oban.Pro</code>, not because I
think they don’t deserve a space here, but because <a href="https://sorentwo.com">Shannon and Parker</a>
already did a fantastic job in their post <a href="https://sorentwo.com/2021/05/06/composing-jobs-with-oban-pro.html">Composing Jobs With Oban
Pro</a>, they will take you on a tour of the workers included in
Pro and explore some real-world use-cases where each one shines.</p>
<h2 id="limitations">Limitations</h2>
<p>If you have reached this point, you undoubtedly noticed that I have <em>enjoyed</em> working with Oban so far, and at this point, I could be <em>biased</em>, so I want to take a step back to add some balance and mention some of Oban’s limitations.</p>
<ul>
<li>Oban is “job processing in Elixir, backed by modern PostgreSQL”. I don’t see this as a limitation, but you can’t use Oban if you don’t use PostgreSQL in your stack. Thankfully that’s not my case for many years :)</li>
<li>If you use PostgreSQL, but you have an overloaded database currently, I don’t recommend adding more pressure with Oban. You have a bigger problem that you need to solve as soon as possible—you need to figure out what’s overloading your database and fix it!</li>
<li>If you still want to use Oban, but you’re thinking of adding a new PostgreSQL database just for the Oban workers, keep in mind that you will lose one of the most relevant features from Oban, which is the built-in transactional control.</li>
<li>If you want good performance in Oban Web, you should keep your <code class="language-plaintext highlighter-rouge">oban_jobs</code> table lean. I’ve noted some delays in the UI at around ten million records.</li>
<li>If you’re trying to ingest high throughput event streams, Oban probably isn’t the solution you need. You can process more than 15K jobs per second with Oban on a single node; of course, that number will highly depend on what your worker does. But you should increase that throughput significantly if you use the <em>Batch Worker</em> behaviour. Still, sometimes that’s not enough, and you should use a <em>message broker</em>, like <a href="https://github.com/rabbitmq">rabbitmq</a> in conjunction with <a href="https://elixir-broadway.org">Broadway</a> or <a href="https://github.com/elixir-lang/gen_stage">GenStage</a>. Still, the initial learning curve of the latter tools could be higher, and you also need to make other considerations, like considering the state of your processes while you deploy.</li>
</ul>
<h2 id="wishlist">Wishlist</h2>
<p>In the same vein as the previous section, there are a few things that I would like to see in Oban:</p>
<ul>
<li>Autoload regulation, suppose that your deployment in production shares the API server with your Oban Workers; these processes will compete with each other for resources. With a regulation framework built-in, your queues could be constrained a bit more if the load in your nodes is too high. If you don’t have a rate limit in your queue, you could automatically increase the <em>concurrency limit</em> if the load in the node is low. An excellent reference to this topic can be found in the paper <a href="https://github.com/uwiger/jobs/blob/master/doc/erlang07g-wiger.pdf">Generic Load Regulation Framework for Erlang</a> by Ulf Wiger. As far as I can tell, this feature is already in the <em>roadmap</em>.</li>
<li>A callback in the <code class="language-plaintext highlighter-rouge">{Fixed,Dynamic}Pruner</code> plugins; that way, before proceeding with the deletion, you can store or transfer those records into cold storage or somewhere else. I recently <a href="https://github.com/sorentwo/oban/issues/633">opened an issue</a> about this feature.</li>
</ul>
<p>Do you have things that you would like to see in Oban? If that’s the case, and you want to share those wishes with me, you can reach me at <a href="https://twitter.com/milmazz"><code class="language-plaintext highlighter-rouge">@milmazz</code></a> on Twitter, or you can find me in the <code class="language-plaintext highlighter-rouge">#oban</code> channel on the <a href="https://elixir-slackin.herokuapp.com/">Elixir Slack</a>.</p>
<h2 id="community">Community</h2>
<p>You can connect with the Oban authors, contributors, and other Oban users through any of these channels:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">#oban</code> channel on the <a href="https://elixir-slackin.herokuapp.com/">Elixir Slack</a>. Here you can see new announcements about new Oban releases.</li>
<li>Follow <code class="language-plaintext highlighter-rouge">@sorentwo</code> on Twitter for tips, announcements, and news about Oban</li>
<li>If you want to contribute to the OSS project, feel free to do it at <a href="https://github.com/sorentwo/oban">https://github.com/sorentwo/oban</a>. You might start with the <a href="https://github.com/sorentwo/oban/issues">issues list</a>. I’ve been able to contribute more than a dozen times, but on each occasion that I had a doubt, the authors were welcoming and offered good references, which is my general experience in the Elixir community, so don’t be shy and join the team :)</li>
</ul>
<h2 id="conclusion">Conclusion</h2>
<p>Oban is a solid solution to handle your background jobs in Elixir. It keeps true to what they offer in the README, like fewer dependencies, isolated queues, transactional control, unique, scheduled, and recurrent jobs, telemetry integration, and much more.</p>
<p>After all this, I think it’s clear that if you or your company need a
<em>background job system</em> in Elixir, I recommend buying the <a href="https://getoban.pro/pricing">Oban Web+Pro
license</a>, keep in mind that when you buy the license, you’re helping
<a href="https://sorentwo.com">Shannon and Parker</a>, to keep investing their time building more
features for the open-source release of <a href="https://getoban.pro">Oban</a>.</p>
<p>If you have any pattern that you follow when you use Oban and you want to share that with me, you can reach me at <a href="https://twitter.com/milmazz"><code class="language-plaintext highlighter-rouge">@milmazz</code></a> on Twitter, or you can find me in the <code class="language-plaintext highlighter-rouge">#oban</code> channel on the <a href="https://elixir-slackin.herokuapp.com/">Elixir Slack</a>.</p>
<p>That’s all folks! Thanks for reading.</p>
<h2 id="acknowledgments">Acknowledgments</h2>
<p>Thank you Parker Selbert for reviewing drafts of this post.</p>Milton Mazzarrime@milmazz.unoAfter working for years on different organizations, one common theme is scheduling background jobs. In this article, I’ll share my experience with Oban, an open-source job processing package for Elixir. I’ll also cover some features, like real-time monitoring with Oban Web and complex workflow management with Oban Pro.Improve the codebase of an acquired product2020-04-01T13:27:31-05:002020-04-01T13:27:31-05:00https://milmazz.uno/article/2020/04/01/improve-the-codebase-of-an-acquired-product<p>In this article I’ll share my experience improving the codebase of an acquired
product, this couldn’t be possible without the help of a fantastic team. Before
diving into the initial diagnostic and strategies that we took to tackle
<em>technical debt</em>, I’ll share some background around the acquisition. Let’s
start.</p>
<!--more-->
<h2 id="background">Background</h2>
<p>I wasn’t involved in the acquisition process. I started working with this
client around a year after that decision; that’s why in this article, I do not
mention those details. Maybe the only thing that I can share is that Elixir and
Phoenix power the product, and possibly at the time of the acquisition, fulfilled
its goal and covered some <a href="https://www.investopedia.com/terms/o/opportunitycost.asp">opportunity costs</a> for the client.</p>
<p>I’ll assume that the previous developers made their best effort under a set of
constraints (time, team-size, among others) that I don’t know how good or bad
they were. This codebase is now part of the team, and it’s our team goal to
take ownership of this codebase, to accomplish that we establish some common
goals:</p>
<ul>
<li>Do not break current working features given that the system is in production,
and it’s serving millions of requests per day</li>
<li>Add new features based on requirements or stories, the usual thing</li>
<li>Improve the codebase along the way</li>
</ul>
<p>After the acquisition process, the client knew that the codebase was a
“dumpster fire” (their words, not mine). Still, it was generating revenue, so
throwing away that product was out of the question. Thankfully, the client
allowed some time to improve the codebase.</p>
<p>At the same time, we delivered new features, even accepted the fact that we had
to delay some deployments until we thought it was the right time to do it.
Again, I mention this because this kind of agreement is difficult to get
sometimes.</p>
<h2 id="diagnostics">Diagnostics</h2>
<p>The repository followed a poncho structure; the state of some of the
applications were the following:</p>
<ul>
<li>Zero (none, nada) unit tests, this was <em>awful</em> because it made the whole
refactoring process more painful and slow.</li>
<li>No documentation in the modules, public functions, only a brief <code class="language-plaintext highlighter-rouge">README</code> file
describing the goal of the application, and that was all.</li>
<li>A lot of 3rd party dependencies, the most painful part here was the
dependency of different <em>storage services</em>. For example, Airtable, Mongo,
PostgreSQL.</li>
<li>The goals of some project dependencies were the same, like HTTP clients, JSON
parsers, job queue processors, among others. Even a team-partner said
something like: “…background worker libraries are like Pokemon, and I think
we have collected them all”, so that could give you an idea of our situation.</li>
<li>We found large portions of duplicate implementations, for example, HTTP
clients implementations embedded in some of the poncho applications instead
of having that HTTP client implementation as a dependency.</li>
</ul>
<h2 id="strategies">Strategies</h2>
<p>The team decided to tackle these improvements in a series of continuous steps.</p>
<ul>
<li>Move the other “storage services” into PostgreSQL one table at a time. The
current team has a better knowledge of this ORDBMS, and it’s been offering
excellent results.</li>
<li>Delete unused code; this was a low-hanging-fruit for us using some tools that
I’ll mention later in this article.</li>
<li>Separate the implementation of 3rd party HTTP clients from the main
components. So, reaching a given 3rd party service must be done via a
separate application.</li>
<li>Improve some design decisions</li>
<li>Introduce error monitoring, that allowed us to discover, triage, and
prioritize errors in real-time.</li>
</ul>
<h2 id="tooling-support">Tooling support</h2>
<p>As I mentioned before, removing unreachable or unused code was easy for us, and
with that, we removed a considerable burden when trying to understand how the
applications work. Here the critical player in my workflow was <a href="https://github.com/joshuaclayton/unused">unused</a>,
which is a command-line tool, written in Haskell, that helps you to identify
unused code in multiple languages and that includes Elixir.</p>
<p>Before we proceeded with the code removal, we were confident that the code we
were removing wouldn’t break the program, one way we used to gain more
confidence was <code class="language-plaintext highlighter-rouge">mix xref callers CALLEE</code>, which prints all the callers of the
given <code class="language-plaintext highlighter-rouge">CALLEE</code><sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>.</p>
<p>Then, we started adding unit tests to each project. Unit testing is a
first-class concern on our team, so, before proceeding with a <em>storage service
migration</em> (e.g., Moving a Mongo collection into a PostgreSQL table), we added
unit tests, using the <a href="https://hexdocs.pm/ex_unit/ExUnit.html">ex_unit</a> framework, around the modules and functions
that we were planning to migrate. While doing this, we did some test coverage
analysis with <code class="language-plaintext highlighter-rouge">mix test --cover</code> to identify untested areas of the code.</p>
<p>Adding unit tests around the areas we were planning to migrate increased our
confidence in the following steps, and also helped us to discover other bugs
along the way. So, that was a win-win situation for us.</p>
<p>Another great addition was the <em>error monitoring</em> process, before doing that we
were almost blind about the things that were failing in production, well,
that’s not entirely accurate, we had access to the logs, but nobody in the team
was checking those constantly.</p>
<p>The team was already using Sentry in other applications, so we decided to use
Sentry here too. We started linking Sentry with our GitHub repository. We proceeded to tackle one bug at a time in multiple applications, giving priority
to the events with a higher number of occurrences. To provide you with an idea of the
situation at the beginning, some of the applications had so many errors that
Sentry responded with HTTP Errors 429 (Too Many Requests), that’s why some
team-members coined the term “dumpster fire” to this legacy application. Today
the situation is another story, so I recommend adding an <em>error monitoring</em>
tool as soon as possible in your projects, and this includes your staging
environments too.</p>
<p>We used other tools too, like <code class="language-plaintext highlighter-rouge">dialyzer</code>, it’s true, <code class="language-plaintext highlighter-rouge">dialyzer</code> can be
intricate sometimes. Still, I would say that it could be beneficial to discover
bugs in your code and also incorrect types and specifications (a.k.a.
<a href="https://hexdocs.pm/elixir/typespecs.html">typespecs</a>).</p>
<p>Last but not least, <a href="https://github.com/rrrene/credo">credo</a> has been useful to run static code analysis in
our codebase and to promote some consistency and readability.</p>
<p>While we haven’t included <a href="https://github.com/elixir-lang/ex_doc">ex_doc</a> in our workflow yet, we have added a bunch
of documentation in our code, and it’s already paying dividends; newer team
members mentioned that the onboarding process is more accessible thanks to that
documentation.</p>
<h2 id="design">Design</h2>
<p>While we were refactoring, we made many decisions to improve the current
codebase, among them, one of the most recent was that we identified that the
company was losing money when an agent/user started a session (LOGON) in a 3rd
party system. Still, the agent wasn’t READY to operate. The interaction with
the 3rd party service was something like this:</p>
<ul>
<li>Every second a process collected data from the 3rd party service because they
didn’t offer a webhook that allowed us to subscribe to those events</li>
<li>Then, the producer sent all the collected messages to some other modules via
Phoenix Channels, among the one that was in charge of forcing those agents to
be READY to operate</li>
<li>The process that put READY those agents was only one instance of
a GenServer</li>
</ul>
<p>So, given the previous scenario and given that a GenServer can process only one
message at a time, we had some agents in the NOT_READY state waiting for the
FORCE_READY operation. What we did, in this case, was to split the input per
agent IDs and then dynamically create GenServers where its state was all around
the agent/user, once the agent finished its session, we stopped that GenServer
instance. This change allowed us to increase our FORCE_READY operations thanks
to the concurrency and prepared us for a new feature, which kicked users after
a configurable time of inactivity.</p>
<p>Previously the kicking process was done by our support staff manually. We added
the “kicking inactive users” feature while we were doing a work re-retreat (the
company is 100% remote). You can’t imagine the happiness shown by our support
staff after deploying that feature and seeing how the application was
automatically kicking inactive users. Sometimes, as a developer, we lose the
perspective and impact that we can make on people’s lives; it was a humbling
experience.</p>
<h2 id="rewards">Rewards</h2>
<ul>
<li>Staff & support teams are enjoying the fruits of these improvements while
doing less manual processes. Of course, we still have some areas that we need
to polish. We’re getting there</li>
<li>Improved the onboarding experience for newer developers, while also improving
the experience for older developers</li>
<li>Our confidence while doing changes have increased</li>
<li>Deliver new features more quickly</li>
<li>Savings, while we haven’t completed the whole migration from some storage
services, our estimate is to save the company more than 20K a month after
finishing the whole process, which, according to our plan, is at the end of
the next month.</li>
</ul>
<h2 id="wrapping-up">Wrapping Up</h2>
<p>Elixir and Phoenix, even in the worst-case scenarios, have demonstrated that it
allows some companies to deliver excellent results. But, it’s the team culture,
and following effective practices, that enables to improve the user experience,
while enhancing the performance and the quality of the code.</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>Another tool that you should probably check out is <a href="https://github.com/hauleth/mix_unused"><code class="language-plaintext highlighter-rouge">Mix Unused</code></a>. I haven’t used it yet, because I recently knew about its existence, but I plan to do it sooner than later. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Milton Mazzarrime@milmazz.unoIn this article I’ll share my experience improving the codebase of an acquired product, this couldn’t be possible without the help of a fantastic team. Before diving into the initial diagnostic and strategies that we took to tackle technical debt, I’ll share some background around the acquisition. Let’s start.Elixir’s MIME library review2018-11-23T13:40:31-06:002018-11-23T13:40:31-06:00https://milmazz.uno/article/2018/11/23/elixir-mime-library-review<p>Elixir’s <a href="https://github.com/elixir-plug/mime">MIME</a> is a read-only and immutable library that embeds the <a href="https://pagure.io/mailcap/blob/master/f/mime.types">MIME
type database</a>, so, users can map MIME (Multipurpose Internet Mail
Extensions) types to extensions and vice-versa. It’s a really compact project
and includes nice features, which I’ll try to explain in case you’re not
familiar with the library. Then, I’ll focus on MIME’s internals or how was
built, and also how MIME illustrates in an elegant way so many features of
Elixir itself.
<!--more-->
One of the goals, maybe the main one, of this library is to offer a <em>performant
lookup</em> of the MIME database at runtime, that’s why new MIME types can only be
added at compile-time via configuration, but we’ll talk about this option
later. First, let’s review its public API.</p>
<h2 id="api">API</h2>
<p>MIME offers a short set of functions, which cover the most relevant cases when
you work with MIME types.</p>
<p>Let’s review real quick the MIME library API, most of the examples were taken
from the MIME’s <a href="https://hexdocs.pm/mime/MIME.html">documentation page</a>.</p>
<h3 id="extensionsstringt--stringt"><code class="language-plaintext highlighter-rouge">extensions(String.t()) :: [String.t()]</code></h3>
<p>Returns the extensions associated with the given MIME type.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">></span> <span class="no">MIME</span><span class="o">.</span><span class="n">extensions</span><span class="p">(</span><span class="s2">"application/json"</span><span class="p">)</span>
<span class="p">[</span><span class="s2">"json"</span><span class="p">]</span>
<span class="n">iex</span><span class="o">></span> <span class="no">MIME</span><span class="o">.</span><span class="n">extensions</span><span class="p">(</span><span class="s2">"foo/bar"</span><span class="p">)</span>
<span class="p">[]</span>
</code></pre></div></div>
<h3 id="typestringt--stringt"><code class="language-plaintext highlighter-rouge">type(String.t()) :: String.t()</code></h3>
<p>Returns the MIME type related to the given file extension.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">></span> <span class="no">MIME</span><span class="o">.</span><span class="n">type</span><span class="p">(</span><span class="s2">"txt"</span><span class="p">)</span>
<span class="s2">"text/plain"</span>
</code></pre></div></div>
<h3 id="from_pathpatht--stringt"><code class="language-plaintext highlighter-rouge">from_path(Path.t()) :: String.t()</code></h3>
<p>Guesses the MIME type based on the path’s extension.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">></span> <span class="no">MIME</span><span class="o">.</span><span class="n">from_path</span><span class="p">(</span><span class="s2">"index.html"</span><span class="p">)</span>
<span class="s2">"text/html"</span>
</code></pre></div></div>
<h3 id="has_typestringt--boolean"><code class="language-plaintext highlighter-rouge">has_type?(String.t()) :: boolean</code></h3>
<p>Returns whether an extension has a MIME type associated.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">></span> <span class="no">MIME</span><span class="o">.</span><span class="n">has_type?</span><span class="p">(</span><span class="s2">"txt"</span><span class="p">)</span>
<span class="no">true</span>
<span class="n">iex</span><span class="o">></span> <span class="no">MIME</span><span class="o">.</span><span class="n">has_type?</span><span class="p">(</span><span class="s2">"foobarbaz"</span><span class="p">)</span>
<span class="no">false</span>
</code></pre></div></div>
<h3 id="validstringt--boolean"><code class="language-plaintext highlighter-rouge">valid?(String.t()) :: boolean</code></h3>
<p>Returns whether a MIME type is registered.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">></span> <span class="no">MIME</span><span class="o">.</span><span class="n">valid?</span><span class="p">(</span><span class="s2">"text/plain"</span><span class="p">)</span>
<span class="no">true</span>
<span class="n">iex</span><span class="o">></span> <span class="no">MIME</span><span class="o">.</span><span class="n">valid?</span><span class="p">(</span><span class="s2">"foo/bar"</span><span class="p">)</span>
<span class="no">false</span>
</code></pre></div></div>
<h2 id="who-is-using-mime-library">Who is using MIME library?</h2>
<p>At the time of this writing, and according to the statistics available from the
<a href="https://hex.pm/">Hex package manager</a>, the <a href="https://hex.pm/packages?search=depends%3Amime">MIME library has 21 dependents
projects</a>, among those projects you can find: <a href="https://hex.pm/packages/plug">Plug</a>,
<a href="https://hex.pm/packages/phoenix">Phoenix</a>, <a href="https://hex.pm/packages/tesla">Tesla</a>, <a href="https://hex.pm/packages/swoosh">Swoosh</a>, etc., and have been
downloaded almost 6 million times. But more importantly, at least to me, is how
the MIME library is implemented, its code is really concise, it’s around 200
SLOC (Source Lines Of Code) including comments, and embed captivating concepts.</p>
<h2 id="how-was-the-mime-library-built">How was the MIME library built?</h2>
<p>Now, let’s start looking into how the MIME library was built.</p>
<p>Inside of the <code class="language-plaintext highlighter-rouge">MIME.Application.quoted/1</code> function you can find the following
section:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># file: lib/mime/application.ex</span>
<span class="n">mime_file</span> <span class="o">=</span> <span class="no">Application</span><span class="o">.</span><span class="n">app_dir</span><span class="p">(</span><span class="ss">:mime</span><span class="p">,</span> <span class="s2">"priv/mime.types"</span><span class="p">)</span>
<span class="nv">@compile</span> <span class="ss">:no_native</span>
<span class="nv">@external_resource</span> <span class="n">mime_file</span>
<span class="n">stream</span> <span class="o">=</span> <span class="no">File</span><span class="o">.</span><span class="n">stream!</span><span class="p">(</span><span class="n">mime_file</span><span class="p">)</span>
<span class="n">mapping</span> <span class="o">=</span>
<span class="n">for</span> <span class="n">line</span> <span class="o"><-</span> <span class="n">stream</span><span class="p">,</span>
<span class="ow">not</span> <span class="no">String</span><span class="o">.</span><span class="n">starts_with?</span><span class="p">(</span><span class="n">line</span><span class="p">,</span> <span class="p">[</span><span class="s2">"#"</span><span class="p">,</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">"</span><span class="p">]),</span>
<span class="p">[</span><span class="n">type</span> <span class="o">|</span> <span class="n">exts</span><span class="p">]</span> <span class="o">=</span> <span class="n">line</span> <span class="o">|></span> <span class="no">String</span><span class="o">.</span><span class="n">trim</span><span class="p">()</span> <span class="o">|></span> <span class="no">String</span><span class="o">.</span><span class="n">split</span><span class="p">(),</span>
<span class="n">exts</span> <span class="o">!=</span> <span class="p">[],</span>
<span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="n">type</span><span class="p">,</span> <span class="n">exts</span><span class="p">}</span>
</code></pre></div></div>
<p>You can notice that the <code class="language-plaintext highlighter-rouge">MIME</code> library transforms the data located in the
<code class="language-plaintext highlighter-rouge">priv/mime.types</code> file, which is a copy of the IANA (Internet Assigned Numbers
Authority) database in text format and describes what Internet media types are
sent to the client for the given file extension(s). Keep in mind that sending
the correct media type to the client is important so they know how to handle
the content of the file.</p>
<p>Here is an example of the <code class="language-plaintext highlighter-rouge">priv/mime.types</code> file content:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># file: priv/mime.types
# IANA types
# MIME type Extensions
application/3gpp-ims+xml
application/ATXML atxml
application/atom+xml atom
application/atomcat+xml atomcat
application/octet-stream bin lha lzh exe class so dll img iso
application/pdf pdf
</code></pre></div></div>
<p>To do the transformation, the <code class="language-plaintext highlighter-rouge">MIME</code> library reads the file line by line (via
<a href="https://hexdocs.pm/elixir/File.html#stream!/3"><code class="language-plaintext highlighter-rouge">File.stream!/3</code></a>) at compile time, and ignores empty lines or lines
that start with a comment (<code class="language-plaintext highlighter-rouge">#</code>). After that, it removes the leading and
trailing whitespace and then splits that line or string into a list of
substrings, the <em>head</em> of this list represents the mime type and the <em>tail</em> of
the list represents the extensions, that’s why at the end of the <code class="language-plaintext highlighter-rouge">for</code>
comprehension you can see an extra filter to ignore mime types that does not
have any extensions associated (e.g. <code class="language-plaintext highlighter-rouge">application/3gpp-ims+xml</code>), this is an
optimization that reduces the compilation time. Finally, it creates a list of
<code class="language-plaintext highlighter-rouge">{type, extensions}</code> tuples. The result of this transformation is stored in a
binding called <code class="language-plaintext highlighter-rouge">mapping</code>.</p>
<p>Is important to note the usage of two Module attributes, the first one is
<code class="language-plaintext highlighter-rouge">@external_resource</code>, which as its name implies, specifies an external resource
for the given module, this attribute is used for tools like <code class="language-plaintext highlighter-rouge">Mix</code> to know if
the current module needs to be recompiled in the case that any external
resource is updated. Lastly, the <code class="language-plaintext highlighter-rouge">@compile</code> attribute defines options for the
module compilation, this is used to configure both Elixir and Erlang compilers.</p>
<p>Once the <code class="language-plaintext highlighter-rouge">MIME</code> library has transformed the data and stored the result in
<code class="language-plaintext highlighter-rouge">mapping</code>, it creates two private helper functions:</p>
<p>The first private helper function is <code class="language-plaintext highlighter-rouge">ext_to_mime/1</code>, which returns the MIME
type given an extension:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">@spec</span> <span class="n">ext_to_mime</span><span class="p">(</span><span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">())</span> <span class="p">::</span> <span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">()</span> <span class="o">|</span> <span class="no">nil</span>
<span class="k">defp</span> <span class="n">ext_to_mime</span><span class="p">(</span><span class="n">type</span><span class="p">)</span>
<span class="n">for</span> <span class="p">{</span><span class="n">type</span><span class="p">,</span> <span class="n">exts</span><span class="p">}</span> <span class="o"><-</span> <span class="n">mapping</span><span class="p">,</span>
<span class="n">ext</span> <span class="o"><-</span> <span class="n">exts</span> <span class="k">do</span>
<span class="k">defp</span> <span class="n">ext_to_mime</span><span class="p">(</span><span class="kn">unquote</span><span class="p">(</span><span class="n">ext</span><span class="p">)),</span> <span class="k">do</span><span class="p">:</span> <span class="kn">unquote</span><span class="p">(</span><span class="n">type</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">ext_to_mime</span><span class="p">(</span><span class="n">_ext</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="no">nil</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">MIME</code> library creates thousands of <code class="language-plaintext highlighter-rouge">ext_to_mime/1</code> function clauses inside
of the <code class="language-plaintext highlighter-rouge">for</code> comprehension, this is a clear example of the power of
meta-programming and how the MIME library relies on <em>pattern matching</em> to be
<em>performant</em>. And this is possible because of Elixir <code class="language-plaintext highlighter-rouge">quote</code> and <code class="language-plaintext highlighter-rouge">unquote</code>
mechanisms provide a feature called <em>unquote fragments</em>, that way is easy to
create function on-the-fly at compile time.</p>
<p>To give you a better idea, here is a section of the final result:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defp</span> <span class="n">ext_to_mime</span><span class="p">(</span><span class="s2">"atom"</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="s2">"application/atom+xml"</span>
<span class="k">defp</span> <span class="n">ext_to_mime</span><span class="p">(</span><span class="s2">"pdf"</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="s2">"application/pdf"</span>
<span class="k">defp</span> <span class="n">ext_to_mime</span><span class="p">(</span><span class="s2">"dll"</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="s2">"application/octet-stream"</span>
<span class="k">defp</span> <span class="n">ext_to_mime</span><span class="p">(</span><span class="s2">"class"</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="s2">"application/octet-stream"</span>
<span class="c1"># ...</span>
<span class="k">defp</span> <span class="n">ext_to_mime</span><span class="p">(</span><span class="n">_ext</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="no">nil</span>
</code></pre></div></div>
<p>To complete the function declaration you see a catch-all function clause, which
will be used if any match is not found.</p>
<p>The second private helper function declaration is <code class="language-plaintext highlighter-rouge">mime_to_ext/1</code>, this
function expects a MIME type and will return a list of extensions or <code class="language-plaintext highlighter-rouge">nil</code>.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">@spec</span> <span class="n">mime_to_ext</span><span class="p">(</span><span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">())</span> <span class="p">::</span> <span class="n">list</span><span class="p">(</span><span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">())</span> <span class="o">|</span> <span class="no">nil</span>
<span class="k">defp</span> <span class="n">mime_to_ext</span><span class="p">(</span><span class="n">type</span><span class="p">)</span>
<span class="n">for</span> <span class="p">{</span><span class="n">type</span><span class="p">,</span> <span class="n">exts</span><span class="p">}</span> <span class="o"><-</span> <span class="n">mapping</span> <span class="k">do</span>
<span class="k">defp</span> <span class="n">mime_to_ext</span><span class="p">(</span><span class="kn">unquote</span><span class="p">(</span><span class="n">type</span><span class="p">)),</span> <span class="k">do</span><span class="p">:</span> <span class="kn">unquote</span><span class="p">(</span><span class="n">exts</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">mime_to_ext</span><span class="p">(</span><span class="n">_type</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="no">nil</span>
</code></pre></div></div>
<p>The result of the transformation should be similar to:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defp</span> <span class="n">mime_to_ext</span><span class="p">(</span><span class="s2">"application/atom+xml"</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="p">[</span><span class="s2">"atom"</span><span class="p">]</span>
<span class="k">defp</span> <span class="n">mime_to_ext</span><span class="p">(</span><span class="s2">"application/octet-stream"</span><span class="p">),</span>
<span class="k">do</span><span class="p">:</span> <span class="p">[</span><span class="s2">"bin"</span><span class="p">,</span> <span class="s2">"lha"</span><span class="p">,</span> <span class="s2">"lzh"</span><span class="p">,</span> <span class="s2">"exe"</span><span class="p">,</span> <span class="s2">"class"</span><span class="p">,</span> <span class="s2">"so"</span><span class="p">,</span> <span class="s2">"dll"</span><span class="p">,</span> <span class="s2">"img"</span><span class="p">,</span> <span class="s2">"iso"</span><span class="p">]</span>
<span class="k">defp</span> <span class="n">mime_to_ext</span><span class="p">(</span><span class="s2">"application/pdf"</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="p">[</span><span class="s2">"pdf"</span><span class="p">]</span>
<span class="c1"># ...</span>
<span class="k">defp</span> <span class="n">mime_to_ext</span><span class="p">(</span><span class="n">_type</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="no">nil</span>
</code></pre></div></div>
<p>From here, is easy to build the public functions:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">@spec</span> <span class="n">valid?</span><span class="p">(</span><span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">())</span> <span class="p">::</span> <span class="n">boolean</span>
<span class="k">def</span> <span class="n">valid?</span><span class="p">(</span><span class="n">type</span><span class="p">)</span> <span class="k">do</span>
<span class="n">is_list</span><span class="p">(</span><span class="n">mime_to_ext</span><span class="p">(</span><span class="n">type</span><span class="p">))</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">extensions</span><span class="p">(</span><span class="n">type</span><span class="p">)</span> <span class="k">do</span>
<span class="n">mime_to_ext</span><span class="p">(</span><span class="n">type</span><span class="p">)</span> <span class="o">||</span> <span class="p">[]</span>
<span class="k">end</span>
<span class="nv">@default_type</span> <span class="s2">"application/octet-stream"</span>
<span class="nv">@spec</span> <span class="n">type</span><span class="p">(</span><span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">())</span> <span class="p">::</span> <span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">()</span>
<span class="k">def</span> <span class="n">type</span><span class="p">(</span><span class="n">file_extension</span><span class="p">)</span> <span class="k">do</span>
<span class="n">ext_to_mime</span><span class="p">(</span><span class="n">file_extension</span><span class="p">)</span> <span class="o">||</span> <span class="nv">@default_type</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">has_type?</span><span class="p">(</span><span class="n">file_extension</span><span class="p">)</span> <span class="k">do</span>
<span class="n">is_binary</span><span class="p">(</span><span class="n">ext_to_mime</span><span class="p">(</span><span class="n">file_extension</span><span class="p">))</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">from_path</span><span class="p">(</span><span class="n">path</span><span class="p">)</span> <span class="k">do</span>
<span class="k">case</span> <span class="no">Path</span><span class="o">.</span><span class="n">extname</span><span class="p">(</span><span class="n">path</span><span class="p">)</span> <span class="k">do</span>
<span class="s2">"."</span> <span class="o"><></span> <span class="n">ext</span> <span class="o">-></span> <span class="n">type</span><span class="p">(</span><span class="n">downcase</span><span class="p">(</span><span class="n">ext</span><span class="p">,</span> <span class="s2">""</span><span class="p">))</span>
<span class="n">_</span> <span class="o">-></span> <span class="nv">@default_type</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">downcase</span><span class="p">(</span><span class="o"><<</span><span class="n">h</span><span class="p">,</span> <span class="n">t</span><span class="p">::</span><span class="n">binary</span><span class="o">>></span><span class="p">,</span> <span class="n">acc</span><span class="p">)</span> <span class="ow">when</span> <span class="n">h</span> <span class="ow">in</span> <span class="sx">?A</span><span class="o">..</span><span class="sx">?Z</span><span class="p">,</span>
<span class="k">do</span><span class="p">:</span> <span class="n">downcase</span><span class="p">(</span><span class="n">t</span><span class="p">,</span> <span class="o"><<</span><span class="n">acc</span><span class="p">::</span><span class="n">binary</span><span class="p">,</span> <span class="n">h</span> <span class="o">+</span> <span class="mi">32</span><span class="o">>></span><span class="p">)</span>
<span class="k">defp</span> <span class="n">downcase</span><span class="p">(</span><span class="o"><<</span><span class="n">h</span><span class="p">,</span> <span class="n">t</span><span class="p">::</span><span class="n">binary</span><span class="o">>></span><span class="p">,</span> <span class="n">acc</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="n">downcase</span><span class="p">(</span><span class="n">t</span><span class="p">,</span> <span class="o"><<</span><span class="n">acc</span><span class="p">::</span><span class="n">binary</span><span class="p">,</span> <span class="n">h</span><span class="o">>></span><span class="p">)</span>
<span class="k">defp</span> <span class="n">downcase</span><span class="p">(</span><span class="o"><<>></span><span class="p">,</span> <span class="n">acc</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="n">acc</span>
</code></pre></div></div>
<p>That’s it!, at least with these functions, the MIME library cover the main
features. But wait, there is more, do you remember that at the beginning I
mentioned the following:</p>
<blockquote>
<p>One of the goals, maybe the main one, of this library is to be provide a
<em>performant lookup</em> of the MIME database at runtime, that’s why new MIME types
can only be added at compile-time via configuration, but we’ll talk about
this option later…</p>
</blockquote>
<p>So, this means that we can add a MIME type like <code class="language-plaintext highlighter-rouge">application/wasm</code> for the
extension: <code class="language-plaintext highlighter-rouge">wasm</code>, which have been added to the <a href="https://www.iana.org/assignments/provisional-standard-media-types/provisional-standard-media-types.xhtml">provisional standard media
type registry</a> but is not official yet.</p>
<p>Via configuration you can do the following:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># file: config/config.exs</span>
<span class="kn">use</span> <span class="no">Mix</span><span class="o">.</span><span class="no">Config</span>
<span class="n">config</span> <span class="ss">:mime</span><span class="p">,</span> <span class="ss">:types</span><span class="p">,</span> <span class="p">%{</span><span class="s2">"application/wasm"</span> <span class="o">=></span> <span class="p">[</span><span class="s2">"wasm"</span><span class="p">]}</span>
</code></pre></div></div>
<p>Then, MIME <strong>needs to be recompiled</strong>, using <code class="language-plaintext highlighter-rouge">Mix</code> you can do the following:</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="go">mix deps.clean mime --build
mix deps.get
</span></code></pre></div></div>
<p>You can test the result via <code class="language-plaintext highlighter-rouge">IEx</code>:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">></span> <span class="no">MIME</span><span class="o">.</span><span class="n">type</span><span class="p">(</span><span class="s2">"wasm"</span><span class="p">)</span>
<span class="s2">"application/wasm"</span>
<span class="n">iex</span><span class="o">></span> <span class="no">MIME</span><span class="o">.</span><span class="n">extensions</span><span class="p">(</span><span class="s2">"application/wasm"</span><span class="p">)</span>
<span class="p">[</span><span class="s2">"wasm"</span><span class="p">]</span>
</code></pre></div></div>
<p>Now you may be wondering, how does it work? and that’s an excellent question,
let’s try to find an answer to that.</p>
<p>In the previous function declarations of <code class="language-plaintext highlighter-rouge">ext_to_mime/1</code> and <code class="language-plaintext highlighter-rouge">mime_to_ext/1</code>
I’ve omitted two function clauses on purpose, which are specifically related
with the custom types handling, let’s see the whole declaration for those two
functions now:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">@spec</span> <span class="n">ext_to_mime</span><span class="p">(</span><span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">())</span> <span class="p">::</span> <span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">()</span> <span class="o">|</span> <span class="no">nil</span>
<span class="k">defp</span> <span class="n">ext_to_mime</span><span class="p">(</span><span class="n">type</span><span class="p">)</span>
<span class="n">for</span> <span class="p">{</span><span class="n">type</span><span class="p">,</span> <span class="n">exts</span><span class="p">}</span> <span class="o"><-</span> <span class="n">custom_types</span><span class="p">,</span>
<span class="n">ext</span> <span class="o"><-</span> <span class="no">List</span><span class="o">.</span><span class="n">wrap</span><span class="p">(</span><span class="n">exts</span><span class="p">)</span> <span class="k">do</span>
<span class="k">defp</span> <span class="n">ext_to_mime</span><span class="p">(</span><span class="kn">unquote</span><span class="p">(</span><span class="n">ext</span><span class="p">)),</span> <span class="k">do</span><span class="p">:</span> <span class="kn">unquote</span><span class="p">(</span><span class="n">type</span><span class="p">)</span>
<span class="k">end</span>
<span class="n">for</span> <span class="p">{</span><span class="n">type</span><span class="p">,</span> <span class="n">exts</span><span class="p">}</span> <span class="o"><-</span> <span class="n">mapping</span><span class="p">,</span>
<span class="n">ext</span> <span class="o"><-</span> <span class="n">exts</span> <span class="k">do</span>
<span class="k">defp</span> <span class="n">ext_to_mime</span><span class="p">(</span><span class="kn">unquote</span><span class="p">(</span><span class="n">ext</span><span class="p">)),</span> <span class="k">do</span><span class="p">:</span> <span class="kn">unquote</span><span class="p">(</span><span class="n">type</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">ext_to_mime</span><span class="p">(</span><span class="n">_ext</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="no">nil</span>
<span class="nv">@spec</span> <span class="n">mime_to_ext</span><span class="p">(</span><span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">())</span> <span class="p">::</span> <span class="n">list</span><span class="p">(</span><span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">())</span> <span class="o">|</span> <span class="no">nil</span>
<span class="k">defp</span> <span class="n">mime_to_ext</span><span class="p">(</span><span class="n">type</span><span class="p">)</span>
<span class="n">for</span> <span class="p">{</span><span class="n">type</span><span class="p">,</span> <span class="n">exts</span><span class="p">}</span> <span class="o"><-</span> <span class="n">custom_types</span> <span class="k">do</span>
<span class="k">defp</span> <span class="n">mime_to_ext</span><span class="p">(</span><span class="kn">unquote</span><span class="p">(</span><span class="n">type</span><span class="p">)),</span> <span class="k">do</span><span class="p">:</span> <span class="kn">unquote</span><span class="p">(</span><span class="no">List</span><span class="o">.</span><span class="n">wrap</span><span class="p">(</span><span class="n">exts</span><span class="p">))</span>
<span class="k">end</span>
<span class="n">for</span> <span class="p">{</span><span class="n">type</span><span class="p">,</span> <span class="n">exts</span><span class="p">}</span> <span class="o"><-</span> <span class="n">mapping</span> <span class="k">do</span>
<span class="k">defp</span> <span class="n">mime_to_ext</span><span class="p">(</span><span class="kn">unquote</span><span class="p">(</span><span class="n">type</span><span class="p">)),</span> <span class="k">do</span><span class="p">:</span> <span class="kn">unquote</span><span class="p">(</span><span class="n">exts</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">mime_to_ext</span><span class="p">(</span><span class="n">_type</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="no">nil</span>
</code></pre></div></div>
<p>Now you can see that these custom MIME types come first. But wait a minute,
where is <code class="language-plaintext highlighter-rouge">custom_types</code> binding coming from?, well that’s the first argument of
the <code class="language-plaintext highlighter-rouge">MIME.Application.quoted/1</code> function.</p>
<p>There is another function that uses the <code class="language-plaintext highlighter-rouge">custom_types</code> binding, and that’s
<code class="language-plaintext highlighter-rouge">compiled_custom_types/0</code>, which as its name implies, returns the custom types
compiled into the MIME library.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">compiled_custom_types</span> <span class="k">do</span>
<span class="kn">unquote</span><span class="p">(</span><span class="no">Macro</span><span class="o">.</span><span class="n">escape</span><span class="p">(</span><span class="n">custom_types</span><span class="p">))</span>
<span class="k">end</span>
</code></pre></div></div>
<p>The last thing that I want to mention related to the function
<code class="language-plaintext highlighter-rouge">MIME.Application.quoted/1</code> is that this function returns a <em>quoted</em>
expression:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">quoted</span><span class="p">(</span><span class="n">custom_types</span><span class="p">)</span> <span class="k">do</span>
<span class="kn">quote</span> <span class="ss">bind_quoted:</span> <span class="p">[</span><span class="ss">custom_types:</span> <span class="no">Macro</span><span class="o">.</span><span class="n">escape</span><span class="p">(</span><span class="n">custom_types</span><span class="p">)]</span> <span class="k">do</span>
<span class="n">mime_file</span> <span class="o">=</span> <span class="no">Application</span><span class="o">.</span><span class="n">app_dir</span><span class="p">(</span><span class="ss">:mime</span><span class="p">,</span> <span class="s2">"priv/mime.types"</span><span class="p">)</span>
<span class="nv">@compile</span> <span class="ss">:no_native</span>
<span class="c1"># ...</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>You can see here that the <code class="language-plaintext highlighter-rouge">bind_quoted</code> option is passing a binding to the
macro, please keep in mind that the <code class="language-plaintext highlighter-rouge">bind_quoted</code> option is recommended every
time you want to inject a value into the <code class="language-plaintext highlighter-rouge">quote</code>.</p>
<p>If you execute the function <code class="language-plaintext highlighter-rouge">MIME.Application.quoted/1</code> in a <code class="language-plaintext highlighter-rouge">IEx</code> session you
will get something like this:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">></span> <span class="no">MIME</span><span class="o">.</span><span class="no">Application</span><span class="o">.</span><span class="n">quoted</span><span class="p">(%{})</span>
<span class="p">{</span><span class="ss">:__block__</span><span class="p">,</span> <span class="p">[],</span>
<span class="p">[</span>
<span class="p">{:</span><span class="o">=</span><span class="p">,</span> <span class="p">[],</span> <span class="p">[{</span><span class="ss">:custom_types</span><span class="p">,</span> <span class="p">[],</span> <span class="no">MIME</span><span class="o">.</span><span class="no">Application</span><span class="p">},</span> <span class="p">{</span><span class="ss">:%{}</span><span class="p">,</span> <span class="p">[],</span> <span class="p">[]}]},</span>
<span class="p">{</span><span class="ss">:__block__</span><span class="p">,</span> <span class="p">[],</span>
<span class="p">[</span>
<span class="p">{:</span><span class="err">@</span><span class="p">,</span> <span class="p">[</span><span class="ss">context:</span> <span class="no">MIME</span><span class="o">.</span><span class="no">Application</span><span class="p">,</span> <span class="kn">import</span><span class="p">:</span> <span class="no">Kernel</span><span class="p">],</span>
<span class="p">[</span>
<span class="p">{</span><span class="ss">:moduledoc</span><span class="p">,</span> <span class="p">[</span><span class="ss">context:</span> <span class="no">MIME</span><span class="o">.</span><span class="no">Application</span><span class="p">],</span>
<span class="c1"># ...</span>
</code></pre></div></div>
<p>So, our beloved <code class="language-plaintext highlighter-rouge">MIME.Application.quoted/1</code> function is actually returning an
Elixir data structure. But, who consumes that data structure? Let’s check the
<code class="language-plaintext highlighter-rouge">lib/mime.ex</code> file contents:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># file: lib/mime.ex</span>
<span class="n">quoted</span> <span class="o">=</span> <span class="no">MIME</span><span class="o">.</span><span class="no">Application</span><span class="o">.</span><span class="n">quoted</span><span class="p">(</span><span class="no">Application</span><span class="o">.</span><span class="n">get_env</span><span class="p">(</span><span class="ss">:mime</span><span class="p">,</span> <span class="ss">:types</span><span class="p">,</span> <span class="p">%{}))</span>
<span class="no">Module</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="no">MIME</span><span class="p">,</span> <span class="n">quoted</span><span class="p">,</span> <span class="n">__ENV__</span><span class="p">)</span>
</code></pre></div></div>
<p>Believe me, that’s all the content on <code class="language-plaintext highlighter-rouge">lib/mime.ex</code> at the moment. In the first
line, you can see a call to <code class="language-plaintext highlighter-rouge">MIME.Application.quoted/1</code> passing as argument the
custom MIME types defined via configuration or an empty map as a fallback, the
result of that invocation is stored in the <code class="language-plaintext highlighter-rouge">quoted</code> binding. Then, the second
line will create a module with the given name of <code class="language-plaintext highlighter-rouge">MIME</code> and it will be defined
by the previous <em>quoted expression</em>, keep in mind that the function
<code class="language-plaintext highlighter-rouge">Module.create/3</code>, compared with <code class="language-plaintext highlighter-rouge">Kernel.defmodule/2</code>, is preferred when the
module body is a <em>quoted expression</em> and another advantage is that
<code class="language-plaintext highlighter-rouge">Module.create/3</code> allow you to control the environment variables used when
defining the module.</p>
<h2 id="automatic-recompilation">Automatic recompilation</h2>
<p>When we started talking about how to add custom MIME types via configuration,
we also mentioned that <strong>we need to recompile the library</strong>. So, what happens
in the case you forget about doing that? Well, your changes will not take
effect until the dependency is manually recompiled, and before the release
<code class="language-plaintext highlighter-rouge">1.3.0</code> you didn’t see any <em>warning</em> about it.</p>
<p>Since version <code class="language-plaintext highlighter-rouge">1.3.0</code> of the MIME library, the recompilation process is
automatic if the compile-time database is out of date.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># file: lib/mime/application.ex</span>
<span class="k">defmodule</span> <span class="no">MIME</span><span class="o">.</span><span class="no">Application</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Application</span>
<span class="kn">require</span> <span class="no">Logger</span>
<span class="k">def</span> <span class="n">start</span><span class="p">(</span><span class="n">_</span><span class="p">,</span> <span class="n">_</span><span class="p">)</span> <span class="k">do</span>
<span class="n">app</span> <span class="o">=</span> <span class="no">Application</span><span class="o">.</span><span class="n">fetch_env!</span><span class="p">(</span><span class="ss">:mime</span><span class="p">,</span> <span class="ss">:types</span><span class="p">)</span>
<span class="k">if</span> <span class="n">app</span> <span class="o">!=</span> <span class="no">MIME</span><span class="o">.</span><span class="n">compiled_custom_types</span><span class="p">()</span> <span class="k">do</span>
<span class="no">Logger</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="sd">"""
The :mime library has been compiled with the following custom types:
#{inspect(MIME.compiled_custom_types())}
But it is being started with the following types:
#{inspect(app)}
We are going to dynamically recompile it during boot,
but please clean the :mime dependency to make sure it is recompiled:
$ mix deps.clean mime --build
"""</span><span class="p">)</span>
<span class="no">Module</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="no">MIME</span><span class="p">,</span> <span class="n">quoted</span><span class="p">(</span><span class="n">app</span><span class="p">),</span> <span class="n">__ENV__</span><span class="p">)</span>
<span class="k">end</span>
<span class="no">Supervisor</span><span class="o">.</span><span class="n">start_link</span><span class="p">([],</span> <span class="ss">strategy:</span> <span class="ss">:one_for_one</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">quoted</span><span class="p">(</span><span class="n">custom_types</span><span class="p">)</span> <span class="k">do</span>
<span class="c1"># ...</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>So, what this means is that the <code class="language-plaintext highlighter-rouge">MIME</code> library at boot-time, will log an error
and will try to dynamically recompile the <code class="language-plaintext highlighter-rouge">MIME</code> module if the custom mime
types of the user environment are different from the ones returned by
<code class="language-plaintext highlighter-rouge">MIME.compiled_custom_types/0</code>, which is great! but, as the log messages
says, it’s recommended to clean the <code class="language-plaintext highlighter-rouge">:mime</code> dependency to make sure it’s
recompiled.</p>
<h2 id="summary">Summary</h2>
<p>Elixir <code class="language-plaintext highlighter-rouge">MIME</code> is a short but powerful library, its goal is clear, and in just
around 200 SLOC you can see a lot of nice concepts, like <em>meta-programming</em>,
<em>file streams</em>, <em>pattern matching</em>, <em>macros</em>, <em>unquote fragments</em>, <em>dynamic
module creation</em>, <em>dynamic recompilation at boot-time</em>, among other really cool
stuff.</p>
<p>That’s all folks! Thanks for reading.</p>Milton Mazzarrime@milmazz.unoElixir’s MIME is a read-only and immutable library that embeds the MIME type database, so, users can map MIME (Multipurpose Internet Mail Extensions) types to extensions and vice-versa. It’s a really compact project and includes nice features, which I’ll try to explain in case you’re not familiar with the library. Then, I’ll focus on MIME’s internals or how was built, and also how MIME illustrates in an elegant way so many features of Elixir itself.Follow-up: Function currying in Elixir2017-09-19T16:00:00-05:002017-09-19T16:00:00-05:00https://milmazz.uno/article/2017/09/19/function-currying-in-elixir<blockquote>
<p><em>NOTE:</em> This article is a follow-up examination after the blog post
<a href="http://blog.patrikstorm.com/function-currying-in-elixir">Function currying in Elixir</a> by <a href="https://twitter.com/stormpat">@stormpat</a></p>
</blockquote>
<p>In his article, Patrik Storm, shows how to implement <em>function currying</em> in
Elixir, which could be really neat in some situations. For those who haven’t
read Patrik’s post, first, let us clarify what is <em>function currying</em>.</p>
<p><em>Currying</em> is the process of transforming a function that takes multiple
arguments (<em>arity</em>) into a function that takes <em>only one</em> argument and returns
another function if any arguments are still required. When the last required
argument is given, the function automatically executes and computes the result.
<!--more-->
As a first step, let us apply <em>function currying</em> manually:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="o">></span> <span class="n">greet</span> <span class="o">=</span> <span class="k">fn</span> <span class="n">greeting</span><span class="p">,</span> <span class="n">name</span> <span class="o">-></span> <span class="no">IO</span><span class="o">.</span><span class="n">puts</span> <span class="s2">"</span><span class="si">#{</span><span class="n">greeting</span><span class="si">}</span><span class="s2">, </span><span class="si">#{</span><span class="n">name</span><span class="si">}</span><span class="s2">"</span> <span class="k">end</span>
<span class="c1">#Function<12.52032458/2 in :erl_eval.expr/5></span>
<span class="n">iex</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span><span class="o">></span> <span class="n">greet</span><span class="o">.</span><span class="p">(</span><span class="s2">"Hello"</span><span class="p">,</span> <span class="s2">"John"</span><span class="p">)</span> <span class="c1"># uncurried function</span>
<span class="no">Hello</span><span class="p">,</span> <span class="no">John</span>
<span class="ss">:ok</span>
<span class="n">iex</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span><span class="o">></span> <span class="n">greetCurry</span> <span class="o">=</span> <span class="k">fn</span> <span class="n">greeting</span> <span class="o">-></span> <span class="k">fn</span> <span class="n">name</span> <span class="o">-></span> <span class="no">IO</span><span class="o">.</span><span class="n">puts</span> <span class="s2">"</span><span class="si">#{</span><span class="n">greeting</span><span class="si">}</span><span class="s2">, </span><span class="si">#{</span><span class="n">name</span><span class="si">}</span><span class="s2">"</span> <span class="k">end</span> <span class="k">end</span>
<span class="c1">#Function<6.52032458/1 in :erl_eval.expr/5></span>
<span class="n">iex</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span><span class="o">></span> <span class="n">greetCurry</span><span class="o">.</span><span class="p">(</span><span class="s2">"Hello"</span><span class="p">)</span><span class="o">.</span><span class="p">(</span><span class="s2">"John"</span><span class="p">)</span>
<span class="no">Hello</span><span class="p">,</span> <span class="no">John</span>
<span class="ss">:ok</span>
</code></pre></div></div>
<p>To get a general solution, Patrik uses a nice approach that combines <em>pattern
matching</em> and <em>tail-call optimization</em>, let’s dive into his implementation:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># file: curry.exs</span>
<span class="k">defmodule</span> <span class="no">Curry</span> <span class="k">do</span>
<span class="k">def</span> <span class="n">curry</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="n">_</span><span class="p">,</span> <span class="n">arity</span><span class="p">}</span> <span class="o">=</span> <span class="ss">:erlang</span><span class="o">.</span><span class="n">fun_info</span><span class="p">(</span><span class="n">fun</span><span class="p">,</span> <span class="ss">:arity</span><span class="p">)</span>
<span class="n">curry</span><span class="p">(</span><span class="n">fun</span><span class="p">,</span> <span class="n">arity</span><span class="p">,</span> <span class="p">[])</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">curry</span><span class="p">(</span><span class="n">fun</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">arguments</span><span class="p">)</span> <span class="k">do</span>
<span class="n">apply</span><span class="p">(</span><span class="n">fun</span><span class="p">,</span> <span class="no">Enum</span><span class="o">.</span><span class="n">reverse</span> <span class="n">arguments</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">curry</span><span class="p">(</span><span class="n">fun</span><span class="p">,</span> <span class="n">arity</span><span class="p">,</span> <span class="n">arguments</span><span class="p">)</span> <span class="k">do</span>
<span class="k">fn</span> <span class="n">arg</span> <span class="o">-></span> <span class="n">curry</span><span class="p">(</span><span class="n">fun</span><span class="p">,</span> <span class="n">arity</span> <span class="o">-</span> <span class="mi">1</span><span class="p">,</span> <span class="p">[</span><span class="n">arg</span> <span class="o">|</span> <span class="n">arguments</span><span class="p">])</span> <span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>The main points in this <code class="language-plaintext highlighter-rouge">Curry</code> module are the following:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">Curry.curry/1</code> represents our entry point, this function use
<a href="https://erlang.org/doc/man/erlang.html#fun_info-2"><code class="language-plaintext highlighter-rouge">:erlang.func_info/2</code></a> to know the arity (number of arguments) of the given
function <code class="language-plaintext highlighter-rouge">fun</code>. Then, we pass the control to the function <code class="language-plaintext highlighter-rouge">Curry.curry/3</code></li>
<li>The recursive function <code class="language-plaintext highlighter-rouge">Curry.curry/3</code> will return <em>anonymous functions</em> that
only takes just one argument.</li>
<li>When the last required argument is given we will use <a href="https://hexdocs.pm/elixir/master/Kernel.html#apply/2"><code class="language-plaintext highlighter-rouge">Kernel.apply/2</code></a> to
invoke the given function <code class="language-plaintext highlighter-rouge">fun</code> with the list of arguments <code class="language-plaintext highlighter-rouge">args</code>.</li>
</ul>
<p>Let’s show how we can use <em>function currying</em>, I’ll use the same examples
that Patrik did in his post but using <code class="language-plaintext highlighter-rouge">ExUnit</code> instead:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># file: curried.exs</span>
<span class="k">defmodule</span> <span class="no">Curried</span> <span class="k">do</span>
<span class="kn">import</span> <span class="no">Curry</span>
<span class="k">def</span> <span class="n">match</span> <span class="n">term</span> <span class="k">do</span>
<span class="n">curry</span><span class="p">(</span><span class="k">fn</span> <span class="n">what</span> <span class="o">-></span> <span class="p">(</span><span class="no">Regex</span><span class="o">.</span><span class="n">match?</span><span class="p">(</span><span class="n">term</span><span class="p">,</span> <span class="n">what</span><span class="p">))</span> <span class="k">end</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">filter</span> <span class="n">f</span> <span class="k">do</span>
<span class="n">curry</span><span class="p">(</span><span class="k">fn</span> <span class="n">list</span> <span class="o">-></span> <span class="no">Enum</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">list</span><span class="p">,</span> <span class="n">f</span><span class="p">)</span> <span class="k">end</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">replace</span> <span class="n">what</span> <span class="k">do</span>
<span class="n">curry</span><span class="p">(</span><span class="k">fn</span> <span class="n">replacement</span><span class="p">,</span> <span class="n">word</span> <span class="o">-></span>
<span class="no">Regex</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">what</span><span class="p">,</span> <span class="n">word</span><span class="p">,</span> <span class="n">replacement</span><span class="p">)</span>
<span class="k">end</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Our unit tests:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># file curry_test.exs</span>
<span class="no">ExUnit</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="no">Code</span><span class="o">.</span><span class="n">require_file</span><span class="p">(</span><span class="s2">"curry.exs"</span><span class="p">,</span> <span class="n">__DIR__</span><span class="p">)</span>
<span class="no">Code</span><span class="o">.</span><span class="n">require_file</span><span class="p">(</span><span class="s2">"curried.exs"</span><span class="p">,</span> <span class="n">__DIR__</span><span class="p">)</span>
<span class="k">defmodule</span> <span class="no">CurryTest</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">Case</span>
<span class="n">test</span> <span class="s2">"applying all the params at once or one step at a time should produce same results"</span> <span class="k">do</span>
<span class="n">curried</span> <span class="o">=</span> <span class="no">Curry</span><span class="o">.</span><span class="n">curry</span><span class="p">(</span><span class="k">fn</span> <span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">c</span><span class="p">,</span> <span class="n">d</span> <span class="o">-></span> <span class="n">a</span> <span class="o">*</span> <span class="n">b</span> <span class="o">+</span> <span class="n">div</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">d</span><span class="p">)</span> <span class="k">end</span><span class="p">)</span>
<span class="n">five_squared</span> <span class="o">=</span> <span class="n">curried</span><span class="o">.</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span><span class="o">.</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
<span class="n">assert</span> <span class="n">five_squared</span><span class="o">.</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span><span class="o">.</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span> <span class="o">==</span> <span class="n">curried</span><span class="o">.</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span><span class="o">.</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span><span class="o">.</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span><span class="o">.</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
<span class="k">end</span>
<span class="n">test</span> <span class="s2">"curry allow to create composable functions"</span> <span class="k">do</span>
<span class="n">has_spaces</span> <span class="o">=</span> <span class="no">Curried</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="sr">~r/\s+/</span><span class="p">)</span>
<span class="n">sentences</span> <span class="o">=</span> <span class="no">Curried</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">has_spaces</span><span class="p">)</span>
<span class="n">disallowed</span> <span class="o">=</span> <span class="no">Curried</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="sr">~r/[jruesbtni]/</span><span class="p">)</span>
<span class="n">censored</span> <span class="o">=</span> <span class="n">disallowed</span><span class="o">.</span><span class="p">(</span><span class="s2">"*"</span><span class="p">)</span>
<span class="n">allowed</span> <span class="o">=</span> <span class="n">sentences</span><span class="o">.</span><span class="p">([</span><span class="s2">"justin bibier"</span><span class="p">,</span> <span class="s2">"and sentences"</span><span class="p">,</span> <span class="s2">"are"</span><span class="p">,</span> <span class="s2">"allowed"</span><span class="p">])</span>
<span class="n">assert</span> <span class="s2">"****** ******"</span> <span class="o">==</span> <span class="n">allowed</span> <span class="o">|></span> <span class="no">List</span><span class="o">.</span><span class="n">first</span><span class="p">()</span> <span class="o">|></span> <span class="n">censored</span><span class="o">.</span><span class="p">()</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Now we can run our tests as follows:</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>elixir curry_test.exs
<span class="c">..
</span><span class="go">
Finished in 0.2 seconds (0.2s on load, 0.00s on tests)
2 tests, 0 failures
Randomized with seed 604000
</span></code></pre></div></div>
<p>It is working, but I feel we can improve a few things, in this case, our <code class="language-plaintext highlighter-rouge">curry</code>
function only takes into account that the arguments are given from left to
right. What about if we want to give the parameters from right to left? Let’s
introduce <code class="language-plaintext highlighter-rouge">curryRight</code>:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># file: curry.exs</span>
<span class="k">defmodule</span> <span class="no">Curry</span> <span class="k">do</span>
<span class="k">def</span> <span class="n">curry</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="ow">when</span> <span class="n">is_function</span><span class="p">(</span><span class="n">fun</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="n">curry</span><span class="p">(</span><span class="n">fun</span><span class="p">,</span> <span class="ss">:left</span><span class="p">)</span>
<span class="k">def</span> <span class="n">curryRight</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="ow">when</span> <span class="n">is_function</span><span class="p">(</span><span class="n">fun</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="n">curry</span><span class="p">(</span><span class="n">fun</span><span class="p">,</span> <span class="ss">:right</span><span class="p">)</span>
<span class="k">defp</span> <span class="n">curry</span><span class="p">(</span><span class="n">fun</span><span class="p">,</span> <span class="n">direction</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="n">_</span><span class="p">,</span> <span class="n">arity</span><span class="p">}</span> <span class="o">=</span> <span class="ss">:erlang</span><span class="o">.</span><span class="n">fun_info</span><span class="p">(</span><span class="n">fun</span><span class="p">,</span> <span class="ss">:arity</span><span class="p">)</span>
<span class="n">curry</span><span class="p">(</span><span class="n">fun</span><span class="p">,</span> <span class="n">arity</span><span class="p">,</span> <span class="p">[],</span> <span class="n">direction</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">curry</span><span class="p">(</span><span class="n">fun</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">args</span><span class="p">,</span> <span class="ss">:left</span><span class="p">)</span> <span class="k">do</span>
<span class="n">apply</span><span class="p">(</span><span class="n">fun</span><span class="p">,</span> <span class="no">Enum</span><span class="o">.</span><span class="n">reverse</span><span class="p">(</span><span class="n">args</span><span class="p">))</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">curry</span><span class="p">(</span><span class="n">fun</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">args</span><span class="p">,</span> <span class="ss">:right</span><span class="p">)</span> <span class="k">do</span>
<span class="n">apply</span><span class="p">(</span><span class="n">fun</span><span class="p">,</span> <span class="n">args</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">curry</span><span class="p">(</span><span class="n">fun</span><span class="p">,</span> <span class="n">arity</span><span class="p">,</span> <span class="n">args</span><span class="p">,</span> <span class="n">direction</span><span class="p">)</span> <span class="k">do</span>
<span class="o">&</span><span class="n">curry</span><span class="p">(</span><span class="n">fun</span><span class="p">,</span> <span class="n">arity</span> <span class="o">-</span> <span class="mi">1</span><span class="p">,</span> <span class="p">[</span><span class="nv">&1</span> <span class="o">|</span> <span class="n">args</span><span class="p">],</span> <span class="n">direction</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Then, our <code class="language-plaintext highlighter-rouge">Curried</code> module, which holds support functions, is much simpler if we
do the following:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># file: curried.exs</span>
<span class="k">defmodule</span> <span class="no">Curried</span> <span class="k">do</span>
<span class="kn">import</span> <span class="no">Curry</span>
<span class="k">def</span> <span class="n">match</span><span class="p">(</span><span class="n">term</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="n">curry</span><span class="p">(</span><span class="o">&</span><span class="no">Regex</span><span class="o">.</span><span class="n">match?</span><span class="o">/</span><span class="mi">2</span><span class="p">)</span><span class="o">.</span><span class="p">(</span><span class="n">term</span><span class="p">)</span>
<span class="k">def</span> <span class="n">filter</span><span class="p">(</span><span class="n">f</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="n">curryRight</span><span class="p">(</span><span class="o">&</span><span class="no">Enum</span><span class="o">.</span><span class="n">filter</span><span class="o">/</span><span class="mi">2</span><span class="p">)</span><span class="o">.</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="k">def</span> <span class="n">replace</span><span class="p">(</span><span class="n">what</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="n">curry</span><span class="p">(</span><span class="o">&</span><span class="no">Regex</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="nv">&1</span><span class="p">,</span> <span class="nv">&3</span><span class="p">,</span> <span class="nv">&2</span><span class="p">))</span><span class="o">.</span><span class="p">(</span><span class="n">what</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Now, without any change in our unit tests, we can verify that everything is
working as before.</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>elixir curry_test.exs
<span class="c">..
</span><span class="go">
Finished in 0.2 seconds (0.2s on load, 0.00s on tests)
2 tests, 0 failures
Randomized with seed 561000
</span></code></pre></div></div>
<h2 id="do-we-need-to-apply-curry-to-everything">Do we need to apply curry to everything?</h2>
<p>No, it will always depend of your case, first, let’s see how worse can be if
apply <em>currying</em> manually and then we will try to find another way to this whole
process as a <em>data transformation workflow</em>.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># file: manual_currying.exs</span>
<span class="k">defmodule</span> <span class="no">ManualCurrying</span> <span class="k">do</span>
<span class="k">def</span> <span class="n">match</span><span class="p">(</span><span class="n">term</span><span class="p">)</span> <span class="k">do</span>
<span class="k">fn</span> <span class="n">what</span> <span class="o">-></span> <span class="no">Regex</span><span class="o">.</span><span class="n">match?</span><span class="p">(</span><span class="n">term</span><span class="p">,</span> <span class="n">what</span><span class="p">)</span> <span class="k">end</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">filter</span><span class="p">(</span><span class="n">f</span><span class="p">)</span> <span class="k">do</span>
<span class="k">fn</span> <span class="n">list</span> <span class="o">-></span> <span class="no">Enum</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">list</span><span class="p">,</span> <span class="n">f</span><span class="p">)</span> <span class="k">end</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">replace</span><span class="p">(</span><span class="n">what</span><span class="p">)</span> <span class="k">do</span>
<span class="k">fn</span> <span class="n">replacement</span> <span class="o">-></span>
<span class="k">fn</span> <span class="n">word</span> <span class="o">-></span>
<span class="no">Regex</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">what</span><span class="p">,</span> <span class="n">word</span><span class="p">,</span> <span class="n">replacement</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Our unit tests:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># file manual_currying_test.exs</span>
<span class="no">ExUnit</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="no">Code</span><span class="o">.</span><span class="n">require_file</span><span class="p">(</span><span class="s2">"manual_currying.exs"</span><span class="p">,</span> <span class="n">__DIR__</span><span class="p">)</span>
<span class="k">defmodule</span> <span class="no">MunualCurryingTest</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">Case</span>
<span class="kn">import</span> <span class="no">ManualCurrying</span>
<span class="n">test</span> <span class="s2">"applying all the params at once or one step at a time should produce same results"</span> <span class="k">do</span>
<span class="n">curried</span> <span class="o">=</span>
<span class="k">fn</span> <span class="n">a</span> <span class="o">-></span>
<span class="k">fn</span> <span class="n">b</span> <span class="o">-></span>
<span class="k">fn</span> <span class="n">c</span> <span class="o">-></span>
<span class="k">fn</span> <span class="n">d</span> <span class="o">-></span>
<span class="n">a</span> <span class="o">*</span> <span class="n">b</span> <span class="o">+</span> <span class="n">div</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">d</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">five_squared</span> <span class="o">=</span> <span class="n">curried</span><span class="o">.</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span><span class="o">.</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
<span class="n">assert</span> <span class="n">five_squared</span><span class="o">.</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span><span class="o">.</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span> <span class="o">==</span> <span class="n">curried</span><span class="o">.</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span><span class="o">.</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span><span class="o">.</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span><span class="o">.</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
<span class="k">end</span>
<span class="n">test</span> <span class="s2">"curry allow to create composable functions"</span> <span class="k">do</span>
<span class="n">has_spaces</span> <span class="o">=</span> <span class="n">match</span><span class="p">(</span><span class="sr">~r/\s+/</span><span class="p">)</span>
<span class="n">sentences</span> <span class="o">=</span> <span class="n">filter</span><span class="p">(</span><span class="n">has_spaces</span><span class="p">)</span>
<span class="n">disallowed</span> <span class="o">=</span> <span class="n">replace</span><span class="p">(</span><span class="sr">~r/[jruesbtni]/</span><span class="p">)</span>
<span class="n">censored</span> <span class="o">=</span> <span class="n">disallowed</span><span class="o">.</span><span class="p">(</span><span class="s2">"*"</span><span class="p">)</span>
<span class="n">allowed</span> <span class="o">=</span> <span class="n">sentences</span><span class="o">.</span><span class="p">([</span><span class="s2">"justin bibier"</span><span class="p">,</span> <span class="s2">"and sentences"</span><span class="p">,</span> <span class="s2">"are"</span><span class="p">,</span> <span class="s2">"allowed"</span><span class="p">])</span>
<span class="n">assert</span> <span class="s2">"****** ******"</span> <span class="o">==</span> <span class="n">allowed</span> <span class="o">|></span> <span class="n">hd</span><span class="p">()</span> <span class="o">|></span> <span class="n">censored</span><span class="o">.</span><span class="p">()</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>But, if you just one to execute this just one time, maybe we can do better
thinking everything as a <em>data transformation workflow</em>, and actually this is
the more succint way:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="s2">"****** ******"</span> <span class="o">==</span>
<span class="p">[</span><span class="s2">"justin bibier"</span><span class="p">,</span> <span class="s2">"and sentences"</span><span class="p">,</span> <span class="s2">"are"</span><span class="p">,</span> <span class="s2">"allowed"</span><span class="p">]</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="o">&</span><span class="no">Regex</span><span class="o">.</span><span class="n">match?</span><span class="p">(</span><span class="sr">~r/\s+/</span><span class="p">,</span> <span class="nv">&1</span><span class="p">))</span>
<span class="o">|></span> <span class="n">hd</span><span class="p">()</span>
<span class="o">|></span> <span class="no">String</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="sr">~r/[jruesbtni]/</span><span class="p">,</span> <span class="s2">"*"</span><span class="p">)</span>
</code></pre></div></div>
<h2 id="wrapping-up">Wrapping up</h2>
<p><em>Function currying</em> is an interesting technique that allow us to reuse
functions, for example, we can create a module with small functions that behave
consistently without so much effort. Although, we need to keep in mind the
arguments order when we want to apply <code class="language-plaintext highlighter-rouge">function currying</code>. Sometimes for
functions like <a href="https://hexdocs.pm/elixir/master/Enum.html#map/2"><code class="language-plaintext highlighter-rouge">Enum.map/2</code></a>, <a href="https://hexdocs.pm/elixir/master/Enum.html#reduce/2"><code class="language-plaintext highlighter-rouge">Enum.reduce/2</code></a>, <a href="https://hexdocs.pm/elixir/master/Enum.html#filter/2"><code class="language-plaintext highlighter-rouge">Enum.filter/2</code></a>,
etc. it would be better or easier to use <code class="language-plaintext highlighter-rouge">curryRight</code> than <code class="language-plaintext highlighter-rouge">curry</code>, normally
our decision will depend on the arguments that will change constantly, because
we want to put those at the end of the execution path.</p>
<p>As a final note, it could be a interesting exercise to implement <code class="language-plaintext highlighter-rouge">uncurry</code>,
which is a function that converts a <em>curried function</em> to a function with
<em>arity</em> <em>n</em>, that way we can convert these two types in either direction.</p>
<h2 id="references">References</h2>
<ul>
<li><a href="http://blog.patrikstorm.com/function-currying-in-elixir">Function currying in Elixir</a> by <a href="https://twitter.com/stormpat">Patrik Storm</a></li>
<li><a href="https://erlang.org/doc/man/erlang.html#fun_info-2">:erlang.fun_info/2</a></li>
<li><a href="https://hexdocs.pm/elixir/master/Kernel.html#apply/2">Kernel.apply/2</a></li>
</ul>Milton Mazzarrime@milmazz.unoNOTE: This article is a follow-up examination after the blog post Function currying in Elixir by @stormpat In his article, Patrik Storm, shows how to implement function currying in Elixir, which could be really neat in some situations. For those who haven’t read Patrik’s post, first, let us clarify what is function currying. Currying is the process of transforming a function that takes multiple arguments (arity) into a function that takes only one argument and returns another function if any arguments are still required. When the last required argument is given, the function automatically executes and computes the result.Asynchronous Tasks with Elixir2016-09-03T03:00:00-05:002016-09-03T03:00:00-05:00https://milmazz.uno/article/2016/09/03/asynchronous-tasks-with-elixir<p>One of my first contributions into <a href="https://github.com/elixir-lang/ex_doc">ExDoc</a>, the tool used to produce HTML
documentation for Elixir projects, was to improve the documentation build
process performance. My first approach for this was to build each module page
concurrently, manually sending and receiving messages between processes. Then,
as you can see in the Pull Request details, <a href="https://github.com/ericmj">Eric Meadows-Jönsson</a>
pointed out that I should look at the <a href="http://elixir-lang.org/docs/stable/elixir/Task.html">Task</a> module. In this article, I’ll try
to show you the path that I followed to do that contribution.
<!--more-->
The original source code was something like this:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">run</span><span class="p">(</span><span class="n">modules</span><span class="p">,</span> <span class="n">config</span><span class="p">)</span> <span class="k">do</span>
<span class="c1"># ...</span>
<span class="n">generate_list</span><span class="p">(</span><span class="n">modules</span><span class="p">,</span> <span class="n">all</span><span class="p">,</span> <span class="n">output</span><span class="p">,</span> <span class="n">config</span><span class="p">,</span> <span class="n">has_readme</span><span class="p">)</span>
<span class="n">generate_list</span><span class="p">(</span><span class="n">exceptions</span><span class="p">,</span> <span class="n">all</span><span class="p">,</span> <span class="n">output</span><span class="p">,</span> <span class="n">config</span><span class="p">,</span> <span class="n">has_readme</span><span class="p">)</span>
<span class="n">generate_list</span><span class="p">(</span><span class="n">protocols</span><span class="p">,</span> <span class="n">all</span><span class="p">,</span> <span class="n">output</span><span class="p">,</span> <span class="n">config</span><span class="p">,</span> <span class="n">has_readme</span><span class="p">)</span>
<span class="c1"># ...</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">generate_list</span><span class="p">(</span><span class="n">nodes</span><span class="p">,</span> <span class="n">all</span><span class="p">,</span> <span class="n">output</span><span class="p">,</span> <span class="n">config</span><span class="p">,</span> <span class="n">has_readme</span><span class="p">)</span> <span class="k">do</span>
<span class="no">Enum</span><span class="o">.</span><span class="n">each</span> <span class="n">nodes</span><span class="p">,</span> <span class="o">&</span><span class="n">generate_module_page</span><span class="p">(</span><span class="nv">&1</span><span class="p">,</span> <span class="n">all</span><span class="p">,</span> <span class="n">output</span><span class="p">,</span> <span class="n">config</span><span class="p">,</span> <span class="n">has_readme</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">generate_module_page</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">modules</span><span class="p">,</span> <span class="n">output</span><span class="p">,</span> <span class="n">config</span><span class="p">,</span> <span class="n">has_readme</span><span class="p">)</span> <span class="k">do</span>
<span class="n">content</span> <span class="o">=</span> <span class="no">Templates</span><span class="o">.</span><span class="n">module_page</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">config</span><span class="p">,</span> <span class="n">modules</span><span class="p">,</span> <span class="n">has_readme</span><span class="p">)</span>
<span class="no">File</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s2">"</span><span class="si">#{</span><span class="n">output</span><span class="si">}</span><span class="s2">/</span><span class="si">#{</span><span class="n">node</span><span class="o">.</span><span class="n">id</span><span class="si">}</span><span class="s2">.html"</span><span class="p">,</span> <span class="n">content</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div></div>
<p>You can see that we can improve the build performance if we generate each module
page concurrently. So, let’s do that in a moment!</p>
<p>For the purposes of this article, let me simplify the example above. So, please
assume that the following was the original piece of code:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># source: demo.exs</span>
<span class="k">defmodule</span> <span class="no">AsyncTaskDemo</span> <span class="k">do</span>
<span class="k">def</span> <span class="n">run</span><span class="p">(</span><span class="n">nodes</span><span class="p">,</span> <span class="n">output</span><span class="p">)</span> <span class="k">do</span>
<span class="k">if</span> <span class="no">File</span><span class="o">.</span><span class="n">exists?</span> <span class="n">output</span> <span class="k">do</span>
<span class="no">File</span><span class="o">.</span><span class="n">rm_rf!</span> <span class="n">output</span>
<span class="k">end</span>
<span class="no">File</span><span class="o">.</span><span class="n">mkdir_p!</span> <span class="n">output</span>
<span class="n">generate_list</span><span class="p">(</span><span class="n">nodes</span><span class="p">,</span> <span class="n">output</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">generate_list</span><span class="p">(</span><span class="n">nodes</span><span class="p">,</span> <span class="n">output</span><span class="p">)</span> <span class="k">do</span>
<span class="no">Enum</span><span class="o">.</span><span class="n">each</span> <span class="n">nodes</span><span class="p">,</span> <span class="o">&</span><span class="n">generate_module_page</span><span class="p">(</span><span class="nv">&1</span><span class="p">,</span> <span class="n">output</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">generate_module_page</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">output</span><span class="p">)</span> <span class="k">do</span>
<span class="n">name</span> <span class="o">=</span> <span class="no">String</span><span class="o">.</span><span class="n">capitalize</span><span class="p">(</span><span class="n">node</span><span class="p">)</span>
<span class="n">content</span> <span class="o">=</span> <span class="no">EEx</span><span class="o">.</span><span class="n">eval_string</span> <span class="s2">"Hello <%= name %>"</span><span class="p">,</span> <span class="p">[</span><span class="ss">name:</span> <span class="n">name</span><span class="p">]</span>
<span class="no">File</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s2">"</span><span class="si">#{</span><span class="n">output</span><span class="si">}</span><span class="s2">/</span><span class="si">#{</span><span class="n">node</span><span class="si">}</span><span class="s2">.txt"</span><span class="p">,</span> <span class="n">content</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>As a second step, lets set up our test suite, in this case, we want to test a
single file <code class="language-plaintext highlighter-rouge">demo.exs</code>.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># source: async_test.exs</span>
<span class="no">ExUnit</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="no">Code</span><span class="o">.</span><span class="n">require_file</span><span class="p">(</span><span class="s2">"demo.exs"</span><span class="p">,</span> <span class="n">__DIR__</span><span class="p">)</span>
<span class="k">defmodule</span> <span class="no">AsyncTaskDemoTest</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">Case</span>
<span class="n">test</span> <span class="s2">"generate node pages"</span> <span class="k">do</span>
<span class="n">nodes</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"john"</span><span class="p">,</span> <span class="s2">"jane"</span><span class="p">]</span>
<span class="n">output</span> <span class="o">=</span> <span class="s2">"doc"</span>
<span class="no">AsyncTaskDemo</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">nodes</span><span class="p">,</span> <span class="n">output</span><span class="p">)</span>
<span class="n">files</span> <span class="o">=</span> <span class="no">File</span><span class="o">.</span><span class="n">ls!</span> <span class="n">output</span>
<span class="n">assert</span> <span class="n">files</span> <span class="o">==</span> <span class="p">[</span><span class="s2">"jane.txt"</span><span class="p">,</span> <span class="s2">"john.txt"</span><span class="p">]</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">for</span> <span class="n">f</span> <span class="o"><-</span> <span class="n">files</span> <span class="k">do</span>
<span class="no">File</span><span class="o">.</span><span class="n">read!</span> <span class="no">Path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">output</span><span class="p">,</span> <span class="n">f</span><span class="p">)</span>
<span class="k">end</span>
<span class="n">assert</span> <span class="n">result</span> <span class="o">==</span> <span class="p">[</span><span class="s2">"Hello Jane"</span><span class="p">,</span> <span class="s2">"Hello John"</span><span class="p">]</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>If we run our test suite we can see that everything is right:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>elixir async_test.exs
<span class="nb">.</span>
Finished <span class="k">in </span>0.1 seconds <span class="o">(</span>0.07s on load, 0.07s on tests<span class="o">)</span>
1 <span class="nb">test</span>, 0 failures
Randomized with seed 114000
</code></pre></div></div>
<p>Ok, now it’s time to introduce the concept of asynchronous tasks with
<a href="http://elixir-lang.org/docs/stable/elixir/Kernel.html#spawn/1"><code class="language-plaintext highlighter-rouge">Kernel.spawn/1</code></a>:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defp</span> <span class="n">generate_list</span><span class="p">(</span><span class="n">nodes</span><span class="p">,</span> <span class="n">output</span><span class="p">)</span> <span class="k">do</span>
<span class="no">Enum</span><span class="o">.</span><span class="n">each</span> <span class="n">nodes</span><span class="p">,</span> <span class="o">&</span><span class="n">generate_module_page_async</span><span class="p">(</span><span class="nv">&1</span><span class="p">,</span> <span class="n">output</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">generate_module_page_async</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">output</span><span class="p">)</span> <span class="k">do</span>
<span class="n">spawn</span><span class="p">(</span><span class="k">fn</span> <span class="o">-></span>
<span class="n">generate_module_page</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">output</span><span class="p">)</span>
<span class="k">end</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">generate_module_page</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">output</span><span class="p">)</span> <span class="k">do</span>
<span class="c1"># ...</span>
<span class="k">end</span>
</code></pre></div></div>
<p>At this point, you’ll notice that now <code class="language-plaintext highlighter-rouge">generate_list/2</code> calls a new function
that we named <code class="language-plaintext highlighter-rouge">generate_module_page_async/2</code>, this function will spawn new
processes, each process will generate a module page.</p>
<p>One problem with the earlier approach is that our program is not waiting for the
results of each invocation of the <code class="language-plaintext highlighter-rouge">generate_module_page/2</code> function. Basically,
we’re doing a <em>fire and forget</em> concurrent execution, this means that the caller
process doesn’t receive any feedback from the spawned function. If we run our
test we’ll see that is failing:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>elixir async_test.exs
1<span class="o">)</span> <span class="nb">test </span>generate node pages <span class="o">(</span>AsyncTaskDemoTest<span class="o">)</span>
async_test.exs:8
Assertion with <span class="o">==</span> failed
code: files <span class="o">==</span> <span class="o">[</span><span class="s2">"jane.txt"</span>, <span class="s2">"john.txt"</span><span class="o">]</span>
left: <span class="o">[]</span>
right: <span class="o">[</span><span class="s2">"jane.txt"</span>, <span class="s2">"john.txt"</span><span class="o">]</span>
stacktrace:
async_test.exs:15: <span class="o">(</span><span class="nb">test</span><span class="o">)</span>
Finished <span class="k">in </span>0.07 seconds <span class="o">(</span>0.05s on load, 0.02s on tests<span class="o">)</span>
1 <span class="nb">test</span>, 1 failure
Randomized with seed 47515
</code></pre></div></div>
<p>We can fix this error doing the following:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># source: demo.exs</span>
<span class="k">defp</span> <span class="n">generate_list</span><span class="p">(</span><span class="n">nodes</span><span class="p">,</span> <span class="n">output</span><span class="p">)</span> <span class="k">do</span>
<span class="n">nodes</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="o">&</span><span class="n">generate_module_page_async</span><span class="p">(</span><span class="nv">&1</span><span class="p">,</span> <span class="n">output</span><span class="p">))</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="k">fn</span> <span class="n">_</span> <span class="o">-></span>
<span class="k">receive</span> <span class="k">do</span>
<span class="ss">:ok</span> <span class="o">-></span> <span class="ss">:ok</span>
<span class="k">end</span>
<span class="k">end</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">generate_module_page_async</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">output</span><span class="p">)</span> <span class="k">do</span>
<span class="n">caller</span> <span class="o">=</span> <span class="n">self</span><span class="p">()</span>
<span class="n">spawn</span><span class="p">(</span><span class="k">fn</span> <span class="o">-></span>
<span class="n">send</span><span class="p">(</span><span class="n">caller</span><span class="p">,</span> <span class="n">generate_module_page</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">output</span><span class="p">))</span>
<span class="k">end</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">generate_module_page</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">output</span><span class="p">)</span> <span class="k">do</span>
<span class="c1"># ...</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Let’s run our tests:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>elixir async_test.exs
<span class="nb">.</span>
Finished <span class="k">in </span>0.09 seconds <span class="o">(</span>0.06s on load, 0.03s on tests<span class="o">)</span>
1 <span class="nb">test</span>, 0 failures
Randomized with seed 474778
</code></pre></div></div>
<p>Until now, we’re assuming that the <a href="http://elixir-lang.org/docs/stable/elixir/File.html#write/3"><code class="language-plaintext highlighter-rouge">File.write/3</code></a> always returns
<code class="language-plaintext highlighter-rouge">:ok</code>. If for some reason <a href="http://elixir-lang.org/docs/stable/elixir/File.html#write/3"><code class="language-plaintext highlighter-rouge">File.write/3</code></a> returns an <code class="language-plaintext highlighter-rouge">{:error,
reason}</code> message we’ll get stuck. One way to solve this issue is by doing the
following:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># source: demo.exs</span>
<span class="k">defp</span> <span class="n">generate_list</span><span class="p">(</span><span class="n">nodes</span><span class="p">,</span> <span class="n">output</span><span class="p">)</span> <span class="k">do</span>
<span class="n">nodes</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="o">&</span><span class="n">generate_module_page_async</span><span class="p">(</span><span class="nv">&1</span><span class="p">,</span> <span class="n">output</span><span class="p">))</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="k">fn</span> <span class="n">_</span> <span class="o">-></span>
<span class="k">receive</span> <span class="k">do</span>
<span class="ss">:ok</span> <span class="o">-></span> <span class="ss">:ok</span>
<span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="n">reason</span><span class="p">}</span> <span class="o">-></span> <span class="no">IO</span><span class="o">.</span><span class="n">puts</span> <span class="ss">:stderr</span><span class="p">,</span> <span class="s2">"</span><span class="si">#{</span><span class="n">reason</span><span class="si">}</span><span class="s2">"</span>
<span class="k">end</span>
<span class="k">end</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Finally, if we don’t receive any message at all, we set a timeout after 5
seconds:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">defp</span> <span class="n">generate_list</span><span class="p">(</span><span class="n">nodes</span><span class="p">,</span> <span class="n">output</span><span class="p">)</span> <span class="k">do</span>
<span class="n">nodes</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="o">&</span><span class="n">generate_module_page_async</span><span class="p">(</span><span class="nv">&1</span><span class="p">,</span> <span class="n">output</span><span class="p">))</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="k">fn</span> <span class="n">_</span> <span class="o">-></span>
<span class="k">receive</span> <span class="k">do</span>
<span class="ss">:ok</span> <span class="o">-></span> <span class="ss">:ok</span>
<span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="n">reason</span><span class="p">}</span> <span class="o">-></span> <span class="no">IO</span><span class="o">.</span><span class="n">puts</span> <span class="ss">:stderr</span><span class="p">,</span> <span class="s2">"</span><span class="si">#{</span><span class="n">reason</span><span class="si">}</span><span class="s2">"</span>
<span class="k">after</span> <span class="mi">5000</span> <span class="o">-></span>
<span class="no">IO</span><span class="o">.</span><span class="n">puts</span> <span class="ss">:stderr</span><span class="p">,</span> <span class="s2">"Timeout"</span>
<span class="k">end</span>
<span class="k">end</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div></div>
<p>With all these changes, we’re ready to send our Pull Request, but wait, there is
a better way to do this.</p>
<h2 id="elixir-way-task-module">Elixir way: Task Module</h2>
<p>As I mentioned before at the beginning of this article, <a href="https://github.com/ericmj">Eric</a> pointed
out that I should look at the <a href="http://elixir-lang.org/docs/stable/elixir/Task.html">Task</a> module documentation, and he was
absolutely right, this module offers a really good abstraction and now it’s
really easy to run simple processes.</p>
<p>Applying the <code class="language-plaintext highlighter-rouge">Task.async/1</code> to our earlier example we cut down our source code
to:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defp</span> <span class="n">generate_list</span><span class="p">(</span><span class="n">nodes</span><span class="p">,</span> <span class="n">output</span><span class="p">)</span> <span class="k">do</span>
<span class="n">nodes</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="o">&</span><span class="no">Task</span><span class="o">.</span><span class="n">async</span><span class="p">(</span><span class="k">fn</span> <span class="o">-></span>
<span class="n">generate_module_page</span><span class="p">(</span><span class="nv">&1</span><span class="p">,</span> <span class="n">output</span><span class="p">)</span>
<span class="k">end</span><span class="p">))</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="o">&</span><span class="no">Task</span><span class="o">.</span><span class="n">await</span><span class="o">/</span><span class="mi">1</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">Task.async/1</code> creates a separate process that runs the <code class="language-plaintext highlighter-rouge">generate_module_page/2</code>
function, then, we collect each task descriptor (returned by <code class="language-plaintext highlighter-rouge">Task.async/1</code>),
which is passed as the first value to <code class="language-plaintext highlighter-rouge">Task.await/2</code>, this call waits for our
background process to finish and returns its value, in this case, the result of
<code class="language-plaintext highlighter-rouge">File.write/3</code>.</p>
<p>You may ask yourself, <em>how is it that with the concurrent version we can improve
the overall performance?</em>, well, that depends, first we need to take into
account that our <em>concurrent program</em> will take advantage of a <em>parallel
computer</em> (several processing units), if we run our program on a computer with
only one CPU core, then, parallelism cannot happen.</p>
<p>Assume for a moment that the <code class="language-plaintext highlighter-rouge">generate_module_page</code> function always takes more
than 2 seconds:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">defp</span> <span class="n">generate_module_page</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">output</span><span class="p">)</span> <span class="k">do</span>
<span class="ss">:timer</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">2000</span><span class="p">)</span>
<span class="n">name</span> <span class="o">=</span> <span class="no">String</span><span class="o">.</span><span class="n">capitalize</span><span class="p">(</span><span class="n">node</span><span class="p">)</span>
<span class="n">content</span> <span class="o">=</span> <span class="no">EEx</span><span class="o">.</span><span class="n">eval_string</span> <span class="s2">"Hello <%= name %>"</span><span class="p">,</span> <span class="p">[</span><span class="ss">name:</span> <span class="n">name</span><span class="p">]</span>
<span class="no">File</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s2">"</span><span class="si">#{</span><span class="n">output</span><span class="si">}</span><span class="s2">/</span><span class="si">#{</span><span class="n">node</span><span class="si">}</span><span class="s2">.txt"</span><span class="p">,</span> <span class="n">content</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Then, with the following code we can test the performance improvements using a
<em>parallel computer</em>:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># performance.exs</span>
<span class="no">Code</span><span class="o">.</span><span class="n">require_file</span><span class="p">(</span><span class="s2">"demo.exs"</span><span class="p">,</span> <span class="n">__DIR__</span><span class="p">)</span>
<span class="n">nodes</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"egg"</span><span class="p">,</span> <span class="s2">"bacon"</span><span class="p">,</span> <span class="s2">"spam"</span><span class="p">,</span> <span class="s2">"sausage"</span><span class="p">,</span> <span class="s2">"beans"</span><span class="p">,</span> <span class="s2">"brandy"</span><span class="p">,</span> <span class="s2">"foo"</span><span class="p">,</span> <span class="s2">"baz"</span><span class="p">]</span>
<span class="n">output</span> <span class="o">=</span> <span class="s2">"doc"</span>
<span class="n">before</span> <span class="o">=</span> <span class="no">System</span><span class="o">.</span><span class="n">monotonic_time</span><span class="p">()</span>
<span class="no">AsyncTaskDemo</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">nodes</span><span class="p">,</span> <span class="n">output</span><span class="p">)</span>
<span class="n">later</span> <span class="o">=</span> <span class="no">System</span><span class="o">.</span><span class="n">monotonic_time</span><span class="p">()</span>
<span class="n">diff</span> <span class="o">=</span> <span class="n">later</span> <span class="o">-</span> <span class="n">before</span>
<span class="n">seconds</span> <span class="o">=</span> <span class="no">System</span><span class="o">.</span><span class="n">convert_time_unit</span><span class="p">(</span><span class="n">diff</span><span class="p">,</span> <span class="ss">:native</span><span class="p">,</span> <span class="ss">:seconds</span><span class="p">)</span>
<span class="no">IO</span><span class="o">.</span><span class="n">puts</span> <span class="s2">"Diff: </span><span class="si">#{</span><span class="n">seconds</span><span class="si">}</span><span class="s2"> seconds. </span><span class="si">#{</span><span class="n">diff</span><span class="si">}</span><span class="s2"> :native time unit"</span>
</code></pre></div></div>
<p>The results are the following:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Sequential</span>
<span class="nv">$ </span>elixir performance.exs
Diff: 16 seconds. 16122888704 :native <span class="nb">time </span>unit
<span class="c"># concurrent</span>
<span class="nv">$ </span>elixir performance.exs
Diff: 2 seconds. 2052834417 :native <span class="nb">time </span>unit
</code></pre></div></div>
<p>The result of our concurrent version is eightfold faster than the sequential
version :)</p>
<h2 id="wrapping-up">Wrapping up</h2>
<p>Is always good to know how concurrency works in Erlang & Elixir, where you can
create new lightweight processes with <code class="language-plaintext highlighter-rouge">spawn</code>, and then send/receive messages
to/from those processes, you can also use some abstractions given by OTP (<em>Open
Telecom Platform</em>), in general, that’s the way you can accomplish concurrency in
Erlang, but sometimes, you want to run simple processes, something like
background jobs, in those cases, is good to know about the <a href="http://elixir-lang.org/docs/stable/elixir/Task.html">Task</a> module,
which is a really good Elixir abstraction that keep us isolated from the details
and let’s concentrate on our goals.</p>
<p>As <a href="https://github.com/josevalim">José Valim</a> later tweeted, this was another entry on the “hard
things made easier with Elixir” series.</p>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Another entry on the "hard things made easier with Elixir" series: <a href="https://t.co/luQ8gJaBpE">https://t.co/luQ8gJaBpE</a> :)</p>— José Valim (@josevalim) <a href="https://twitter.com/josevalim/status/611451815616507904">June 18, 2015</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<h2 id="references">References</h2>
<ul>
<li><a href="http://elixir-lang.org/docs/stable/elixir/Kernel.html#spawn/1">Kernel.spawn/1</a></li>
<li><a href="http://elixir-lang.org/docs/stable/elixir/Task.html">Task module</a></li>
<li><a href="http://elixir-lang.org/docs/stable/elixir/File.html#write/3">File.write/3</a></li>
</ul>
<h2 id="acknowledgments">Acknowledgments</h2>
<p>Thank you to <a href="https://github.com/josevalim">José Valim</a>, <a href="https://github.com/sebasmagri">Sebastián Magrí</a> and Ana
Rangel for reviewing drafts of this post.</p>Milton Mazzarrime@milmazz.unoOne of my first contributions into ExDoc, the tool used to produce HTML documentation for Elixir projects, was to improve the documentation build process performance. My first approach for this was to build each module page concurrently, manually sending and receiving messages between processes. Then, as you can see in the Pull Request details, Eric Meadows-Jönsson pointed out that I should look at the Task module. In this article, I’ll try to show you the path that I followed to do that contribution.How to document your Javascript code2014-08-27T12:00:00-05:002014-08-27T12:00:00-05:00https://milmazz.uno/article/2014/08/27/how-to-document-your-javascript-code<p>Someone that knows something about Java probably knows about <a href="http://en.wikipedia.org/wiki/Javadoc">JavaDoc</a>. If you
know something about Python you probably document your code following the rules
defined for <a href="http://sphinx-doc.org">Sphinx</a> (Sphinx uses <a href="http://docutils.sf.net/rst.html">reStructuredText</a> as its markup
language). Or in C, you follow the rules defined for <a href="http://www.stack.nl/~dimitri/doxygen/">Doxygen</a> (Doxygen also
supports other programming languages such as Objective-C, Java, C#, PHP, etc.).
But, what happens when we are coding in JavaScript? How can we document our
source code?</p>
<p>As a developer that interacts with other members of a team, the need to document
all your intentions must become a habit. If you follow some basic rules and
stick to them you can gain benefits like the automatic generation of
documentation in formats like HTML, PDF, and so on.</p>
<p>I must confess that I’m relatively new to JavaScript, but one of the first
things that I implement is the source code documentation. I’ve been using
<a href="http://usejsdoc.org/">JSDoc</a> for documenting all my JavaScript code, it’s easy, and you only need
to follow a short set of rules.</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="cm">/**
* @file Working with Tags
* @author Milton Mazzarri <me@milmazz.uno>
* @version 0.1
*/</span>
<span class="kd">var</span> <span class="nx">Tag</span> <span class="o">=</span> <span class="nx">$</span><span class="p">(</span><span class="kd">function</span><span class="p">(){</span>
<span class="cm">/**
* The Tag definition.
*
* @param {String} id - The ID of the Tag.
* @param {String} description - Concise description of the tag.
* @param {Number} min - Minimum value accepted for trends.
* @param {Number} max - Maximum value accepted for trends.
* @param {Object} plc - The ID of the {@link PLC} object where this tag belongs.
*/</span>
<span class="kd">var</span> <span class="nx">Tag</span> <span class="o">=</span> <span class="kd">function</span><span class="p">(</span><span class="nx">id</span><span class="p">,</span> <span class="nx">description</span><span class="p">,</span> <span class="nx">min</span><span class="p">,</span> <span class="nx">max</span><span class="p">,</span> <span class="nx">plc</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">id</span> <span class="o">=</span> <span class="nx">id</span><span class="p">;</span>
<span class="nx">description</span> <span class="o">=</span> <span class="nx">description</span><span class="p">;</span>
<span class="nx">trend_min</span> <span class="o">=</span> <span class="nx">min</span><span class="p">;</span>
<span class="nx">trend_max</span> <span class="o">=</span> <span class="nx">max</span><span class="p">;</span>
<span class="nx">plc</span> <span class="o">=</span> <span class="nx">plc</span><span class="p">;</span>
<span class="p">};</span>
<span class="k">return</span> <span class="p">{</span>
<span class="cm">/**
* Get the current value of the tag.
*
* @see [Example]{@link http://example.com}
* @returns {Number} The current value of the tag.
*/</span>
<span class="na">getValue</span><span class="p">:</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="nb">Math</span><span class="p">.</span><span class="nx">random</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">};</span>
<span class="p">}());</span>
</code></pre></figure>
<p>In the previous example, I have documented the index of the file, showing the
author and version, you can also include other things such as a copyright and
license note. I have also documented the <em>class definition</em> including parameters
and methods specifying the name, and type with a concise description.</p>
<p>After you process your source code with <a href="http://usejsdoc.org/">JSDoc</a> the result looks like the
following:</p>
<p><img src="https://milmazz.uno/images/2014-08-27-how-to-document-your-javascript-code/jsdoc3.png" alt="usejsdoc" /></p>
<p>In the previous image you see the documentation in HTML format, also you see a
table that displays the parameters with appropriate links to your source code,
and finally, JSDoc implements a very nice style to your document.</p>
<p>If you need further details I recommend you check out the <a href="http://usejsdoc.org/index.html">JSDoc documentation</a>.</p>Milton Mazzarrime@milmazz.unoSomeone that knows something about Java probably knows about JavaDoc. If you know something about Python you probably document your code following the rules defined for Sphinx (Sphinx uses reStructuredText as its markup language). Or in C, you follow the rules defined for Doxygen (Doxygen also supports other programming languages such as Objective-C, Java, C#, PHP, etc.). But, what happens when we are coding in JavaScript? How can we document our source code? As a developer that interacts with other members of a team, the need to document all your intentions must become a habit. If you follow some basic rules and stick to them you can gain benefits like the automatic generation of documentation in formats like HTML, PDF, and so on. I must confess that I’m relatively new to JavaScript, but one of the first things that I implement is the source code documentation. I’ve been using JSDoc for documenting all my JavaScript code, it’s easy, and you only need to follow a short set of rules. /** * @file Working with Tags * @author Milton Mazzarri <me@milmazz.uno> * @version 0.1 */ var Tag = $(function(){ /** * The Tag definition. * * @param {String} id - The ID of the Tag. * @param {String} description - Concise description of the tag. * @param {Number} min - Minimum value accepted for trends. * @param {Number} max - Maximum value accepted for trends. * @param {Object} plc - The ID of the {@link PLC} object where this tag belongs. */ var Tag = function(id, description, min, max, plc) { id = id; description = description; trend_min = min; trend_max = max; plc = plc; }; return { /** * Get the current value of the tag. * * @see [Example]{@link http://example.com} * @returns {Number} The current value of the tag. */ getValue: function() { return Math.random; } }; }()); In the previous example, I have documented the index of the file, showing the author and version, you can also include other things such as a copyright and license note. I have also documented the class definition including parameters and methods specifying the name, and type with a concise description. After you process your source code with JSDoc the result looks like the following: In the previous image you see the documentation in HTML format, also you see a table that displays the parameters with appropriate links to your source code, and finally, JSDoc implements a very nice style to your document. If you need further details I recommend you check out the JSDoc documentation.Grunt: The Javascript Task Manager2014-06-28T16:00:00-05:002014-06-28T16:00:00-05:00https://milmazz.uno/article/2014/06/28/grunt-javascript-task-manager<p>When you play the Web Developer role, sometimes you may have to endure some
repetitive tasks like minification, unit testing, compilation, linting,
beautify or unpack Javascript code and so on. To solve this problems, and in
the meantime, try to keep your mental health in a good shape, you desperately
need to find a way to automate this tasks. <a href="http://gruntjs.com">Grunt</a> offers you an easy way to
accomplish this kind of automation.</p>
<!--more-->
<p>In this article I’ll try to explain how to automate some tasks with Grunt, but
I recommend that you should take some time to read Grunt’s documentation and
enjoy the experience by yourself.</p>
<p>So, in the following sections I’ll try to show you how to accomplish this
tasks:</p>
<ul>
<li>Concatenate and create a minified version of your CSS, JavaScript and HTML
files.</li>
<li>Automatic generation of the documentation for JavaScript with JSDoc.</li>
<li>Linting your JavaScript code.</li>
<li>Reformat and reindent (prettify) your JavaScript code.</li>
</ul>
<p>You can install <a href="http://gruntjs.com">Grunt</a> via <a href="https://www.npmjs.org">npm</a> (Node Package Manager), so, to
install Grunt you need to install <a href="http://nodejs.org">Node.js</a> first.</p>
<p>Now that you have <a href="http://nodejs.org">Node.js</a> and <code class="language-plaintext highlighter-rouge">npm</code> installed is a good time to install
<em>globally</em> the Grunt CLI (Command Line Interface) package.</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span><span class="nb">sudo </span>npm <span class="nb">install</span> <span class="nt">-g</span> grunt-cli</code></pre></figure>
<p>Once <code class="language-plaintext highlighter-rouge">grunt-cli</code> is installed you need to go to the root directory of your
project and create a <code class="language-plaintext highlighter-rouge">package.json</code> file, to accomplish this you can do the
following:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span><span class="nb">cd </span>example_project
<span class="nv">$ </span>npm init</code></pre></figure>
<p>The previous command will ask you a series of questions in order to create the
<code class="language-plaintext highlighter-rouge">package.json</code> file, <code class="language-plaintext highlighter-rouge">package.json</code> basically store metadata for projects
published as <code class="language-plaintext highlighter-rouge">npm</code> modules. It’s important to remember to add this file to your
source code versioning tool to ease the installation process of the development
dependencies among your partners via <code class="language-plaintext highlighter-rouge">npm install</code> command.</p>
<p>At this point we can install Grunt and their respective plugins in the
existing <code class="language-plaintext highlighter-rouge">package.json</code> with:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span>npm <span class="nb">install </span>grunt <span class="nt">--save-dev</span></code></pre></figure>
<p>And the plugins that you need can be installed as follows:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span>npm <span class="nb">install</span> <grunt-plugin-name> <span class="nt">--save-dev</span></code></pre></figure>
<p>Please note that the <code class="language-plaintext highlighter-rouge">--save-dev</code> parameter will change your <code class="language-plaintext highlighter-rouge">devDependencies</code>
section in your <code class="language-plaintext highlighter-rouge">package.json</code>. So, be sure to commit the updated <code class="language-plaintext highlighter-rouge">package.json</code>
file with your project whenever you consider appropriate.</p>
<h2 id="code-documentation">Code documentation</h2>
<p>If you document your code following the syntax rules defined on <a href="http://usejsdoc.org">JSDoc</a> 3,
e.g.:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"> <span class="cm">/**
* A callback function returning array defining where the ticks
* are laid out on the axis.
*
* @function tickPositioner
* @see {@link http://api.highcharts.com/highstock#yAxis.tickPositioner}
* @param {String} xOrY - Is it X or Y Axis?
* @param {Number} min - Minimum value
* @param {Number} max - Maximum value
* @returns {Array} - Where the ticks are laid out on the axis.
*/</span>
<span class="kd">function</span> <span class="nx">tickPositioner</span><span class="p">(</span><span class="nx">xOrY</span><span class="p">,</span> <span class="nx">min</span><span class="p">,</span> <span class="nx">max</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// do something</span>
<span class="k">return</span> <span class="nx">tickPositions</span><span class="p">;</span>
<span class="p">}</span></code></pre></figure>
<p>If you need more information about <a href="http://usejsdoc.org">JSDoc</a>, read their documentation, it’s
easy to catch up.</p>
<p>The next step to automate the generation of the code documentation is to install
first the <a href="https://github.com/krampstudio/grunt-jsdoc">grunt-jsdoc</a> plugin as follows:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span>npm <span class="nb">install </span>grunt-jsdoc <span class="nt">--save-dev</span></code></pre></figure>
<p>Once <code class="language-plaintext highlighter-rouge">grunt-jsdoc</code> is installed you must create your <code class="language-plaintext highlighter-rouge">Gruntfile.js</code> in the
root directory of your project and then add the <code class="language-plaintext highlighter-rouge">jsdoc</code> entry to the options
of the <code class="language-plaintext highlighter-rouge">initConfig</code> method.</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="nx">module</span><span class="p">.</span><span class="nx">exports</span> <span class="o">=</span> <span class="kd">function</span><span class="p">(</span><span class="nx">grunt</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// Project configuration.</span>
<span class="nx">grunt</span><span class="p">.</span><span class="nx">initConfig</span><span class="p">({</span>
<span class="na">pkg</span><span class="p">:</span> <span class="nx">grunt</span><span class="p">.</span><span class="nx">file</span><span class="p">.</span><span class="nx">readJSON</span><span class="p">(</span><span class="dl">'</span><span class="s1">package.json</span><span class="dl">'</span><span class="p">),</span>
<span class="na">jsdoc</span> <span class="p">:</span> <span class="p">{</span>
<span class="na">dist</span> <span class="p">:</span> <span class="p">{</span>
<span class="na">src</span><span class="p">:</span> <span class="p">[</span><span class="dl">'</span><span class="s1">src/*.js</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">test/*.js</span><span class="dl">'</span><span class="p">],</span>
<span class="na">dest</span><span class="p">:</span> <span class="dl">'</span><span class="s1">doc</span><span class="dl">'</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">});</span>
<span class="p">};</span></code></pre></figure>
<p>Then, you need to load the plugin after the <code class="language-plaintext highlighter-rouge">initConfig</code> method in the
<code class="language-plaintext highlighter-rouge">Gruntfile.js</code>:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="c1">// Load the plugin that provides the 'jsdoc' task.</span>
<span class="nx">grunt</span><span class="p">.</span><span class="nx">loadNpmtasks</span><span class="p">(</span><span class="dl">'</span><span class="s1">grunt-jsdoc</span><span class="dl">'</span><span class="p">);</span></code></pre></figure>
<p>The resulting <code class="language-plaintext highlighter-rouge">Gruntfile.js</code> until now is:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="nx">module</span><span class="p">.</span><span class="nx">exports</span> <span class="o">=</span> <span class="kd">function</span><span class="p">(</span><span class="nx">grunt</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// Project configuration.</span>
<span class="nx">grunt</span><span class="p">.</span><span class="nx">initConfig</span><span class="p">({</span>
<span class="na">pkg</span><span class="p">:</span> <span class="nx">grunt</span><span class="p">.</span><span class="nx">file</span><span class="p">.</span><span class="nx">readJSON</span><span class="p">(</span><span class="dl">'</span><span class="s1">package.json</span><span class="dl">'</span><span class="p">),</span>
<span class="na">jsdoc</span> <span class="p">:</span> <span class="p">{</span>
<span class="na">dist</span> <span class="p">:</span> <span class="p">{</span>
<span class="na">src</span><span class="p">:</span> <span class="p">[</span><span class="dl">'</span><span class="s1">src/*.js</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">test/*.js</span><span class="dl">'</span><span class="p">],</span>
<span class="na">dest</span><span class="p">:</span> <span class="dl">'</span><span class="s1">doc</span><span class="dl">'</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">});</span>
<span class="c1">// Load the plugin that provides the 'jsdoc' task.</span>
<span class="nx">grunt</span><span class="p">.</span><span class="nx">loadNpmtasks</span><span class="p">(</span><span class="dl">'</span><span class="s1">grunt-jsdoc</span><span class="dl">'</span><span class="p">);</span>
<span class="p">};</span></code></pre></figure>
<p>To generate the documentation, you need to call the <code class="language-plaintext highlighter-rouge">jsdoc</code> task as follows:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span>grunt jsdoc</code></pre></figure>
<p>Immediately you can see, inside the <code class="language-plaintext highlighter-rouge">doc</code> directory, the available documentation
in HTML format with some beautiful styles by default.</p>
<p><img src="https://milmazz.uno/images/grunt/jsdoc.png" alt="JSDoc" /></p>
<h2 id="linting-your-javascript-code">Linting your JavaScript code</h2>
<p>In order to find suspicious, non-portable or potential problems in JavaScript
code or simply to enforce your team’s coding convention, whatever may be the
reason, I recommend that you should include a static code analysis tool in
your toolset.</p>
<p>The first step is to define your set of rules. I prefer to specify the set of
rules in an independent file called <code class="language-plaintext highlighter-rouge">.jshintrc</code> located at the root directory of
the project, let’s see an example:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="c1">// file: .jshintrc</span>
<span class="p">{</span>
<span class="dl">"</span><span class="s2">globals</span><span class="dl">"</span><span class="p">:</span> <span class="p">{</span>
<span class="dl">"</span><span class="s2">Highcharts</span><span class="dl">"</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span> <span class="c1">// This is only necessary if you use Highcharts</span>
<span class="dl">"</span><span class="s2">module</span><span class="dl">"</span><span class="p">:</span> <span class="kc">true</span> <span class="c1">// Gruntfile.js</span>
<span class="p">},</span>
<span class="dl">"</span><span class="s2">bitwise</span><span class="dl">"</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">browser</span><span class="dl">"</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">camelcase</span><span class="dl">"</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">curly</span><span class="dl">"</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">eqeqeq</span><span class="dl">"</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">forin</span><span class="dl">"</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">freeze</span><span class="dl">"</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">immed</span><span class="dl">"</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">indent</span><span class="dl">"</span><span class="p">:</span> <span class="mi">4</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">jquery</span><span class="dl">"</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">latedef</span><span class="dl">"</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">newcap</span><span class="dl">"</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">noempty</span><span class="dl">"</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">nonew</span><span class="dl">"</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">quotmark</span><span class="dl">"</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">trailing</span><span class="dl">"</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">undef</span><span class="dl">"</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">unused</span><span class="dl">"</span><span class="p">:</span> <span class="kc">true</span>
<span class="p">}</span></code></pre></figure>
<p>If you need more details about the checks that offer every rule mentioned above,
please, read the page that contains a list of
<a href="http://www.jshint.com/docs/options/">all options supported by JSHint</a></p>
<p>To install the <a href="https://github.com/gruntjs/grunt-contrib-jshint">grunt-contrib-jshint</a> plugin, , please do as follows:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span>npm <span class="nb">install </span>grunt-contrib-jshint <span class="nt">--save-dev</span></code></pre></figure>
<p>Next, proceed to add the <code class="language-plaintext highlighter-rouge">jshint</code> entry to the options of the <code class="language-plaintext highlighter-rouge">initConfig</code>
method in the <code class="language-plaintext highlighter-rouge">Gruntfile.js</code> as follows:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="nx">jshint</span><span class="p">:</span> <span class="p">{</span>
<span class="nl">options</span><span class="p">:</span> <span class="p">{</span>
<span class="na">jshintrc</span><span class="p">:</span> <span class="dl">'</span><span class="s1">.jshintrc</span><span class="dl">'</span>
<span class="p">},</span>
<span class="nx">all</span><span class="p">:</span> <span class="p">[</span><span class="dl">'</span><span class="s1">Gruntfile.js</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">src/*.js</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">test/*.js</span><span class="dl">'</span><span class="p">]</span>
<span class="p">}</span></code></pre></figure>
<p>Then, as it’s done in the previous section, you need to load the plugin after
the <code class="language-plaintext highlighter-rouge">initConfig</code> method in the <code class="language-plaintext highlighter-rouge">Grunfile.js</code></p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="c1">// Load the plugin that provides the 'jshint' task.</span>
<span class="nx">grunt</span><span class="p">.</span><span class="nx">loadNpmTasks</span><span class="p">(</span><span class="dl">'</span><span class="s1">grunt-contrib-jshint</span><span class="dl">'</span><span class="p">);</span></code></pre></figure>
<p>To validate your JavaScript code against the previous set of rules you need to
call the <code class="language-plaintext highlighter-rouge">jshint</code> task:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span>grunt jshint</code></pre></figure>
<p>Note that if you need more information or explanations about the errors that you
may receive after the previous command I suggest that you should visit
<a href="http://jslinterrors.com">JSLint Error Explanations</a> site.</p>
<h2 id="code-style">Code Style</h2>
<p>As I mentioned in the previous section, I suggest that you should maintain an
independent file where you define the set of rules about your coding styles:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="c1">// file: .jsbeautifyrc</span>
<span class="p">{</span>
<span class="dl">"</span><span class="s2">indent_size</span><span class="dl">"</span><span class="p">:</span> <span class="mi">4</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">indent_char</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2"> </span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">indent_level</span><span class="dl">"</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">indent_with_tabs</span><span class="dl">"</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">preserve_newlines</span><span class="dl">"</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">max_preserve_newlines</span><span class="dl">"</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">jslint_happy</span><span class="dl">"</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">brace_style</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">collapse</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">keep_array_indentation</span><span class="dl">"</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">keep_function_indentation</span><span class="dl">"</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">space_before_conditional</span><span class="dl">"</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">break_chained_methods</span><span class="dl">"</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">eval_code</span><span class="dl">"</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">unescape_strings</span><span class="dl">"</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">wrap_line_length</span><span class="dl">"</span><span class="p">:</span> <span class="mi">0</span>
<span class="p">}</span></code></pre></figure>
<p>Next, proceed to add the <code class="language-plaintext highlighter-rouge">jsbeautifier</code> entry to the options of the <code class="language-plaintext highlighter-rouge">initConfig</code>
method in the <code class="language-plaintext highlighter-rouge">Gruntfile.js</code> as follows:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="nx">jsbeautifier</span><span class="p">:</span> <span class="p">{</span>
<span class="nl">modify</span><span class="p">:</span> <span class="p">{</span>
<span class="na">src</span><span class="p">:</span> <span class="dl">'</span><span class="s1">index.js</span><span class="dl">'</span><span class="p">,</span>
<span class="na">options</span><span class="p">:</span> <span class="p">{</span>
<span class="na">config</span><span class="p">:</span> <span class="dl">'</span><span class="s1">.jsbeautifyrc</span><span class="dl">'</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="nx">verify</span><span class="p">:</span> <span class="p">{</span>
<span class="nl">src</span><span class="p">:</span> <span class="p">[</span><span class="dl">'</span><span class="s1">index.js</span><span class="dl">'</span><span class="p">],</span>
<span class="nx">options</span><span class="p">:</span> <span class="p">{</span>
<span class="nl">mode</span><span class="p">:</span> <span class="dl">'</span><span class="s1">VERIFY_ONLY</span><span class="dl">'</span><span class="p">,</span>
<span class="nx">config</span><span class="p">:</span> <span class="dl">'</span><span class="s1">.jsbeautifyrc</span><span class="dl">'</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<p>The next step it to load the plugin after the <code class="language-plaintext highlighter-rouge">initConfig</code> method:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="c1">// Load the plugin that provides the 'jsbeautifier' task.</span>
<span class="nx">grunt</span><span class="p">.</span><span class="nx">loadNpmTasks</span><span class="p">(</span><span class="dl">'</span><span class="s1">grunt-jsbeautifier</span><span class="dl">'</span><span class="p">);</span></code></pre></figure>
<p>To adjust your JS files according to the previous set of rules you need to call
the <code class="language-plaintext highlighter-rouge">jsbeautifier</code> task:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span>grunt jsbeautifier:modify</code></pre></figure>
<h2 id="concat-and-minified-css-html-js">Concat and minified CSS, HTML, JS</h2>
<p>To reduce the size of your CSS, HTML and JS files do as follows:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span>npm <span class="nb">install </span>grunt-contrib-uglify <span class="nt">--save-dev</span>
<span class="nv">$ </span>npm <span class="nb">install </span>grunt-contrib-htmlmin <span class="nt">--save-dev</span>
<span class="nv">$ </span>npm <span class="nb">install </span>grunt-contrib-cssmin <span class="nt">--save-dev</span></code></pre></figure>
<p>Next, add the <code class="language-plaintext highlighter-rouge">htmlmin</code>, <code class="language-plaintext highlighter-rouge">cssmin</code> and <code class="language-plaintext highlighter-rouge">uglify</code> entries to the options
of the <code class="language-plaintext highlighter-rouge">initConfig</code> method in the <code class="language-plaintext highlighter-rouge">Gruntfile.js</code> as follows:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="nx">htmlmin</span><span class="p">:</span> <span class="p">{</span>
<span class="nl">dist</span><span class="p">:</span> <span class="p">{</span>
<span class="na">options</span><span class="p">:</span> <span class="p">{</span>
<span class="na">removeComments</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="na">collapseWhitespace</span><span class="p">:</span> <span class="kc">true</span>
<span class="p">},</span>
<span class="na">files</span><span class="p">:</span> <span class="p">{</span>
<span class="dl">'</span><span class="s1">dist/index.html</span><span class="dl">'</span><span class="p">:</span> <span class="dl">'</span><span class="s1">src/index.html</span><span class="dl">'</span><span class="p">,</span> <span class="c1">// 'destination': 'source'</span>
<span class="dl">'</span><span class="s1">dist/contact.html</span><span class="dl">'</span><span class="p">:</span> <span class="dl">'</span><span class="s1">src/contact.html</span><span class="dl">'</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="nx">cssmin</span><span class="p">:</span> <span class="p">{</span>
<span class="nl">add_banner</span><span class="p">:</span> <span class="p">{</span>
<span class="na">options</span><span class="p">:</span> <span class="p">{</span>
<span class="na">banner</span><span class="p">:</span> <span class="dl">'</span><span class="s1">/* My minified css file */</span><span class="dl">'</span>
<span class="p">},</span>
<span class="na">files</span><span class="p">:</span> <span class="p">{</span>
<span class="dl">'</span><span class="s1">path/to/output.css</span><span class="dl">'</span><span class="p">:</span> <span class="p">[</span><span class="dl">'</span><span class="s1">path/to/**/*.css</span><span class="dl">'</span><span class="p">]</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="nx">uglify</span><span class="p">:</span> <span class="p">{</span>
<span class="nl">options</span><span class="p">:</span> <span class="p">{</span>
<span class="na">banner</span><span class="p">:</span> <span class="dl">'</span><span class="s1">/* <%= grunt.template.today("yyyy-mm-dd") %> */</span><span class="se">\n</span><span class="dl">'</span><span class="p">,</span>
<span class="na">separator</span><span class="p">:</span> <span class="dl">'</span><span class="s1">,</span><span class="dl">'</span><span class="p">,</span>
<span class="na">compress</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="p">},</span>
<span class="nx">chart</span><span class="p">:</span> <span class="p">{</span>
<span class="nl">src</span><span class="p">:</span> <span class="p">[</span><span class="dl">'</span><span class="s1">src/js/*.js</span><span class="dl">'</span><span class="p">],</span>
<span class="nx">dest</span><span class="p">:</span> <span class="dl">'</span><span class="s1">dist/js/example.min.js</span><span class="dl">'</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<p>The next step it to load the plugins after the <code class="language-plaintext highlighter-rouge">initConfig</code> method:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="c1">// Load the plugin that provides the 'uglify' task.</span>
<span class="nx">grunt</span><span class="p">.</span><span class="nx">loadNpmTasks</span><span class="p">(</span><span class="dl">'</span><span class="s1">grunt-contrib-uglify</span><span class="dl">'</span><span class="p">);</span>
<span class="c1">// Load the plugin that provides the 'htmlmin' task.</span>
<span class="nx">grunt</span><span class="p">.</span><span class="nx">loadNpmTasks</span><span class="p">(</span><span class="dl">'</span><span class="s1">grunt-contrib-htmlmin</span><span class="dl">'</span><span class="p">);</span>
<span class="c1">// Load the plugin that provides the 'cssmin' task.</span>
<span class="nx">grunt</span><span class="p">.</span><span class="nx">loadNpmTasks</span><span class="p">(</span><span class="dl">'</span><span class="s1">grunt-contrib-cssmin</span><span class="dl">'</span><span class="p">);</span></code></pre></figure>
<p>To adjust your files according to the previous set of rules you need to call
the <code class="language-plaintext highlighter-rouge">htmlmin</code>, <code class="language-plaintext highlighter-rouge">cssmin</code> or <code class="language-plaintext highlighter-rouge">uglify</code> tasks:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span>grunt htmlmin
<span class="nv">$ </span>grunt cssmin
<span class="nv">$ </span>grunt uglify</code></pre></figure>
<p>After loading all of your Grunt tasks you can create some aliases.</p>
<p>If you want to create an alias that runs the three previous tasks in one step do
as follows:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="c1">// Wrapper around htmlmin, cssmin and uglify tasks.</span>
<span class="nx">grunt</span><span class="p">.</span><span class="nx">registerTask</span><span class="p">(</span><span class="dl">'</span><span class="s1">minified</span><span class="dl">'</span><span class="p">,</span> <span class="p">[</span>
<span class="dl">'</span><span class="s1">htmlmin</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">cssmin</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">uglify</span><span class="dl">'</span>
<span class="p">]);</span></code></pre></figure>
<p>So, to execute the three tasks in one step you can do the following:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span>grunt minified</code></pre></figure>
<p>The previous command is a wrapper around the <code class="language-plaintext highlighter-rouge">htmlmin</code>, <code class="language-plaintext highlighter-rouge">cssmin</code> and <code class="language-plaintext highlighter-rouge">uglify</code>
tasks defined before.</p>
<p>Wrapping all together we get the following [Grunfile.js][]</p>
<h2 id="other-plugins">Other plugins</h2>
<p>Grunt offers you a very large amount of <a href="http://gruntjs.com/plugins">plugins</a>, they have a dedicated
section to list them!</p>
<p>I recommend that you should take some time to check out the following
<em>plugins</em>:</p>
<ul>
<li><a href="https://www.npmjs.org/package/grunt-newer">grunt-newer</a> lets you configure Grunt tasks to run with newer files
only.</li>
<li><a href="https://www.npmjs.org/package/grunt-contrib-watch">grunt-contrib-watch</a> lets you run predefined tasks whenever file patterns
are added, changed or deleted.</li>
<li><a href="https://www.npmjs.org/package/grunt-contrib-imagemin">grunt-contrib-imagemin</a> let you minify images.</li>
</ul>
<h2 id="conclusion">Conclusion</h2>
<p>Certainly, Grunt do a great job when we talk about task automation, also,
automating this repetitive tasks is fun and reduce the errors associated with
tasks that in the past we used to run manually over and over again. That been
said, I definitely recommend that you should try Grunt, you will get in exchange
an improvement in your development workflow, also with it you can forget to
endure repetitive tasks that we hate to do, as a result you will get more
free time to focus on getting things done.</p>
<p>Last but not least, another option for Grunt is <a href="http://gulpjs.com">Gulp</a>, I recently read about
it, if you look at their documentation you notice that their syntax is more
clean than Grunt, also is very concise and allows chaining calls, following the
concept of streams that pipes offer in *nix like systems, so, I’ll give it a try
in my next project.</p>Milton Mazzarrime@milmazz.unoWhen you play the Web Developer role, sometimes you may have to endure some repetitive tasks like minification, unit testing, compilation, linting, beautify or unpack Javascript code and so on. To solve this problems, and in the meantime, try to keep your mental health in a good shape, you desperately need to find a way to automate this tasks. Grunt offers you an easy way to accomplish this kind of automation.The DRY principle2014-06-24T19:00:00-05:002014-06-24T19:00:00-05:00https://milmazz.uno/article/2014/06/24/the-dry-principle<p>The <a href="http://en.wikipedia.org/wiki/DRY_principle">DRY</a> (<em>Don’t Repeat Yourself</em>) principle it basically consist in the
following:</p>
<blockquote>
<p>Every piece of knowledge must have a single, unambiguous, authoritative
representation within a system.</p>
</blockquote>
<p>That said, it’s almost clear that the DRY principle is against the code
duplication, something that in the long-term affect the maintenance phase, it
doesn’t facilitate the improvement or code refactoring and, in some
cases, it can generate some contradictions, among other problems.
<!--more-->
Recently I have inherited a project, and one of the things that I noticed in
some part of the source code are the following:</p>
<script src="https://gist.github.com/milmazz/ed5d4aab90ea70deda9c.js?file=original.js"> </script>
<p>After a glance, you can save some bytes and apply the <em>facade</em> and <em>module</em>
pattern without breaking the API compatibility in this way:</p>
<script src="https://gist.github.com/milmazz/ed5d4aab90ea70deda9c.js?file=first_iteration.js"> </script>
<p>But, if you have read the <a href="http://jquery.com">jQuery</a> documentation it’s obvious that the
previous code portions are against the DRY principle, basically, this functions
expose some shorthands for the <code class="language-plaintext highlighter-rouge">$.ajax</code> method from the jQuery library. That
said, it’s important to clarify that <a href="http://jquery.com">jQuery</a>, from version 1.0, offers some
shorthands, they are: <code class="language-plaintext highlighter-rouge">$.get()</code>, <code class="language-plaintext highlighter-rouge">$.getJSON()</code>, <code class="language-plaintext highlighter-rouge">$.post()</code>, among others.</p>
<p>So, in this particular case, I prefer to break the backward compatibility and
delete the previous code portions. After that, I changed the code that used the
previous functions and from this point I only used the shorthand that jQuery
offers, some examples may clarify this thought:</p>
<script src="https://gist.github.com/milmazz/ed5d4aab90ea70deda9c.js?file=final.js"> </script>
<p>Another advantage of using the shorthand methods that jQuery provides is that
you can work with Deferreds, one last thing that we must take in consideration
is that <code class="language-plaintext highlighter-rouge">jqXHR.success()</code> and <code class="language-plaintext highlighter-rouge">jqXHR.error()</code> callback methods are deprecated
as of jQuery 1.8.</p>
<p>Anyway, I wanted to share my experience in this case. Also, remark that we need
to take care of some principles at the moment we develop software and avoid to
reinvent the wheel or do <em>overengineering</em>.</p>
<p>Last but not least, one way to read <em>offline</em> documentation that I tend to use
is <a href="http://kapeli.com/dash">Dash</a> or <a href="http://zealdocs.org">Zeal</a>, I can reach a bunch of documentation without the need
of an Internet connection, give it a try!</p>Milton Mazzarrime@milmazz.unoThe DRY (Don’t Repeat Yourself) principle it basically consist in the following: Every piece of knowledge must have a single, unambiguous, authoritative representation within a system. That said, it’s almost clear that the DRY principle is against the code duplication, something that in the long-term affect the maintenance phase, it doesn’t facilitate the improvement or code refactoring and, in some cases, it can generate some contradictions, among other problems.libturpial needs your help2014-03-10T16:00:00-05:002014-03-10T16:00:00-05:00https://milmazz.uno/article/2014/03/10/libturpial-needs-your-help<p>Do you want to begin to contribute into <a href="https://github.com/satanas/libturpial">libturpial</a> codebase but you don’t
know where to start?, that’s ok, it also happens to me sometimes, but today, the
reality is that we need your help to fix some errors reported by our style
checker (<a href="https://pypi.python.org/pypi/flake8">flake8</a>)<!--more-->, this errors are:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">E126</code>: continuation line over-indented for hanging indent</li>
<li><code class="language-plaintext highlighter-rouge">E128</code>: continuation line under-indented for visual indent</li>
<li><code class="language-plaintext highlighter-rouge">E231</code>: missing whitespace after <code class="language-plaintext highlighter-rouge">:</code></li>
<li><code class="language-plaintext highlighter-rouge">E251</code>: unexpected spaces around keyword / parameter equals</li>
<li><code class="language-plaintext highlighter-rouge">E261</code>: at least two spaces before inline comment</li>
<li><code class="language-plaintext highlighter-rouge">E301</code>: expected 1 blank line</li>
<li><code class="language-plaintext highlighter-rouge">E302</code>: expected 2 blank lines</li>
<li><code class="language-plaintext highlighter-rouge">E303</code>: too many blank lines</li>
<li><code class="language-plaintext highlighter-rouge">E501</code>: line too long</li>
<li><code class="language-plaintext highlighter-rouge">E711</code>: comparison to <code class="language-plaintext highlighter-rouge">None</code> should be <code class="language-plaintext highlighter-rouge">if cond is not None:</code></li>
<li><code class="language-plaintext highlighter-rouge">E712</code>: comparison to <code class="language-plaintext highlighter-rouge">False</code> should be <code class="language-plaintext highlighter-rouge">if cond is False:</code> or <code class="language-plaintext highlighter-rouge">if not cond:</code></li>
<li><code class="language-plaintext highlighter-rouge">F401</code>: imported but unused</li>
<li><code class="language-plaintext highlighter-rouge">F403</code>: unable to detect undefined names</li>
<li><code class="language-plaintext highlighter-rouge">F821</code>: undefined name</li>
<li><code class="language-plaintext highlighter-rouge">F841</code>: local variable is assigned to but never used</li>
<li><code class="language-plaintext highlighter-rouge">F999</code>: syntax error in doctest</li>
<li><code class="language-plaintext highlighter-rouge">W291</code>: trailing whitespace</li>
<li><code class="language-plaintext highlighter-rouge">W391</code>: blank line at end of file</li>
<li><code class="language-plaintext highlighter-rouge">W601</code>: <code class="language-plaintext highlighter-rouge">.has_key()</code> is deprecated, use <code class="language-plaintext highlighter-rouge">in</code></li>
</ul>
<p>As you can see, some errors are really easy to fix but are too many for us, so
please, we desperately need your help, help us!</p>
<p>The following is how we expect the community can contribute code into
<code class="language-plaintext highlighter-rouge">libturpial</code> via <a href="https://help.github.com/articles/using-pull-requests">Pull Requests</a>.</p>
<p>1) If you don’t have a <a href="https://github.com">Github</a> account, please create one first.</p>
<p>2) Fork our project</p>
<p>3) Install <a href="http://git-scm.com/download">Git</a>, after that you should setup your name and email:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"> <span class="nv">$ </span>git config <span class="nt">--global</span> user.name <span class="s2">"Your Name, not your Github nickname"</span>
<span class="nv">$ </span>git config <span class="nt">--global</span> user.email <span class="s2">"you@example.com"</span>
</code></pre></figure>
<p>4) Set your local repository</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"> <span class="nv">$ </span>git clone https://github.com/your_github_nickname/libturpial.git
</code></pre></figure>
<p>5) Set <code class="language-plaintext highlighter-rouge">satanas/libturpial</code> as your <code class="language-plaintext highlighter-rouge">upstream</code> remote, this basically tells Git
that your are referencing libturpial’s repository as your main source.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"> <span class="nv">$ </span>git remote add upstream https://github.com/satanas/libturpial.git
</code></pre></figure>
<p>5.1) Update your local copy</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"> <span class="nv">$ </span>git fetch upstream
</code></pre></figure>
<p>6) Verify that no one else has been working on the same bug, you can check that
in our <a href="https://github.com/satanas/libturpial/issues">issues</a> list, in this list you can also check
<a href="https://help.github.com/articles/using-pull-requests">Pull Requests</a> pending for the <a href="http://en.wikipedia.org/wiki/Benevolent_Dictator_for_Life">BDFL</a> approval.</p>
<p>7) Working on a bug</p>
<p>7.1) In the first place, install <a href="https://pypi.python.org/pypi/tox">tox</a></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"> <span class="nv">$ </span>pip <span class="nb">install </span>tox
</code></pre></figure>
<p>7.2) Then, create a branch that identifies the bug that you will begin to work
on:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"> <span class="nv">$ </span>git checkout <span class="nt">-b</span> E231 upstream/development
</code></pre></figure>
<p>In this example we are working on the bugs of the type: <strong>E231</strong> (as indicated
by our style checker, <code class="language-plaintext highlighter-rouge">flake8</code>)</p>
<p>7.3) Make some local changes and commit, repeat.</p>
<p>7.3.1) Delete the error code that you will begin to work from the <code class="language-plaintext highlighter-rouge">ignore</code> list
located at the <code class="language-plaintext highlighter-rouge">flake8</code> section in the <code class="language-plaintext highlighter-rouge">tox.ini</code> file (located at the root of
the project)</p>
<p>Example:</p>
<figure class="highlight"><pre><code class="language-ini" data-lang="ini"> <span class="c"># Original list:
</span> <span class="py">ignore</span> <span class="p">=</span> <span class="s">E126,E128,E231,E251,E261,E301</span>
<span class="c"># After we decide to work in the E231 error:
</span> <span class="py">ignore</span> <span class="p">=</span> <span class="s">E126,E128,E251,E261,E301</span>
</code></pre></figure>
<p>7.3.2) Execute <code class="language-plaintext highlighter-rouge">tox -e py27</code> to check the current errors.</p>
<p>7.3.3) Fix, fix, fix…</p>
<p>7.3.4) Commit your changes</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"> <span class="nv">$ </span>git commit <span class="nt">-m</span> <span class="s2">"Fixed errors 'E231' according to flake8."</span>
</code></pre></figure>
<p>7.4) In the case that you fixed all errors of the same type, please delete the
corresponding line in the <code class="language-plaintext highlighter-rouge">tox.ini</code> file (located at the root of the project)</p>
<p>7.4.1) Don’t forget to commit that.</p>
<p>8) Publish your work</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"> <span class="nv">$ </span>git push origin E231
</code></pre></figure>
<p>Please, adjust the name <code class="language-plaintext highlighter-rouge">E231</code> to something more appropiate in your case.</p>
<p>9) Create a Pull Request and don’t hesitate to bug the main maintainer until
your changes get merged and published. Last but not least, don’t forget to add
yourself as an author in the AUTHORS file, located at the root of the project.</p>
<p>10) Enjoy your work! :-)</p>
<p>If you want to help us more than you did already you can check our <a href="https://github.com/satanas/libturpial/issues">issues</a>
list, also, you can check out the <a href="https://github.com/satanas/Turpial">Turpial</a> project, our light, fast and
beautiful microblogging client written in Python, currently supports
<a href="https://twitter.com">Twitter</a>, and <a href="https://identi.ca">identi.ca</a>.</p>
<p>You can find more details on our <a href="http://libturpial.readthedocs.org/en/latest/">guide</a>, also, in case
of doubt don’t hesitate to reach us.</p>
<h2 id="about-libturpial">About libturpial</h2>
<p><em>libturpial</em> is a Python library that handles multiple microblogging protocols.
It implements a lot of features and aims to support all the features for each
protocol. At the moment it supports <a href="https://twitter.com">Twitter</a> and <a href="https://identi.ca">Identi.ca</a> and is the
backend used for <em>Turpial</em>.</p>
<h2 id="about-turpial">About Turpial</h2>
<p>Turpial is an alternative client for microblogging with multiple interfaces. At
the moment it supports Twitter and Identi.ca and works with Gtk and Qt
interfaces.</p>Milton Mazzarrime@milmazz.unoDo you want to begin to contribute into libturpial codebase but you don’t know where to start?, that’s ok, it also happens to me sometimes, but today, the reality is that we need your help to fix some errors reported by our style checker (flake8)