Improve the codebase of an acquired product

In this article I’ll share my experience improving the codebase of an acquired product, this couldn’t be possible without the help of a fantastic team. Before diving into the initial diagnostic and strategies that we took to tackle technical debt, I’ll share some background around the acquisition. Let’s start.

7 min read

Elixir’s MIME library review

Elixir’s MIME is a read-only and immutable library that embeds the MIME type database, so, users can map MIME (Multipurpose Internet Mail Extensions) types to extensions and vice-versa. It’s a really compact project and includes nice features, which I’ll try to explain in case you’re not familiar with the library. Then, I’ll focus on MIME’s internals or how was built, and also how MIME illustrates in an elegant way so many features of Elixir itself.

One of the goals, maybe the main one, of this library is to offer a performant lookup of the MIME database at runtime, that’s why new MIME types can only be added at compile-time via configuration, but we’ll talk about this option later. First, let’s review its public API.

API

MIME offers a short set of functions, which cover the most relevant cases when you work with MIME types.

Let’s review real quick the MIME library API, most of the examples were taken from the MIME’s documentation page.

extensions(String.t()) :: [String.t()]

Returns the extensions associated with the given MIME type.

iex> MIME.extensions("application/json")
["json"]
iex> MIME.extensions("foo/bar")
[]

type(String.t()) :: String.t()

Returns the MIME type related to the given file extension.

iex> MIME.type("txt")
"text/plain"

from_path(Path.t()) :: String.t()

Guesses the MIME type based on the path’s extension.

iex> MIME.from_path("index.html")
"text/html"

has_type?(String.t()) :: boolean

Returns whether an extension has a MIME type associated.

iex> MIME.has_type?("txt")
true
iex> MIME.has_type?("foobarbaz")
false

valid?(String.t()) :: boolean

Returns whether a MIME type is registered.

iex> MIME.valid?("text/plain")
true
iex> MIME.valid?("foo/bar")
false

Who is using MIME library?

At the time of this writing, and according to the statistics available from the Hex package manager, the MIME library has 21 dependents projects, among those projects you can find: Plug, Phoenix, Tesla, Swoosh, etc., and have been downloaded almost 6 million times. But more importantly, at least to me, is how the MIME library is implemented, its code is really concise, it’s around 200 SLOC (Source Lines Of Code) including comments, and embed captivating concepts.

How was the MIME library built?

Now, let’s start looking into how the MIME library was built.

Inside of the MIME.Application.quoted/1 function you can find the following section:

# file: lib/mime/application.ex
mime_file = Application.app_dir(:mime, "priv/mime.types")
@compile :no_native
@external_resource mime_file
stream = File.stream!(mime_file)

mapping =
  for line <- stream,
      not String.starts_with?(line, ["#", "\n"]),
      [type | exts] = line |> String.trim() |> String.split(),
      exts != [],
      do: {type, exts}

You can notice that the MIME library transforms the data located in the priv/mime.types file, which is a copy of the IANA (Internet Assigned Numbers Authority) database in text format and describes what Internet media types are sent to the client for the given file extension(s). Keep in mind that sending the correct media type to the client is important so they know how to handle the content of the file.

Here is an example of the priv/mime.types file content:

# file: priv/mime.types
# IANA types

# MIME type         Extensions
application/3gpp-ims+xml
application/ATXML       atxml
application/atom+xml        atom
application/atomcat+xml       atomcat
application/octet-stream    bin lha lzh exe class so dll img iso
application/pdf         pdf

To do the transformation, the MIME library reads the file line by line (via File.stream!/3) at compile time, and ignores empty lines or lines that start with a comment (#). After that, it removes the leading and trailing whitespace and then splits that line or string into a list of substrings, the head of this list represents the mime type and the tail of the list represents the extensions, that’s why at the end of the for comprehension you can see an extra filter to ignore mime types that does not have any extensions associated (e.g. application/3gpp-ims+xml), this is an optimization that reduces the compilation time. Finally, it creates a list of {type, extensions} tuples. The result of this transformation is stored in a binding called mapping.

Is important to note the usage of two Module attributes, the first one is @external_resource, which as its name implies, specifies an external resource for the given module, this attribute is used for tools like Mix to know if the current module needs to be recompiled in the case that any external resource is updated. Lastly, the @compile attribute defines options for the module compilation, this is used to configure both Elixir and Erlang compilers.

Once the MIME library has transformed the data and stored the result in mapping, it creates two private helper functions:

The first private helper function is ext_to_mime/1, which returns the MIME type given an extension:

@spec ext_to_mime(String.t()) :: String.t() | nil
defp ext_to_mime(type)

for {type, exts} <- mapping,
    ext <- exts do
  defp ext_to_mime(unquote(ext)), do: unquote(type)
end

defp ext_to_mime(_ext), do: nil

The MIME library creates thousands of ext_to_mime/1 function clauses inside of the for comprehension, this is a clear example of the power of meta-programming and how the MIME library relies on pattern matching to be performant. And this is possible because of Elixir quote and unquote mechanisms provide a feature called unquote fragments, that way is easy to create function on-the-fly at compile time.

To give you a better idea, here is a section of the final result:

defp ext_to_mime("atom"), do: "application/atom+xml"
defp ext_to_mime("pdf"), do: "application/pdf"
defp ext_to_mime("dll"), do: "application/octet-stream"
defp ext_to_mime("class"), do: "application/octet-stream"
# ...
defp ext_to_mime(_ext), do: nil

To complete the function declaration you see a catch-all function clause, which will be used if any match is not found.

The second private helper function declaration is mime_to_ext/1, this function expects a MIME type and will return a list of extensions or nil.

@spec mime_to_ext(String.t()) :: list(String.t()) | nil
defp mime_to_ext(type)

for {type, exts} <- mapping do
  defp mime_to_ext(unquote(type)), do: unquote(exts)
end

defp mime_to_ext(_type), do: nil

The result of the transformation should be similar to:

defp mime_to_ext("application/atom+xml"), do: ["atom"]
defp mime_to_ext("application/octet-stream"),
  do:  ["bin", "lha", "lzh", "exe", "class", "so", "dll", "img", "iso"]
defp mime_to_ext("application/pdf"), do: ["pdf"]
# ...
defp mime_to_ext(_type), do: nil

From here, is easy to build the public functions:

@spec valid?(String.t()) :: boolean
def valid?(type) do
  is_list(mime_to_ext(type))
end

def extensions(type) do
  mime_to_ext(type) || []
end

@default_type "application/octet-stream"

@spec type(String.t()) :: String.t()
def type(file_extension) do
  ext_to_mime(file_extension) || @default_type
end

def has_type?(file_extension) do
  is_binary(ext_to_mime(file_extension))
end

def from_path(path) do
  case Path.extname(path) do
    "." <> ext -> type(downcase(ext, ""))
    _ -> @default_type
  end
end

defp downcase(<<h, t::binary>>, acc) when h in ?A..?Z,
  do: downcase(t, <<acc::binary, h + 32>>)

defp downcase(<<h, t::binary>>, acc), do: downcase(t, <<acc::binary, h>>)
defp downcase(<<>>, acc), do: acc

That’s it!, at least with these functions, the MIME library cover the main features. But wait, there is more, do you remember that at the beginning I mentioned the following:

One of the goals, maybe the main one, of this library is to be provide a performant lookup of the MIME database at runtime, that’s why new MIME types can only be added at compile-time via configuration, but we’ll talk about this option later…

So, this means that we can add a MIME type like application/wasm for the extension: wasm, which have been added to the provisional standard media type registry but is not official yet.

Via configuration you can do the following:

# file: config/config.exs
use Mix.Config

config :mime, :types, %{"application/wasm" => ["wasm"]}

Then, MIME needs to be recompiled, using Mix you can do the following:

mix deps.clean mime --build
mix deps.get

You can test the result via IEx:

iex> MIME.type("wasm")
"application/wasm"
iex> MIME.extensions("application/wasm")
["wasm"]

Now you may be wondering, how does it work? and that’s an excellent question, let’s try to find an answer to that.

In the previous function declarations of ext_to_mime/1 and mime_to_ext/1 I’ve omitted two function clauses on purpose, which are specifically related with the custom types handling, let’s see the whole declaration for those two functions now:

@spec ext_to_mime(String.t()) :: String.t() | nil
defp ext_to_mime(type)

for {type, exts} <- custom_types,
    ext <- List.wrap(exts) do
  defp ext_to_mime(unquote(ext)), do: unquote(type)
end

for {type, exts} <- mapping,
    ext <- exts do
  defp ext_to_mime(unquote(ext)), do: unquote(type)
end

defp ext_to_mime(_ext), do: nil

@spec mime_to_ext(String.t()) :: list(String.t()) | nil
defp mime_to_ext(type)

for {type, exts} <- custom_types do
  defp mime_to_ext(unquote(type)), do: unquote(List.wrap(exts))
end

for {type, exts} <- mapping do
  defp mime_to_ext(unquote(type)), do: unquote(exts)
end

defp mime_to_ext(_type), do: nil

Now you can see that these custom MIME types come first. But wait a minute, where is custom_types binding coming from?, well that’s the first argument of the MIME.Application.quoted/1 function.

There is another function that uses the custom_types binding, and that’s compiled_custom_types/0, which as its name implies, returns the custom types compiled into the MIME library.

def compiled_custom_types do
  unquote(Macro.escape(custom_types))
end

The last thing that I want to mention related to the function MIME.Application.quoted/1 is that this function returns a quoted expression:

def quoted(custom_types) do
  quote bind_quoted: [custom_types: Macro.escape(custom_types)] do
    mime_file = Application.app_dir(:mime, "priv/mime.types")
    @compile :no_native
    # ...
  end
end

You can see here that the bind_quoted option is passing a binding to the macro, please keep in mind that the bind_quoted option is recommended every time you want to inject a value into the quote.

If you execute the function MIME.Application.quoted/1 in a IEx session you will get something like this:

iex> MIME.Application.quoted(%{})
{:__block__, [],
 [
   {:=, [], [{:custom_types, [], MIME.Application}, {:%{}, [], []}]},
   {:__block__, [],
    [
      {:@, [context: MIME.Application, import: Kernel],
       [
         {:moduledoc, [context: MIME.Application],
          # ...

So, our beloved MIME.Application.quoted/1 function is actually returning an Elixir data structure. But, who consumes that data structure? Let’s check the lib/mime.ex file contents:

# file: lib/mime.ex
quoted = MIME.Application.quoted(Application.get_env(:mime, :types, %{}))
Module.create(MIME, quoted, __ENV__)

Believe me, that’s all the content on lib/mime.ex at the moment. In the first line, you can see a call to MIME.Application.quoted/1 passing as argument the custom MIME types defined via configuration or an empty map as a fallback, the result of that invocation is stored in the quoted binding. Then, the second line will create a module with the given name of MIME and it will be defined by the previous quoted expression, keep in mind that the function Module.create/3, compared with Kernel.defmodule/2, is preferred when the module body is a quoted expression and another advantage is that Module.create/3 allow you to control the environment variables used when defining the module.

Automatic recompilation

When we started talking about how to add custom MIME types via configuration, we also mentioned that we need to recompile the library. So, what happens in the case you forget about doing that? Well, your changes will not take effect until the dependency is manually recompiled, and before the release 1.3.0 you didn’t see any warning about it.

Since version 1.3.0 of the MIME library, the recompilation process is automatic if the compile-time database is out of date.

# file: lib/mime/application.ex
defmodule MIME.Application do
  use Application
  require Logger

  def start(_, _) do
    app = Application.fetch_env!(:mime, :types)

    if app != MIME.compiled_custom_types() do
      Logger.error("""
      The :mime library has been compiled with the following custom types:

          #{inspect(MIME.compiled_custom_types())}

      But it is being started with the following types:

          #{inspect(app)}

      We are going to dynamically recompile it during boot,
      but please clean the :mime dependency to make sure it is recompiled:

          $ mix deps.clean mime --build

      """)

      Module.create(MIME, quoted(app), __ENV__)
    end

    Supervisor.start_link([], strategy: :one_for_one)
  end

  def quoted(custom_types) do
  # ...
  end
end

So, what this means is that the MIME library at boot-time, will log an error and will try to dynamically recompile the MIME module if the custom mime types of the user environment are different from the ones returned by MIME.compiled_custom_types/0, which is great! but, as the log messages says, it’s recommended to clean the :mime dependency to make sure it’s recompiled.

Summary

Elixir MIME is a short but powerful library, its goal is clear, and in just around 200 SLOC you can see a lot of nice concepts, like meta-programming, file streams, pattern matching, macros, unquote fragments, dynamic module creation, dynamic recompilation at boot-time, among other really cool stuff.

That’s all folks! Thanks for reading.

9 min read

Follow-up: Function currying in Elixir

NOTE: This article is a follow-up examination after the blog post Function currying in Elixir by @stormpat

In his article, Patrik Storm, shows how to implement function currying in Elixir, which could be really neat in some situations. For those who haven’t read Patrik’s post, first, let us clarify what is function currying.

Currying is the process of transforming a function that takes multiple arguments (arity) into a function that takes only one argument and returns another function if any arguments are still required. When the last required argument is given, the function automatically executes and computes the result.

As a first step, let us apply function currying manually:

iex(1)> greet = fn greeting, name -> IO.puts "#{greeting}, #{name}" end
#Function<12.52032458/2 in :erl_eval.expr/5>
iex(2)> greet.("Hello", "John") # uncurried function
Hello, John
:ok
iex(3)> greetCurry = fn greeting -> fn name -> IO.puts "#{greeting}, #{name}" end end
#Function<6.52032458/1 in :erl_eval.expr/5>
iex(4)> greetCurry.("Hello").("John")
Hello, John
:ok

To get a general solution, Patrik uses a nice approach that combines pattern matching and tail-call optimization, let’s dive into his implementation:

# file: curry.exs
defmodule Curry do
  def curry(fun) do
    {_, arity} = :erlang.fun_info(fun, :arity)
    curry(fun, arity, [])
  end

  def curry(fun, 0, arguments) do
    apply(fun, Enum.reverse arguments)
  end

  def curry(fun, arity, arguments) do
    fn arg -> curry(fun, arity - 1, [arg | arguments]) end
  end
end

The main points in this Curry module are the following:

  • Curry.curry/1 represents our entry point, this function use :erlang.func_info/2 to know the arity (number of arguments) of the given function fun. Then, we pass the control to the function Curry.curry/3
  • The recursive function Curry.curry/3 will return anonymous functions that only takes just one argument.
  • When the last required argument is given we will use Kernel.apply/2 to invoke the given function fun with the list of arguments args.

Let’s show how we can use function currying, I’ll use the same examples that Patrik did in his post but using ExUnit instead:

# file: curried.exs
defmodule Curried do
  import Curry

  def match term do
    curry(fn what -> (Regex.match?(term, what)) end)
  end

  def filter f do
    curry(fn list -> Enum.filter(list, f) end)
  end

  def replace what do
    curry(fn replacement, word ->
      Regex.replace(what, word, replacement)
    end)
  end
end

Our unit tests:

# file curry_test.exs
ExUnit.start()

Code.require_file("curry.exs", __DIR__)
Code.require_file("curried.exs", __DIR__)

defmodule CurryTest do
  use ExUnit.Case

  test "applying all the params at once or one step at a time should produce same results" do
    curried = Curry.curry(fn a, b, c, d -> a * b + div(c, d) end)
    five_squared = curried.(5).(5)

    assert five_squared.(10).(2) == curried.(5).(5).(10).(2)
  end

  test "curry allow to create composable functions" do
    has_spaces = Curried.match(~r/\s+/)
    sentences = Curried.filter(has_spaces)
    disallowed = Curried.replace(~r/[jruesbtni]/)
    censored = disallowed.("*")

    allowed = sentences.(["justin bibier", "and sentences", "are", "allowed"])

    assert "****** ******" == allowed |> List.first() |> censored.()
  end
end

Now we can run our tests as follows:

$ elixir curry_test.exs
..

Finished in 0.2 seconds (0.2s on load, 0.00s on tests)
2 tests, 0 failures

Randomized with seed 604000

It is working, but I feel we can improve a few things, in this case, our curry function only takes into account that the arguments are given from left to right. What about if we want to give the parameters from right to left? Let’s introduce curryRight:

# file: curry.exs
defmodule Curry do
  def curry(fun) when is_function(fun), do: curry(fun, :left)

  def curryRight(fun) when is_function(fun), do: curry(fun, :right)

  defp curry(fun, direction) do
    {_, arity} = :erlang.fun_info(fun, :arity)
    curry(fun, arity, [], direction)
  end

  defp curry(fun, 0, args, :left) do
    apply(fun, Enum.reverse(args))
  end

  defp curry(fun, 0, args, :right) do
    apply(fun, args)
  end

  defp curry(fun, arity, args, direction) do
    &curry(fun, arity - 1, [&1 | args], direction)
  end
end

Then, our Curried module, which holds support functions, is much simpler if we do the following:

# file: curried.exs
defmodule Curried do
  import Curry

  def match(term), do: curry(&Regex.match?/2).(term)

  def filter(f), do: curryRight(&Enum.filter/2).(f)

  def replace(what), do: curry(&Regex.replace(&1, &3, &2)).(what)
end

Now, without any change in our unit tests, we can verify that everything is working as before.

$ elixir curry_test.exs
..

Finished in 0.2 seconds (0.2s on load, 0.00s on tests)
2 tests, 0 failures

Randomized with seed 561000

Do we need to apply curry to everything?

No, it will always depend of your case, first, let’s see how worse can be if apply currying manually and then we will try to find another way to this whole process as a data transformation workflow.

# file: manual_currying.exs
defmodule ManualCurrying do
  def match(term) do
    fn what -> Regex.match?(term, what) end
  end

  def filter(f) do
    fn list -> Enum.filter(list, f) end
  end

  def replace(what) do
    fn replacement ->
      fn word ->
        Regex.replace(what, word, replacement)
      end
    end
  end
end

Our unit tests:

# file manual_currying_test.exs
ExUnit.start()

Code.require_file("manual_currying.exs", __DIR__)

defmodule MunualCurryingTest do
  use ExUnit.Case
  import ManualCurrying

  test "applying all the params at once or one step at a time should produce same results" do
    curried =
      fn a ->
        fn b ->
          fn c ->
            fn d ->
              a * b + div(c, d)
            end
          end
        end
      end

    five_squared = curried.(5).(5)

    assert five_squared.(10).(2) == curried.(5).(5).(10).(2)
  end

  test "curry allow to create composable functions" do
    has_spaces = match(~r/\s+/)
    sentences = filter(has_spaces)
    disallowed = replace(~r/[jruesbtni]/)
    censored = disallowed.("*")

    allowed = sentences.(["justin bibier", "and sentences", "are", "allowed"])

    assert "****** ******" == allowed |> hd() |> censored.()
  end
end

But, if you just one to execute this just one time, maybe we can do better thinking everything as a data transformation workflow, and actually this is the more succint way:

"****** ******" ==
  ["justin bibier", "and sentences", "are", "allowed"]
  |> Enum.filter(&Regex.match?(~r/\s+/, &1))
  |> hd()
  |> String.replace(~r/[jruesbtni]/, "*")

Wrapping up

Function currying is an interesting technique that allow us to reuse functions, for example, we can create a module with small functions that behave consistently without so much effort. Although, we need to keep in mind the arguments order when we want to apply function currying. Sometimes for functions like Enum.map/2, Enum.reduce/2, Enum.filter/2, etc. it would be better or easier to use curryRight than curry, normally our decision will depend on the arguments that will change constantly, because we want to put those at the end of the execution path.

As a final note, it could be a interesting exercise to implement uncurry, which is a function that converts a curried function to a function with arity n, that way we can convert these two types in either direction.

References

5 min read

Asynchronous Tasks with Elixir

One of my firsts contributions into ExDoc, the tool used to produce HTML documentation for Elixir projects, was to improve the documentation build process performance. My first approach for this was to build each module page concurrently, manually sending and receiving messages between processes. Then, as you can see in the Pull Request details, Eric Meadows-Jönsson pointed out that I should look at the Task module. In this article, I’ll try to show you the path that I followed to do that contribution.

The original source code was something like this:

def run(modules, config) do
  # ...
  generate_list(modules, all, output, config, has_readme)
  generate_list(exceptions, all, output, config, has_readme)
  generate_list(protocols, all, output, config, has_readme)
  # ...
end

defp generate_list(nodes, all, output, config, has_readme) do
  Enum.each nodes, &generate_module_page(&1, all, output, config, has_readme)
end

defp generate_module_page(node, modules, output, config, has_readme) do
  content = Templates.module_page(node, config, modules, has_readme)
  File.write("#{output}/#{node.id}.html", content)
end

You can see that we can improve the build performance if we generate each module page concurrently. So, let’s do that in a moment!

For the purposes of this article, let me simplify the example above. So, please assume that the following was the original piece of code:

# source: demo.exs
defmodule AsyncTaskDemo do
  def run(nodes, output) do
    if File.exists? output do
      File.rm_rf! output
    end
    File.mkdir_p! output

    generate_list(nodes, output)
  end

  defp generate_list(nodes, output) do
    Enum.each nodes, &generate_module_page(&1, output)
  end

  defp generate_module_page(node, output) do
    name = String.capitalize(node)
    content = EEx.eval_string "Hello <%= name %>", [name: name]
    File.write("#{output}/#{node}.txt", content)
  end
end

As a second step, lets set up our test suite, in this case, we want to test a single file demo.exs.

# source: async_test.exs
ExUnit.start()

Code.require_file("demo.exs", __DIR__)

defmodule AsyncTaskDemoTest do
  use ExUnit.Case

  test "generate node pages" do
    nodes = ["john", "jane"]
    output = "doc"
    AsyncTaskDemo.run(nodes, output)

    files = File.ls! output

    assert files == ["jane.txt", "john.txt"]

    result = for f <- files do
       File.read! Path.join(output, f)
    end

    assert result == ["Hello Jane", "Hello John"]
  end
end

If we run our test suite we can see that everything is right:

$ elixir async_test.exs
.

Finished in 0.1 seconds (0.07s on load, 0.07s on tests)
1 test, 0 failures

Randomized with seed 114000

Ok, now it’s time to introduce the concept of asynchronous tasks with Kernel.spawn/1:

defp generate_list(nodes, output) do
  Enum.each nodes, &generate_module_page_async(&1, output)
end

defp generate_module_page_async(node, output) do
  spawn(fn ->
    generate_module_page(node, output)
  end)
end

defp generate_module_page(node, output) do
  # ...
end

At this point, you’ll notice that now generate_list/2 calls a new function that we named generate_module_page_async/2, this function will spawn new processes, each process will generate a module page.

One problem with the earlier approach is that our program is not waiting for the results of each invocation of the generate_module_page/2 function. Basically, we’re doing a fire and forget concurrent execution, this means that the caller process doesn’t receive any feedback from the spawned function. If we run our test we’ll see that is failing:

$ elixir async_test.exs

  1) test generate node pages (AsyncTaskDemoTest)
     async_test.exs:8
     Assertion with == failed
     code:  files == ["jane.txt", "john.txt"]
     left:  []
     right: ["jane.txt", "john.txt"]
     stacktrace:
       async_test.exs:15: (test)

Finished in 0.07 seconds (0.05s on load, 0.02s on tests)
1 test, 1 failure

Randomized with seed 47515

We can fix this error doing the following:

# source: demo.exs
  defp generate_list(nodes, output) do
    nodes
    |> Enum.map(&generate_module_page_async(&1, output))
    |> Enum.map(fn _ ->
      receive do
        :ok -> :ok
      end
    end)
  end

  defp generate_module_page_async(node, output) do
    caller = self()
    spawn(fn ->
      send(caller, generate_module_page(node, output))
    end)
  end

  defp generate_module_page(node, output) do
    # ...
  end

Let’s run our tests:

$ elixir async_test.exs
.

Finished in 0.09 seconds (0.06s on load, 0.03s on tests)
1 test, 0 failures

Randomized with seed 474778

Until now, we’re assuming that the File.write/3 always returns :ok. If for some reason File.write/3 returns an {:error, reason} message we’ll get stuck. One way to solve this issue is by doing the following:

# source: demo.exs
  defp generate_list(nodes, output) do
    nodes
    |> Enum.map(&generate_module_page_async(&1, output))
    |> Enum.map(fn _ ->
      receive do
        :ok -> :ok
        {:error, reason} -> IO.puts :stderr, "#{reason}"
      end
    end)
  end

Finally, if we don’t receive any message at all, we set a timeout after 5 seconds:

  defp generate_list(nodes, output) do
    nodes
    |> Enum.map(&generate_module_page_async(&1, output))
    |> Enum.map(fn _ ->
      receive do
        :ok -> :ok
        {:error, reason} -> IO.puts :stderr, "#{reason}"
      after 5000 ->
        IO.puts :stderr, "Timeout"
      end
    end)
  end

With all these changes, we’re ready to send our Pull Request, but wait, there is a better way to do this.

Elixir way: Task Module

As I mentioned before at the beginning of this article, Eric pointed out that I should look at the Task module documentation, and he was absolutely right, this module offers a really good abstraction and now it’s really easy to run simple processes.

Applying the Task.async/1 to our earlier example we cut down our source code to:

defp generate_list(nodes, output) do
  nodes
  |> Enum.map(&Task.async(fn ->
       generate_module_page(&1, output)
     end))
  |> Enum.map(&Task.await/1)
end

Task.async/1 creates a separate process that runs the generate_module_page/2 function, then, we collect each task descriptor (returned by Task.async/1), which is passed as the first value to Task.await/2, this call waits for our background process to finish and returns its value, in this case, the result of File.write/3.

You may ask yourself, how is it that with the concurrent version we can improve the overall performance?, well, that depends, first we need to take into account that our concurrent program will take advantage of a parallel computer (several processing units), if we run our program on a computer with only one CPU core, then, parallelism cannot happen.

Assume for a moment that the generate_module_page function always takes more than 2 seconds:

  defp generate_module_page(node, output) do
    :timer.sleep(2000)
    name = String.capitalize(node)
    content = EEx.eval_string "Hello <%= name %>", [name: name]
    File.write("#{output}/#{node}.txt", content)
  end

Then, with the following code we can test the performance improvements using a parallel computer:

# performance.exs
Code.require_file("demo.exs", __DIR__)

nodes = ["egg", "bacon", "spam", "sausage", "beans", "brandy", "foo", "baz"]
output = "doc"

before = System.monotonic_time()

AsyncTaskDemo.run(nodes, output)

later = System.monotonic_time()
diff = later - before
seconds = System.convert_time_unit(diff, :native, :seconds)

IO.puts "Diff: #{seconds} seconds. #{diff} :native time unit"

The results are the following:

# Sequential
$ elixir performance.exs
Diff: 16 seconds. 16122888704 :native time unit
# concurrent
$ elixir performance.exs
Diff: 2 seconds. 2052834417 :native time unit

The result of our concurrent version is eightfold faster than the sequential version :)

Wrapping up

Is always good to know how concurrency works in Erlang & Elixir, where you can create new lightweight processes with spawn, and then send/receive messages to/from those processes, you can also use some abstractions given by OTP (Open Telecom Platform), in general, that’s the way you can accomplish concurrency in Erlang, but sometimes, you want to run simple processes, something like background jobs, in those cases, is good to know about the Task module, which is a really good Elixir abstraction that keep us isolated from the details and let’s concentrate on our goals.

As José Valim later tweeted, this was another entry on the “hard things made easier with Elixir” series.

References

Acknowledgments

Thank you to José Valim, Sebastián Magrí and Ana Rangel for reviewing drafts of this post.

6 min read

How to document your Javascript code

Someone that knows something about Java probably knows about JavaDoc. If you know something about Python you probably document your code following the rules defined for Sphinx (Sphinx uses reStructuredText as its markup language). Or in C, you follow the rules defined for Doxygen (Doxygen also supports other programming languages such as Objective-C, Java, C#, PHP, etc.). But, what happens when we are coding in JavaScript? How can we document our source code?

As a developer that interacts with other members of a team, the need to document all your intentions must become a habit. If you follow some basic rules and stick to them you can gain benefits like the automatic generation of documentation in formats like HTML, PDF, and so on.

I must confess that I’m relatively new to JavaScript, but one of the first things that I implement is the source code documentation. I’ve been using JSDoc for documenting all my JavaScript code, it’s easy, and you only need to follow a short set of rules.

/**
 * @file Working with Tags
 * @author Milton Mazzarri <[email protected]>
 * @version 0.1
 */

var Tag = $(function(){
  /**
   * The Tag definition.
   *
   * @param {String} id - The ID of the Tag.
   * @param {String} description - Concise description of the tag.
   * @param {Number} min - Minimum value accepted for trends.
   * @param {Number} max - Maximum value accepted for trends.
   * @param {Object} plc - The ID of the {@link PLC} object where this tag belongs.
   */
  var Tag = function(id, description, min, max, plc) {
    id = id;
    description = description;
    trend_min = min;
    trend_max = max;
    plc = plc;
  };

  return {
    /**
     * Get the current value of the tag.
     *
     * @see [Example]{@link http://example.com}
     * @returns {Number} The current value of the tag.
     */
    getValue: function() {
      return Math.random;
    }
  };
 }());
 

In the previous example, I have documented the index of the file, showing the author and version, you can also include other things such as a copyright and license note. I have also documented the class definition including parameters and methods specifying the name, and type with a concise description.

After you process your source code with JSDoc the result looks like the following:

usejsdoc

In the previous image you see the documentation in HTML format, also you see a table that displays the parameters with appropriate links to your source code, and finally, JSDoc implements a very nice style to your document.

If you need further details I recommend you check out the JSDoc documentation.

2 min read