What’s going on here?

As usual: First try to find what’s happening on your own. I’ll wait…

OK, let me show you what I have found so far.

The first thing to notice is this rather obscure part of the Erlang documentation:

A list comprehension:

[Expr(E) || E <- List]

is basically translated to a local function:

'lc^0'([E|Tail], Expr) ->

[Expr(E)|'lc^0'(Tail, Expr)];

'lc^0'([], _Expr) -> [].

Although it’s not exactly the root cause of the behavior seen above, it looks like a promising starting point. My hypothesis is that the code that translates LCs into functions has some particularities on how it treats function applications in generators or filters. Let’s see if we can prove that…

Compiling a File

As we saw above, different things happen when the code is on a file or evaluated on a shell. Let’s first check how list comprehensions are translated when a module is compiled from a file.

In this case, x:c(). fails with case_clause , while x:d(). and x:e(nothing) just return an empty list. Adding to the weirdness, Erlang even produces warnings for d/0 but not for c/0 (see below).

The trick to understand what’s going on is to check how code looks like in core bsm, to do that we compile the module like this:

$ erlc +dcbsm x.erl

x.erl:6: Warning: no clause will ever match

x.erl:6: Warning: the guard for this clause evaluates to 'false'

Despite the warnings, that command generated the following file.

If we translate the important functions to something more Erlang-y, we get:

d() -> [].

In this case, the compiler is smart enough to know that the LC will always result in an empty list (hence the warnings above) and so the compiled code is just that: a function that returns [].

e(X) ->

case '<>' of

'<>' when X =:= true -> ['x'];

'<>' -> []

end.

…or…

e(X) ->

if X =:= true -> ['x']

; true -> []

end.

In other words, Erlang is transforming the LC in a simple case/if so that if X is true it returns a list with one element, otherwise it returns an empty list. More or less what we expected considering what we already knew so far.

c() ->

case id(nothing) of

true -> ['x'];

false -> []

% Generated code has another clause but that clause is the one

% added to case statements without catch-all clauses.

end.

In this case Erlang compiler uses pattern-matching directly on the result of the function application instead of using guards. And it explicitly matches on boolean values. Why? I don’t know. Second clause could’ve been a catch-all clause, to be consistent with the guard behavior.

Evaluating Expressions in the Shell

As we’ve seen a while back, the shell doesn’t compile expressions like the compiler. It evaluates them through erl_eval:expr(…).

With the same procedure we used previously, we’ll try to see how the expressions to be evaluated look like in our case. To achieve that, we put the expression in an anonymous function and use erlang:fun_info/1 on it.

1> F = fun() -> [x || nothing], [x || x:id(nothing)] end.

#Fun<erl_eval.20.99386804>

2> F().

** exception error: bad filter nothing

3> rp(erlang:fun_info(F)).

[{pid,<0.166.0>},

{module,erl_eval},

{new_index,20},

{new_uniq,<<189,144,182,154,187,236,207,96,89,18,34,161,

52,152,16,216>>},

{index,20},

{uniq,99386804},

{name,'-expr/5-fun-3-'},

{arity,0},

{env,[{[],

{eval,#Fun<shell.21.98304428>},

{value,#Fun<shell.5.98304428>},

[{clause,1,[],[],

[{lc,1,{atom,1,x},[{atom,1,nothing}]},

{lc,1,

{atom,1,x},

[{call,1,

{remote,1,{atom,1,x},{atom,1,id}},

[{atom,1,nothing}]}]}]}]}]},

{type,local}]

ok

I highlighted the important pieces, the ones that represent the two LCs. As you can see here LCs are not translated into cases or ifs or anything else, they have their own tuple representation.

Checking the code on erl_eval, we can see that lc tagged tuples are evaluated with eval_lc/6 . In particular, what matters to us is this clause of eval_lc1 :

eval_lc1(E, [F|Qs], Bs0, Lf, Ef, Acc) ->

CompFun = fun(Bs) -> eval_lc1(E, Qs, Bs, Lf, Ef, Acc) end,

eval_filter(F, Bs0, Lf, Ef, CompFun, Acc).

It will be evaluated once for each lc tuple because they both have just one F in their list of Qs (declarative variable naming was not a priority in erl_eval 🙄). Let’s see what eval_filter/6 looks like…

eval_filter(F, Bs0, Lf, Ef, CompFun, Acc) ->

case erl_lint:is_guard_test(F) of

true ->

case guard_test(F, Bs0, Lf, Ef) of

{value,true,Bs1} -> CompFun(Bs1);

{value,false,_} -> Acc

end;

false ->

case expr(F, Bs0, Lf, Ef, none) of

{value,true,Bs1} -> CompFun(Bs1);

{value,false,_} -> Acc;

{value,V,_} ->

erlang:raise(error, {bad_filter,V}, stacktrace())

end

end.

The tricky part is erl_lint:is_guard_test/2 : nothing is a guard, but x:id(nothing) is not. guard_test/4 evaluates guards within a try...catch statement and it always result in either true or false . But expressions that are not guards are evaluated recursively using expr/5 and therefore they may result in something that’s not a boolean (they might even raise an error). If expr/5 returns something that’s not a boolean, erl_eval raises a bad_filter error.

The question is once more: why this distinction? Why is eval_filter not implemented as follows?

eval_filter(F, Bs0, Lf, Ef, CompFun, Acc) ->

case erl_lint:is_guard_test(F) of

…

false ->

case expr(F, Bs0, Lf, Ef, none) of

{value,true,Bs1} -> CompFun(Bs1);

{value,_,_} -> Acc

end

end.

And… should that case be a try to precisely match the behavior of guard_test/4 ? should eval_filter/6 not distinguish between guards and regular expressions?

If you find out, let me know in the comments, please.

Final Words

In conclusion, while LCs are still one of my favorite features of Erlang, there are a couple of things to remember while working with them: