The Lesson: ETS

The lesson I was delivering that day was about ETS tables. So eventually I get to the point where I explain how to use ets:first/1 and ets:next/1 to iterate over the keys of a table.

For those who don’t know, ets:first/1 …

Returns the first key Key in table Tab. For an ordered_set table, the first key in Erlang term order is returned. For other table types, the first key according to the internal order of the table is returned. If the table is empty, ‘$end_of_table’ is returned.

…while ets:next/2…

Returns the next key Key2, following key Key1 in table Tab. For table type ordered_set, the next key in Erlang term order is returned. For other table types, the next key according to the internal order of the table is returned. If no next key exists, ‘$end_of_table’ is returned.

The Question

As with a couple of other classical OTP functions (more examples below), the students spotted an edge case very easily. Precisely after I have shown my slide and before I could even get to the console to show them how these two functions can be used in a dummy example, one of them asked…

What happens if ‘$end_of_table’ is one of the keys in my table?

It didn’t actually take me by surprise. I’ve made the same kind of observation myself more than once. Each time I find a function that returns some special result (in this case ‘$end_of_table’) to indicate an abnormal status (in this case the fact that there is no next key), I wonder: what if that result is not so special after all? But even when I had seen this special result, I never checked what happened in this particular scenario.

My response to the student was, of course: Let’s see! And what was going to be a boring recursive function showing how to use ets:first/1 and ets:next/2 turned out to be a nice live coding session that eventually morphed into this blog post.

So What Happens?

Covering the Bases

First, we need to check the basic stuff:

1> spawn(fun() -> ets:new(wat, [named_table, public]), receive _ -> ok end end).

<0.63.0>

2> ets:first(wat).

'$end_of_table'

3> ets:next(wat, '$end_of_table').

** exception error: bad argument

in function ets:next/2

called as ets:next(wat,'$end_of_table')

6> ets:next(wat, a).

** exception error: bad argument

in function ets:next/2

called as ets:next(wat,a)

6> ets:insert(wat, {a, 1}).

true

7> ets:first(wat).

a

8> ets:next(wat, a).

'$end_of_table'

9> ets:next(wat, '$end_of_table').

** exception error: bad argument

in function ets:next/2

called as ets:next(wat,'$end_of_table')

10>

We first spawn a process to hold a public table called wat. We do that to avoid exceptions in the console killing our table (because ets tables are linked to the process who created them).

Then we check that everything works as expected:

If you ask for the first key of an empty table, you get ‘$end_of_table’.

If you ask for the next key of an inexistent key, you get an exception.

if you ask for the next key of the last key in your table, you get ‘$end_of_table’.

Adding the Strange Key

So far, so good. Now, let’s try with our crazy key ‘$end_of_table’ atom…

10> ets:insert(wat, {'$end_of_table', 2}).

true

11> ets:first(wat).

'$end_of_table'

12> ets:next(wat, '$end_of_table').

a

13> ets:next(wat, a).

'$end_of_table'

14>

Suddenly, our table is empty! (At least, according to the docs, right?). Well, to be fair, now ‘$end_of_table’ is actually a key in our table. So, what happens if we ask for the next key after it? Of course, we get the next one: a. And after a we get ‘$end_of_table’, since a is, in fact, the last key in that table.

Making Things Even More Complex

Then again, why is ‘$end_of_table’ the first key and not a? That is actually related to the internal order of the table (as stated in the docs). Let’s add a key that occupies a lower position than ‘$end_of_table’ to our table:

14> ets:insert(wat, {'_a', 0}).

true

15> ets:first(wat).

'_a'

16> ets:next(wat, '_a').

'$end_of_table'

17> ets:next(wat, '$end_of_table').

a

18> ets:next(wat, a).

'$end_of_table'

19>

As you can see, there is no way to be sure that ‘$end_of_table’ will be the first or the last element in your table, which might have been convenient (e.g. if you want to create a ring).

The Impact

Well, now that we’ve played for a bit, let’s see what can actually happen if we add an element to an ets table with ‘$end_of_table’ as the key…

There might be more, but I would like to show you 2 ways in which you can use ets:first/1 and ets:next/2 to traverse an ets table. Check them out in this handy module:

For matching/1,2 we let the iterator end when it finds the value ‘$end_of_table’ instead of a key.

For catching/1,2 we don’t rely on that value, we determine we reached the end of the table when there is no next key (that is, when ets:next/2 fails with a badarg error).

In the simplest scenario, both work as expected:

1> spawn(fun() -> ets:new(wat, [named_table, public]), receive _ -> ok end end).

<0.129.0>

2> iterator:catching(wat).

done

3> iterator:matching(wat).

done

4> ets:insert(wat, [{'_a', 0}, {b, 1}, {c, 2}]).

true

5> iterator:matching(wat).

'_a': [{'_a',0}]

b: [{b,1}]

c: [{c,2}]

done

6> iterator:catching(wat).

'_a': [{'_a',0}]

b: [{b,1}]

c: [{c,2}]

done

7>

Once we add ‘$end_of_table’, none of them work anymore:

7> ets:insert(wat, {'$end_of_table', 3}).

true

8> iterator:matching(wat).

'_a': [{'_a',0}]

done

9> iterator:catching(wat).

'_a': [{'_a',0}]

'$end_of_table': [{'$end_of_table',3}]

b: [{b,1}]

c: [{c,2}]

'$end_of_table': [{'$end_of_table',3}]

b: [{b,1}]

c: [{c,2}]

'$end_of_table': [{'$end_of_table',3}]

b: [{b,1}]

…

So, if you want to break a system and somehow you find out its turning user input into atoms, using those atoms as keys on a ets table and later traversing it using ets:first/1 and ets:next/2, now you know how to break it.