JTC1/SC22/WG21 N3853

Document number: N3853 Date: 2014-01-17 Project: Programming Language C++, Evolution Working Group Reply-to: Stephan T. Lavavej <stl@microsoft.com> Range-Based For-Loops: The Next Generation I. Introduction This is a proposal to add the syntax "for (elem : range)". In addition to being easier for novices to learn and being clearer for everyone to read, this will encourage correctness and efficiency. II. The Problem C++11's range-based for-loops are awesome: they're less verbose than traditional for-loops, they handle both arrays and containers in a generic and extensible manner, and they allow users to focus on the elements they're interested in instead of iterators/pointers/indices. Unfortunately, range-based for-loops are currently vulnerable to misuse: they require users to specify element types, and most users don't know how to do that perfectly, even with auto. "for (auto elem : range)" is very tempting and very bad. It produces "auto elem = *__begin;" (see 6.5.4 [stmt.ranged]/1), which copies each element, which is bad because: * It might not compile - for example, unique_ptr elements aren't copyable. This is problematic both for users who won't understand the resulting compiler errors, and for users writing generic code that'll happily compile until someone instantiates it for movable-only elements. * It might misbehave at runtime - for example, "elem = val;" will modify the copy, but not the original element in the range. Additionally, &elem will be invalidated after each iteration. * It might be inefficient - for example, unnecessarily copying std::string. These problems are effectively regressions from C++98/03, whose traditional iterator/pointer/index loops don't copy elements. "for (auto& elem : range)" is much better, although it's not quite perfect. It works with const/modifiable elements, and allows them to be observed/mutated in-place. However, it won't compile for proxy objects - for example, if the range is vector<bool>, then *__begin will be a temporary object returned by value (of type vector<bool>::reference), and auto& refuses to bind to temporaries. It also won't compile for "move ranges", see Q5 below. Again, this is problematic due to incomprehensible compiler errors and stealth build breaks. "for (const auto& elem : range)" observes elements in-place, and mostly works for proxy objects, but it obviously can't mutate elements in-place. Note that this isn't auto's fault, and manually specifying element types is equally bad or even worse: * "for (string elem : range)" still copies elements. * "for (auto& elem : range)" works with const elements (if the loop's body doesn't attempt to mutate them, of course), but "for (string& elem : range)" gives up that ability. * "for (const Elem& elem : range)" is pernicious if the Elem type is incorrectly specified. If the range is map<K, V>, then "for (const pair<K, V>& elem : range)" will convert-copy each element (triggering the previously-described correctness/efficiency problems) because the map's elements are actually pair<const K, V>. There are two reasons why it's easy to unintentionally copy elements. First, given "string s1 = s2;" or "string s3 = v[0];", every C++98/03 programmer can see the copies. Given "auto t1 = t2;" or "auto t3 = v[0];", it's reasonably simple for new-to-C++11 programmers to learn how to see the copies, because the syntax is similar. (It still must be learned that t3 is copied even if v[0] returns string&.) However, "for (auto elem : range)" and "for (string elem : range)" conceal elem's initialization, making it harder to see the copies. Second, although there are other contexts that conceal copies (e.g. taking parameters by value), range-based for-loops are special. Programmers never think of elements as being separate from their ranges. In traditional for-loops, *iter, *ptr, and ptr[idx] all refer to elements in-place. (Even proxy objects attempt to simulate this.) In the rare event that users want to copy elements while looping, they do so explicitly in the loop body. What should we do about this problem? We can teach users to write "for (auto& elem : range)" or "for (const auto& elem : range)", but they're imperfect (for proxies or mutation, respectively), and they require novices to be introduced to auto and references earlier than otherwise necessary. As explained immediately below, "for (auto&& elem : range)" is superior, but it's terrible to teach to novices. Even experienced C++98/03 programmers can find it difficult to learn how rvalue references, the template argument deduction tweak, and reference collapsing interact to produce what Scott Meyers calls "universal references" (see [1]), and auto&& is far less commonly seen than T&& in perfect forwarding. III. The Solution The most effective way to encourage users to write good code is to offer them the path of least resistance: writing less syntax. Therefore, this proposal introduces the syntax "for (elem : range)", with the semantics of "for (auto&& elem : range)". auto&& binds to everything (including proxies, but see Q3 below) without copying, solving the compiletime correctness, runtime correctness, and efficiency problems. "for (elem : range)" is easier for novices to learn, because it mentions only the fundamental topics of loops, elements, and ranges. More advanced topics like automatic type deduction, references, and rvalue references can be introduced at appropriate times, instead of having to be mentioned as soon as the novice has a container to loop through. "for (elem : range)" is also clearer for everyone to read, because it eliminates the visual noise of "auto&&" (or "auto&", or "const auto&"). In addition to reducing overall verbosity, making common cases terse has the bonus effect of making uncommon cases stand out due to their remaining verbosity. (For example, writing v.push_back(t) instead of v.insert(v.end(), t) makes any front-insertions or mid-insertions stand out.) It will still be possible to write range-based for-loops that copy elements (possibly to different types, e.g. widening uint8_t to uint64_t), but such unusual code will be more noticeable. IV. Impact On The Standard This proposal is a pure extension. It doesn't affect existing user code because "for (elem : range)" was previously ill-formed. (Even if elem had already been declared, the grammar simply did not permit a plain identifier to appear before the colon.) I am not aware of any interactions between this proposal and other proposals. V. Standardese 1. In 6.5 [stmt.iter]/1 and A.5 [gram.stmt], after: iteration-statement: [...] for ( for-range-declaration : for-range-initializer ) statement add: for ( identifier : for-range-initializer ) statement 2. At the end of 6.5.4 [stmt.ranged], add a new paragraph: A range-based for statement of the form for ( identifier : for-range-initializer ) statement is equivalent to for ( auto&& identifier : for-range-initializer ) statement VI. Questions And Answers Q1. What about constness? A1. This is controlled by the range's type. If the range is vector<int>, elem will be modifiable. If the range is "const vector<int>", elem will be const. If the range is a braced-init-list, elem will be const (because initializer_list<E>::begin() returns "const E *", 18.9 [support.initlist]/1). Users with modifiable ranges who really want to view their elements as const can continue to say "for (const auto& elem : range)". Q2. What's decltype(elem)? A2. decltype observes the "declared type" after template argument deduction and reference collapsing. range: vector<int> ; decltype(elem): int& range: const vector<int> ; decltype(elem): const int& range: { "cat"s, "dog"s } ; decltype(elem): const string& range: vector<bool> ; decltype(elem): vector<bool>::reference&& range: const vector<bool> ; decltype(elem): bool&& Q3. Oh, that's interesting. Tell me more about proxies. A3. That's not a question, but proxies are the most difficult thing to deal with here. auto&& binds to proxy objects, effectively suspending them in midair. This allows vector<bool> elements to be read from and written to, and "const vector<bool>" elements to be read from. Most user code will work. However, decltype(elem) above indicates how this breaks down slightly in certain situations. First, for vector<bool>, &elem will compile and point to a proxy object that will be destroyed after each iteration. Second, for "const vector<bool>", in addition to &elem also compiling and being invalidated, "elem = false;" will compile (and modify the temporary bool suspended in midair, not the container; if vector<bool>::const_reference was a separate proxy type instead of bool, this wouldn't compile). Argh. Note that *vb.begin() is a prvalue, so &*vb.begin() won't compile, and "*vb.begin() = false;" won't compile for "const vector<bool>" (see [2]). These issues are caused by replacing a prvalue *__begin with an lvalue elem, so this is inherent to the range-based for-loop as specified by C++11, where the for-range-declaration introduces a named variable for the element. Q4. What can we do about proxies? A4. If the EWG considers these issues with proxies to be important to solve, there are a few options. (Note that here I'm considering alternative semantics for the new syntax in this proposal. I am not suggesting that the semantics of C++11's range-based for-loop should be changed, as that would alter the meaning of existing code.) First, we could simply make "for (elem : range)" ill-formed when *__begin is a prvalue. However, that would lead to stealth build breaks (consider generic code taking vector<T>, where T might be bool someday), and it would prevent perfectly reasonable uses of proxies from compiling. Second, we could special-case prvalues. I believe that the Standardese for this would look like: "If *__begin is a glvalue, then use 'auto&& identifier'. Otherwise (i.e. if *__begin is a prvalue), replace every occurrence of identifier in statement with (*__begin)." Careful wordsmithing would be required to phrase "every occurrence", since things like ::elem should not be affected. This would make elem appear to be a named identifier that's actually a prvalue, but that's not completely unprecedented ("this" is a prvalue, 9.3.2 [class.this]/1), and I believe that's an acceptable cost for solving the proxy problem. Note that such replacement shouldn't be performed for lvalues because dereferencing the iterator may be expensive, and it especially shouldn't be performed for xvalues. Third, I briefly considered what would happen if we kept using auto&& for everything, but added Standardese saying "If *__begin is a prvalue, then identifier is a prvalue in statement." This would fix vector<bool>, but it would horribly damage other proxies (e.g. moving from std::string), so this is impossible. Fourth, we could choose to do nothing beyond using auto&&, which works for most uses of proxies. Compilers would be free to add coarse-grained warnings about prvalue elements, or fine-grained warnings about specific dangers like &elem for proxies. Q5. What about xvalues? A5. If __begin is a move_iterator<string *>, then *__begin will return string&&. This can be bound to "auto&& elem" (observe that auto& wouldn't compile), producing "string&& elem". This follows the rule that named rvalue references are lvalues, so the body of "for (elem : move_range)" sees elem as an lvalue and can mention it repeatedly without accidentally stealing from it. move(elem) must be used to explicitly move from the element. Q6. Is there any precedent for using auto&& like this? A6. Yep! The range-based for-loop already uses "auto && __range = range-init;" to handle lvalue/xvalue/prvalue ranges. Note that __range is affected by a limitation of the Core Language (it'll prolong the lifetime of a prvalue range, but not the lifetimes of any temporaries in subexpressions of range-init), which does not affect "auto&& elem" (*__begin doesn't contain such subexpressions). Q7. What about shadowing? A7. Because "for (elem : range)" is equivalent to "for (auto&& elem : range)", they handle shadowing identically. That is, they always generate "auto&& elem = *__begin;" within the for-loop's body, even when there's an outer declaration of elem. Compilers can and should warn about this. (If Q4's prvalue replacement is chosen, compilers can and should still warn about shadowing, even though elem will not actually exist as a variable.) Q8. What about attributes? A8. 6.5 [stmt.iter]/1 says: for-range-declaration: attribute-specifier-seq[opt] decl-specifier-seq declarator which permits attributes. The Standardese in this proposal, "for ( identifier : for-range-initializer ) statement", currently doesn't permit attributes. I have no objection to permitting attributes, if the EWG/CWG can tell me where to put them in the grammar. C++11's range-based for-loop will still be available, so I don't believe it's critical to permit attributes in the shorter form. Q9. Does this interact with anything that the Ranges Study Group is working on? A9. Despite appearances, this proposal actually deals with elements, not ranges. It's unaffected by anything outside of "for-range-declaration = *__begin;", such as changing how extensibility via ADL works, allowing __begin and __end to have different types, fixing the Core Language limitation about temporaries in subexpressions, providing range adapters or algorithms in the STL, etc. Q10. Is there any precedent for allowing "for (elem : range)" to declare elem without mentioning a type? A10. Earlier, I would have answered no, and strongly argued that it's fine because programmers never think of elements as being separate from their ranges. Now the answer is yes! Lambda init-captures permit "[x = 9 * 9 * 9] { return x + 10 * 10 * 10; }", which declares a non-static data member without mentioning its type (5.1.2 [expr.prim.lambda]/11). (Note that this is done via "auto init-capture ;" which differs from this proposal's auto&& as previously explained.) Although init-captures happen in a different context, they are similar to this proposal's syntax in a specific way. The init-capture grammar of "identifier initializer" places the newly-introduced identifier immediately before the initializer which provides its state and type information. This proposal's "identifier : for-range-initializer" places the newly-introduced identifier immediately before the range, which provides the element's state and type information. Q11. Has this been implemented? A11. Hi, Clark! Not yet. Q12. After "for (elem : range) { body; }", can elem be accessed? A12. No. It's out of scope, behaving identically to C++11's range-based for-loop. Q13. Can't we just use a macro for this? A13. C++11's range-based for-loop is a first-class citizen of the Core Language, unlike BOOST_FOREACH (see [3]). The central argument of this proposal is: "for (auto elem : range)" is very tempting and very bad. A proper solution to this problem must also be a first-class citizen of the Core Language; it cannot possibly be a macro. Q14. What about something like "for (auto iter : iterator_range(v))", where the body refers to *iter? A14. That would attempt to solve this problem on the wrong side of the colon. C++11's range-based for-loop is focused on elements, which are conceptually simpler to reason about than iteration variables. Inserting an adapter to make the range consist of iterators instead of elements defeats the entire purpose. Such an adapter may be useful in very specific situations (probably as part of a system of adapters), but it is not a solution to the problem presented here. It would require typing much more syntax, when the problem is that "auto elem" is more tempting than "auto& elem". VII. Acknowledgements Thanks to Ajay Vijayvargiya, Angel Hernandez Matos, Artur Laksberg, Bill Plauger, Billy O'Neal, Chris Guzak, Eric Niebler, Gabriel Dos Reis, Herb Sutter, Ian Bearman, Jon Kalb, Jonathan Caves, Marc Gregoire, Scott Meyers, and Sridhar Madhugiri for reviewing this proposal. VIII. References All of the Standardese citations in this proposal are to Working Paper N3797: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3797.pdf [1] "Universal References in C++11" by Scott Meyers: http://isocpp.org/blog/2012/11/universal-references-in-c11-scott-meyers [2] "Even Evil Has Standards" by TVTropes: http://tvtropes.org/pmwiki/pmwiki.php/Main/EvenEvilHasStandards For "Even Standards Have Evil", see 23.3.7 [vector.bool]. [3] "Boost.Foreach" by Eric Niebler: http://www.boost.org/doc/libs/1_55_0/doc/html/foreach.html (end)