Many functions in Python use or generate in-memory lists for iteration. This is fine for relatively small list, but for large lists it becomes a problem.

Lazy Lists

A lazy list is when the next item is supplied when needed, and not before. If we stop processing before reaching the end of the list then those unused items will never be generated or use up resources. It is also possible to run an endless list. Lazy lists are implemented by supplying an iterator to the list, rather than the whole list itself.

Simple Example

def simp(): yield 10 yield 20 yield 30 it=simp() print(it.next()) # 10 print(it.next()) # 20 print(it.next()) # 30 1 2 3 4 5 6 7 8 9 def simp ( ) : yield 10 yield 20 yield 30 it = simp ( ) print ( it . next ( ) ) # 10 print ( it . next ( ) ) # 20 print ( it . next ( ) ) # 30

The function simp returns a generator , calling next() return the next value

We can use the generator like a list object:

for item in simp(): print(item) 1 2 for item in simp ( ) : print ( item )

Instead of building a big list, just generate the next value:

For example, return a list:

def nogen(num): res=[] i=0 while i<num: i+=1 res.append(i) return res s=nogen(10); print (s) # [1, 2, 3, 4, 5, 6, 7, 8, 9] for i in nogen(5): print(i) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 def nogen ( num ) : res = [ ] i = 0 while i < num : i += 1 res . append ( i ) return res s = nogen ( 10 ) ; print ( s ) # [1, 2, 3, 4, 5, 6, 7, 8, 9] for i in nogen ( 5 ) : print ( i )

And with generator:

def withgen(num): i=0; while i<num: i+=1 yield i s=withgen(10); print (s) # <generator object withgen at 0x1093786e0> for i in s: print(i) 1 2 3 4 5 6 7 8 9 10 11 12 def withgen ( num ) : i = 0 ; while i < num : i += 1 yield i s = withgen ( 10 ) ; print ( s ) # <generator object withgen at 0x1093786e0> for i in s : print ( i )

Note: I didn’t use a for loop because in python 2 range() function returns a list and in python 3 returns a generator (use xrange() in python 2 to get a generator)

Endless sequence

Sometimes we receive data from a stream and want to handle it like a sequence , with generator its easy:

def simp(): num = 0; while True: # endless loop num+=10 yield num it=simp() print(it.next()) # 10 print(it.next()) # 20 print(it.next()) # 30 1 2 3 4 5 6 7 8 9 10 def simp ( ) : num = 0 ; while True : # endless loop num += 10 yield num it = simp ( ) print ( it . next ( ) ) # 10 print ( it . next ( ) ) # 20 print ( it . next ( ) ) # 30

You can replace it with a function that receives a message from a socket and return it etc.

Co-routines via Enhanced Generators

You can use the send() generator method to sends a value back to the generator function, which can be picked-up as the return value from yield().

When the send() method is not used, yield returns None.

Example

def simp(): firstnum = 0 while True: num = firstnum; while True: num+=10 firstnum = yield num if firstnum: break it=simp() print(it.next()) # 10 print(it.next()) # 20 print(it.next()) # 30 print(it.send(200)) # 210 print(it.next()) # 220 print(it.next()) # 230 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 def simp ( ) : firstnum = 0 while True : num = firstnum ; while True : num += 10 firstnum = yield num if firstnum : break it = simp ( ) print ( it . next ( ) ) # 10 print ( it . next ( ) ) # 20 print ( it . next ( ) ) # 30 print ( it . send ( 200 ) ) # 210 print ( it . next ( ) ) # 220 print ( it . next ( ) ) # 230

As you can see, using send we change the next value. The inner loop is finished, num gets the value we sent and continue

List Comprehensions as Generators

An alternative syntax to yield, we can use list comprehension , use parentheses instead of squared brackets. You can’t send value to the generator

Example

def fn(): return (item for item in [1,2,3,4,5]) x=fn(); # returns print(x.next()) # 1 print(x.next()) # 2 print(x.next()) # 3 1 2 3 4 5 6 7 8 9 def fn ( ) : return ( item for item in [ 1 , 2 , 3 , 4 , 5 ] ) x = fn ( ) ; # returns print ( x . next ( ) ) # 1 print ( x . next ( ) ) # 2 print ( x . next ( ) ) # 3

Implementing a Custom Iterable Class

You can write a new class that behaves like an iterator. For example if your class holds a data structure and we want to provide a way to iterate over its internal structure without knowing how it implemented.

Example

class simpIter(object): def __init__(self, limit): self.limit = limit self.all = 0 self.nums = [] def __iter__(self): return self # Python 3 compatibility def __next__(self): return self.next() def next(self): if self.all < self.limit: cur = self.all self.all += 1 return cur else: raise StopIteration() it = simpIter(5) for n in it: print n s = sum(simpIter(10)) print (s) # 45 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 class simpIter ( object ) : def __init__ ( self , limit ) : self . limit = limit self . all = 0 self . nums = [ ] def __iter__ ( self ) : return self # Python 3 compatibility def __next__ ( self ) : return self . next ( ) def next ( self ) : if self . all < self . limit : cur = self . all self . all += 1 return cur else : raise StopIteration ( ) it = simpIter ( 5 ) for n in it : print n s = sum ( simpIter ( 10 ) ) print ( s ) # 45

We use a simple list inside the class but we can change it to any other data structure without worry as long as we reimplement the inner class functions