Effective Python Memo Item 9 Consider generator expressions for large comprehensions

** Notes on Effective Python ** Item 9 Consider a generator formula for large comprehensions (p18 ~ 20)

Use generators when working with large lists

Processing comprehensions with large amounts of data has some problems.

1. (Depending on the process) may generate a new list
2. When calling a loop statement, etc., everything is expanded in memory, which may cause a memory crash.

The generator also solves these problems. The advantage of the generator is that it doesn't call the whole sequence every time. Since it is called one at a time and processed, there is almost no concern about memory overrun.

Generator expressions are similar in syntax to comprehensions and are written in () instead of [].

my_lists = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
it = (x for x in my_lists)
print(it)

>>>
<generator object <genexpr> at 0x103fe93a8>

※Caution It is the generator object that is returned by the generator expression, not the value. To generate a value, use the next () function to generate the value. Again, the generator doesn't generate any values!

print(next(it))
print(next(it))

>>>
1
2

You can also immediately assign the iterator returned by one generator to another generator.

my_lists = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
it = (x for x in my_lists)
roots = ((x, x ** 0.5) for x in it)
print(next(roots))
print(next(roots))

>>>
(1, 1.0)
(2, 1.4142135623730951)

Summary

We haven't planned the processing time this time, but due to the characteristics of the generator, it will not take much time. Therefore, it is a very convenient function for processing large-scale data.