pwnlib.util.iters — Extension of standard module itertools

This module includes and extends the standard module itertools.

pwnlib.util.iters.bruteforce(func, alphabet, length, method='upto', start=None)

Bruteforce func to return True. func should take a string input and return a bool(). func will be called with strings from alphabet until it returns True or the search space has been exhausted.

The argument start can be used to split the search space, which is useful if multiple CPU cores are available.

Parameters

• func (function) – The function to bruteforce.

• alphabet – The alphabet to draw symbols from.

• length – Longest string to try.

• method – If ‘upto’ try strings of length 1 .. length, if ‘fixed’ only try strings of length length and if ‘downfrom’ try strings of length length .. 1.

• start – a tuple (i, N) which splits the search space up into N pieces and starts at piece i (1..N). None is equivalent to (1, 1).

Returns

A string s such that func(s) returns True or None if the search space was exhausted.

Example

>>> bruteforce(lambda x: x == 'yes', string.ascii_lowercase, length=5)
'yes'

pwnlib.util.iters.chained(func)

A decorator chaining the results of func. Useful for generators.

Parameters

func (function) – The function being decorated.

Returns

A generator function whoose elements are the concatenation of the return values from func(*args, **kwargs).

Example

>>> @chained
... def g():
...     for x in count():
...         yield (x, -x)
>>> take(6, g())
[0, 0, 1, -1, 2, -2]
>>> @chained
... def g2():
...     for x in range(3):
...         yield (x, -x)
>>> list(g2())
[0, 0, 1, -1, 2, -2]

pwnlib.util.iters.consume(n, iterator)

Advance the iterator n steps ahead. If n is :const:`None, consume everything.

Parameters

• n (int) – Number of elements to consume.

• iterator (iterator) – An iterator.

Returns

None.

Examples

>>> i = count()
>>> consume(5, i)
>>> next(i)
5
>>> i = iter([1, 2, 3, 4, 5])
>>> consume(2, i)
>>> list(i)
[3, 4, 5]
>>> def g():
...     for i in range(2):
...         yield i
...         print(i)
>>> consume(None, g())
0
1

pwnlib.util.iters.cyclen(n, iterable) → iterator

Repeats the elements of iterable n times.

Parameters

• n (int) – The number of times to repeat iterable.

• iterable – An iterable.

Returns

An iterator whoose elements are the elements of iterator repeated n times.

Examples

>>> take(4, cyclen(2, [1, 2]))
[1, 2, 1, 2]
>>> list(cyclen(10, []))
[]

pwnlib.util.iters.dotproduct(x, y) → int

Computes the dot product of x and y.

Parameters

• x (iterable) – An iterable.

• x – An iterable.

Returns

The dot product of x and y, i.e. – x[0] * y[0] + x[1] * y[1] + ....

Example

>>> dotproduct([1, 2, 3], [4, 5, 6])
... # 1 * 4 + 2 * 5 + 3 * 6 == 32
32

pwnlib.util.iters.flatten(xss) → iterator

Flattens one level of nesting; when xss is an iterable of iterables, returns an iterator whoose elements is the concatenation of the elements of xss.

Parameters

xss – An iterable of iterables.

Returns

An iterator whoose elements are the concatenation of the iterables in xss.

Examples

>>> list(flatten([[1, 2], [3, 4]]))
[1, 2, 3, 4]
>>> take(6, flatten([[43, 42], [41, 40], count()]))
[43, 42, 41, 40, 0, 1]

pwnlib.util.iters.group(n, iterable, fill_value=None) → iterator

Similar to pwnlib.util.lists.group(), but returns an iterator and uses itertools fast build-in functions.

Parameters

• n (int) – The group size.

• iterable – An iterable.

• fill_value – The value to fill into the remaining slots of the last group if the n does not divide the number of elements in iterable.

Returns

An iterator whoose elements are n-tuples of the elements of iterable.

Examples

>>> list(group(2, range(5)))
[(0, 1), (2, 3), (4, None)]
>>> take(3, group(2, count()))
[(0, 1), (2, 3), (4, 5)]
>>> [''.join(x) for x in group(3, 'ABCDEFG', 'x')]
['ABC', 'DEF', 'Gxx']

pwnlib.util.iters.iter_except(func, exception)

Calls func repeatedly until an exception is raised. Works like the build-in iter() but uses an exception instead of a sentinel to signal the end.

Parameters

• func (callable) – The function to call.

• exception (Exception) – The exception that signals the end. Other exceptions will not be caught.

Returns

An iterator whoose elements are the results of calling func() until an exception matching exception is raised.

Examples

>>> s = {1, 2, 3}
>>> i = iter_except(s.pop, KeyError)
>>> next(i)
1
>>> next(i)
2
>>> next(i)
3
>>> next(i)
Traceback (most recent call last):
    ...
StopIteration

pwnlib.util.iters.lexicographic(alphabet) → iterator

The words with symbols in alphabet, in lexicographic order (determined by the order of alphabet).

Parameters

alphabet – The alphabet to draw symbols from.

Returns

An iterator of the words with symbols in alphabet, in lexicographic order.

Example

>>> take(8, map(lambda x: ''.join(x), lexicographic('01')))
['', '0', '1', '00', '01', '10', '11', '000']

pwnlib.util.iters.lookahead(n, iterable) → object

Inspects the upcoming element at index n without advancing the iterator. Raises IndexError if iterable has too few elements.

Parameters

• n (int) – Index of the element to return.

• iterable – An iterable.

Returns

The element in iterable at index n.

Examples

>>> i = count()
>>> lookahead(4, i)
4
>>> next(i)
0
>>> i = count()
>>> nth(4, i)
4
>>> next(i)
5
>>> lookahead(4, i)
10

pwnlib.util.iters.mbruteforce(func, alphabet, length, method='upto', start=None, threads=None)

Same functionality as bruteforce(), but multithreaded.

Parameters

• func – same as for bruteforce()

• alphabet – same as for bruteforce()

• length – same as for bruteforce()

• method – same as for bruteforce()

• start – same as for bruteforce()

• threads – Amount of threads to spawn, default is the amount of cores.

Example

>>> mbruteforce(lambda x: x == 'hello', string.ascii_lowercase, length = 10)
'hello'
>>> mbruteforce(lambda x: x == 'hello', 'hlo', 5, 'downfrom') is None
True
>>> mbruteforce(lambda x: x == 'no', string.ascii_lowercase, length=2, method='fixed')
'no'
>>> mbruteforce(lambda x: x == '9999', string.digits, length=4, threads=1, start=(2, 2))
'9999'

pwnlib.util.iters.nth(n, iterable, default=None) → object

Returns the element at index n in iterable. If iterable is a iterator it will be advanced.

Parameters

• n (int) – Index of the element to return.

• iterable – An iterable.

• default (objext) – A default value.

Returns

The element at index n in iterable or default if iterable has too few elements.

Examples

>>> nth(2, [0, 1, 2, 3])
2
>>> nth(2, [0, 1], 42)
42
>>> i = count()
>>> nth(42, i)
42
>>> nth(42, i)
85

pwnlib.util.iters.pad(iterable, value=None) → iterator

Pad an iterable with value, i.e. returns an iterator whoose elements are first the elements of iterable then value indefinitely.

Parameters

• iterable – An iterable.

• value – The value to pad with.

Returns

An iterator whoose elements are first the elements of iterable then value indefinitely.

Examples

>>> take(3, pad([1, 2]))
[1, 2, None]
>>> i = pad(iter([1, 2, 3]), 42)
>>> take(2, i)
[1, 2]
>>> take(2, i)
[3, 42]
>>> take(2, i)
[42, 42]

pwnlib.util.iters.pairwise(iterable) → iterator

Parameters

iterable – An iterable.

Returns

An iterator whoose elements are pairs of neighbouring elements of iterable.

Examples

>>> list(pairwise([1, 2, 3, 4]))
[(1, 2), (2, 3), (3, 4)]
>>> i = starmap(operator.add, pairwise(count()))
>>> take(5, i)
[1, 3, 5, 7, 9]

pwnlib.util.iters.powerset(iterable, include_empty=True) → iterator

The powerset of an iterable.

Parameters

• iterable – An iterable.

• include_empty (bool) – Whether to include the empty set.

Returns

The powerset of iterable as an interator of tuples.

Examples

>>> list(powerset(range(3)))
[(), (0,), (1,), (2,), (0, 1), (0, 2), (1, 2), (0, 1, 2)]
>>> list(powerset(range(2), include_empty = False))
[(0,), (1,), (0, 1)]

pwnlib.util.iters.quantify(iterable, pred=bool) → int

Count how many times the predicate pred is True.

Parameters

• iterable – An iterable.

• pred – A function that given an element from iterable returns either True or False.

Returns

The number of elements in iterable for which pred returns True.

Examples

>>> quantify([1, 2, 3, 4], lambda x: x % 2 == 0)
2
>>> quantify(['1', 'two', '3', '42'], str.isdigit)
3

pwnlib.util.iters.random_combination(iterable, r) → tuple

Parameters

• iterable – An iterable.

• r (int) – Size of the combination.

Returns

A random element from itertools.combinations(iterable, r = r).

Examples

>>> random_combination(range(2), 2)
(0, 1)
>>> random_combination(range(10), r = 2) in combinations(range(10), r = 2)
True

pwnlib.util.iters.random_combination_with_replacement(iterable, r)

random_combination(iterable, r) -> tuple

Parameters

• iterable – An iterable.

• r (int) – Size of the combination.

Returns

A random element from itertools.combinations_with_replacement(iterable, r = r).

Examples

>>> cs = {(0, 0), (0, 1), (1, 1)}
>>> random_combination_with_replacement(range(2), 2) in cs
True
>>> i = combinations_with_replacement(range(10), r = 2)
>>> random_combination_with_replacement(range(10), r = 2) in i
True

pwnlib.util.iters.random_permutation(iterable, r=None)

random_product(iterable, r = None) -> tuple

Parameters

• iterable – An iterable.

• r (int) – Size of the permutation. If None select all elements in iterable.

Returns

A random element from itertools.permutations(iterable, r = r).

Examples

>>> random_permutation(range(2)) in {(0, 1), (1, 0)}
True
>>> random_permutation(range(10), r = 2) in permutations(range(10), r = 2)
True

pwnlib.util.iters.random_product(*args, repeat=1) → tuple

Parameters

• args – One or more iterables

• repeat (int) – Number of times to repeat args.

Returns

A random element from itertools.product(*args, repeat = repeat).

Examples

>>> args = (range(2), range(2))
>>> random_product(*args) in {(0, 0), (0, 1), (1, 0), (1, 1)}
True
>>> args = (range(3), range(3), range(3))
>>> random_product(*args, repeat = 2) in product(*args, repeat = 2)
True

pwnlib.util.iters.repeat_func(func, *args, **kwargs) → iterator

Repeatedly calls func with positional arguments args and keyword arguments kwargs. If no keyword arguments is given the resulting iterator will be computed using only functions from itertools which are very fast.

Parameters

• func (function) – The function to call.

• args – Positional arguments.

• kwargs – Keyword arguments.

Returns

An iterator whoose elements are the results of calling func(*args, **kwargs) repeatedly.

Examples

>>> def f(x):
...     x[0] += 1
...     return x[0]
>>> i = repeat_func(f, [0])
>>> take(2, i)
[1, 2]
>>> take(2, i)
[3, 4]
>>> def f(**kwargs):
...     return kwargs.get('x', 43)
>>> i = repeat_func(f, x = 42)
>>> take(2, i)
[42, 42]
>>> i = repeat_func(f, 42)
>>> take(2, i)
Traceback (most recent call last):
    ...
TypeError: f() takes exactly 0 arguments (1 given)

pwnlib.util.iters.roundrobin(*iterables)

Take elements from iterables in a round-robin fashion.

Parameters

*iterables – One or more iterables.

Returns

An iterator whoose elements are taken from iterables in a round-robin fashion.

Examples

>>> ''.join(roundrobin('ABC', 'D', 'EF'))
'ADEBFC'
>>> ''.join(take(10, roundrobin('ABC', 'DE', repeat('x'))))
'ADxBExCxxx'

pwnlib.util.iters.tabulate(func, start=0) → iterator

Parameters

• func (function) – The function to tabulate over.

• start (int) – Number to start on.

Returns

An iterator with the elements func(start), func(start + 1), ....

Examples

>>> take(2, tabulate(str))
['0', '1']
>>> take(5, tabulate(lambda x: x**2, start = 1))
[1, 4, 9, 16, 25]

pwnlib.util.iters.take(n, iterable) → list

Returns first n elements of iterable. If iterable is a iterator it will be advanced.

Parameters

• n (int) – Number of elements to take.

• iterable – An iterable.

Returns

A list of the first n elements of iterable. If there are fewer than n elements in iterable they will all be returned.

Examples

>>> take(2, range(10))
[0, 1]
>>> i = count()
>>> take(2, i)
[0, 1]
>>> take(2, i)
[2, 3]
>>> take(9001, [1, 2, 3])
[1, 2, 3]

pwnlib.util.iters.unique_everseen(iterable, key=None) → iterator

Get unique elements, preserving order. Remember all elements ever seen. If key is not None then for each element elm in iterable the element that will be rememberes is key(elm). Otherwise elm is remembered.

Parameters

• iterable – An iterable.

• key – A function to map over each element in iterable before remembering it. Setting to None is equivalent to the identity function.

Returns

An iterator of the unique elements in iterable.

Examples

>>> ''.join(unique_everseen('AAAABBBCCDAABBB'))
'ABCD'
>>> ''.join(unique_everseen('ABBCcAD', str.lower))
'ABCD'

pwnlib.util.iters.unique_justseen(iterable, key=None)

unique_everseen(iterable, key = None) -> iterator

Get unique elements, preserving order. Remember only the elements just seen. If key is not None then for each element elm in iterable the element that will be rememberes is key(elm). Otherwise elm is remembered.

Parameters

• iterable – An iterable.

• key – A function to map over each element in iterable before remembering it. Setting to None is equivalent to the identity function.

Returns

An iterator of the unique elements in iterable.

Examples

>>> ''.join(unique_justseen('AAAABBBCCDAABBB'))
'ABCDAB'
>>> ''.join(unique_justseen('ABBCcAD', str.lower))
'ABCAD'

pwnlib.util.iters.unique_window(iterable, window, key=None)

unique_everseen(iterable, window, key = None) -> iterator

Get unique elements, preserving order. Remember only the last window elements seen. If key is not None then for each element elm in iterable the element that will be rememberes is key(elm). Otherwise elm is remembered.

Parameters

• iterable – An iterable.

• window (int) – The number of elements to remember.

• key – A function to map over each element in iterable before remembering it. Setting to None is equivalent to the identity function.

Returns

An iterator of the unique elements in iterable.

Examples

>>> ''.join(unique_window('AAAABBBCCDAABBB', 6))
'ABCDA'
>>> ''.join(unique_window('ABBCcAD', 5, str.lower))
'ABCD'
>>> ''.join(unique_window('ABBCcAD', 4, str.lower))
'ABCAD'

pwnlib.util.iters.chain()

Alias for itertools.chain().

pwnlib.util.iters.combinations()

Alias for itertools.combinations()

pwnlib.util.iters.combinations_with_replacement()

Alias for itertools.combinations_with_replacement()

pwnlib.util.iters.compress()

Alias for itertools.compress()

pwnlib.util.iters.count()

Alias for itertools.count()

pwnlib.util.iters.cycle()

Alias for itertools.cycle()

pwnlib.util.iters.dropwhile()

Alias for itertools.dropwhile()

pwnlib.util.iters.groupby()

Alias for itertools.groupby()

pwnlib.util.iters.filter()

Alias for python3-style filter()

pwnlib.util.iters.filterfalse()

Alias for itertools.filterfalse()

pwnlib.util.iters.map()

Alias for python3-style map()

pwnlib.util.iters.islice()

Alias for itertools.islice()

pwnlib.util.iters.zip()

Alias for python3-style zip()

pwnlib.util.iters.zip_longest()

Alias for itertools.zip_longest()

pwnlib.util.iters.permutations()

Alias for itertools.permutations()

pwnlib.util.iters.product()

Alias for itertools.product()

pwnlib.util.iters.repeat()

Alias for itertools.repeat()

pwnlib.util.iters.starmap()

Alias for itertools.starmap()

pwnlib.util.iters.takewhile()

Alias for itertools.takewhile()

pwnlib.util.iters.tee()

Alias for itertools.tee()