Operators

The Python CollectionComposer exposes the map, flat_map and filter operators. Each operator takes a step name and a function, which can be a lambda, a function or a method.

from asgarde import CollectionComposer

map

result = (CollectionComposer.of(input_teams)
          .map('Map with country', lambda name: TeamInfo(name=name, country=team_countries[name], city=''))
          .map('Map with city', to_team_with_city))

flat_map

The function returns an iterable:

result = (CollectionComposer.of(teams)
          .flat_map('To players', lambda team: team.players))

Note

The outputs of an element are materialized before being emitted: a generator failing in the middle gives a failure only, not partial outputs.

filter

The function returns a boolean:

result = (CollectionComposer.of(teams_info)
          .filter('Filter french teams', lambda info: info.country == 'France'))

Result

outputs: PCollection[TeamInfo] = result.outputs    # Output of the last step
failures: PCollection[Failure] = result.failures   # Failures of all the steps

Only Exception subclasses are caught: KeyboardInterrupt, SystemExit and MemoryError are raised to the runner.