[This is preliminary documentation and is subject to change.]
The WorkGenerator framework contains base classes for implementing a work queue of items to be distributed dynamically to a set
of Naiad workers. The queue is implemented as a Naiad loop; when new work arrives at a coordinator, it sends "wakeup" messages
to all the workers, which respond with "ready" messages. Each time the coordinator receives a ready message from a worker, it matches
an outstanding work item to that worker and sends it back. When the worker completes its work item, it responds with another ready message.
Work items are thus sent out as workers become ready, rather than requiring coordination across workers. The framework allows
workers to send identifying information that the coordinator can use for matching. For example the Hdfs and WebHdfs frameworks
instantiate work generators in which the workers identify themselves by the IP addresses of the computer they are running on,
allowing file blocks to be preferentially read on the worker computer that is hosting them.
Classes
| Class | Description | |
|---|---|---|
 | ExtensionMethods |
Extension methods for the work generator
|
| MatchingCoordinatorTInput, TCategory, TWorkStub, TWorkDescription, TWorkerDescription |
A coordinator that assigns work items to zero or more matching workers. When a worker is free, it is assigned a matching item if
there is one, otherwise it is assigned an item that matched to no workers if there is one, otherwise it is assigned an item from
a worker that has the maximal number of remaining unassigned items.
|
| MatchingCoordinatorTInput, TCategory, TWorkStub, TWorkDescription, TWorkerDescriptionMatch |
A work item and the list of categories that match the work item.
|
 | MatchingCoordinatorTInput, TCategory, TWorkStub, TWorkDescription, TWorkerDescriptionMatchQueue |
Queue of work items that match to a particular worker.
|
Structures
| Structure | Description | |
|---|---|---|
 | WorkItemTWorkDescription |
Internal record describing a work item, sent from the Coordinator to a Worker. This is public only for the benefit
of the serialization code
|
| WorkRequestTWorkerDescription |
Internal record describing a worker that is ready to receive a work item, sent by the Worker to the Coordinator when
the Worker has processed a WorkItem. This is public only for the benefit of the serialization code
|
Interfaces
| Interface | Description | |
|---|---|---|
 | ICoordinatorTInput, TWorkDescription, TWorkerDescription |
Interface specifying the behavior of a coordinator vertex. A coordinator takes a stream of input records via calls to AddInput,
and for each input it generates a set of work items. As workers become free, the coordinator is informed, via AssignWork, and
responds with a work item for that worker, or nothing if there is no more work for that worker to perform.
|
| IWorkerTWorkDescription, TWorkerDescription, TOutput |
Interface specifying the behavior of a worker vertex. A worker takes a stream of work items to perform. Before it is
assigned its first work item, and after each item has been performed, the worker is asked to describe itself; the
coordinator uses this description to match work items to workers. The act of performing a work item causes a sequence of
output items to be generated
|
Enumerations
| Enumeration | Description | |
|---|---|---|
 | WorkRecordType |
Internal enumeration distinguishing work item records. This is public only for the benefit of the serialization code
|