Microsoft.Research.Naiad.Frameworks.WorkGenerator NamespaceNaiad Help

[This is preliminary documentation and is subject to change.]

The WorkGenerator framework contains base classes for implementing a work queue of items to be distributed dynamically to a set of Naiad workers. The queue is implemented as a Naiad loop; when new work arrives at a coordinator, it sends "wakeup" messages to all the workers, which respond with "ready" messages. Each time the coordinator receives a ready message from a worker, it matches an outstanding work item to that worker and sends it back. When the worker completes its work item, it responds with another ready message. Work items are thus sent out as workers become ready, rather than requiring coordination across workers. The framework allows workers to send identifying information that the coordinator can use for matching. For example the Hdfs and WebHdfs frameworks instantiate work generators in which the workers identify themselves by the IP addresses of the computer they are running on, allowing file blocks to be preferentially read on the worker computer that is hosting them.
Classes

  ClassDescription
Public classExtensionMethods
Extension methods for the work generator
Public classMatchingCoordinatorTInput, TCategory, TWorkStub, TWorkDescription, TWorkerDescription
A coordinator that assigns work items to zero or more matching workers. When a worker is free, it is assigned a matching item if there is one, otherwise it is assigned an item that matched to no workers if there is one, otherwise it is assigned an item from a worker that has the maximal number of remaining unassigned items.
Public classMatchingCoordinatorTInput, TCategory, TWorkStub, TWorkDescription, TWorkerDescriptionMatch
A work item and the list of categories that match the work item.
Protected classMatchingCoordinatorTInput, TCategory, TWorkStub, TWorkDescription, TWorkerDescriptionMatchQueue
Queue of work items that match to a particular worker.
Structures

  StructureDescription
Public structureWorkItemTWorkDescription
Internal record describing a work item, sent from the Coordinator to a Worker. This is public only for the benefit of the serialization code
Public structureWorkRequestTWorkerDescription
Internal record describing a worker that is ready to receive a work item, sent by the Worker to the Coordinator when the Worker has processed a WorkItem. This is public only for the benefit of the serialization code
Interfaces

  InterfaceDescription
Public interfaceICoordinatorTInput, TWorkDescription, TWorkerDescription
Interface specifying the behavior of a coordinator vertex. A coordinator takes a stream of input records via calls to AddInput, and for each input it generates a set of work items. As workers become free, the coordinator is informed, via AssignWork, and responds with a work item for that worker, or nothing if there is no more work for that worker to perform.
Public interfaceIWorkerTWorkDescription, TWorkerDescription, TOutput
Interface specifying the behavior of a worker vertex. A worker takes a stream of work items to perform. Before it is assigned its first work item, and after each item has been performed, the worker is asked to describe itself; the coordinator uses this description to match work items to workers. The act of performing a work item causes a sequence of output items to be generated
Enumerations

  EnumerationDescription
Public enumerationWorkRecordType
Internal enumeration distinguishing work item records. This is public only for the benefit of the serialization code