[This is preliminary documentation and is subject to change.]
base class for Dfs coordinators, that expands directories into sets of files to read, and keeps track
of the mapping from IPAddresses to IPEndpoints
Inheritance Hierarchy
Microsoft.Research.Naiad.Frameworks.WorkGeneratorMatchingCoordinatorUri, IPEndPoint, TWorkDescription, TWorkDescription, IPAddress
Microsoft.Research.Naiad.Frameworks.Storage.DfsDfsBaseCoordinatorTWorkDescription
Microsoft.Research.Naiad.Frameworks.Storage.DfsDfsBlockCoordinatorTItem
Microsoft.Research.Naiad.Frameworks.Storage.DfsDfsFileCoordinator
Namespace: Microsoft.Research.Naiad.Frameworks.Storage.Dfs
Assembly: Microsoft.Research.Naiad.Storage (in Microsoft.Research.Naiad.Storage.dll) Version: 0.5.0.0 (0.5.0.0)
Syntax
C#
public abstract class DfsBaseCoordinator<TWorkDescription> : MatchingCoordinator<Uri, IPEndPoint, TWorkDescription, TWorkDescription, IPAddress[]>
Type Parameters
- TWorkDescription
- the type used by workers to identify themselves, e.g. including their IP addresses
The DfsBaseCoordinatorTWorkDescription type exposes the following members.
Constructors
Name | Description | |
---|---|---|
DfsBaseCoordinatorTWorkDescription |
return a new coordinator for a dfs reader
|
Methods
Name | Description | |
---|---|---|
EnumerateFileWork |
given an hdfs file, return a sequence of work items, each with a set of matching categories
| |
EnumerateWork |
given an input string of a file or directory, expand it into a set of files, and then expand each file
into a set of matches using the derived-class implementation of EnumerateFileWork
(Overrides MatchingCoordinatorTInput, TCategory, TWorkStub, TWorkDescription, TWorkerDescriptionEnumerateWork(TInput).) | |
MapWorkerToQueue |
Called when a worker announces that it is ready for another work item, to find a work item on the
worker's matching queue, if any. The worker may have multiple IP addresses, so it returns them all,
and if any matches an address the datanode is listening on, then the worker is matched to that datanode
(Overrides MatchingCoordinatorTInput, TCategory, TWorkStub, TWorkDescription, TWorkerDescriptionMapWorkerToQueue(TWorkerDescription, TCategory).) | |
NotifyQueueAddition |
called whenever a queue is added: keep our index of addresses up to date
(Overrides MatchingCoordinatorTInput, TCategory, TWorkStub, TWorkDescription, TWorkerDescriptionNotifyQueueAddition(TCategory).) | |
NotifyQueueRemoval |
called whenever a queue is removed: keep our index of addresses up to date
(Overrides MatchingCoordinatorTInput, TCategory, TWorkStub, TWorkDescription, TWorkerDescriptionNotifyQueueRemoval(TCategory).) |
Fields
Name | Description | |
---|---|---|
client |
The client for doing basic DFS operations.
|
Remarks
See Also