[This is preliminary documentation and is subject to change.]
base coordinator for workers that read an entire hdfs file at a time, rather than split the file into blocks.
For each file the coordinator tries to match it to a worker that holds a large proportion of the relevant data
Inheritance Hierarchy
Microsoft.Research.Naiad.Frameworks.WorkGeneratorMatchingCoordinatorUri, IPEndPoint, DfsBlock, DfsBlock, IPAddress
Microsoft.Research.Naiad.Frameworks.Storage.DfsDfsBaseCoordinatorDfsBlock
Microsoft.Research.Naiad.Frameworks.Storage.DfsDfsFileCoordinator
Namespace: Microsoft.Research.Naiad.Frameworks.Storage.Dfs
Assembly: Microsoft.Research.Naiad.Storage (in Microsoft.Research.Naiad.Storage.dll) Version: 0.5.0.0 (0.5.0.0)
Syntax
C#
public class DfsFileCoordinator : DfsBaseCoordinator<DfsBlock>
The DfsFileCoordinator type exposes the following members.
Constructors
| Name | Description | |
|---|---|---|
 | DfsFileCoordinator |
create a new coordinator for file-at-a-time dfs reads
|
Methods
| Name | Description | |
|---|---|---|
 | EnumerateFileWork |
given a file, determine how much of that file's data are stored on each datanode. Return every datanode that stores a
threshold percentage of the file's data as a candidate match for that file
(Overrides DfsBaseCoordinatorTWorkDescriptionEnumerateFileWork(HdfsFile).) |
| EnumerateWork |
given an input string of a file or directory, expand it into a set of files, and then expand each file
into a set of matches using the derived-class implementation of EnumerateFileWork
(Inherited from DfsBaseCoordinatorTWorkDescription.) |
| ExpandWorkItem |
if the work item was matched to a worker on the same computer, fill in the datanode endpoint before sending the work item
(Overrides MatchingCoordinatorTInput, TCategory, TWorkStub, TWorkDescription, TWorkerDescriptionExpandWorkItem(Boolean, TCategory, TWorkStub).) |
| MapWorkerToQueue |
Called when a worker announces that it is ready for another work item, to find a work item on the
worker's matching queue, if any. The worker may have multiple IP addresses, so it returns them all,
and if any matches an address the datanode is listening on, then the worker is matched to that datanode
(Inherited from DfsBaseCoordinatorTWorkDescription.) |
| NotifyQueueAddition |
called whenever a queue is added: keep our index of addresses up to date
(Inherited from DfsBaseCoordinatorTWorkDescription.) |
| NotifyQueueRemoval |
called whenever a queue is removed: keep our index of addresses up to date
(Inherited from DfsBaseCoordinatorTWorkDescription.) |
Fields
| Name | Description | |
|---|---|---|
 | client |
The client for doing basic DFS operations.
(Inherited from DfsBaseCoordinatorTWorkDescription.) |
See Also