[This is preliminary documentation and is subject to change.]
Implementation of a MatchingCoordinator that manages work for reading dfs files in parallel, split by blocks
Inheritance Hierarchy
Microsoft.Research.Naiad.Frameworks.WorkGeneratorMatchingCoordinatorUri, IPEndPoint, TItem, TItem, IPAddress
Microsoft.Research.Naiad.Frameworks.Storage.DfsDfsBaseCoordinatorTItem
Microsoft.Research.Naiad.Frameworks.Storage.DfsDfsBlockCoordinatorTItem
Microsoft.Research.Naiad.Frameworks.Storage.DfsDfsTextCoordinator
Namespace: Microsoft.Research.Naiad.Frameworks.Storage.Dfs
Assembly: Microsoft.Research.Naiad.Storage (in Microsoft.Research.Naiad.Storage.dll) Version: 0.5.0.0 (0.5.0.0)
Syntax
C#
public abstract class DfsBlockCoordinator<TItem> : DfsBaseCoordinator<TItem> where TItem : DfsBlock
Type Parameters
- TItem
- The concrete type of work item, which may include metadata specific to a particular serializer or file type
The DfsBlockCoordinatorTItem type exposes the following members.
Constructors
Name | Description | |
---|---|---|
DfsBlockCoordinatorTItem |
return a new coordinator for a block-based HDFS reader
|
Methods
Name | Description | |
---|---|---|
EnumerateFileWork |
called to convert an input file into a list of blocks
(Overrides DfsBaseCoordinatorTWorkDescriptionEnumerateFileWork(HdfsFile).) | |
EnumerateWork |
given an input string of a file or directory, expand it into a set of files, and then expand each file
into a set of matches using the derived-class implementation of EnumerateFileWork
(Inherited from DfsBaseCoordinatorTWorkDescription.) | |
ExpandWorkItem |
Called when a work item is going to be sent to a worker. If usedMatchingQueue is true then dataNode
is the endpoint of the dataNode that matches the worker
(Overrides MatchingCoordinatorTInput, TCategory, TWorkStub, TWorkDescription, TWorkerDescriptionExpandWorkItem(Boolean, TCategory, TWorkStub).) | |
MakeBaseMatches |
For a given file, map a sequence of locations to a sequence of "base" matches, where the workStub
component of the match has any filetype-specific metadata filled in and the categories component of the
match has been set to the locations of that block. The HdfsBlock fields will
all be filled in later after this sequence has been returned
| |
MapWorkerToQueue |
Called when a worker announces that it is ready for another work item, to find a work item on the
worker's matching queue, if any. The worker may have multiple IP addresses, so it returns them all,
and if any matches an address the datanode is listening on, then the worker is matched to that datanode
(Inherited from DfsBaseCoordinatorTWorkDescription.) | |
NotifyQueueAddition |
called whenever a queue is added: keep our index of addresses up to date
(Inherited from DfsBaseCoordinatorTWorkDescription.) | |
NotifyQueueRemoval |
called whenever a queue is removed: keep our index of addresses up to date
(Inherited from DfsBaseCoordinatorTWorkDescription.) |
Fields
Name | Description | |
---|---|---|
client |
The client for doing basic DFS operations.
(Inherited from DfsBaseCoordinatorTWorkDescription.) |
See Also