DfsBaseCoordinatorTWorkDescription ClassNaiad Help

[This is preliminary documentation and is subject to change.]

base class for Dfs coordinators, that expands directories into sets of files to read, and keeps track of the mapping from IPAddresses to IPEndpoints
Inheritance Hierarchy

SystemObject
  Microsoft.Research.Naiad.Frameworks.WorkGeneratorMatchingCoordinatorUri, IPEndPoint, TWorkDescription, TWorkDescription, IPAddress
    Microsoft.Research.Naiad.Frameworks.Storage.DfsDfsBaseCoordinatorTWorkDescription
      Microsoft.Research.Naiad.Frameworks.Storage.DfsDfsBlockCoordinatorTItem
      Microsoft.Research.Naiad.Frameworks.Storage.DfsDfsFileCoordinator

Namespace: Microsoft.Research.Naiad.Frameworks.Storage.Dfs
Assembly: Microsoft.Research.Naiad.Storage (in Microsoft.Research.Naiad.Storage.dll) Version: 0.5.0.0 (0.5.0.0)
Syntax

C#
public abstract class DfsBaseCoordinator<TWorkDescription> : MatchingCoordinator<Uri, IPEndPoint, TWorkDescription, TWorkDescription, IPAddress[]>

Type Parameters

TWorkDescription
the type used by workers to identify themselves, e.g. including their IP addresses

The DfsBaseCoordinatorTWorkDescription type exposes the following members.

Constructors

  NameDescription
Public methodDfsBaseCoordinatorTWorkDescription
return a new coordinator for a dfs reader
Top
Methods

  NameDescription
Protected methodEnumerateFileWork
given an hdfs file, return a sequence of work items, each with a set of matching categories
Protected methodEnumerateWork
given an input string of a file or directory, expand it into a set of files, and then expand each file into a set of matches using the derived-class implementation of EnumerateFileWork
(Overrides MatchingCoordinatorTInput, TCategory, TWorkStub, TWorkDescription, TWorkerDescriptionEnumerateWork(TInput).)
Protected methodMapWorkerToQueue
Called when a worker announces that it is ready for another work item, to find a work item on the worker's matching queue, if any. The worker may have multiple IP addresses, so it returns them all, and if any matches an address the datanode is listening on, then the worker is matched to that datanode
(Overrides MatchingCoordinatorTInput, TCategory, TWorkStub, TWorkDescription, TWorkerDescriptionMapWorkerToQueue(TWorkerDescription, TCategory).)
Protected methodNotifyQueueAddition
called whenever a queue is added: keep our index of addresses up to date
(Overrides MatchingCoordinatorTInput, TCategory, TWorkStub, TWorkDescription, TWorkerDescriptionNotifyQueueAddition(TCategory).)
Protected methodNotifyQueueRemoval
called whenever a queue is removed: keep our index of addresses up to date
(Overrides MatchingCoordinatorTInput, TCategory, TWorkStub, TWorkDescription, TWorkerDescriptionNotifyQueueRemoval(TCategory).)
Top
Fields

  NameDescription
Protected fieldclient
The client for doing basic DFS operations.
Top
Remarks

This is currently somewhat HDFS-specific, and would probably be generalized if another file system were added
See Also

Reference