[This is preliminary documentation and is subject to change.]
IWorker implementation for the dfs worker that reads data from blocks. This is further specialized to different file
formats by passing in functions for syncing to record boundaries and deserializing data
Inheritance Hierarchy
Microsoft.Research.Naiad.Frameworks.Storage.DfsDfsBlockWorkerTItem, TOutput
Microsoft.Research.Naiad.Frameworks.Storage.DfsDfsTextWorker
Namespace: Microsoft.Research.Naiad.Frameworks.Storage.Dfs
Assembly: Microsoft.Research.Naiad.Storage (in Microsoft.Research.Naiad.Storage.dll) Version: 0.5.0.0 (0.5.0.0)
Syntax
C#
public class DfsBlockWorker<TItem, TOutput> : IWorker<TItem, IPAddress[], TOutput> where TItem : DfsBlock
Type Parameters
- TItem
- The work item passed by the matching coordinator, which inherits from DfsBlock but may contain metadata used in syncing to record boundaries or deserializing
- TOutput
- The type of deserialized records produced by the worker
The DfsBlockWorkerTItem, TOutput type exposes the following members.
Constructors
| Name | Description | |
|---|---|---|
 | DfsBlockWorkerTItem, TOutput |
create a worker to read dfs files broken into blocks
|
Methods
| Name | Description | |
|---|---|---|
| DescribeWorker |
the IWorker implementation used to identify this worker to the WebHdfs coordinator, so that it can be sent work items
of blocks stored at the same machine
|
| DoWork |
find the range of valid records in the block (those that start within the block), deserialize them, and return them in batches
|
Fields
| Name | Description | |
|---|---|---|
 | client |
the client used for reading Hdfs data
|
See Also