[This is preliminary documentation and is subject to change.]
create a worker for a file-at-a-time hdfs reader
Namespace: Microsoft.Research.Naiad.Frameworks.Storage.Dfs
Assembly: Microsoft.Research.Naiad.Storage (in Microsoft.Research.Naiad.Storage.dll) Version: 0.5.0.0 (0.5.0.0)
Syntax
C#
public DfsFileWorker( HdfsClientBase client, Func<Stream, IEnumerable<ArraySegment<TOutput>>> deserialize )
Parameters
- client
- Type: HdfsClientBase
Hdfs client used to read files - deserialize
- Type: SystemFuncStream, IEnumerableArraySegmentTOutput
function to take a stream consisting of an entire webhdfs file, and return a sequence of batches, each containing an arraysegment of output records
See Also