[This is preliminary documentation and is subject to change.]
create a worker to read dfs files broken into blocks
Namespace: Microsoft.Research.Naiad.Frameworks.Storage.Dfs
Assembly: Microsoft.Research.Naiad.Storage (in Microsoft.Research.Naiad.Storage.dll) Version: 0.5.0.0 (0.5.0.0)
Syntax
C#
public DfsBlockWorker( int syncRequestLength, Action<TItem, Stream> syncToNextRecord, Func<TItem, Stream, IEnumerable<ArraySegment<TOutput>>> deserialize, HdfsClientBase client )
Parameters
- syncRequestLength
- Type: SystemInt32
size of each dfs request when seeking past the end of the block for the start of the next record. If records are expected to be small this should also be small, to avoid prefetching and buffering a lot of the next block's data - syncToNextRecord
- Type: SystemActionTItem, Stream
action to sync to the start of the next record. The first argument is the block item being read, which may contain metadata about sync markers. The second argument is the stream to scan. - deserialize
- Type: SystemFuncTItem, Stream, IEnumerableArraySegmentTOutput
function to deserialize records in a stream - client
- Type: HdfsClientBase
client used to read hdfs data
See Also