ExtensionMethodsToWebHdfsBinaryTOutput, TWriter, TTime Method (StreamTOutput, TTime, String, Int32, FuncInt32, Int32, TTime, Int32, Uri, FuncStream, TWriter, ActionTWriter, ArraySegmentTOutput, Int64, Int64)Naiad Help

[This is preliminary documentation and is subject to change.]

general method to write a stream of records to a collection of HDFS files, partitioned by time as well as key. Within a given time and part, records are written in an undefined order

Namespace: Microsoft.Research.Naiad.Frameworks.WebHdfs
Assembly: Microsoft.Research.Naiad.WebHdfsSupport (in Microsoft.Research.Naiad.WebHdfsSupport.dll) Version: 0.5.0.0 (0.5.0.0)
Syntax

C#
public static Stream<Uri, TTime> ToWebHdfsBinary<TOutput, TWriter, TTime>(
	this Stream<TOutput, TTime> source,
	string user,
	int webPort,
	Func<int, int, TTime, int, Uri> format,
	Func<Stream, TWriter> writerFunction,
	Action<TWriter, ArraySegment<TOutput>> serialize,
	long blockSize,
	long segmentThreshold
)
where TWriter : class, IDisposable, IFlushable
where TTime : Object, Time<TTime>

Parameters

source
Type: Microsoft.Research.NaiadStreamTOutput, TTime
stream of records to write
user
Type: SystemString
hdfs user
webPort
Type: SystemInt32
webhdfs protocol port
format
Type: SystemFuncInt32, Int32, TTime, Int32, Uri
function to generate a filename given a processId, threadId, time and sequence number
writerFunction
Type: SystemFuncStream, TWriter
function to generate a serializer given a Stream to write to
serialize
Type: SystemActionTWriter, ArraySegmentTOutput
function to serialize a batch of records given a serializer
blockSize
Type: SystemInt64
hdfs block size to use, or -1 for the file system default value
segmentThreshold
Type: SystemInt64
file size to write before closing the file and opening another one

Type Parameters

TOutput
type of the records to write
TWriter
type of the serializer object
TTime
type of the record time

Return Value

Type: StreamUri, TTime
stream of filenames written

Usage Note

In Visual Basic and C#, you can call this method as an instance method on any object of type StreamTOutput, TTime. When you use instance method syntax to call this method, omit the first parameter. For more information, see Extension Methods (Visual Basic) or Extension Methods (C# Programming Guide).
See Also

Reference