ExtensionMethods.ToWebHdfsBinary(TOutput, TWriter, TTime) Method (Stream(TOutput, TTime), String, Int32, Func(Int32, Int32, TTime, Int32, Uri), Func(Stream, TWriter), Action(TWriter, ArraySegment(TOutput)), Int64, Int64)

ExtensionMethods.ToWebHdfsBinary<TOutput, TWriter, TTime> Method (Stream<TOutput, TTime>, String, Int32, Func<Int32, Int32, TTime, Int32, Uri>, Func<Stream, TWriter>, Action<TWriter, ArraySegment<TOutput>>, Int64, Int64)

Naiad Help

[This is preliminary documentation and is subject to change.]

general method to write a stream of records to a collection of HDFS files, partitioned by time as well as key. Within a given time and part, records are written in an undefined order

Namespace: Microsoft.Research.Naiad.Frameworks.WebHdfs
Assembly: Microsoft.Research.Naiad.WebHdfsSupport (in Microsoft.Research.Naiad.WebHdfsSupport.dll) Version: 0.5.0.0 (0.5.0.0)

Syntax

Copy

public static Stream<Uri, TTime> ToWebHdfsBinary<TOutput, TWriter, TTime>(
	this Stream<TOutput, TTime> source,
	string user,
	int webPort,
	Func<int, int, TTime, int, Uri> format,
	Func<Stream, TWriter> writerFunction,
	Action<TWriter, ArraySegment<TOutput>> serialize,
	long blockSize,
	long segmentThreshold
)
where TWriter : class, IDisposable, IFlushable
where TTime : Object, Time<TTime>

Parameters

source: Type: Microsoft.Research.Naiad.Stream<TOutput, TTime>
stream of records to write
user: Type: System.String
hdfs user
webPort: Type: System.Int32
webhdfs protocol port
format: Type: System.Func<Int32, Int32, TTime, Int32, Uri>
function to generate a filename given a processId, threadId, time and sequence number
writerFunction: Type: System.Func<Stream, TWriter>
function to generate a serializer given a Stream to write to
serialize: Type: System.Action<TWriter, ArraySegment<TOutput>>
function to serialize a batch of records given a serializer
blockSize: Type: System.Int64
hdfs block size to use, or -1 for the file system default value
segmentThreshold: Type: System.Int64
file size to write before closing the file and opening another one

Type Parameters

TOutput: type of the records to write
TWriter: type of the serializer object
TTime: type of the record time

Return Value

Type: Stream<Uri, TTime>
stream of filenames written

Usage Note

In Visual Basic and C#, you can call this method as an instance method on any object of type Stream<TOutput, TTime>. When you use instance method syntax to call this method, omit the first parameter. For more information, see Extension Methods (Visual Basic) or Extension Methods (C# Programming Guide).

Reference

ExtensionMethods Class

ToWebHdfsBinary Overload

Microsoft.Research.Naiad.Frameworks.WebHdfs Namespace