ExtensionMethods.WriteWebHdfsBinary(TOutput, TWriter) Method (Stream(TOutput, Epoch), String, Int32, Func(Int32, Int32, Int32, Uri), Func(Stream, TWriter), Action(TWriter, ArraySegment(TOutput)), Int64, Int64)

ExtensionMethods.WriteWebHdfsBinary<TOutput, TWriter> Method (Stream<TOutput, Epoch>, String, Int32, Func<Int32, Int32, Int32, Uri>, Func<Stream, TWriter>, Action<TWriter, ArraySegment<TOutput>>, Int64, Int64)

Naiad Help

[This is preliminary documentation and is subject to change.]

general method to write a stream of records to a collection of HDFS files. The collection is active throughout the computation and is closed when the computation terminates: it concatenates records from all epochs in an undefined order

Namespace: Microsoft.Research.Naiad.Frameworks.WebHdfs
Assembly: Microsoft.Research.Naiad.WebHdfsSupport (in Microsoft.Research.Naiad.WebHdfsSupport.dll) Version: 0.5.0.0 (0.5.0.0)

Syntax

Copy

public static Subscription WriteWebHdfsBinary<TOutput, TWriter>(
	this Stream<TOutput, Epoch> source,
	string user,
	int webPort,
	Func<int, int, int, Uri> format,
	Func<Stream, TWriter> writerFunction,
	Action<TWriter, ArraySegment<TOutput>> serialize,
	long blockSize,
	long segmentThreshold
)
where TWriter : class, IDisposable, IFlushable

Parameters

source: Type: Microsoft.Research.Naiad.Stream<TOutput, Epoch>
stream of records to write
user: Type: System.String
hdfs user
webPort: Type: System.Int32
webhdfs protocol port
format: Type: System.Func<Int32, Int32, Int32, Uri>
function to generate a filename given a processId, threadId and sequence number
writerFunction: Type: System.Func<Stream, TWriter>
function to generate a serializer given a Stream to write to
serialize: Type: System.Action<TWriter, ArraySegment<TOutput>>
function to serialize a batch of records given a serializer
blockSize: Type: System.Int64
hdfs block size to use, or -1 for the file system default value
segmentThreshold: Type: System.Int64
file size to write before closing the file and opening another one

Type Parameters

TOutput: type of the records to write
TWriter: type of the serializer object

Return Value

Type: Subscription
handle to wait on until the computation completes

Usage Note

In Visual Basic and C#, you can call this method as an instance method on any object of type Stream<TOutput, Epoch>. When you use instance method syntax to call this method, omit the first parameter. For more information, see Extension Methods (Visual Basic) or Extension Methods (C# Programming Guide).

Reference

ExtensionMethods Class

WriteWebHdfsBinary Overload

Microsoft.Research.Naiad.Frameworks.WebHdfs Namespace