ExtensionMethods.WriteWebHdfsBinary(TOutput, TWriter) Method (Stream(TOutput, Epoch), String, Int32, Func(Int32, Int32, Int32, Uri), Func(Stream, TWriter), Action(TWriter, ArraySegment(TOutput)), Int64, Int64)

ExtensionMethodsWriteWebHdfsBinaryTOutput, TWriter Method (StreamTOutput, Epoch, String, Int32, FuncInt32, Int32, Int32, Uri, FuncStream, TWriter, ActionTWriter, ArraySegmentTOutput, Int64, Int64)

Naiad Help

[This is preliminary documentation and is subject to change.]

general method to write a stream of records to a collection of HDFS files. The collection is active throughout the computation and is closed when the computation terminates: it concatenates records from all epochs in an undefined order

Namespace: Microsoft.Research.Naiad.Frameworks.WebHdfs
Assembly: Microsoft.Research.Naiad.WebHdfsSupport (in Microsoft.Research.Naiad.WebHdfsSupport.dll) Version: 0.5.0.0 (0.5.0.0)

Syntax

Copy

public static Subscription WriteWebHdfsBinary<TOutput, TWriter>(
	this Stream<TOutput, Epoch> source,
	string user,
	int webPort,
	Func<int, int, int, Uri> format,
	Func<Stream, TWriter> writerFunction,
	Action<TWriter, ArraySegment<TOutput>> serialize,
	long blockSize,
	long segmentThreshold
)
where TWriter : class, IDisposable, IFlushable

Parameters

source: Type: Microsoft.Research.NaiadStreamTOutput, Epoch
stream of records to write
user: Type: SystemString
hdfs user
webPort: Type: SystemInt32
webhdfs protocol port
format: Type: SystemFuncInt32, Int32, Int32, Uri
function to generate a filename given a processId, threadId and sequence number
writerFunction: Type: SystemFuncStream, TWriter
function to generate a serializer given a Stream to write to
serialize: Type: SystemActionTWriter, ArraySegmentTOutput
function to serialize a batch of records given a serializer
blockSize: Type: SystemInt64
hdfs block size to use, or -1 for the file system default value
segmentThreshold: Type: SystemInt64
file size to write before closing the file and opening another one

Type Parameters

TOutput: type of the records to write
TWriter: type of the serializer object

Return Value

Type: Subscription
handle to wait on until the computation completes

Usage Note

In Visual Basic and C#, you can call this method as an instance method on any object of type StreamTOutput, Epoch. When you use instance method syntax to call this method, omit the first parameter. For more information, see Extension Methods (Visual Basic) or Extension Methods (C# Programming Guide).

Reference

ExtensionMethods Class

WriteWebHdfsBinary Overload

Microsoft.Research.Naiad.Frameworks.WebHdfs Namespace