ExtensionMethods.WriteHdfsText Method

ExtensionMethodsWriteHdfsText Method

Naiad Help

[This is preliminary documentation and is subject to change.]

write a sequence of strings as hdfs text files. The collection is active throughout the computation and is closed when the computation terminates: it concatenates records from all epochs in an undefined order

Namespace: Microsoft.Research.Naiad.Frameworks.Hdfs
Assembly: Microsoft.Research.Naiad.HdfsSupport (in Microsoft.Research.Naiad.HdfsSupport.dll) Version: 0.5.0.0 (0.5.0.0)

Syntax

Copy

public static Subscription WriteHdfsText(
	this Stream<string, Epoch> source,
	Uri prefix,
	int bufferSize = 1048576,
	long blockSize = -1,
	long segmentThreshold = 266338304
)

Parameters

source: Type: Microsoft.Research.NaiadStreamString, Epoch
stream of records to write
prefix: Type: SystemUri
webhdfs directory to write the partitioned data into
bufferSize (Optional): Type: SystemInt32
buffer size to use for the text serializer
blockSize (Optional): Type: SystemInt64
hdfs block size to use, or -1 for the file system default value
segmentThreshold (Optional): Type: SystemInt64
file size to write before closing the file and opening another one

Return Value

Type: Subscription
handle to wait on until the computation completes

Usage Note

In Visual Basic and C#, you can call this method as an instance method on any object of type StreamString, Epoch. When you use instance method syntax to call this method, omit the first parameter. For more information, see Extension Methods (Visual Basic) or Extension Methods (C# Programming Guide).

Reference

ExtensionMethods Class

Microsoft.Research.Naiad.Frameworks.Hdfs Namespace