[This is preliminary documentation and is subject to change.]
method to write a stream of records to a collection of HDFS files using the default Naiad binary serializer.
The collection is active throughout the computation and is closed when the computation terminates: it concatenates
records from all epochs in an undefined order
Namespace: Microsoft.Research.Naiad.Frameworks.WebHdfs
Assembly: Microsoft.Research.Naiad.WebHdfsSupport (in Microsoft.Research.Naiad.WebHdfsSupport.dll) Version: 0.5.0.0 (0.5.0.0)
Syntax
C#
public static Subscription WriteWebHdfsBinary<TOutput>( this Stream<TOutput, Epoch> source, string user, int webPort, Uri prefix, int bufferSize = 1048576, long blockSize = -1, long segmentThreshold = 266338304 )
Parameters
- source
- Type: Microsoft.Research.NaiadStreamTOutput, Epoch
stream of records to write - user
- Type: SystemString
hdfs user - webPort
- Type: SystemInt32
webhdfs protocol port - prefix
- Type: SystemUri
webhdfs directory to write the partitioned data into - bufferSize (Optional)
- Type: SystemInt32
buffer size to use in the serializer - blockSize (Optional)
- Type: SystemInt64
hdfs block size to use, or -1 for the file system default value - segmentThreshold (Optional)
- Type: SystemInt64
file size to write before closing the file and opening another one
Type Parameters
- TOutput
- type of the records to write
Return Value
Type: Subscriptionhandle to wait on until the computation completes
Usage Note
In Visual Basic and C#, you can call this method as an instance method on any object of type StreamTOutput, Epoch. When you use instance method syntax to call this method, omit the first parameter. For more information, see Extension Methods (Visual Basic) or Extension Methods (C# Programming Guide).See Also