[This is preliminary documentation and is subject to change.]
method to write a stream of records to a collection of HDFS files using the default Naiad binary serializer.
The collection is active throughout the computation and is closed when the computation terminates: it concatenates
records from all epochs in an undefined order
Namespace: Microsoft.Research.Naiad.Frameworks.WebHdfs
Assembly: Microsoft.Research.Naiad.WebHdfsSupport (in Microsoft.Research.Naiad.WebHdfsSupport.dll) Version: 0.5.0.0 (0.5.0.0)
Syntax
C#
public static Subscription WriteWebHdfsBinary<TOutput>( this Stream<TOutput, Epoch> source, string user, int webPort, Uri prefix, int bufferSize = 1048576, long blockSize = -1, long segmentThreshold = 266338304 )
Parameters
- source
- Type: Microsoft.Research.Naiad.Stream<TOutput, Epoch>
stream of records to write - user
- Type: System.String
hdfs user - webPort
- Type: System.Int32
webhdfs protocol port - prefix
- Type: System.Uri
webhdfs directory to write the partitioned data into - bufferSize (Optional)
- Type: System.Int32
buffer size to use in the serializer - blockSize (Optional)
- Type: System.Int64
hdfs block size to use, or -1 for the file system default value - segmentThreshold (Optional)
- Type: System.Int64
file size to write before closing the file and opening another one
Type Parameters
- TOutput
- type of the records to write
Return Value
Type: Subscriptionhandle to wait on until the computation completes
Usage Note
In Visual Basic and C#, you can call this method as an instance method on any object of type Stream<TOutput, Epoch>. When you use instance method syntax to call this method, omit the first parameter. For more information, see Extension Methods (Visual Basic) or Extension Methods (C# Programming Guide).See Also