[This is preliminary documentation and is subject to change.]
method to write a stream of records to a collection of HDFS files using the default Naiad binary serializer,
partitioned by time as well as key. Within a given time and part, records are written in an undefined order
Namespace: Microsoft.Research.Naiad.Frameworks.WebHdfs
Assembly: Microsoft.Research.Naiad.WebHdfsSupport (in Microsoft.Research.Naiad.WebHdfsSupport.dll) Version: 0.5.0.0 (0.5.0.0)
Syntax
C#
public static Stream<Uri, TTime> ToWebHdfsBinary<TOutput, TTime>( this Stream<TOutput, TTime> source, string user, int webPort, Uri prefix, int bufferSize = 1048576, long blockSize = -1, long segmentThreshold = 266338304 ) where TTime : Object, Time<TTime>
Parameters
- source
- Type: Microsoft.Research.Naiad.Stream<TOutput, TTime>
stream of records to write - user
- Type: System.String
hdfs user - webPort
- Type: System.Int32
webhdfs protocol port - prefix
- Type: System.Uri
webhdfs directory to write the partitioned data into - bufferSize (Optional)
- Type: System.Int32
buffer size to use in the serializer - blockSize (Optional)
- Type: System.Int64
hdfs block size to use, or -1 for the file system default value - segmentThreshold (Optional)
- Type: System.Int64
file size to write before closing the file and opening another one
Type Parameters
- TOutput
- type of the records to write
- TTime
- type of the record time
Return Value
Type: Stream<Uri, TTime>stream of filenames written
Usage Note
In Visual Basic and C#, you can call this method as an instance method on any object of type Stream<TOutput, TTime>. When you use instance method syntax to call this method, omit the first parameter. For more information, see Extension Methods (Visual Basic) or Extension Methods (C# Programming Guide).See Also