ExtensionMethods.ReadWebHdfsCollection(TOutput) Method (Computation, Uri, Func(Stream, IEnumerable(ArraySegment(TOutput))), String, Int32)

ExtensionMethodsReadWebHdfsCollectionTOutput Method (Computation, Uri, FuncStream, IEnumerableArraySegmentTOutput, String, Int32)

Naiad Help

[This is preliminary documentation and is subject to change.]

Read a collection of HDFS files serialized in a custom binary format. Concatenate all the records to the output, in an unspecified order.

Namespace: Microsoft.Research.Naiad.Frameworks.WebHdfs
Assembly: Microsoft.Research.Naiad.WebHdfsSupport (in Microsoft.Research.Naiad.WebHdfsSupport.dll) Version: 0.5.0.0 (0.5.0.0)

Syntax

Copy

public static Stream<TOutput, Epoch> ReadWebHdfsCollection<TOutput>(
	this Computation manager,
	Uri fileOrDirectoryPath,
	Func<Stream, IEnumerable<ArraySegment<TOutput>>> deserialize,
	string user,
	int webPort
)

Parameters

manager: Type: Microsoft.Research.NaiadComputation
Naiad computation
fileOrDirectoryPath: Type: SystemUri
path of the file or directory to read
deserialize: Type: SystemFuncStream, IEnumerableArraySegmentTOutput
custom deserializer function to convert a stream of bytes into a sequence of records
user: Type: SystemString
hdfs user
webPort: Type: SystemInt32
webhdfs protocol port

Type Parameters

TOutput: output record type

Return Value

Type: StreamTOutput, Epoch
stream of records in the hdfs files

Usage Note

In Visual Basic and C#, you can call this method as an instance method on any object of type Computation. When you use instance method syntax to call this method, omit the first parameter. For more information, see Extension Methods (Visual Basic) or Extension Methods (C# Programming Guide).

Reference

ExtensionMethods Class

ReadWebHdfsCollection Overload

Microsoft.Research.Naiad.Frameworks.WebHdfs Namespace