Utils.WriteBySubscription(TOutput, TWriter) Method

UtilsWriteBySubscriptionTOutput, TWriter Method

Naiad Help

[This is preliminary documentation and is subject to change.]

serialize a sequence of records to a collection of files partitioned by process and thread. For each process/thread this writes a sequence of files; each time a file reaches a threshold number of bytes, it is closed and another is opened. This keeps individual files of bounded length, allowing for more parallelism when reading them later

Namespace: Microsoft.Research.Naiad.Frameworks.Storage
Assembly: Microsoft.Research.Naiad.Storage (in Microsoft.Research.Naiad.Storage.dll) Version: 0.5.0.0 (0.5.0.0)

Syntax

Copy

public static Subscription WriteBySubscription<TOutput, TWriter>(
	this Stream<TOutput, Epoch> source,
	Func<int, int, int, Uri> pathFunction,
	Func<Uri, Stream> streamFunction,
	Func<Stream, TWriter> writerFunction,
	Action<TWriter, ArraySegment<TOutput>> serialize,
	long fileLengthThreshold
)
where TWriter : class, IDisposable, IFlushable

Parameters

source: Type: Microsoft.Research.NaiadStreamTOutput, Epoch
stream of records to serialize
pathFunction: Type: SystemFuncInt32, Int32, Int32, Uri
function from processId, threadId and sequence number to filename
streamFunction: Type: SystemFuncUri, Stream
function to create an output stream given a filename
writerFunction: Type: SystemFuncStream, TWriter
function to create a serializer from a stream
serialize: Type: SystemActionTWriter, ArraySegmentTOutput
action to serialize a batch of records
fileLengthThreshold: Type: SystemInt64
length in bytes of a file after which it is closed and a new one is opened

Type Parameters

TOutput: type of record to serialize
TWriter: type of the serializer

Return Value

Type: Subscription
a handle that can be waited on for the computation to complete

Usage Note

In Visual Basic and C#, you can call this method as an instance method on any object of type StreamTOutput, Epoch. When you use instance method syntax to call this method, omit the first parameter. For more information, see Extension Methods (Visual Basic) or Extension Methods (C# Programming Guide).

Reference

Utils Class

Microsoft.Research.Naiad.Frameworks.Storage Namespace