Utils.WriteByTime(TOutput, TClient, TWriter, TTime) Method

UtilsWriteByTimeTOutput, TClient, TWriter, TTime Method

Naiad Help

[This is preliminary documentation and is subject to change.]

serialize a sequence of records to a collection of files partitioned by process, thread and time. For each process/thread/time this writes a sequence of files; each time a file reaches a threshold number of bytes, it is closed and another is opened. This keeps individual files of bounded length, allowing for more parallelism when reading them later

Namespace: Microsoft.Research.Naiad.Frameworks.Storage
Assembly: Microsoft.Research.Naiad.Storage (in Microsoft.Research.Naiad.Storage.dll) Version: 0.5.0.0 (0.5.0.0)

Syntax

Copy

public static Stream<Uri, TTime> WriteByTime<TOutput, TClient, TWriter, TTime>(
	this Stream<TOutput, TTime> source,
	Func<int, int, TTime, int, Uri> pathFunction,
	Func<TClient> clientFunction,
	Func<TClient, Uri, Stream> streamFunction,
	Func<Stream, TWriter> writerFunction,
	Action<TWriter, ArraySegment<TOutput>> serialize,
	long fileLengthThreshold
)
where TWriter : class, IDisposable, IFlushable
where TTime : Object, Time<TTime>

Parameters

source: Type: Microsoft.Research.NaiadStreamTOutput, TTime
stream of records to serialize
pathFunction: Type: SystemFuncInt32, Int32, TTime, Int32, Uri
function from processId, threadId, time and sequence number to filename
clientFunction: Type: SystemFuncTClient
function to return a client to pass as context to the stream function
streamFunction: Type: SystemFuncTClient, Uri, Stream
function to create an output stream given a client and filename
writerFunction: Type: SystemFuncStream, TWriter
function to create a serializer from a stream
serialize: Type: SystemActionTWriter, ArraySegmentTOutput
action to serialize a batch of records
fileLengthThreshold: Type: SystemInt64
length in bytes of a file after which it is closed and a new one is opened

Type Parameters

TOutput: type of record to serialize
TClient: type of client that is passed as a context to the stream function
TWriter: type of the serializer
TTime: type of the time of records

Return Value

Type: StreamUri, TTime
stream of names of files written. The set of names written for a given time is released when the time completes

Usage Note

In Visual Basic and C#, you can call this method as an instance method on any object of type StreamTOutput, TTime. When you use instance method syntax to call this method, omit the first parameter. For more information, see Extension Methods (Visual Basic) or Extension Methods (C# Programming Guide).

Reference

Utils Class

Microsoft.Research.Naiad.Frameworks.Storage Namespace