API Reference

Unless otherwise stated below functions are available at top level import, i.e. after import chunkflow.

Chunk

class chunkflow.chunk.Chunk(array, voxel_offset=None, voxel_size=None, layer_type=None)

chunk of a volume

a chunk of big array with offset implementation using numpy `dispatch< https://docs.scipy.org/doc/numpy/user/basics.dispatch.html#module-numpy.doc.dispatch>`_. and `examples<https://docs.scipy.org/doc/numpy/user/basics.dispatch.html#module-numpy.doc.dispatch>`_.

Parameters:

• array (np.ndarray) – the data

• voxel_offset (Cartesian, optional) – voxel offset. Defaults to None.

• voxel_size (Cartesian, optional) – voxel size. Defaults to None.

• type (str, optional) – type of chunk. [None, image, segmentation, probability_map, affinity_map, unknown]. Defaults to None.

Returns:

a new chunk with array data and global offset

add_overlap(other)

sum up overlaping region with another chunk

Parameters:

other – another chunk

Returns:

sum up result.

property bbox: BoundingBox

the bounding box in the big volume

Type:

getter

blend(patch)

same with add_overlap

connected_component(threshold=None, connectivity=6)

threshold the map chunk and get connected components.

classmethod create(size=Cartesian(z=64, y=64, x=64), dtype=<class 'numpy.uint8'>, voxel_offset=Cartesian(z=0, y=0, x=0), voxel_size=None, pattern='sin', high=255)

create a fake chunk for tests.

Parameters:

• size (tuple, Cartesian, optional) – chunk size or shape. Defaults to (64, 64, 64).

• dtype (type, optional) – data type like numpy. Defaults to np.uint8. options: [uint8, uint16, uint32, uint64, float32]

• voxel_offset (Cartesian, optional) – coordinate of starting voxel. Defaults to Cartesian(0, 0, 0).

• voxel_size (Cartesian, optional) – physical size of each voxel. Defaults to None.

• pattern (str, optional) – ways to create an array. [‘sin’, ‘random’, ‘zero’]. Defaults to ‘sin’.

• high (int, optional) – the high value of random integer array. Defaults to 255.

Raises:

NotImplementedError – not support pattern or data type was used.

Returns:

the random chunk created.

Return type:

Chunk

crop_margin(margin_size=None, output_bbox=None)

_summary_

Parameters:

• margin_size (tuple, optional) – -z,-y,-x,+z,+y,+x. Defaults to None.

• output_bbox (BoundingBox, optional) – _description_. Defaults to None.

Returns:

_description_

Return type:

_type_

cutout(x)

cutout a region of interest from this chunk

Parameters:

slices – the global slices of region of interest

Returns:

another chunk of region of interest

classmethod from_array(array, bbox, voxel_size=None)

Parameters:

• array (ndarray) – ndarray data

• bbox (BoundingBox) – cloudvolume bounding box

• voxel_size (Optional[tuple]) – physical size of each voxel.

Returns:

construct a new Chunk

classmethod from_bbox(bbox, dtype=<class 'numpy.uint8'>, pattern='zero', voxel_size=None)

create a chunk from bounding box

Parameters:

• bbox (BoundingBox) – 3D bounding box

• dtype (type, optional) – data type. Defaults to np.uint8.

• pattern (str, optional) – method to create empty array. [zero, random, sin]. Defaults to ‘zero’.

• voxel_size (tuple, optional) – physical size of a voxel. Defaults to None.

Returns:

_description_

Return type:

_type_

gaussian_filter_2d(sigma=1)

gaussion filter for smoothing or blurring

Parameters:

sigma (Union[int, tuple, list]) – the standard deviation of gaussian filter

maskout(chunk)

Make part of the chunk to be black.

property ndoffset: tuple

make the voxel offset have the same dimension with array

save(patch)

replace a region of interest from another chunk

Parameters:

patch – a small chunk to replace subvolume

shrink(size)

shrink the array from surrounding boundary

Parameters:

• size (tuple) – the shrink size in -Z,-Y, -X, Z, Y, X direction.

• sometimes –

• need (the shrinking might be asymmetric. that's why we) –

• 3. (6 elements rather than) –

property slices: tuple

the global slice in the big volume

Type:

getter

squeeze_channel(axis=0)

given a 4D array, squeeze the channel axis.

Return type:

ndarray

to_h5(file_name, with_offset=True, chunk_size=(8, 8, 8), with_unique=True, compression='gzip', voxel_size=None)

Parameters:

• file_name (str) – output file name. If it is not end with h5, the coordinate will be appended to the file name.

• with_offset (bool) – save the voxel offset or not

• with_unique (bool) – if this is a segmentation chunk, save the unique object ids or not.

• compression – use HDF5 compression or not. Options are gzip, lzf

transpose(only_array=False)

To-Do: support arbitrary axis transpose

validate()

validate the completeness of this chunk, there should not have black boxes.

where(mask)

find the indexes of masked value as an alternative of np.where function

Parameters:

mask (ndarray) – binary ndarray as mask

Return type:

tuple

Returns:

the coordinates in global coordinates.

Image

chunkflow.chunk.validate.validate_by_template_matching(img)

Detect 3d black boxes by template matching. 1. binarize the image. the voxels inside the black box will be false, and the outside will be true 2. The template is 7x7x2 with one section true and the other false. 3. sliding the template through the array, and detect the matching regions. 4. rotate the template to be 7x2x7 and 2x7x7, do the same detection. 5. if we can find multiple matchings in all the x,y,z direction, there is probably a black box. Note that this is always effective. If the black box is large enough to reach both sides, the detection will fail.

Parameters:

img (ndarray) – 3D image volume.

Flow

Operators

class chunkflow.flow.cloud_watch.CloudWatchOperator(log_name='chunkflow', name='cloud-watch')

watch the performance of each operator using AWS CloudWatch.

Parameters:

log_name (str) – the log used in AWS CloudWatch

Returns:

the timer result will be uploaded to AWS CloudWatch and nothing will return here.

class chunkflow.flow.downsample_upload.DownsampleUploadOperator(volume_path, factor=(2, 2, 2), chunk_mip=0, start_mip=None, stop_mip=5, fill_missing=True, name='downsample-upload')

Multiple mip level downsampling including image and segmenation.

For image, the algorithm will be automatically choosen as average pooling. For segmentation, the algorithm will be Will Silversman’s countless algorithm to perform model pooling. The most frequent segmentation ID will be choosen.

The type of chunk was automatically determined from the data type. Image: uint8, floating Segmentation: uint16, uint32, uint64,…

volume_path: (str) path of volume factor: (tuple) downsampling factor in z,y,x order chunk_mip: (int) the mip level of input chunk start_mip: (int) the mip level for starting uploading stop_mip: (int) the mip level for stoping uploading. Note that the indexing follows python indexing, this stop mip will not be included. For example, if you would like to upload mip level 1 to 4, the start mip will be 1, and the stop mip should be 5. fill_missing: (bool) fill missing blocks with zeros or not. See same parameter in cloudvolume.

class chunkflow.flow.mask.MaskOperator(volume_path, mask_mip, chunk_mip, inverse=False, fill_missing=True, name='mask')

class chunkflow.flow.mesh.MeshOperator(output_path, output_format, mip=None, voxel_size=(1, 1, 1), simplification_factor=100, max_simplification_error=8, manifest=False, skip_ids=None, shard=False, name='mesh')

Create mesh files from segmentation.

Parameters:

• output_path (str) – path to store mesh files

• output_format (str) – format of output {‘ply’, ‘obj’, ‘precomputed’}

• voxel_size (tuple) – size of voxels

• simplification_factor (int) – mesh simplification factor.

• max_simplification_error (int) – maximum tolerance error of meshing.

• manifest (bool) – create manifest files or not. This should not be True if you are only doing meshing for a segmentation chunk.

• name (str) – operator name.

• igneous. (Note that some functions are adopted from) –

class chunkflow.flow.neuroglancer.NeuroglancerOperator(name='neuroglancer', port=None, voxel_size=None)

class chunkflow.flow.save_precomputed.SavePrecomputedOperator(volume_path, mip, upload_log=True, create_thumbnail=False, fill_missing=False, name='save-precomputed')

create_chunk_with_zeros(bbox, num_channels, dtype)

Create a fake all zero chunk. this is used in skip some operation based on mask.

class chunkflow.flow.save_pngs.SavePNGsOperator(output_path='./pngs/', dtype='uint8', name='save-pngs')

class chunkflow.flow.view.ViewOperator(name='view')

Lib

chunkflow.lib.load_source(fname)

Imports a module from source.

Parameters:

fname (str) – file path of the python code.

AWS

class chunkflow.lib.aws.cloud_watch.CloudWatch(log_name, credentials=None)

monitor time elapsed of each operator using AWS CloudWatch.

Parameters:

• log_name (str) – the name of the log.

• credentials (Optional[dict]) – aws credential. If it is None, we’ll try to construct it automatically.

put_metric_data(log)

Parameters:

log (dict) – the log containing the operator time elapsed in the timer key.

class chunkflow.lib.aws.sqs_queue.SQSQueue(queue_name, visibility_timeout=3600, wait_if_empty=100, fetch_wait_time_seconds=20, retry_times=30)

upload/fetch messages using AWS Simple Queue Services.

Parameters:

• visibility_timeout (int) – make the task invisible for a while (seconds)

• wait_if_empty (int) – wait for a while and continue fetching task if the queue is empty.

• fetch_wait_time_seconds (int) – the maximum wait time if the fetched queue is empty. The maximum value is 20, which will use the long polling. If we set it to be 0, the message fetch could fail even if the queue has messages. This problem is due to the fact that the message in queue is managed distributedly, and the query was only sent to a few servers. Normally, we should set fetch wait time to use long poll. checkout the AWS documentation

delete(receipt_handle)

Parameters:

receipt_handle (str) – a random string as a handle of the message in queue.

send_message_list(message_list)

Use batch mode to send a bunch of messages quickly.

Parameters:

message_list (list) – a list of input messages. the messages are string.