syconn.exec package

Submodules

• syconn.exec.exec_init module

• syconn.exec.exec_inference module

• syconn.exec.exec_skeleton module

• syconn.exec.exec_syns module

syconn.exec.exec_init module

syconn.exec.exec_init.init_cell_subcell_sds(chunk_size=None, n_folders_fs=10000, n_folders_fs_sc=10000, max_n_jobs=None, load_cellorganelles_from_kd_overlaycubes=False, transf_func_kd_overlay=None, cube_of_interest_bb=None, overwrite=False)

Converts a binary class segmentation mask of sub-cellular structure predictions into an instance segmentation using connected components / watershed. Then, it extracts the properties of sub-cellular structures (voxel count, coordinate, bounding box, mesh, etc.) and their associations with cell fragments by calculating the overlap between every sub-cellular structure and cell fragment instance.

Parameters

• chunk_size (Optional[Tuple[int, int, int]]) – Size of the cube which are processed by each worker.

• n_folders_fs (int) – Number of folders used to create the folder structure in the resulting SegmentationDataset for the cell supervoxels.

• n_folders_fs_sc (int) – Number of folders used to create the folder structure in the resulting SegmentationDataset for the cell organelle supervoxels.

• max_n_jobs (Optional[int]) – Number of parallel jobs.

• load_cellorganelles_from_kd_overlaycubes (bool) – If True, cell organelles will be loaded from KnossosDataset overlay cubes.

• transf_func_kd_overlay (Optional[Dict[Any, Callable]]) – Transformation function applied on the probability map or segmentation data.

• cube_of_interest_bb (Optional[Union[tuple, np.ndarray]]) – Bounding box of the (sub-) volume of the dataset which is processed (minimum and maximum coordinates in mag1 voxels, XYZ).

• overwrite (bool) – If True, will overwrite existing data.

syconn.exec.exec_init.kd_init(co, chunk_size, transf_func_kd_overlay, load_cellorganelles_from_kd_overlaycubes, cube_of_interest_bb, log)

This function initializes a per-object segmentation KnossosDataset for the given supervoxel type based on an initial prediction. The location of the prediction has to be defined in the config.yml file for the object. The results will be stored as a KnossosDataset at “{}/knossosdatasets/{}_seg/”.format(global_params.config.working_dir, co).

Examples

Was used to process sub-cellular structures independently:

ps = [Process(target=kd_init, args=[co, chunk_size, transf_func_kd_overlay,

load_cellorganelles_from_kd_overlaycubes, cube_of_interest_bb, log]) for co in global_params.config[‘process_cell_organelles’]]

for p in ps:

p.start() time.sleep(5)

for p in ps:

p.join() if p.exitcode != 0:

raise Exception(f’Worker {p.name} stopped unexpectedly with exit code {p.exitcode}.’)

Parameters

• co – Type of cell organelle supervoxels, e.g ‘mi’ for mitochondria or ‘vc’ for vesicle clouds.

• chunk_size – Size of the cube which are processed by each worker.

• transf_func_kd_overlay (Optional[Callable]) – Transformation applied on the prob. map or segmentation data.

• load_cellorganelles_from_kd_overlaycubes (bool) – If True, loads cell organelles from kd overlay cubes.

• cube_of_interest_bb (Tuple[ndarray]) – Bounding of the (sub-) volume of the dataset which is processed.

• log (Logger) – Logger object for logging progress and debug information.

syconn.exec.exec_init.run_create_neuron_ssd(apply_ssv_size_threshold=False, ncores_per_job=1, overwrite=False)

This function creates a SuperSegmentationDataset with version 0 at the currently active working directory based on the SV graph at the neuron_svgraph_path. If astrocyte splitting is active, this will be the SV graph after astrocyte removal, if it was disabled it is identical to pruned_svgraph.bz2.

Parameters

• apply_ssv_size_threshold (bool) – If True, applies a filter with minimum bounding box diagonal. This is usually not needed as the filter is applied either in run_create_rag (prior_astrocyte_removal=False) or during the astrocyte separation.

• ncores_per_job (int) – Number of cores per worker for SuperSegmentationDataset.save_dataset_deep.

• overwrite (bool) – If True, overwrites existing data.

Notes

• This is a memory intensive step, consider increasing ncores_per_job.

• Requires init_cell_subcell_sds and optionally run_astrocyte_splitting.

• Networkx requires a lot of memory for >1e9 edges, graph_tool and igraph are not usable for this either. Currently the workaround is to not use the graph information when storing the cell SV graph, but only the connected component as a graph with N-1 edges (N nodes). [TODO]

syconn.exec.exec_init.run_create_rag(graph_node_dtype=None)

Stores pruned Region Adjacency Graph (RAG) or pruned supervoxel (SV) graph based on the configuration. If ‘prior_astrocyte_removal’ is set to True in the configuration, it stores the pruned RAG at global_params.config.pruned_svgraph_path, required for all glia removal steps. run_astrocyte_splitting() will finally store the neuron SV graph. Otherwise, it stores the pruned SV graph at pruned_svgraph_path, required by run_create_neuron_ssd().

Parameters

graph_node_dtype (Optional) – Data type of the graph nodes. Defaults to np.uint64.

syconn.exec.exec_init.sd_init(co, max_n_jobs, log=None)

This function initializes a SegmentationDataset of a given supervoxel type.

Parameters

• co (str) – Cellular organelle identifier (e.g. ‘mi’, ‘vc’, …).

• max_n_jobs (int) – Number of parallel jobs.

• log (Optional[Logger]) – Logger object for logging progress and debug

• information. –

syconn.exec.exec_inference module

syconn.exec.exec_inference.run_astrocyte_prediction()

This function predicts astrocyte supervoxels based on the img2scalar Cellular Morphology Network (CMN). It requires the functions syconn.exec_init.init_cell_subcell_sds and run_astrocyte_rendering to have been executed beforehand.

Parameters

None –

Notes

Requires init_cell_subcell_sds() and run_astrocyte_rendering().

syconn.exec.exec_inference.run_astrocyte_prediction_pts(max_n_jobs_gpu=None)

This function predicts astrocyte and neuron supervoxels using point cloud-based convolutional networks. It requires the function syconn.exec_init.init_cell_subcell_sds to have been executed beforehand.

Note: Post-processing currently requires locking. To prevent locking, an additional map-reduce step is required to write the final probabilities of all SVs in a “per-storage” (per chunk attribute dict) fashion.

Parameters

• max_n_jobs_gpu (Optional[int]) – The maximum number of parallel GPU jobs. If not specified, it

• None. (defaults to) –

syconn.exec.exec_inference.run_astrocyte_splitting()

This function uses the pruned Region Adjacency Graph (RAG) at global_params.config.pruned_svgraph_path (stored as edge list .bz2 file) which is computed in syconn.exec.exec_init.init_cell_subcell_sds to split astrocyte fragments from neuron reconstructions and separate those and entire glial cells from the neuron supervoxel graph.

It stores neuron SV graph at syconn.handler.config.DynConfig.neuron_svgraph_path which is then used by syconn.exec.exec_init.run_create_neuron_ssd.

Note: This function requires syconn.exec_init.init_cell_subcell_sds, run_astrocyte_rendering and run_astrocyte_prediction to have been executed beforehand.

Parameters

None –

syconn.exec.exec_inference.run_cell_embedding(max_n_jobs=None)

This function infers cell embeddings for all neuron reconstructions based on a triplet-loss trained cellular morphology learning network (tCMN). The point-based model is trained with the pts_loader_scalar (used for cell types). Multi-views functionality is not implemented.

Parameters

• max_n_jobs (Optional[int]) – Number of parallel jobs. If not provided, it defaults to the

• 4. (total number of GPUs available multiplied by) –

Notes

This function requires the following functions to be run beforehand: - run_create_neuron_ssd from syconn.exec.exec_init - run_neuron_rendering - run_skeleton_generation from syconn.exec.skeleton

syconn.exec.exec_inference.run_celltype_prediction(max_n_jobs_gpu=None)

This function runs the cell type inference based on the img2scalar CMN.

Parameters

• max_n_jobs_gpu (Optional[int]) – Number of parallel GPU jobs. If not provided, it defaults to the total

• enabled (number of GPUs available multiplied by 4 if batch jobs are) –

• defaults (otherwise it) –

• 2. (to) –

Notes

This function requires the following functions to be run beforehand: - run_create_neuron_ssd from syconn.exec.exec_init - run_neuron_rendering

syconn.exec.exec_inference.run_morphology_embedding(max_n_jobs=None)

This function infers local morphology embeddings for all neuron reconstructions based on a triplet-loss trained cellular morphology learning network (tCMN). The point-based model is trained with the pts_loader_scalar (used for cell types).

Parameters

• max_n_jobs (Optional[int]) – Number of parallel jobs. If not provided, it defaults to the total number

• 4. (of GPUs available multiplied by) –

Notes

This function requires the following functions to be run beforehand: - run_create_neuron_ssd from syconn.exec.exec_init - run_neuron_rendering - run_skeleton_generation from syconn.exec.skeleton

syconn.exec.exec_inference.run_semsegaxoness_prediction(max_n_jobs_gpu=None)

This function infers and maps the semantic segmentation of the 2D projections onto the cell reconstruction mesh (ssv.label_dict(‘vertex’)) via semseg_of_sso_nocache. It generates the following skeleton attributes by semsegaxoness2skel and makes them available in SuperSegmentationObject.skeleton:

• “axoness”: Vertex predictions mapped to skeleton (see global_params.config['compartments']['map_properties_semsegax']).

• “axoness_avg10000”: Sliding window average along skeleton (10um traversal length)

• “axoness_avg10000_comp_maj”: Majority vote on connected components after removing the soma

Parameters

• max_n_jobs_gpu (Optional[int]) – Number of parallel GPU jobs. If not provided, it defaults to the total

• enabled (number of GPUs available multiplied by 4 if batch jobs are) –

• 1. (otherwise it defaults to) –

syconn.exec.exec_inference.run_semsegspiness_prediction(max_n_jobs_gpu=None)

This function stores semantic spine labels inside ssv.label_dict(‘vertex’)[‘spiness].

Parameters

• max_n_jobs_gpu (Optional[int]) – Number of parallel GPU jobs. If not provided, it

• if (defaults to the total number of GPUs available multiplied by 4) –

• enabled (batch jobs are) –

• 1. (otherwise it defaults to) –

syconn.exec.exec_skeleton module

syconn.exec.exec_skeleton.map_myelin_global(max_n_jobs=None)

This function maps myelin to cell reconstruction skeletons globally. It is a standalone function that can be used for mapping right after skeleton generation. See kwarg map_myelin in run_skeleton_generation().

Parameters

max_n_jobs (Optional[int]) – Number of parallel jobs.

syconn.exec.exec_skeleton.run_kimimaro_skeletonization(max_n_jobs=None, map_myelin=None, cube_size=None, cube_of_interest_bb=None, ds=None, ncores_skelgen=2)

This function generates the cell reconstruction skeletons using the kimimaro tool. The kimimaro tool is used for volume-based skeletonization. The functions are in proc.sekelton, GSUB_kimimaromerge, QSUB_kimimaroskelgen.

Parameters

• max_n_jobs (Optional[int]) – Number of parallel jobs.

• map_myelin (Optional[bool]) – If True, maps myelin predictions at every skeleton['nodes'] in skeleton.

• cube_size (np.ndarray) – Cube size used within each worker. This should be as big as possible to prevent un-centered skeletons in cell compartments with big diameters. In mag 1 voxels.

• cube_of_interest_bb (Optional[tuple]) – Partial volume of the data set. Bounding box in mag 1 voxels: (lower coord, upper coord).

• ds (Optional[np.ndarray]) – Downsampling.

• ncores_skelgen (int) – Number of cores used during skeleton generation.

syconn.exec.exec_skeleton.run_skeleton_generation(cube_of_interest_bb=None, map_myelin=None, ncores_skelgen=2)

This function generates the skeleton of the cell reconstruction. It decides whether to use the kimimaro tool for volume-based skeletonization or the fallback method for SSV-based skeletonization based on the global parameters.

Parameters

• cube_of_interest_bb (Optional[Union[tuple, np.ndarray]]) – Partial volume of the data set. Bounding box in mag 1 voxels: (lower coord, upper coord).

• map_myelin (Optional[bool]) – If True, maps myelin predictions at every skeleton['nodes'] in skeleton.

• ncores_skelgen (int) – Number of cores used during skeleton generation.

syconn.exec.exec_skeleton.run_skeleton_generation_fallback(max_n_jobs=None, map_myelin=None)

This function generates the cell reconstruction skeletons using the fallback method. The fallback method is SSV-based skeletonization on mesh vertices.

Parameters

• max_n_jobs (Optional[int]) – Number of parallel jobs.

• map_myelin (Optional[bool]) – If True, maps myelin predictions at every skeleton['nodes'] in skeleton.

syconn.exec.exec_syns module

syconn.exec.exec_syns.run_cs_ssv_generation(n_folders_fs=10000, overwrite=False)

This function creates agglomerated contact site objects between cells. For this, ‘cs’ objects need to be extracted. It takes two parameters: n_folders_fs and overwrite.

Parameters

• n_folders_fs (int) – Number of folders used to create the folder structure in each SegmentationDataset.

• overwrite (bool) – If True, existing SegmentationDataset objects of type ‘cs_ssv’ will be overwritten.

syconn.exec.exec_syns.run_matrix_export()

This function exports the connectivity matrix as a .csv file to the ‘connectivity_matrix’ folder in the current working directory. It also collects various synapse properties from previous analysis steps, including:

• ‘partner_axoness’: Cell compartment type (axon: 1, dendrite: 0, soma: 2, en-passant bouton: 3, terminal bouton: 4) of the partner neurons.

• ‘partner_spiness’: Spine compartment predictions (0: dendritic shaft, 1: spine head, 2: spine neck, 3: other) of both neurons.

• ‘partner_spineheadvol’: Spinehead volume in µm^3 of pre- and post-synaptic partners.

• ‘partner_celltypes’: Celltype of the both neurons.

• ‘latent_morph’: Local morphology embeddings of the pre- and post- synaptic partners.

Examples

See SegmentationDataset for examples.

syconn.exec.exec_syns.run_spinehead_volume_calc()

This function calculates spine head volumes based on a watershed segmentation which is run on 3D spine label masks propagated from cell surface predictions. Spine head volumes are stored in the SSV attribute dictionary with the key ‘partner_spineheadvol’ in µm^3.

Subsequent call to collect_properties_from_ssv_partners() will add this property to the attribute dict of all syn_ssv. Calling syconn.proc.sd_proc.dataset_analysis() accordingly collects all syn_ssv properties and makes them available as numpy arrays. These two steps are performed in run_matrix_export().

syconn.exec.exec_syns.run_syn_generation(chunk_size=(512, 512, 512), n_folders_fs=10000, max_n_jobs=None, cube_of_interest_bb=None, overwrite=False, transf_func_sj_seg=None)

This function runs the synapse generation process. It creates SegmentationDataset objects with the following versions:

• ‘cs’: Contact site objects between supervoxels.

• ‘syn’: Objects representing the overlap between ‘cs’ and the initial synaptic junction predictions. These objects effectively represent synapse fragments between supervoxels.

• ‘syn_ssv’: Final synapse objects. Agglomerated ‘syn’ objects based on the supervoxel graph.

• ‘cs_ssv’: Final contact site objects. Agglomerated ‘cs’ objects based on the supervoxel graph.

  Only processed if ['cell_contacts']['generate_cs_ssv'] is set to True in the config.

Parameters

• chunk_size (Optional[Tuple[int, int, int]]) – The size of processed cubes.

• n_folders_fs (int) – Number of folders used to create the folder structure in each SegmentationDataset.

• max_n_jobs (Optional[int]) – Number of parallel jobs.

• cube_of_interest_bb (Union[tuple, ndarray, None]) – Defines the bounding box of the cube to process. By default this is set to (np.zoers(3); kd.boundary).

• overwrite (bool) –

• transf_func_sj_seg (Optional[Callable]) – Method that converts the cell organelle segmentation into a binary mask of background vs. sj foreground.