Working with datasets

We have 4 main goals for this tutorial:

1. Create a Dataset
   1. From data in memory
   2. From a snapshot
2. Run packingcubes on the dataset
3. Save the sorted dataset
4. Load the dataset

For the tutorial, you'll need

Install and Import Dependencies¶

As always for a tutorial, we'll need to install and import the dependencies we'll need:

$ pip install packingcubes

$ pixi add packingcubes --pypi

Now import the modules we need

import numpy as np 

import packingcubes

Creating a Dataset¶

The first thing we need to do for anything with packingcubes that will be used long term is create a Dataset object. We'll demonstrate both of the primary ways to do that here.

From a Snapshot¶

Creating a dataset from a snapshot is pretty straightforward if it's Gadget-based¹.

dataset = packingcubes.GadgetishHDF5Dataset(filepath="./snapshot_090.hdf5")

We need to choose which particles we want to look at

dataset.particle_type = "PartType0" # the gas particles

Then we can check the particle positions are loaded:

dataset.positions

From positions in memory¶

We'll start by generating some random data. We'll make 1000 particles with coordinates ranging from 0 to 100.

positions = np.random.uniform(size=(1000,3)) * 100

Then to create a dataset, just use

inmem_dataset = packingcubes.InMemory(positions=positions)

Run packingcubes¶

We now want to cubify our datasets.

It's actually the same command (Cubes), regardless of which kind of dataset you're using. However, we'll need to specify that we only want the gas particles at this time for the GadgetishHDF5Dataset²³.

cubes = packingcubes.Cubes(dataset, particle_type="PartType0")
inmem_cubes = packingcubes.Cubes(inmem_dataset)

These cubes are ready for searching!

cubes.get_particle_indices_in_sphere(
    center=[13548,23147,1684], radius=1000,
)

inmem_cubes.get_particle_indices_in_sphere(
    center=[64,73,15], radius=20,
)

Saving a Dataset¶

Generally, you'll only want to do the initial cubing once and save the results. (Though it is fast enough you could regenerate it each time for "small" datasets).

Saving the cubes structure doesn't save the sorted particle position information to disk. You'll need to separately save it:

dataset.save(output_file="sorted_positions.hdf5")
inmem_dataset.save(output_file="inmem_positions.hdf5")

Loading a sorted dataset¶

If you have an already sorted dataset, like we now do, you have two options:

1. Specify the sorted dataset as your dataset filepath - Simpler, but will only contain the fields you've already sorted (so just positions)
2. Pass the sorted dataset to the sorted_filepath parameter - An extra step, but will check the sorted dataset for any fields first before loading from the original.

We'll pick option 2.

dataset_reloaded = packingcubes.GadgetishHDF5Dataset(
    filepath="snapshot_090.hdf5",
    sorted_filepath="sorted_positions.hdf5",
)

dataset_reloaded.positions

Note the sorted positions!

Sorting additional fields¶

Once the dataset is sorted, we can include additional fields to be linked to the particles. The process takes two steps:

1. Define a mapping between names and fields. Names are what you want the field to be referenceable as (say mass), and fields are the actual data. All datasets support two types: N length vectors and NxM sized matrices. For HDF5-based Datasets, a string corresponding to the name of the HDF5 dataset on disk is also supported (e.g., if you provide field_name, packingcubes will look for a dataset in the form current_particle_type/field_name). Example:

   calculated_field = calculation()
   extra_fields = {"Mass":"Mass", "vx":"Velocity_x", "cf":calculated_field}
   

2. Provide the mapping to the dataset. This will sort the data using the shuffle list in a one-time pass. Note that this means this step must be done after the cubing process! Otherwise the field will retain the original order.

   dataset.process_extra_fields(extra_fields)
   

The extra fields will now be available as dataset attributes:

mass = dataset.Mass
vx = dataset.vx
dcf = dataset.cf

dataset.process_extra_fields(
    {
        "sf":(already_sorted, True),
        "sdf":("already_sorted_on_disk",True),
        "nsf":"not_sorted",
    }
)

All-in-one¶

As mentioned previously, the Cubes command combines a number of the above steps into one command. So if you don't need any additional flexibility, or direct access to the dataset positions/shuffle list/etc., you can include the dataset saving and extra fields in the initial Cubes call via the sorted_filepath, save_dataset, and extras parameters, like so:

cubes = packingcubes.Cubes(
    "./snapshot_090.hdf5", 
    particle_type="PartType0",
    sorted_filepath="sorted_positions.hdf5",
    save_dataset=True,
    extras={"Mass":"Mass","vx":"Velocity_x"},
)

The Dataset is then available via cubes.dataset.