Parthenon Set-Dimensional Arrays

We provide type aliases to Kokkos::View. They are

template <typename T>
using ParArray1D = Kokkos::View<T *, LayoutWrapper, DevMemSpace>;
template <typename T>
using ParArray2D = Kokkos::View<T **, LayoutWrapper, DevMemSpace>;
template <typename T>
using ParArray3D = Kokkos::View<T ***, LayoutWrapper, DevMemSpace>;
template <typename T>
using ParArray4D = Kokkos::View<T ****, LayoutWrapper, DevMemSpace>;
template <typename T>
using ParArray5D = Kokkos::View<T *****, LayoutWrapper, DevMemSpace>;
template <typename T>
using ParArray6D = Kokkos::View<T ******, LayoutWrapper, DevMemSpace>;

where LayoutWrapper is currently hardcoded to Kokkos::LayoutRight. DevMemSpace is the memory space associated with the default execution space. If UVM is enabled, it is Kokkos::CudaUVMSpace.

Parthenon Arbitrary-Dimensional Arrays

ParArrayND is a wrapper around a rank 6 Kokkos::View. It provides a rank-agnostic way to create, manage, and carry Kokkos::Views. Since it is built on Kokkos::View, it is reference counted, works on GPUs, and is almost as performant as Kokkos::View.

The index and size convention is the same as in Athena++. The fastest moving indexes come last.

Constructors

To construct, call it with ParArrayND(label,dimensions...), where dimensions is some number of ints specifying the shape of the array. e.g.,

ParArrayND<Real> myArray("a 3d array", 6, 5, 4);

If you don’t know what to name your array, try using the PARARRAY_TEMP macro as the name. It is provided to name the array based on the file and line where it is created.

Rank counting

The rank of the object is indexed from 1, not 0. In other words, the code

ParArrayND<Real> myArray("a 3d array", 6, 5, 4);
std::cout << myArray.GetDim(3) << " "
          << myArray.GetDim(2) << " "
          << myArray.GetDim(1) << std::endl;

prints out 6 5 4.

Accessors

Similarly, you can access it using operator() with up to six integers specifying the indices. Unspecified (slow-moving) indices are assumed to be zero.

Slicing

You can slice a ParArrayND using Kokkos::subview as described in Kokkos documentation. We have overloaded Kokkos::subview to support this.

Get

A Kokkos::View with the dimensionality you want can be extracted with ParArrayND.Get<D>();. This returns a rank-D view. Dimensions higher than D are set to zero.

Type Subtleties

Note that the type returned by ParArrayND.Get<D>() is not not a simple type. For example, the type returned by ParArrayND<Real>.Get<4>(); is:

Kokkos::View<double****, Kokkos::LayoutRight, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, Kokkos::MemoryTraits<0> >

while the type for ParArray4D<Real> is

Kokkos::View<double*****, Kokkos::LayoutRight, Kokkos::HostSpace>

These two types are compatible. So you can set one equal to the other and do implicit casts. I.e., the following works:

void doNothingByValue(ParArray4D<Real> array) {}
ParArrayND<Real> ndarray;
auto b = ndarray.Get<4>();
doNothingByValue(b);

However, implicit casts on reference variables are not performed. So the following fails:

void doNothingByReference(ParArray4D<Real>& array) {}
ParArrayND<Real> ndarray;
auto b = ndarray.Get<4>();
doNothingByReference(b);

To avoid this issue, you can:

  • Explicitly typecast ParArrayND when using it in conjunction with Kokkos views.

  • Pass views by reference

  • Template appropriate functions on array type

For more details, see here.

Mirrors and Deep Copies

ParArrayND requires mirrors and deep copies, just like the Kokkos views it wraps. You can get one via, e.g.,

auto my_mirror = my_array.GetMirror(my_memory_space());

you can then deep copy into my_mirror with

my_mirror.DeepCopy(my_array);

ParArrayND provides two convenience functions, GetHostMirror() and GetDeviceMirror() which put a mirror on the host and device respectively. In addition, GetHostMirrorAndCopy() creates a new ParArrayND on the host with identical layout and deep copies the content, e.g.,

auto my_host_array = my_array.getHostMirrorAndCopy();

A note on templates

Strictly, ParArrayND is a specialization of ParArrayGeneric, which wraps an arbitrary container. The specializations and type aliases available are as follows:

template<typename T, typename Layout=LayoutWrapper>
using device_view_t = Kokkos::View<T******,Layout,DevMemSpace>;

template<typename T, typename Layout=LayoutWrapper>
using host_view_t = typename device_view_t<T,Layout>::HostMirror;

template<typename T, typename Layout=LayoutWrapper>
using ParArrayND = ParArrayGeneric<device_view_t<T,Layout>>;

template<typename T, typename Layout=LayoutWrapper>
using ParArrayHost = ParArrayGeneric<host_view_t<T,Layout>>;

Examples of use

See the unit test for how to use ParArrayND.