.. _pararrays: Parthenon Set-Dimensional Arrays ================================ We provide type aliases to ``Kokkos::View``. They are .. code:: c++ template <typename T> using ParArray1D = Kokkos::View<T *, LayoutWrapper, DevMemSpace>; template <typename T> using ParArray2D = Kokkos::View<T **, LayoutWrapper, DevMemSpace>; template <typename T> using ParArray3D = Kokkos::View<T ***, LayoutWrapper, DevMemSpace>; template <typename T> using ParArray4D = Kokkos::View<T ****, LayoutWrapper, DevMemSpace>; template <typename T> using ParArray5D = Kokkos::View<T *****, LayoutWrapper, DevMemSpace>; template <typename T> using ParArray6D = Kokkos::View<T ******, LayoutWrapper, DevMemSpace>; where ``LayoutWrapper`` is currently hardcoded to ``Kokkos::LayoutRight``. ``DevMemSpace`` is the memory space associated with the default execution space. If UVM is enabled, it is ``Kokkos::CudaUVMSpace``. Parthenon Arbitrary-Dimensional Arrays ====================================== ``ParArrayND`` is a wrapper around a rank 6 ``Kokkos::View``. It provides a rank-agnostic way to create, manage, and carry ``Kokkos::Views``. Since it is built on ``Kokkos::View``, it is reference counted, works on GPUs, and is almost as performant as ``Kokkos::View``. The index and size convention is the same as in ``Athena++``. The fastest moving indexes come last. Constructors ------------ To construct, call it with ``ParArrayND(label,dimensions...)``, where dimensions is some number of ints specifying the shape of the array. e.g., .. code:: c++ ParArrayND<Real> myArray("a 3d array", 6, 5, 4); If you don’t know what to name your array, try using the ``PARARRAY_TEMP`` macro as the name. It is provided to name the array based on the file and line where it is created. Rank counting ------------- The rank of the object is indexed from 1, not 0. In other words, the code .. code:: c++ ParArrayND<Real> myArray("a 3d array", 6, 5, 4); std::cout << myArray.GetDim(3) << " " << myArray.GetDim(2) << " " << myArray.GetDim(1) << std::endl; prints out ``6 5 4``. Accessors --------- Similarly, you can access it using ``operator()`` with up to six integers specifying the indices. Unspecified (slow-moving) indices are assumed to be zero. Slicing ------- You can slice a ``ParArrayND`` using ``Kokkos::subview`` as described in Kokkos documentation. We have overloaded ``Kokkos::subview`` to support this. Get --- A ``Kokkos::View`` with the dimensionality you want can be extracted with ``ParArrayND.Get<D>();``. This returns a rank-D view. Dimensions higher than D are set to zero. Type Subtleties ~~~~~~~~~~~~~~~ Note that the type returned by ``ParArrayND.Get<D>()`` is not not a simple type. For example, the type returned by ``ParArrayND<Real>.Get<4>();`` is: .. code:: c++ Kokkos::View<double****, Kokkos::LayoutRight, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, Kokkos::MemoryTraits<0> > while the type for ``ParArray4D<Real>`` is .. code:: c++ Kokkos::View<double*****, Kokkos::LayoutRight, Kokkos::HostSpace> These two types are compatible. So you can set one equal to the other and do implicit casts. I.e., the following works: .. code:: c++ void doNothingByValue(ParArray4D<Real> array) {} ParArrayND<Real> ndarray; auto b = ndarray.Get<4>(); doNothingByValue(b); However, implicit casts on reference variables are not performed. So the following fails: .. code:: c++ void doNothingByReference(ParArray4D<Real>& array) {} ParArrayND<Real> ndarray; auto b = ndarray.Get<4>(); doNothingByReference(b); To avoid this issue, you can: - Explicitly typecast ``ParArrayND`` when using it in conjunction with Kokkos views. - Pass views by reference - Template appropriate functions on array type For more details, see `here <https://github.com/parthenon-hpc-lab/parthenon/issues/143>`__. Mirrors and Deep Copies ----------------------- ``ParArrayND`` requires mirrors and deep copies, just like the ``Kokkos`` views it wraps. You can get one via, e.g., .. code:: c++ auto my_mirror = my_array.GetMirror(my_memory_space()); you can then deep copy into ``my_mirror`` with .. code:: c++ my_mirror.DeepCopy(my_array); ``ParArrayND`` provides two convenience functions, ``GetHostMirror()`` and ``GetDeviceMirror()`` which put a mirror on the host and device respectively. In addition, ``GetHostMirrorAndCopy()`` creates a new ``ParArrayND`` on the host with identical layout and deep copies the content, e.g., .. code:: c++ auto my_host_array = my_array.getHostMirrorAndCopy(); A note on templates ------------------- Strictly, ``ParArrayND`` is a specialization of ``ParArrayGeneric``, which wraps an arbitrary container. The specializations and type aliases available are as follows: .. code:: c++ template<typename T, typename Layout=LayoutWrapper> using device_view_t = Kokkos::View<T******,Layout,DevMemSpace>; template<typename T, typename Layout=LayoutWrapper> using host_view_t = typename device_view_t<T,Layout>::HostMirror; template<typename T, typename Layout=LayoutWrapper> using ParArrayND = ParArrayGeneric<device_view_t<T,Layout>>; template<typename T, typename Layout=LayoutWrapper> using ParArrayHost = ParArrayGeneric<host_view_t<T,Layout>>; Examples of use --------------- See the `unit test <https://github.com/parthenon-hpc-lab/parthenon/blob/develop/tst/unit/test_pararrays.cpp>`__ for how to use ``ParArrayND``.