6. CkDyn: Dynamic Chares for Language Bindings
This manual is for authors of language bindings: a Python, Julia or Rust
package, or a C++ program, that defines Charm++ chare classes at run time
instead of in a .ci file. It describes the C API in
src/ck-core/ckdyn.h from the binding’s side: what to call, in which
order, and what the runtime expects of the binding’s code. How the runtime
implements each feature is in the internals chapter,
doc/internals/ckdyn.md; this manual points to it where the mechanism
matters to a binding author.
6.1. Introduction
6.1.1. What CkDyn is
CkDyn lets a program register chare classes at run time. A class is a
descriptor: a name, a kind (main chare, chare, group, nodegroup, array or
array map) and a table of entry methods, each with a name and attributes.
The runtime reaches the objects of these classes through one table of C
function pointers that the binding supplies, the hook table. Every entry
method of every dynamic class receives the same message type, CkDynMsg:
a header followed by typed slots (integers, floats, byte strings, typed
n-dimensional buffers, futures, callbacks, section ids, proxies) and an
opaque tail that the binding fills with its own serialized data.
The objects behind a dynamic class are ordinary Charm++ objects: six fixed C++ classes stand in for every dynamic class of their kind, and each holds an opaque handle that the binding owns. Everything the runtime does for a charmxi class it therefore does for a dynamic class:
point-to-point sends, broadcasts, inline and expedited sends, message priorities;
groups (also constrained to a PE list), nodegroups (with exclusive entries), arrays of one to six index dimensions, array maps, dynamic insertion;
reductions with the built-in reducers and with reducers the binding defines;
array and group sections over CkMulticast, with multicasts and section reductions;
migration, load balancing (AtSync and anytime migration), checkpoint to disk and restart;
quiescence detection;
zero-copy transfer of host buffers and, in a CUDA or HIP build, device buffers;
tracing: Projections shows the binding’s class and method names and the index of each array element.
6.1.2. Who it is for
CkDyn is for code that cannot run charmxi: an interpreted or JIT-compiled
language whose classes exist only at run time, or a C++ program whose set of
chare classes is decided by its input. The API is C: plain structs with
fixed-width fields, function pointers and extern "C" functions.
ckdyn.h includes no Charm++ header, so a binding builds without the
Charm++ include tree, and a language with a C foreign-function interface
(Julia ccall, Rust extern "C", Python with any extension tool) can
call it directly.
6.1.3. What a binding provides and what it gets
A binding provides:
A hook table (
CkDynHooks): functions that construct, destroy, deliver to, pack and unpack the binding’s objects, plus hooks for futures, reductions with binding-defined reducers, array maps, load balancing resumption, section membership, readonly data and errors (The hook table).A class descriptor per class (
CkDynClassDesc) with an entry table (CkDynEntryDesc), registered withCkDynRegisterClass(Start-up and Registration).A message convention: how its values map onto slots and the tail. CkDyn defines the slot types; which slots an entry method expects is up to the binding (Messages).
It gets, per class, a Charm++ chare index and a contiguous range of entry
indices, and from then on the functions of ckdyn.h: create objects,
send, broadcast, contribute, create sections, set futures, migrate,
checkpoint, trace.
6.1.4. The two existing bindings
The C++ test binding,
tests/charm++/ckdyn/cbinding.handcbinding.C(about 450 lines). A class is a table of C++ function pointers; an object is a small struct. The acceptance testtests/charm++/ckdyn/ckdyntest.Cuses it to exercise every feature without Python. It is the reference for this manual’s examples (Writing a Binding).charm4py 2, the Python binding: the extension module
src/ckdyn_py.cppof the charm4py repository (branchthin), built with nanobind. It shows the patterns a binding for an interpreted language needs: taking the interpreter lock in every hook, running entry methods on coroutines (greenlets), zero-copy numpy views over retained messages, and its own encodings inside the slot types.
6.1.5. Build requirements
CkDyn is compiled into libck on every Charm++ target built on
Reconverse; nothing in it runs until a program calls CkDynInit or
CkDynRegisterClass. It is not available on the frozen classic Converse
line.
Two build shapes are in use:
A C++ program links CkDyn from
libckand starts like any Charm++ program. Anycharm++build works, for example:./build charm++ reconverse-linux-x86_64 --with-production -j8
The program includes
ckdyn.hand is linked withcharmc; it has a.cifile only for its own main module (the C++ test binding has one, with aninitnodethat registers the dynamic classes).A shared-library binding (Python, Julia, Rust) loads
libcharmas a shared library and starts the runtime itself withCkDynStart. Thecharm4pybuild target produces that library:./build charm4py reconverse-linux-x86_64 -j8 --with-production \ --disable-fortran --with-refnum-type="unsigned int"
Both shapes support every feature, including registration after start-up (Late registration).
Reference number width. Futures travel in the envelope’s reference
number, whose type is CMK_REFNUM_TYPE. The default build has a 16-bit
reference number, which limits future ids to 1 to 65534. A binding that
needs more builds Charm++ with --with-refnum-type="unsigned int", which
allows ids from 1 to 232-2, and checks CkDynRefnumBytes() at
start-up (charm4py refuses to start unless it returns 4).
Device support. DEVICE slots, device posts and
CkDynHapiAddCallback need a build with CUDA or HIP;
CkDynHasDeviceSupport() reports it, and the device functions abort in
other builds.
6.1.6. API version and ABI rules
CKDYN_API_VERSION (currently 2) is bumped on any incompatible change to
ckdyn.h. Additions (a new function, such as CkDynContributeShaped) do
not bump it: an older binding keeps working with a newer library, while a
binding that uses an addition needs a library that has it and fails to load
with an undefined symbol against an older one, before CkDynApiVersion()
can be called. A binding pins the Charm++ commit it is built with
(charm4py’s charm_version.txt). Three checks keep a binding and a libcharm consistent:
CkDynApiVersion()returns the version the library was built with; a binding compares it with theCKDYN_API_VERSIONit was compiled against before calling anything else.The hook table starts with
versionandsizefields. The binding setsversion = CKDYN_API_VERSIONandsize = sizeof(CkDynHooks)as it compiled it.CkDynInitaborts if either differs from the library’s, so a binding compiled against a header with a different table is rejected even when the version number matches. A class’s own hook table (CkDynClassDesc.hooks) is checked the same way at registration.The structs that CkDyn copies as bytes into messages have fixed-width integer fields only, and the library fixes their sizes at compile time:
CkDynTarget40 bytes,CkDynSectionID16,CkDynCallback88,CkDynHdr24,CkDynSlot24. The enums only supply constants; a binding in another language declares these structs with the same field widths and order and checks the sizes.
6.2. Concepts
6.2.1. Objects and handles
An object of a dynamic class is a Charm++ object of one of six fixed C++
classes, one per kind, that holds the binding’s handle: a void *
returned by the construct, mainchare or unpack hook. CkDyn
stores the handle and passes it back to every hook that concerns the object;
it never dereferences it. A handle must be non-NULL, because CkDyn uses
NULL to mark a group branch that was not constructed (a branch outside a
constrained group’s PE list).
The binding decides what a handle points to: a C struct, a C++ object, a
reference to an interpreter object (charm4py passes a PyObject * that
holds a reference). The object’s lifetime ends with the destruct hook,
which CkDyn calls when the object is deleted, when it migrates away from
the PE, and at exit.
6.2.2. Targets
A CkDynTarget names one object of any kind. It is a plain 40-byte
struct that a binding can store, copy, compare and put into messages:
typedef struct CkDynTarget {
int32_t kind; /* CkDynKind */
int32_t chare_idx; /* the Charm++ chare index of its class */
union {
struct { int32_t pe; void *ptr; } chare; /* MAINCHARE, CHARE */
struct { int32_t gid; int32_t pe_or_node; } group; /* GROUP, NODEGROUP, ARRAYMAP */
struct { int32_t aid; int32_t ndims; int32_t index[CKDYN_MAX_DIMS]; } array;
} u;
} CkDynTarget;
CKDYN_MAINCHARE,CKDYN_CHARE: theCkChareID(PE and object pointer on that PE).CkDynCreateCharereturns a virtual id with a negative PE that is usable for sends and callbacks at once; theconstructhook receives the real id. Both name the same chare.CKDYN_GROUP,CKDYN_ARRAYMAP: the group id and a PE;CKDYN_NODEGROUP: the group id and a node.CKDYN_ARRAY: the array id, the dimension count (1 to 6) and the index. Indices of four to six dimensions are stored as 16-bit values by Charm++, so each component must fit in 16 bits.
Every hook that creates an object (construct, mainchare,
unpack) receives the object’s target; a binding keeps it with the
object, because most CkDyn functions that act on a local object
(CkDynContribute, CkDynAtSync, CkDynCoroResume, …) take the
target, not the handle. Fill unused fields with zeros (memset) before
setting the fields of a kind, so that targets compare equal byte for byte.
6.2.3. Class descriptors and entry descriptors
typedef struct CkDynEntryDesc {
const char *name; /* "Class.method"; copied at registration */
uint32_t flags; /* CKDYN_EP_* */
uint32_t user_id; /* opaque to CkDyn; passed back on every delivery */
} CkDynEntryDesc;
typedef struct CkDynClassDesc {
const char *name; /* "Class"; copied at registration */
CkDynKind kind;
int ndims; /* arrays: 1..6, or 0 for any */
int num_entries; /* entries[0] is the constructor */
const CkDynEntryDesc *entries;
void *binding_data; /* opaque; passed to every per-class hook */
const struct CkDynHooks *hooks; /* NULL: the table given to CkDynInit */
} CkDynClassDesc;
entries[0] is always the constructor. The names are the ones
Projections and error messages show; the "Class.method" form matches
what charmxi registers. user_id is any 32-bit value the binding wants
back on delivery: the C++ test binding uses the method’s position in its
table, so deliver indexes the table with it.
binding_data is the binding’s per-class pointer (the C++ test binding
passes its CbClass *); every per-class hook receives it as its first
argument. hooks selects a per-class hook table; NULL, the usual choice,
selects the table given to CkDynInit. A per-class table is kept by
pointer and must outlive the program.
Entry attributes (CkDynEntryDesc.flags):
flag |
meaning |
|---|---|
|
the binding runs the method on a coroutine; CkDyn only passes it
back in |
|
a send to a local object runs the method at once, inside the send |
|
every send to the entry is expedited (bypasses the priority queue) |
|
Charm++ immediate entry; nodegroups only |
|
set by CkDyn on every entry; see Ownership and retaining |
|
the entry is not traced |
|
the entry is application work ( |
|
nodegroups: one invocation at a time per branch |
|
documentation only: the entry may receive reduction results |
6.2.4. The hook table
typedef struct CkDynHooks {
uint32_t version; /* CKDYN_API_VERSION */
uint32_t size; /* sizeof(CkDynHooks) as compiled by the binding */
void *global; /* passed to the hooks without a class */
void (*register_classes)(void *global);
void *(*construct)(void *binding_data, const CkDynTarget *t, const CkDynMsgView *ctor_msg);
void (*destruct)(void *binding_data, void *handle);
void (*deliver)(void *binding_data, void *handle, int ep, uint32_t user_id,
uint32_t ep_flags, const CkDynMsgView *msg);
void (*deliver_many)(void *binding_data, void **handles, int n, int ep, uint32_t user_id,
uint32_t ep_flags, const CkDynMsgView *msg);
void (*mainchare)(void *binding_data, void **handle_out, const CkDynTarget *t,
int argc, char **argv);
size_t (*pack_size)(void *binding_data, void *handle, int reason);
void (*pack)(void *binding_data, void *handle, int reason, void *buf, size_t len);
void *(*unpack)(void *binding_data, const CkDynTarget *t, int reason,
const void *buf, size_t len);
size_t (*pack_readonly_size)(void *global);
void (*pack_readonly)(void *global, void *buf, size_t len);
void (*unpack_readonly)(void *global, const void *buf, size_t len);
void (*reduce)(void *global, uint32_t reducer_id, const CkDynMsgView **contribs, int n,
CkDynMsgBuilder *result);
void (*future)(void *global, int pe, uint32_t id, const CkDynMsgView *value);
void (*section_joined)(void *binding_data, void *handle, const CkDynSectionID *sid);
int (*map_proc_num)(void *binding_data, void *handle, int aid, int ndims, const int *index);
void (*resume_from_sync)(void *binding_data, void *handle);
void (*error)(void *global, const char *what);
} CkDynHooks;
Zero the table (memset) and set the hooks the binding uses. A hook that
is NULL is either optional or needed only when a situation calls for it; in
the second case CkDyn aborts with a message naming the hook when the
situation arises (for example a migration without pack_size).
hook |
called when |
what it does |
|---|---|---|
|
at start-up, on rank 0 of each process, for a shared-library binding
that called |
registers the binding’s classes (Start-up and Registration) |
|
an object of kind CHARE, GROUP, NODEGROUP, ARRAY or ARRAYMAP is created, on its PE |
builds the object from the constructor message; returns the handle |
|
a MAINCHARE class’s object is created, on PE 0, at start-up |
stores the handle in |
|
an object is deleted, migrates away, or the program exits |
releases the handle |
|
an entry method is invoked |
runs the method on the object |
|
an array broadcast reaches a PE (optional) |
runs the method on |
|
migration, checkpoint, restart |
serialize an object; see Migration and pup |
|
readonly broadcast at start-up, checkpoint, restart (optional) |
serialize the binding’s global read-only data |
|
a reduction with a binding-defined reducer combines contributions |
|
|
a value for a binding future arrives ( |
resolves the future |
|
a member receives its section’s creation multicast (optional) |
records membership |
|
the array manager asks an array map where an index lives |
returns a PE |
|
after a load balancing step, for array elements created with
|
resumes the element |
|
CkDyn is about to abort because of something the binding did |
reports the error in the binding’s terms; CkDyn aborts after it returns |
The per-class hooks (construct, destruct, deliver,
deliver_many, mainchare, the object pup hooks, section_joined,
map_proc_num, resume_from_sync) receive the class’s
binding_data. The hooks without a class (register_classes, the
readonly hooks, reduce, future, error) receive global and are
always taken from the table given to CkDynInit.
6.2.4.1. Where hooks run
Every hook runs inside the Charm++ scheduler on the PE that owns the object
(for future, the PE that owns the future), except reduce, which runs
inside the reduction manager on whichever PE combines contributions, with no
object context. A hook may call any CkDyn function: send, create objects,
contribute, set futures, create sections. Nested message building works
(a builder per nesting level is allocated as needed).
Rules that apply to every hook:
Views are valid only during the hook. A
CkDynMsgViewand every pointer read from it (slots, data, tail, buffer pointers) are valid until the hook returns. To keep a pointer into a message longer, callCkDynMsgRetainbefore returning (Ownership and retaining).The runtime owns delivered messages. A hook never frees the message it receives.
Do not unwind through CkDyn. Hooks are called from C++ code; a binding in a language with exceptions catches them inside the hook. charm4py converts a Python exception in a hook into an abort with the traceback.
6.2.5. Threading model
A Reconverse process can run several PEs, each a thread. CkDyn itself works with several PEs per process; the C++ test runs four PEs in one process. What a binding must assume:
Hooks of different PEs of one process may run at the same time, on different threads. Per-PE binding state (future tables, coroutine schedulers) must be per thread: the C++ test binding keeps it in a
Ckpvvariable, charm4py inthread_localstorage.The hook table and
globalare shared by all PEs of a process.A nodegroup branch is shared by the PEs of its process; its entries may run on any of them, concurrently, unless the entry is
CKDYN_EP_EXCLUSIVE.reducemay run on any PE and concurrently with other hooks.Registration happens once per process, on rank 0.
Two features need one PE per process: late registration (Late registration), and charm4py 2, whose interpreter lock serializes all Python code in a process (it aborts at start-up when a process has more than one PE; run several processes with one PE each).
6.3. Start-up and Registration
6.3.1. Start-up
A shared-library binding starts the runtime itself:
CkDynHooks h;
memset(&h, 0, sizeof h);
h.version = CKDYN_API_VERSION;
h.size = sizeof(CkDynHooks);
h.register_classes = my_register_classes;
/* ... the other hooks ... */
if (CkDynApiVersion() != CKDYN_API_VERSION) { /* refuse */ }
CkDynInit(&h); /* copies the table, checks version and size */
CkDynStart(argc, argv); /* ConverseInit; never returns */
CkDynInit copies the table. CkDynStart copies argv into storage
it owns and calls ConverseInit; it does not return. On every process,
the runtime then calls register_classes(global) on rank 0, where the
binding registers its classes. Then PE 0 constructs the main chares. A
binding for an interpreted language releases its interpreter lock around
CkDynStart and takes it again in each hook (charm4py does this with
PyEval_SaveThread and PyGILState_Ensure).
A C++ program starts as any Charm++ program. It calls CkDynInit and
registers its classes from an initnode function declared in its .ci
file; register_classes is not called for it. If its main chare is an
ordinary C++ main chare (not a dynamic MAINCHARE class), the main chare’s
constructor calls CkDynCreateManagers() before using CkDyn, to create
CkDyn’s per-PE manager group and its CkMulticast manager. A dynamic main
chare does this itself, before calling the mainchare hook.
6.3.2. Registering a class
int first_ep;
int chare_idx = CkDynRegisterClass(&desc, &first_ep);
CkDynRegisterClass returns the class’s chare index and stores the entry
index of entries[0] (the constructor) in first_ep; entries[i]
has entry index first_ep + i. The binding computes entry indices by
addition and keeps chare_idx and first_ep with its class.
Rules:
Call it on rank 0 of every process (from
register_classesor aninitnode), and in the same order on every process, so that every process assigns the same indices.Call it before start-up completes. A class registered after that needs late registration.
The class and entry names are copied at registration: the binding may free its strings once
CkDynRegisterClassreturns.CkDynClassOf(chare_idx)returns the registered descriptor, or NULL for an index that is not a dynamic class.
A MAINCHARE class is constructed on PE 0 at start-up; a program may register more than one (the C++ test has an ordinary main chare and a dynamic one).
6.3.3. Late registration
CkDynAllowLateRegistration(); /* once, before or during start-up */
int idx = CkDynRegisterClassLate(&desc, &first_ep);
CkDynRegisterClassLate registers a class after start-up, in every build.
CkDynRegisterClass after start-up aborts and names it, also in a
charm4py build of Charm++, whose own registration tables accept late
entries. Its limits:
The process must have one PE; it aborts otherwise, because the runtime’s entry and chare tables are shared by the PEs of a process.
It must run on every process, in the same order, before any message for the class is sent. The C++ test registers from a group broadcast and waits for a reduction before it creates objects of the class.
A MAINCHARE class registered late is never constructed.
Readonly data is broadcast once at start-up; the readonly hooks are registered by
CkDynInit, so they are unaffected.With the trace-summary or trace-utilization modules, executing an entry of a late class aborts (their per-entry tables are sized at start-up); trace-projections works.
A checkpoint holding objects of a late class restarts only if the restarted program registers the same classes, in the same order, at start-up.
6.3.4. Array dimensions
An array class has a fixed dimension count (ndims 1 to 6) or accepts
any (ndims 0): each creation and insert then gives its own count.
Languages whose arrays pick their dimension count at creation (charm4py)
register every array class with 0. CkDynCreateArray checks the count
only for a class registered with a fixed one.
For tracing, a class registered with 0 takes the dimension count of the first index the process sees for it, so Projections records element indices as for a charmxi array. A class used for arrays of different dimension counts in one run keeps the first count.
6.3.5. Limits
Entry indices. Entry indices are 16 bits: at most 65536 per process, the runtime’s own included.
Message types. CkDyn uses two of the runtime’s 256 message types.
Array dimensions. 1 to 6 (
CKDYN_MAX_DIMS); components of 4- to 6-dimensional indices fit in 16 bits.Buffers. At most 32 dimensions (
CKDYN_MAX_BUF_DIMS).Message size. A slot’s length is 32 bits; a payload is at most 2 GB; at most 65535 slots per message.
6.4. Messages
Every entry method of every dynamic class receives a CkDynMsg. Its
payload is laid out as:
the header,
CkDynHdr(24 bytes): magic, entry index, flags, slot count,user_id, tail length, data length, zero-copy buffer count;nslotsslot records,CkDynSlot(24 bytes each): type, element kind, dimension count, buffer flags, a 16-bit binding tag, length, and a value union (an inline scalar or an offset into the data region);the data region, 8-byte aligned items: byte strings, buffer shapes and contents, and the bytes of callback, section and proxy values;
the tail, opaque bytes written by the binding.
A message with zero-copy slots also has a descriptor prefix before the header; the view API hides it.
6.4.1. Slot types
type |
add with |
read with |
|---|---|---|
|
|
the type alone (the binding’s null) |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Slots keep the order in which they were added; slot i of the view is the
i-th one added. The tag field of a BYTES, BUFFER, NCPY or DEVICE slot
is the binding’s: CkDyn neither sets nor reads it. charm4py uses it to tell
a pickled argument from raw bytes and an array.array from a numpy array.
Element kinds (CKDYN_ELEM_*) describe buffer contents: signed and
unsigned integers of 8 to 64 bits, 16-, 32- and 64-bit floats, 64- and
128-bit complex, bool and byte.
6.4.2. The builder
CkDynMsgBuilder *b = CkDynMsgBegin(nslots_hint, data_hint, tail_hint, prio_bits, group_dep);
CkDynMsgSetFlags(b, flags); /* CKDYN_MSG_* */
CkDynMsgSetUserId(b, user_id); /* rarely needed: see foreign reducers */
CkDynMsgAddI64(b, 42);
CkDynMsgAddBuffer(b, &desc);
void *t = CkDynMsgReserveTail(b, len); /* write len bytes at t */
CkDynMsg *m = CkDynMsgFinish(b, ep); /* one allocation */
The hints size the builder’s scratch space; they are not limits. Each PE keeps one builder and reuses it, so its scratch space stays at the largest size any message on that PE needed; it is not returned until the run ends. A negative
nslots_hintcounts as 0.prio_bitsreserves that many priority bits (0: none).group_depis a group id the message must wait for on delivery, or -1. It must be given here because the envelope reserves space for it at allocation. The runtime honors it on chare, group, nodegroup and creation messages, not on point-to-point sends to array elements. Its main use is creating an array with a custom map: the array’s constructor message depends on the map group (array maps).CkDynMsgAddBuffercopies the shape at once but the contents only atCkDynMsgFinish, straight into the message, sodesc.ptrmust stay valid untilFinishorAbandon(except in thereducehook’s result builder, which copies the contents at once). A NULLdesc.ptrwith a nonzero length aborts. Strides, if given, must describe a contiguous layout in C or Fortran order (CKDYN_BUF_FORTRAN); CkDyn aborts otherwise. An empty buffer (an extent of 0) is accepted whatever its strides.CkDynMsgFinish(b, ep)allocates the message once and writesepinto the header. The send functions write the destination entry into the header again, so for a plain send theepgiven toFinishdoes not matter (the C++ test binding passes 0). For a constructor message it must be the class’s constructor entry,first_ep.CkDynMsgFinishMcast(b, ep)finishes the same payload as a section message, forCkDynSendSection; it takes no group dependency.CkDynMsgAbandon(b)drops a builder that will not be finished. A binding calls it on every error path betweenBeginandFinish; charm4py uses a guard object whose destructor abandons the builder. Abandoning runs thedonefunctions of any zero-copy slots.CkDynMsgFree(m)frees a finished message that will not be sent.
Every send, creation, CkDynFutureSet and CkDynCallbackSend takes
ownership of the message: after the call the binding must not touch it.
6.4.3. Header flags
CkDynMsgSetFlags sets the header’s flags:
flag |
meaning |
|---|---|
|
constructor message that acknowledges creation to its first FUTURE slot (Objects) |
|
array element constructor: the element uses AtSync load balancing |
|
group constructor: the last slot is a BYTES slot of |
|
set by CkDyn on broadcasts |
|
defined for bindings and passed through unread. The convention of the
existing bindings: with |
|
bits 12 to 15 belong to the binding; CkDyn neither sets nor reads them |
6.4.4. Priorities and queueing
Reserve priority bits at CkDynMsgBegin, then after Finish write the
priority through CkDynMsgPriorityPtr(m) and set the queueing strategy
with CkDynMsgSetQueueing(m, q), where q is one of the
CKDYN_QUEUEING_* constants of ckdyn.h (their values equal Charm++’s
CK_QUEUEING_*). Without a call the message is queued FIFO.
constants |
priority |
|---|---|
|
none |
|
an |
|
a bit vector |
|
a 64-bit integer |
From the C++ test (a smaller integer runs first):
CkDynMsgBuilder *b = CkDynMsgBegin(1, 0, 0, 8 * sizeof(int), -1);
CkDynMsgAddI64(b, p);
CkDynMsg *m = CkDynMsgFinish(b, 0);
*(int *)CkDynMsgPriorityPtr(m) = p;
CkDynMsgSetQueueing(m, CKDYN_QUEUEING_IFIFO);
CkDynSendGroup(gid, CkDynMyPe(), ep, m, NULL);
6.4.5. Sending
void CkDynSendChare (const CkDynTarget *t, int ep, CkDynMsg *m, const CkDynSendOpts *o);
void CkDynSendGroup (int gid, int pe, int ep, CkDynMsg *m, const CkDynSendOpts *o); /* pe -1: broadcast */
void CkDynSendGroupMulti(int gid, int npes, const int *pes, int ep, CkDynMsg *m, const CkDynSendOpts *o);
void CkDynSendNodeGroup (int gid, int node, int ep, CkDynMsg *m, const CkDynSendOpts *o); /* node -1: broadcast */
void CkDynSendArray (int aid, int ndims, const int *index, int ep, CkDynMsg *m,
const CkDynSendOpts *o); /* index NULL: broadcast */
void CkDynSendTarget (const CkDynTarget *t, int ep, CkDynMsg *m, const CkDynSendOpts *o);
CkDynSendTarget sends to whatever a target names, selecting the function
by its kind; a binding whose proxies hold a CkDynTarget uses it for
every point-to-point send. Groups and array maps use CkDynSendGroup.
Array sends to an element that does not exist yet are buffered, as for any
Charm++ array.
CkDynSendOpts.flags adds per-send options to the entry’s own
attributes (NULL: none):
CKDYN_SEND_INLINE: a send to a local object runs the method inside the send.CKDYN_SEND_EXPEDITED: the message bypasses the priority queue.CKDYN_SEND_IMMEDIATE: Charm++ immediate delivery (nodegroups).CKDYN_SEND_NOTRACE: the message is not recorded in the load balancer’s communication statistics (CK_MSG_LB_NOTRACE, which only array sends consult). It does not affect Projections tracing.
An inline send nests the receiver’s deliver inside the sender’s hook; a
binding whose methods send inline to each other in a loop should drive the
loop from one side, as the C++ test’s inline ping-pong does, so that the
nesting stays shallow. A message with zero-copy slots is never sent inline.
6.4.6. Receiving: the view
Every delivery hook receives a CkDynMsgView:
typedef struct CkDynMsgView {
CkDynMsgKind kind;
const CkDynHdr *hdr; /* NULL for REDRESULT and SYS */
const CkDynSlot *slots;
int nslots;
const char *data; /* data region */
const char *tail; /* tail_len bytes, or NULL */
uint32_t tail_len;
uint32_t refnum; /* the envelope's reference number, 0 if none */
const CkDynSectionID *section; /* non-NULL for a section delivery */
void *msg; /* the runtime message, for retain */
} CkDynMsgView;
An entry method can receive four kinds of message:
kind |
what it is |
|---|---|
|
a |
|
a section multicast, or a section callback’s result; |
|
a built-in reduction result, shown as one BUFFER slot over the
reduction data, plus a FUTURE slot |
|
a message from the runtime with no data (quiescence, checkpoint completion, a HAPI callback): no slots, except the same FUTURE slot |
For DYN and MCAST views, hdr->ep is the entry index, and it is
authoritative (the envelope’s copy can be stale on inline sends). The
deliver hook also receives ep, the entry’s user_id and
ep_flags (the entry’s attributes, with CKDYN_EP_NOKEEP always set).
refnum is the envelope’s reference number; 0, and the all-ones value
(the default reduction user flag), read as none. A binding uses it as the
future id a result belongs to.
The slots of a REDRESULT or SYS view are made by CkDyn in the caller’s stack frame: the view, its slots and pointers read from them are valid only until the hook returns, and retaining the message keeps the reduction data alive but not those slots.
Reading slots:
const void *CkDynViewBufferPtr(const CkDynMsgView *v, int slot, const int64_t **shape_out);
const void *CkDynViewBytesPtr(const CkDynMsgView *v, int slot);
void CkDynViewCallback(const CkDynMsgView *v, int slot, CkDynCallback *out);
void CkDynViewSection(const CkDynMsgView *v, int slot, CkDynSectionID *out);
void CkDynViewProxy(const CkDynMsgView *v, int slot, CkDynTarget *out);
Scalar slots are read from slots[i].v. Accessors abort on a slot index
out of range or a slot of another type.
6.4.7. Ownership and retaining
Every CkDyn entry is registered “no keep”: the runtime frees the message
after the hook returns. A binding that keeps a pointer into the message
beyond the hook (a zero-copy numpy view of a BUFFER slot, a received NCPY
buffer it forwards later) calls CkDynMsgRetain(view) before returning
and CkDynMsgRelease(view->msg) once per retain when done; the message is
freed when the last reference goes. Keep view->msg, not the view: the
view itself is invalid after the hook.
From the C++ test, which echoes a received 16 MB NCPY buffer from the retained message and releases it when the echo’s transfer is done:
CkDynMsgRetain(v);
zSend(replyTo, mode, -1, p, seed, v->msg /* released by the done function */, ...);
charm4py wraps each retain in a Python object whose deallocation calls
CkDynMsgRelease, and gives numpy arrays over BUFFER slots that object as
their base.
6.4.8. Broadcasts
CkDynSendGroup with PE -1, CkDynSendNodeGroup with node -1 and
CkDynSendArray with a NULL index broadcast. CkDynSendGroupMulti
sends one message to a list of group branches. The receiver sees
CKDYN_MSG_BCAST in the header; so does the receiver of a message sent with
CkDynCallbackSend through a broadcast TARGET callback.
6.4.8.1. Array broadcasts and deliver_many
An array broadcast arrives once per PE. Without a deliver_many hook,
CkDyn calls deliver once per local element, and traces and times each
call as for a charmxi element. With deliver_many, CkDyn calls it once
with the handles of all local elements that should receive the broadcast;
the binding can then convert the message’s arguments once and run the method
per element. In that case time is charged to the array manager and tracing
records no event per element, unless the binding brackets each element’s
call with CkDynCoroResume/CkDynCoroSuspend (or
CkDynCallstackPush/Pop for the load alone); charm4py does that:
for (int i = 0; i < n; i++) {
CkDynTarget t = target_of(handles[i]);
CkDynCoroResume(&t, ep, 0);
deliver_one(handles[i], ep, ep_flags, v, args);
CkDynCoroSuspend(&t);
}
A handler that destroys another local element makes that element’s handle invalid for the rest of the call.
Without deliver_many a handler may migrate or destroy other local
elements of the array: CkDyn looks each element up again before its call,
and an element migrated away before its turn receives the broadcast once, on
its new PE. With deliver_many the call covers every local element at
once, and each is counted as having received the broadcast before the call:
an element that a handler migrates away during the call does not receive it
on its new PE, and the binding must not deliver to its handle either.
6.5. Objects
6.5.1. Creating objects
Every creation takes a constructor message finished with the class’s
constructor entry (first_ep); the construct hook receives its view.
void CkDynCreateChare (int chare_idx, int on_pe, CkDynMsg *ctor, CkDynTarget *out);
int CkDynCreateGroup (int chare_idx, CkDynMsg *ctor); /* GROUP or ARRAYMAP */
int CkDynCreateNodeGroup (int chare_idx, CkDynMsg *ctor);
int CkDynCreateArray (int chare_idx, CkDynMsg *ctor, const CkDynArrayOpts *o);
void CkDynArrayInsert (int aid, int ndims, const int *index, int on_pe, CkDynMsg *ctor);
void CkDynArrayDoneInserting(int aid);
void CkDynArrayDestroyElement(int aid, int ndims, const int *index);
The creation functions check that the class has the right kind and abort otherwise.
Main chares. A MAINCHARE class is constructed by the runtime on PE 0 at
start-up through the mainchare hook, with argc and argv; there is
no constructor message. The C++ test binding runs the class’s constructor
method with a NULL view.
Singleton chares. CkDynCreateChare(idx, on_pe, ctor, &out) creates
one chare on on_pe, or, with CKDYN_PE_ANY, where the seed balancer
places it. out receives a virtual chare id usable at once.
Groups and constrained groups. CkDynCreateGroup returns the group id;
a branch is constructed on every PE. To construct branches only on a list of
PEs, set CKDYN_MSG_CONSTRAINED and add the PE list as the last slot, a
BYTES slot of int32 values:
CkDynMsgSetFlags(b, CKDYN_MSG_CONSTRAINED);
/* ... constructor arguments ... */
CkDynMsgAddBytes(b, pes, npes * sizeof(int32_t), 0);
int gid = CkDynCreateGroup(idx, CkDynMsgFinish(b, first_ep));
Branches on other PEs exist with a NULL handle: they take part in reductions, ignore broadcasts and callback results, and are skipped when pupped. A point-to-point send to such a branch is a binding error.
Nodegroups. CkDynCreateNodeGroup constructs one branch per process.
Arrays. CkDynCreateArray takes options:
typedef struct CkDynArrayOpts {
int ndims;
const int *dims; /* NULL: empty array, insert later */
int map_gid; /* -1: default map */
int anytime_migration;
int static_insertion;
int bind_to_aid; /* -1, or an array to bind to */
} CkDynArrayOpts;
With dims the array is created with all its elements; with NULL it starts
empty, elements are added with CkDynArrayInsert (on on_pe, or -1 for
the map’s choice), and CkDynArrayDoneInserting ends insertion. Every
element of an array is of the array’s class, and the insert’s constructor
message must be finished with that class’s first_ep; an insert with
another class’s constructor aborts once the process has seen the array’s
class (it created the array, or holds one of its elements).
CKDYN_MSG_CREATION is for whole-array creation only: on an insert it
would start a reduction over one element.
Array maps. An array map is a class of kind CKDYN_ARRAYMAP, created
with CkDynCreateGroup. The runtime asks its local branch where an index
lives by calling map_proc_num(binding_data, handle, aid, ndims, index),
which returns a PE. An array that uses the map passes its id as map_gid
and as the group_dep of the array’s constructor message, so that no
element is created before the map exists on its PE:
CkDynMsgBuilder *b = CkDynMsgBegin(4, 64, 0, 0, map_gid /* group_dep */);
/* ... */
CkDynArrayOpts o = { 1, dims, map_gid, 0, 0, -1 };
int aid = CkDynCreateArray(idx, CkDynMsgFinish(b, first_ep), &o);
6.5.2. Creation acknowledgement
A constructor message with CKDYN_MSG_CREATION and a FUTURE slot
acknowledges creation without the binding’s help: once the object exists (a
chare), or every branch or element exists (a group, nodegroup or array), the
future in the first FUTURE slot receives a value through the future
hook. From the C++ test:
aidPing = CkDynCreateArray(Ping.chareIdx,
MsgB().flags(CKDYN_MSG_CREATION | CKDYN_MSG_USESATSYNC)
.fut(0, created("Ping")).finish(Ping.ep(0)), &ao);
6.5.3. Destruction
CkDynArrayDestroyElement destroys a local array element; it aborts for an
element that is not on the calling PE. CkDyn calls destruct for the
handle. The C API has no call to destroy a singleton chare or a group.
6.5.4. Local objects
CkDyn gives the binding its handles only through hooks. A binding that needs the local object a target names outside a hook keeps its own table (the C++ test binding stores its object pointers where its methods find them; charm4py keeps a map from targets to Python objects). Functions that act on a local object take its target and abort when the object is not on the calling PE.
6.5.5. Migration and pup
An array element migrates when the load balancer moves it or when the
binding calls CkDynMigrateMe(t, to_pe). CkDynSetMigratable(t, 0)
pins an element. Both act on a local array element only.
AtSync load balancing. Create the elements with
CKDYN_MSG_USESATSYNC in the constructor message. Each element calls
CkDynAtSync(&target); after the load balancing step the runtime calls
resume_from_sync(binding_data, handle) for each element, on its new PE.
The C++ test binding resumes by sending the element a message:
static void hResumeFromSync(void *bd, void *h) {
Obj *o = (Obj *)h;
/* m: the class's "resumed" method */
CkDynSendTarget(&o->t, c->ep(m), MsgB().finish(c->ep(m)), nullptr);
}
CkDynArrayOpts.anytime_migration does not decide whether elements can
migrate: AtSync, CkDynMigrateMe and the load balancer move them either
way. It is Charm++’s CkArrayOptions::setAnytimeMigration, which only
matters together with static_insertion: an array with static insertion
and without anytime migration is assumed to keep its elements where they
were created, and its broadcasts skip the bookkeeping for elements that
move. Anytime migration is on for every array unless the program runs with
+noAnytimeMigration, so anytime_migration = 0 keeps that default and
a nonzero value turns it on for this array under +noAnytimeMigration.
Pup hooks. Packing an object runs two passes:
pack_size(bd, handle, reason | CKDYN_PUP_SIZING)returns the sizen;pack_size(bd, handle, reason)is called again and must return the samen, thenpack(bd, handle, reason, buf, n)writes exactlynbytes.
On the receiving PE, unpack(bd, target, reason, buf, n) returns a new
handle; the old object’s destruct runs on the PE it left. reason is
CKDYN_PUP_MIGRATE or CKDYN_PUP_CHECKPOINT. An object that cannot be
packed in its current state (charm4py: one with a suspended coroutine)
returns CKDYN_PACK_REFUSED from pack_size; CkDyn reports it through
error and aborts. Section membership travels with the element; buffer
posts do not.
Readonly data. If pack_readonly_size or unpack_readonly is set,
CkDynInit registers one readonly for the binding: its bytes are packed
on PE 0 and unpacked on every other PE at start-up (values set while the
main chares are constructed), and they are written to and read from a
checkpoint.
6.5.6. Checkpoint and restart
void CkDynCheckpoint(const char *dirname, const CkDynCallback *done);
CkDynCheckpoint writes a checkpoint of every object to dirname
(main chares, chares, groups, nodegroups, arrays, through the pup hooks with
CKDYN_PUP_CHECKPOINT, and the readonly data). done is called when the
checkpoint is written and, in a program started with +restart dirname,
once the restart completes. On restart:
the program registers the same classes in the same order at start-up;
the runtime restores the objects through
unpack; the main chare is restored too, and itsmaincharehook is not called;CkDynIsRestarting()returns 1;doneis delivered. It may be a TARGET callback to a group, an array element or a main chare entry: a callback to a main chare is routed by chare index, not by object pointer, so it reaches the restored main chare. A FUTURE callback does not survive a restart (the future belonged to the checkpointed run’s coroutine). A SECTION callback aborts inCkDynCheckpoint: sections are not checkpointed, so it could not be delivered after a restart.
Not restored: sections (a binding recreates them; an old section id aborts on use), buffer posts, futures and pending messages. A program that continues after its own checkpoint tells the two runs apart itself; the C++ test’s dynamic main chare checks whether its constructor ran in this process.
A restart may use another PE count than the checkpointed run. Then:
array elements are restored and spread over the new PEs;
every group branch is unpacked from PE 0’s bytes and every nodegroup branch from node 0’s (the Charm++ checkpoint rule):
unpackon every PE receives the bytes PE 0 packed, so per-branch state of the other branches is lost;singleton chares (
CkDynCreateChare) are not restored at all, and nothing reports it: their targets name no object, and a send to one is an error the runtime does not detect. A program that holds singleton chares restarts with the same PE count;the main chare is restored on PE 0 as usual.
6.5.7. Quiescence
CkDynStartQD(&cb) starts quiescence detection; cb receives a SYS view
(with a FUTURE slot when the callback carries a reference number) once no
messages are in flight.
6.6. Futures and Callbacks
6.6.1. Futures
A future is the binding’s: CkDyn knows it only as a pair (pe, id), the PE
whose binding holds it and a 32-bit id the binding chose. Any code can send a
value to a future:
void CkDynFutureSet(int pe, uint32_t id, CkDynMsg *value);
The message goes to CkDyn’s manager on pe, which calls
future(global, pe, id, view). The binding finds the future by id and
resolves it, typically by resuming a coroutine that waits on it. Values also
reach futures from callbacks of kind CKDYN_CB_FUTURE (reductions,
quiescence, creation acknowledgements), as REDRESULT, SYS or DYN views.
Future ids travel as the message’s reference number, so their range is 1 to 28 * CkDynRefnumBytes()-2: 0 means “no reference number” and the all-ones value is the default reduction flag. The C++ test binding keeps ids below 65536 so that it works with the default 16-bit reference number; charm4py requires 32 bits.
The future hook runs inside an untraced entry of CkDyn’s manager. That
is where a binding resumes coroutines, bracketed with
CkDynCoroResume/CkDynCoroSuspend (Coroutines, Tracing and Load Balancing).
A method’s return value as a future value (the CKDYN_MSG_BLOCK
convention): the sender adds a FUTURE slot and sets CKDYN_MSG_BLOCK; the
receiving binding runs the method and sends the result:
int64_t r = c->methods[user_id].fn(o, v);
if (v->hdr && (v->hdr->flags & CKDYN_MSG_BLOCK)) {
int pe; uint32_t id;
cbFindFuture(v, &pe, &id); /* the first FUTURE slot */
CkDynFutureSet(pe, id, MsgB().i64(r).finish(0));
}
6.6.2. Callbacks
A CkDynCallback is an 88-byte plain struct describing where a result
goes. Zero it, then set the fields of its kind:
kind |
fields |
delivers |
|---|---|---|
|
|
to entry |
|
|
to entry |
|
|
to the binding future |
|
nowhere |
|
|
exits the program |
refnum, when not zero, becomes the reference number of the delivered
message, which the receiver reads as view->refnum: a binding passes a
future id this way to an entry method that will forward the result. A
callback can travel in a message (CkDynMsgAddCallback), so an entry can
reply to a callback its caller chose.
CkDynCallbackSend(&cb, m) sends a finished message through a callback;
for a TARGET callback CkDyn writes the entry into the header. Callbacks are
also accepted by CkDynContribute, CkDynContributeSection, section
creation, CkDynStartQD, CkDynCheckpoint and
CkDynHapiAddCallback.
A TARGET callback to a main chare entry is routed by chare index through CkDyn’s manager on the main chare’s PE, so it stays valid after a restart from a checkpoint.
6.6.3. System callbacks
Callbacks that the runtime fires without data (quiescence, checkpoint
completion, HAPI completion, creation of a chare) deliver a SYS view or an
empty DYN message; a reduction delivers a REDRESULT view. A binding that
wants one function for all result kinds reads the view’s kind and its
slots: the C++ test’s resultOf accepts a BUFFER slot (a built-in
reduction), an I64 slot (its foreign reducer) or no slot.
6.6.4. Reduction numbers
CkDynGetReductionNumber(&target) returns the reduction number of a local
group branch, nodegroup branch, array map branch or array element: the number
of reductions it has contributed to. Every member computes the same value
for the same reduction, so a binding can derive from it a future id that
every member agrees on without communication.
6.7. Reductions
6.7.1. Built-in reducers
CkDynReducers(&table) returns the number of built-in reducers and points
table at a static array of CkDynReducerInfo: the reducer’s name as in
CkReduction::reducerType ("sum_int", "max_double", …), its type
value, and the element kind of its data. The table covers nop; sum,
product, max and min for every integer and floating type
(_char to _ulong_long, _float, _double); the logical_*
and bitvec_* reducers; random; and concat, set,
statistics and tuple, whose data is CKDYN_ELEM_BYTE and whose
format the binding handles. random passes one contribution on unchanged,
so its element kind is CKDYN_ELEM_BYTE too. The _char reducers
operate on the platform’s plain char: their element kind is
CKDYN_ELEM_I8 where char is signed (x86, macOS) and
CKDYN_ELEM_U8 where it is unsigned (aarch64 and POWER Linux); use
_uchar or a binding-defined reducer for a type with a fixed sign. The
_long reducers are 32-bit on Windows and 64-bit elsewhere. A binding
looks the types up by name once at start-up:
static int redType(const char *name) {
const CkDynReducerInfo *t;
int n = CkDynReducers(&t);
for (int i = 0; i < n; i++) if (strcmp(t[i].name, name) == 0) return t[i].type;
return -1;
}
6.7.2. Contributing
typedef struct CkDynRedData {
const void *data;
uint32_t len; /* bytes; 0 with data NULL for an empty contribution */
int type; /* a reducer type, or CkDynForeignReducerType() */
} CkDynRedData;
void CkDynContribute(const CkDynTarget *me, const CkDynRedData *d, const CkDynCallback *cb);
CkDynContribute contributes for the local group branch, nodegroup
branch or array element that me names (chares and main chares do not
contribute). The data is copied. Every member of the collection contributes
once per reduction, with the same reducer and callback; nop with no data
is a barrier.
6.7.2.1. Shaped contributions
The built-in reducers combine flat arrays of elements. To get a result with the contributors’ array shape, contribute with the shape:
void CkDynContributeShaped(const CkDynTarget *me, const CkDynRedData *d, const CkDynCallback *cb,
int ndim, const int64_t *shape);
void CkDynContributeSectionShaped(const CkDynTarget *me, const CkDynSectionID *sid, const CkDynRedData *d,
const CkDynCallback *cb, int ndim, const int64_t *shape);
The elements are in C order, and the product of shape times the element
size of the reducer must equal d->len (CkDyn aborts otherwise). With
ndim 2 or more and a built-in reducer on a numeric element kind, the
result’s BUFFER slot (REDRESULT view, or the MCAST view of a SECTION
callback) has that ndim and shape: CkDynViewBufferPtr(v, 0, &shape)
returns it. Every contribution to one reduction must have the same shape;
contributions of two shapes abort with “contributions to one reduction have
different shapes”. One reduction also cannot mix the two forms: a shaped
contribution (ndim 2 or more) and a flat one (CkDynContribute, or
ndim 0 or 1) under a built-in reducer use different Charm++ reducers
(CkDyn::shaped and the built-in one), and the program aborts with
“contributions to one reduction use different reducers” (or, through a
SECTION callback, “contributions to one reduction mix shaped and flat
forms”). The shaped reducer combines the elements in place in the first
message, so a shaped reduction costs what a flat one does apart from the
8 + 8 * ndim header bytes. With ndim 0 or 1, nop, a binding-defined reducer,
or a reducer whose element kind is CKDYN_ELEM_BYTE or
CKDYN_ELEM_NONE, the shape is ignored and the call is
CkDynContribute (or CkDynContributeSection): the result slot is flat
(ndim 0), as before. Both functions were added within API version 2;
the plain contribute functions are unchanged.
double a[2][3]; /* this member's values */
int64_t shape[2] = { 2, 3 };
CkDynRedData d = { a, sizeof a, redType("sum_double") };
CkDynContributeShaped(&o->t, &d, &cb, 2, shape);
/* in the target entry: slot 0 has ndim 2 and shape (2, 3) */
const int64_t *shp;
const double *p = (const double *)CkDynViewBufferPtr(v, 0, &shp);
6.7.3. Results
The callback receives the result:
With a built-in reducer, a REDRESULT view: slot 0 is a BUFFER slot over the reduction data, with
elemfrom the reducer table andlenthe data’s byte count;ndimis 0, or the contributions’ shape for shaped contributions. When the reference number is set (a FUTURE callback, or a TARGET callback withrefnum),view->refnumholds it and a FUTURE slot(CkDynMyPe(), refnum)follows.With a binding-defined reducer, a DYN view of the payload the
reducehook built;view->refnumcarries the reference number, and there is no synthesized FUTURE slot.Through a SECTION callback, an MCAST view on every member of the section (Sections).
A FUTURE callback hands the same view to the future hook. From the C++
test, an array reduction to element 0’s entry:
double x = myIndex(o) + 1.0;
CkDynRedData d = { &x, sizeof x, redType("sum_double") };
CkDynContribute(&o->t, &d, &cb);
/* in the target entry */
const double *p = (const double *)CkDynViewBufferPtr(v, 0, NULL); /* slot 0 */
6.7.4. Binding-defined reducers
CkDyn registers one Charm++ reducer for all binding-defined reducers;
CkDynForeignReducerType() returns its type. A contribution to it is a
CkDyn payload (header, slots, data region, tail) whose header user_id is
the binding’s own reducer id. Build it as a message, take its payload, and
free the message after contributing:
CkDynMsgBuilder *b = CkDynMsgBegin(1, 0, 0, 0, -1);
CkDynMsgSetUserId(b, MY_REDUCER_SUM_I64);
CkDynMsgAddI64(b, value);
CkDynMsg *m = CkDynMsgFinish(b, 0);
uint32_t len;
const void *p = CkDynMsgPayload(m, &len);
CkDynRedData d = { p, len, CkDynForeignReducerType() };
CkDynContribute(&me, &d, &cb);
CkDynMsgFree(m);
A delivered DYN message can also be contributed as is: its payload starts at
view->hdr and is sizeof(CkDynHdr) + nslots * sizeof(CkDynSlot) +
data_len + tail_len bytes long.
The runtime combines contributions by calling
reduce(global, reducer_id, contribs, n, result) with the n
contributions as DYN views and an open builder. The hook adds the combined
value’s slots (and tail) to result; CkDyn writes the payload, with the
same user_id, and passes it up the reduction tree, where it may be
combined again. The hook does not finish or abandon result. It may set
header flags on result with CkDynMsgSetFlags, for example a bit of
CKDYN_MSG_BINDING_MASK that marks the message as a reduction result for
the receiving binding (charm4py does this). reduce runs on whichever PE
combines, with no object context, and can run concurrently with other hooks
of the process.
Everything the hook adds to result is copied before the add call
returns, BUFFER contents included: the hook may build its combined value
in a temporary and free it as soon as CkDynMsgAddBuffer returns. A
reduce result has the limits of any message (65535 slots, 2 GB with header
and envelope) and carries no NCPY or DEVICE slot; CkDyn aborts otherwise.
The C++ test binding’s reducer, without the BUFFER slot it adds to check
this:
static void hReduce(void *, uint32_t rid, const CkDynMsgView **contribs, int n,
CkDynMsgBuilder *result) {
int64_t sum = 0;
for (int i = 0; i < n; i++) sum += contribs[i]->slots[0].v.i64;
CkDynMsgAddI64(result, sum);
}
Zero-copy slots are not allowed in a contribution.
6.8. Sections
A section is a subset of one array, of one group, or of several arrays (a combined section), with multicasts and section reductions carried by CkMulticast.
6.8.1. Creating and deleting
void CkDynSectionCreateArray(int aid, int nelems, int ndims, const int *indices, int bfactor,
const CkDynCallback *done, CkDynSectionID *out);
void CkDynSectionCreateGroup(int gid, int npes, const int *pes, int bfactor,
const CkDynCallback *done, CkDynSectionID *out);
void CkDynSectionCombine(int nparts, const CkDynSectionID *parts, const CkDynCallback *done,
CkDynSectionID *out);
void CkDynSectionDelete(const CkDynSectionID *sid);
Creation runs on the calling PE, which becomes the section’s root.
indices holds nelems indices of ndims components each; duplicate
members are removed. bfactor is the spanning tree’s branching factor
(the C++ test passes 0, CkMulticast’s default). out receives the id at
once, but the section is ready only when done fires: CkDyn multicasts
the id to every member, each member stores its section information (and the
section_joined hook, if set, is called), and the members contribute an
empty reduction to done. Send to the section and contribute to it only
after ``done`` fires.
CkDynSectionCombine makes a section whose members are the union of the
parts, each member once. Parts may be array sections or combined sections,
from any root, all with one index dimension count; a group part aborts. A
combine takes each part’s member list when the part’s root receives the
request; a part that is itself a combined section not yet ready answers once
it is ready. Deleting such a part while a combine waits on it aborts, naming
the waiting combine: delete a part only after the combines that name it are
ready.
CkDynSectionDelete deletes a section (from any PE; the request goes to
the root). Later use of the id aborts with a message about a stale
generation.
Section operations from one PE are applied at the root in program order.
A send (CkDynSendSection, also through a section callback), the member
query of a combine and a delete issued on one PE reach the root in any order
on the network, but the root applies them in the order that PE issued them,
so a send followed by a delete delivers the message. Operations from
different PEs have no order between them.
6.8.2. Section ids
typedef struct CkDynSectionID {
int32_t root_pe; /* the PE that created the section */
int32_t idx; /* index in the root's table */
int32_t coll_id; /* array or group id (-1 for a combined section) */
uint8_t kind; /* CKDYN_ARRAY, CKDYN_GROUP, or 0 for a combined section */
uint8_t reserved; /* restarts before the id was made, modulo 256 */
uint16_t gen; /* generation */
} CkDynSectionID;
The id is 16 bytes, travels in messages (CkDynMsgAddSection,
CkDynViewSection) and is valid on every PE. The generation makes a
deleted section’s id distinct from a later section that reuses its table
slot; generation 0 is never valid. reserved holds the number of times
the program has been restarted from a checkpoint (modulo 256) when the id
was made. Sections are not checkpointed, and after +restart the first
sections a program creates get the same root, index and generation as
sections of the checkpointed run; every use of an id (send, contribute,
CkDynSectionRedNo, combine, delete) compares reserved with the
current count and aborts with “stale section id from before restart” when
they differ. A binding treats the id as 16 opaque bytes and copies all of
them.
6.8.3. Multicasts and section reductions
CkDynMcastMsg *m = CkDynMsgFinishMcast(b, ep);
CkDynSendSection(&sid, ep, m, NULL);
CkDynSendSection sends from any PE (a send from another PE than the root
takes one extra hop to the root). Send options are ignored. Members receive
an MCAST view whose section field points at the id. Every member of a
combined section over arrays of different classes must have the entry: a
section message for an entry of another class than the receiving element
aborts.
void CkDynContributeSection(const CkDynTarget *me, const CkDynSectionID *sid,
const CkDynRedData *d, const CkDynCallback *cb);
int CkDynSectionRedNo(const CkDynTarget *me, const CkDynSectionID *sid);
A member contributes to a section reduction with CkDynContributeSection;
built-in and binding-defined reducers both work. The member must be local
and must have received the creation multicast. CkDynSectionRedNo returns
the number of section reductions the member has contributed to, which every
member computes identically: a binding derives a collective future id from
it.
After CkDynSectionDelete the root multicasts the deletion to the members,
with the number of multicasts it sent to the section before it. A member
drops the section once that deletion and all those multicasts have reached
it; from then on CkDynContributeSection and CkDynSectionRedNo with
the id abort. The handler of a multicast sent before the deletion can still
use the id, whichever arrives first. A contribution made before the deletion
reaches the member is reduced as usual.
A section callback (CKDYN_CB_SECTION) multicasts a result to every
member of a section, as an MCAST view with refnum set. Unlike Charm++’s
bcastSection callback it can be put into a message and pupped. A
built-in reduction result arrives as one BUFFER slot; a binding-defined
reducer’s result keeps its payload; a SYS result (quiescence, checkpoint)
arrives empty.
A whole-collection reduction (CkDynContribute from array elements,
group or nodegroup branches) may have a section callback, for example a
section of the same array: every contributor passes the same
CkDynCallback. CkDyn sends the contributions through an internal reducer
that carries the section, the entry and the reference number along with the
data and applies the reducer the binding gave (built-in or binding-defined),
so that every PE contributes with the same Charm++ callback; the result is
multicast from the section’s root PE.
6.8.4. Migration, restart and restrictions
A member that migrates keeps its section membership and its reduction numbers. After its next contribution, the next multicast reaches members point to point and CkMulticast rebuilds the spanning tree for the one after.
Sections are not checkpointed: after
+restarta binding recreates them. An id from before the restart, for example one held in an object’s restored state, aborts on its next use (see Section ids). The check repeats after 256 restarts.A deleted section’s CkMulticast trees stay allocated. Deleting a section costs one multicast to its members.
Section messages take no group dependency and no zero-copy slots.
A group section must name a dynamic group; an empty member list aborts.
Each PE keeps one relay entry per (section id, entry) pair used in a section callback created on it (
CkDynContributeneeds none), plus one per distinct reference number of a quiescence or checkpoint section callback. They are never freed. With a 16-bit reference number a PE can hold at most 65534.
6.9. Zero-copy and Devices
6.9.1. Host buffers: NCPY slots
typedef void (*CkDynBufDone)(void *done_arg);
void CkDynMsgAddNcpy(CkDynMsgBuilder *b, const CkDynBufDesc *d, CkDynBufDone done, void *arg);
An NCPY slot carries a host buffer that the runtime transfers without
copying it into the message: within a process the receiver’s runtime copies
it once, across processes it is fetched with an RDMA get. d->ptr must
stay valid until done(arg) runs. done runs exactly once, from the
scheduler of the sending PE, after the data has left the source buffer, or
after CkDynMsgAbandon or CkDynMsgFree of an unsent message.
The receiver reads the data with CkDynViewBufferPtr; the data lives as
long as the message, so a retain keeps it (Ownership and retaining).
Restrictions: a message with zero-copy slots (header nbuf > 0) is sent
point to point only (a send to a chare, a group or nodegroup branch, an
array element, a future, or through a TARGET callback to one element). A
broadcast, a section message, a multi-branch send, a constructor message, a
reduction contribution, or a broadcast or section callback aborts, naming
the slot. NCPY and DEVICE slots cannot share a message. A message holds at
most 4096 zero-copy slots. Sending such a message inline is not possible;
CkDyn drops the inline flag.
6.9.2. Posted buffers
void CkDynPostBuffer(const CkDynTarget *t, int ep, int slot, const CkDynBufDesc *d);
void CkDynPostDeviceBuffer(const CkDynTarget *t, int ep, int slot, const CkDynDevDesc *d);
int CkDynUnpostBuffer(const CkDynTarget *t, int ep, int slot);
#define CKDYN_POST_ARG 0x40000000
A post names, on the calling PE, the destination of slot slot of the
next message to the local object t’s entry ep. The posted length must
equal the slot’s. One post per (object, entry, slot) at a time; posts are
dropped when the object migrates or is destroyed; nodegroups cannot post. A
host post may be made any time before the message is delivered to the
object, even after the send was issued: the runtime copies the slot from the
delivered message into the posted buffer (one copy on the receiving PE), and
CkDynViewBufferPtr returns the posted buffer. A device post receives the
transfer directly.
CkDynUnpostBuffer(t, ep, slot) withdraws the post for (t, ep, slot)
if it is still outstanding and returns 1, or returns 0 when there is none. A
post is consumed only by a message whose slot is NCPY or DEVICE; a message
that carries that slot as a BUFFER or BYTES value leaves it in place, so a
binding that posted for a call whose argument then arrived by value
withdraws the post before posting again for the next call.
slot in a post or unpost may be OR’d with CKDYN_POST_ARG; it then
counts arguments, not slots: in a message whose header has
CKDYN_MSG_BLOCK and whose slot 0 is a FUTURE slot, argument k is slot
k + 1, otherwise slot k. The binding then need not know at post time
whether the sender will block on the call. An unpost names the slot in the
same form as its post.
6.9.3. Device buffers: DEVICE slots
typedef struct CkDynDevDesc {
const void *ptr; /* device memory, C-contiguous */
uint64_t len; /* bytes */
uint8_t elem;
uint8_t ndim;
uint16_t tag;
int device; /* device ordinal */
uint64_t stream; /* the cudaStream_t / hipStream_t the data is valid on (0: default) */
const int64_t *shape;
} CkDynDevDesc;
void CkDynMsgAddDevice(CkDynMsgBuilder *b, const CkDynDevDesc *d, CkDynBufDone done, void *arg);
A DEVICE slot is transferred device to device: through the stream within a
process, through the IPC buffer on one host, by RDMA otherwise. The rules of
NCPY slots apply, and done runs once the source may be reused. On the
receiver, CkDynViewBufferPtr returns device memory on the device named
by the slot’s reserved field (the receiver’s device). Unless the slot was
delivered into a posted buffer, that memory comes from the runtime’s device
pool and belongs to the message: it is freed after the hook returns, or at
the last CkDynMsgRelease if the binding retained the message. A binding
that hands the data to user code either copies it or retains the message for
as long as the user holds it.
A device send needs to know the destination PE: a DEVICE slot to a singleton
chare created with CKDYN_PE_ANY that is not yet constructed aborts. For an
array element CkDyn uses the sending PE’s last known location of the element
and chooses the transfer mode for that PE’s process. If the element has moved
to another process by the time the message arrives (or the sender’s location
was out of date), the receiver aborts (“Destination process does not match
the one the sender determined”); a move to another PE of the same process is
fine. A binding that sends DEVICE slots to array elements does so while
they do not migrate between processes (between load balancing steps, for
example).
CkDynHasDeviceSupport() returns 1 in a CUDA or HIP build; every device
function aborts in other builds.
6.9.4. HAPI callbacks
CkDynHapiAddCallback(stream, &cb) delivers an empty message to cb,
from the calling PE’s scheduler, when stream reaches this point. Quiescence
is not detected while such a callback is pending. A binding uses it to resume
a coroutine that waits for a kernel or a copy.
6.9.5. Persistent device buffers
typedef struct CkDynDevPersist CkDynDevPersist; /* opaque, valid on the PE that opened it */
CkDynDevPersist *CkDynDevPersistOpen(const CkDynDevDesc *d, const CkDynCallback *done);
void CkDynDevPersistClose(CkDynDevPersist *p);
size_t CkDynDevPersistHandleSize(void);
void CkDynDevPersistHandle(const CkDynDevPersist *p, void *out);
void CkDynDevPersistGet(CkDynDevPersist *dst, const void *src_handle, uint32_t refnum);
void CkDynDevPersistPut(const void *dst_handle, CkDynDevPersist *src, uint32_t refnum);
int CkDynDevPersistMode(const CkDynDevPersist *p, const void *handle);
A persistent device buffer is a device buffer opened once on one PE and then
used for any number of one-sided transfers, without a message or a memory
registration per transfer. CkDynDevPersistOpen opens the buffer d
names (ptr, len, device and stream are used; the stream
orders each transfer after earlier work on it); the memory is registered with
the network the first time a transfer needs RDMA, so a run in one process, or
one that uses only IPC, registers nothing. CkDynDevPersistHandle writes the
buffer’s handle, CkDynDevPersistHandleSize() bytes, which the binding
sends to other PEs in an ordinary message (a BYTES slot, for example).
CkDynDevPersistGet(dst, src_handle, refnum) copies the source buffer
named by the handle into dst, which must be open on the calling PE.
CkDynDevPersistPut(dst_handle, src, refnum) copies the local open buffer
src into the buffer named by the handle. The transfer moves the source’s
length; the destination must be at least as long. Either side may be on any
PE. done, the callback given to CkDynDevPersistOpen of the local
buffer, receives an empty message on the issuing PE once per completed
transfer: a get is complete when the bytes are in dst, a put when they
are in the remote buffer. For a FUTURE callback a non-zero refnum of the
call is the future id; for the other kinds the message carries the
callback’s own reference number. The caller keeps both buffers valid and
does not read the destination before done.
The transfer mode is chosen per pair of buffers, and
CkDynDevPersistMode returns it: 0, a device copy when both buffers are in
one process; 1, CUDA or HIP IPC when they are in different processes on one
host and the peer’s IPC handle opens in this process (+gpushm is not
needed; it concerns DEVICE slots); 2, RDMA on the registered memory
otherwise. Between hosts a put is carried out as a get issued by the
destination’s PE, so that done means the data has arrived. RDMA needs a
machine layer that can register device memory; the underlying
CkDevicePersistent aborts an RDMA transfer whose buffer cannot be
registered, or whose buffer’s owner has closed it. With +gpupersistverbose
the first use of each mode and operation in a process prints one line
([pe] CkDevicePersistent> get: ...).
The local buffer keeps what a transfer with a peer’s handle set up: its
reference on this process’s IPC mapping of the peer’s allocation and the
peer’s RDMA registration, so that later transfers with that handle reuse them.
A handle of the same peer buffer after the peer reopened it replaces the
earlier one. The same peer buffer means the same PE and device pointer: two
buffers open at once at one address (a whole array and a prefix of it, say)
replace each other’s copy at every use, and each RDMA transfer then requests
the registration again, so open one buffer per address.
CkDynDevPersistClose releases those, deregisters the local
memory if a transfer registered it, and frees the local record; it aborts
while a transfer of that buffer is outstanding. A handle stays usable only
while its buffer is open, and only in the run that opened it (a handle saved
in a checkpoint does not name a buffer after a restart). Persistent buffers
are not migrated or checkpointed. Every function aborts in a build without
CUDA or HIP.
charm4py binds them as charm.DevicePersistent.
Before a program frees the memory of a persistent buffer, every other process
on the same host that transferred with the buffer’s handle must have closed,
with CkDynDevPersistClose, each of its buffers that did so. A transfer
through IPC maps the peer’s whole allocation into the process that issues it,
and the mapping stays open until every buffer of that process that transferred
with a handle of the allocation has been closed; closing the peer’s own buffer
does not close mappings in other processes. CUDA documents freeing an
allocation (cudaFree) before the processes that imported it close their
mappings as undefined behavior, and the rule here is the same for HIP. A pool
allocator that returns cached blocks to the driver (CuPy’s
free_all_blocks, torch.cuda.empty_cache) frees memory in this sense.
There is no call that releases one peer’s copy without closing the local
buffer, so a binding that exposes persistent buffers passes this rule on to
its users.
6.10. Coroutines, Tracing and Load Balancing
6.10.1. Coroutines
A binding that runs entry methods on coroutines (charm4py’s greenlets) faces
two attribution problems once a coroutine suspends on a future and is
resumed later from the future hook: its time is charged to CkDyn’s
manager group, which the load balancer cannot move, and tracing shows
nothing for it. The binding therefore brackets every run of a coroutine:
CkDynCoroResume(&target, ep, coro_id); /* right after switching into the coroutine */
/* ... the coroutine runs until it waits again or finishes ... */
CkDynCoroSuspend(&target); /* right before switching out */
CkDynCoroResume pushes the target’s object on the running-object stack,
which charges the time to it, and, when tracing is on and ep is traced,
begins an execute event for ep. For an array element the event carries
the element’s index; for other kinds the object id is made from
coro_id. CkDynCoroSuspend ends both. Call the pair outside any
traced entry (from the future hook, which runs in an untraced entry, or
around each element of a deliver_many call) so that events do not nest.
charm4py’s switch function:
CkDynCoroResume(&c.t, c.ep, c.id);
r = PyGreenlet_Switch(w, argt, nullptr);
CkDynCoroSuspend(&c.t);
CkDynCallstackPush(&t) and CkDynCallstackPop(&t) do the
running-object half alone: load balancer timing and, in a CUDA build with
load balancing, the attribution of GPU kernels launched in between to that
object. charm4py uses them for a coroutine entry inside deliver_many,
whose trace events come from its own switch. Each call changes the stack by
exactly one entry: Push of an object that is not local pushes nothing that
the load balancer charges, and Pop after the object destroyed itself in
between (CkDynArrayDestroyElement) still pops its entry, so the running
object is again the one before Push.
6.10.2. Load balancing
CkDynAtSync,resume_from_sync,CkDynMigrateMeandCkDynSetMigratable: Migration and pup.CkDynLBInstrument(on)turns load instrumentation on or off.CkDynSetObjLoad(&t, secs)sets the load of a local array element, for a binding that measures time itself.CkDynHasLB()returns 1 when the build has load balancing.
The usual +balancer options select the strategy.
6.10.3. Tracing
Each dynamic class is its own Charm++ chare type and each entry its own
entry index, registered with the binding’s names. Projections therefore
shows Class.method names, and for array elements the element index, as
for a charmxi program. The usual tracing options (+traceroot,
+logsize, +traceoff) apply to a build with tracing enabled.
CKDYN_EP_NOTRACEon an entry leaves its executions out.CKDYN_SEND_NOTRACEon a send does not: it only keeps the message out of the load balancer’s communication statistics.CkDynTraceBeginEntry(ep)/CkDynTraceEndEntry(ep)bracket a region as an execute event ofepwith no object.User events:
CkDynTraceRegisterUserEvent(desc, id)returns the event id;CkDynTraceBeginUserBracketEvent(id)andCkDynTraceEndUserBracketEvent(id)bracket a region.Array broadcasts through
deliver_manyneed the per-element bracket above to appear per element.Projections reads an array element’s index fields from the dimension count of its class in the
.stsfile. A class registered withndims0 (any dimension count) takes the count of the first element constructed in each process, and only PE 0’s process writes the.sts: if the class’s arrays differ in dimension count, or PE 0’s process constructed none of its elements before the file was written, Projections shows the indices of some elements with the wrong number of fields. The count is recorded by every PE thread of a process without synchronization; the threads write the same value for one array. Register a class with a fixedndimswhen its indices matter in Projections.
6.10.4. Other runtime services
ckdyn.h wraps the runtime calls a binding needs besides CkDyn’s own, so
that it includes no Charm++ header: PE and node numbers
(CkDynMyPe, CkDynNumPes, CkDynMyNode, CkDynNumNodes,
CkDynMyRank, CkDynNodeSize, CkDynNodeFirst, CkDynNodeOf),
physical node topology, spanning tree edges (CkDynTopoTreeEdges),
CkDynExit, CkDynAbort, CkDynPrintf, CkDynWallTimer, timed
callbacks (CkDynCallAfter), and CCS handlers (CkDynCcsRegister and
the reply functions). They are listed in API Reference.
6.11. Writing a Binding
This chapter builds a minimal binding from the C++ test binding,
tests/charm++/ckdyn/cbinding.C: one array class with two methods, an
array, a send, a broadcast and a reduction.
6.11.1. A class as a table
The binding’s own class record holds a method table whose element 0 is the constructor, and receives the indices from registration:
typedef int64_t (*CbMethodFn)(Obj *o, const CkDynMsgView *v);
struct CbMethod { const char *name; uint32_t flags; CbMethodFn fn; };
struct CbClass {
const char *name; CkDynKind kind; int ndims;
std::vector<CbMethod> methods; /* methods[0]: constructor */
int chareIdx, firstEp;
int ep(int method) const { return firstEp + method; }
};
static int64_t hiCtor(Obj *, const CkDynMsgView *) { return 0; }
static int64_t hiAdd(Obj *o, const CkDynMsgView *v) { /* add(value, cb) */
int r = (int)v->slots[0].v.i64;
CkDynCallback cb;
CkDynViewCallback(v, 1, &cb);
CkDynRedData d = { &r, sizeof r, redType("sum_int") };
CkDynContribute(&o->t, &d, &cb);
return 0;
}
static int64_t hiPrint(Obj *o, const CkDynMsgView *v) { /* print(x) */
CkDynPrintf("got a value\n");
return 0;
}
CbClass Hi = { "Hi", CKDYN_ARRAY, 1, {
{ "Hi.Hi", 0, hiCtor },
{ "Hi.add", CKDYN_EP_REDNTARGET, hiAdd },
{ "Hi.print", 0, hiPrint },
} };
6.11.2. Registration
Registration turns the table into descriptors. user_id is the method’s
position, so delivery can index the table; binding_data is the class
record:
static void registerClass(CbClass *c) {
int n = (int)c->methods.size();
CkDynEntryDesc *e = new CkDynEntryDesc[n];
for (int i = 0; i < n; i++)
e[i] = { c->methods[i].name, c->methods[i].flags, (uint32_t)i };
CkDynClassDesc d;
memset(&d, 0, sizeof d);
d.name = c->name; d.kind = c->kind; d.ndims = c->ndims;
d.num_entries = n; d.entries = e;
d.binding_data = c;
c->chareIdx = CkDynRegisterClass(&d, &c->firstEp);
}
/* an initnode in the program's .ci file */
void registerAll(void) {
CkDynInit(cbHooks());
registerClass(&Hi);
}
6.11.3. The hooks
An object is a struct holding its class and its target. construct
makes one and runs the constructor method; deliver checks that the entry
belongs to the class and runs the method; destruct frees it:
struct Obj { const CbClass *cls; CkDynTarget t; /* state */ };
static void *hConstruct(void *bd, const CkDynTarget *t, const CkDynMsgView *v) {
const CbClass *c = (const CbClass *)bd;
Obj *o = new Obj;
o->cls = c;
o->t = *t;
c->methods[0].fn(o, v);
return o;
}
static void hDeliver(void *bd, void *h, int ep, uint32_t user_id, uint32_t,
const CkDynMsgView *v) {
const CbClass *c = (const CbClass *)bd;
if (user_id >= c->methods.size() || ep != c->ep((int)user_id))
CkDynAbort("delivery does not match the class table");
c->methods[user_id].fn((Obj *)h, v);
}
static void hDestruct(void *, void *h) { delete (Obj *)h; }
const CkDynHooks *cbHooks(void) {
static CkDynHooks g;
memset(&g, 0, sizeof g);
g.version = CKDYN_API_VERSION;
g.size = sizeof(CkDynHooks);
g.construct = hConstruct;
g.destruct = hDestruct;
g.deliver = hDeliver;
g.future = hFuture; /* required: CkDynCreateManagers aborts without it */
g.pack_size = hPackSize; /* needed once elements migrate */
g.pack = hPack;
g.unpack = hUnpack;
g.error = hError;
return &g;
}
The future hook looks the id up in a per-PE table of callbacks and runs
the one it finds; cbNewFuture(fn) allocates an id and stores fn:
static void hFuture(void *, int pe, uint32_t id, const CkDynMsgView *v) {
PeState *s = CkpvAccess(cbPe);
auto it = s->futures.find(id);
CbFutureFn fn = std::move(it->second);
s->futures.erase(it);
fn(v);
}
6.11.4. Create, send, broadcast, reduce
The program’s main chare calls CkDynCreateManagers() and then uses the
class. MsgB is a small wrapper over the builder functions:
CkDynCreateManagers();
int dims[1] = { 8 }; /* aid below is a global (a readonly in the .ci file) */
CkDynArrayOpts ao = { 1, dims, -1, 0, 0, -1 };
uint32_t created = cbNewFuture([](const CkDynMsgView *) {
/* every element exists: send to element 3 */
int i = 3;
CkDynSendArray(aid, 1, &i, Hi.ep(2), MsgB().i64(42).finish(0), nullptr);
/* broadcast add(1, cb); the sum arrives at a future on PE 0 */
uint32_t f = cbNewFuture([](const CkDynMsgView *v) {
const int *sum = (const int *)CkDynViewBufferPtr(v, 0, nullptr);
CkDynPrintf(*sum == 8 ? "sum ok\n" : "wrong sum\n");
CkDynExit(0);
});
CkDynSendArray(aid, 1, nullptr, Hi.ep(1), MsgB().i64(1).cb(cbFutureCb(0, f)).finish(0), nullptr);
});
aid = CkDynCreateArray(Hi.chareIdx,
MsgB().flags(CKDYN_MSG_CREATION).fut(0, created).finish(Hi.ep(0)), &ao);
Run it as any Reconverse program, for example ./hi +pe 4.
A shared-library binding does the same from register_classes and from its
mainchare hook, with CkDynInit and CkDynStart in its start-up
function.
6.11.5. Checklist
The rules a binding must follow, collected from the chapters above:
Check
CkDynApiVersion()and setversionandsizein every hook table. CheckCkDynRefnumBytes()if future ids may exceed 65534.Register the same classes in the same order on every process, from rank 0.
Keep the hook table and what
binding_datapoints to valid for the program’s lifetime (the descriptor, the entry table and the name strings are copied at registration).Never return NULL from
construct,mainchareorunpack.Abandon a builder on every error path between
CkDynMsgBeginandCkDynMsgFinish; free a finished message that will not be sent withCkDynMsgFree. Never touch a message after passing it to a send, creation,CkDynFutureSetorCkDynCallbackSend.Treat every view and every pointer read from it as valid only during the hook. Call
CkDynMsgRetainbefore keeping one, andCkDynMsgRelease(msg)once per retain.Keep NCPY and DEVICE source buffers valid until their
donefunction runs; expectdoneexactly once, from the sending PE’s scheduler, also for abandoned or freed messages.Return the same size from both
pack_sizepasses and write exactly that many bytes inpack; returnCKDYN_PACK_REFUSEDfor an object that cannot be packed.Keep per-PE state per thread: hooks of different PEs of one process may run at the same time, and
reduceruns on any PE.Expect hooks to nest: an inline send, or a send to a local object with an inline entry, runs
deliverinside the caller’s hook;reducecan run while the binding is building a message.Catch the binding language’s exceptions inside each hook. Report errors with
CkDynAbort; implementerrorto print CkDyn’s errors in the binding’s terms (CkDyn aborts after it returns).Wait for a section’s
donecallback before sending to it or contributing to it; recreate sections after a restart.Bracket every coroutine run with
CkDynCoroResume/CkDynCoroSuspend, and every element of adeliver_manycall if per-element load and tracing matter.A binding for an interpreted language releases its interpreter lock around
CkDynStartand takes it in every hook.
6.12. API Reference
Every function of ckdyn.h, grouped as in the header. Types and constants
are described in the chapters linked from each group. Unless stated
otherwise, a function that names a local object by target aborts when the
object is not on the calling PE.
6.12.1. Version and start-up
int CkDynApiVersion(void)The
CKDYN_API_VERSIONthe library was built with.int CkDynRefnumBytes(void)The width of the envelope reference number in bytes (2 by default, 4 with
--with-refnum-type="unsigned int"); bounds future ids.int CkDynHasDeviceSupport(void)1 in a CUDA or HIP build, else 0.
void CkDynInit(const CkDynHooks *hooks)Copies and checks the hook table; registers the readonly hooks. Before
CkDynStart(shared library) or from aninitnode(C++ program).void CkDynStart(int argc, char **argv)Starts the runtime; never returns. Shared-library bindings only, after
CkDynInit.void CkDynCreateManagers(void)Creates CkDyn’s manager group and multicast manager. Called by a dynamic main chare itself; an ordinary C++ main chare calls it from its constructor. Later calls do nothing.
6.12.2. Registration
See Start-up and Registration.
int CkDynRegisterClass(const CkDynClassDesc *desc, int *first_ep)Registers a class on rank 0 before start-up completes; returns the chare index and stores the constructor’s entry index in
*first_ep. Aborts after start-up (useCkDynRegisterClassLate), on a NULL name, and when an entry index would exceed 65535 (the message header keeps 16 bits).void CkDynAllowLateRegistration(void)Enables
CkDynRegisterClassLate.int CkDynRegisterClassLate(const CkDynClassDesc *desc, int *first_ep)Registers a class after start-up; one PE per process only.
const CkDynClassDesc *CkDynClassOf(int chare_idx)The descriptor of a dynamic class, or NULL.
6.12.3. Message building
See Messages. Each Add function appends one slot.
CkDynMsgBuilder *CkDynMsgBegin(int nslots_hint, size_t data_hint, size_t tail_hint, int prio_bits, int group_dep)Begins a message with
prio_bitspriority bits and an optional group dependency (-1: none).void CkDynMsgSetFlags(CkDynMsgBuilder *b, uint16_t flags)Sets the header flags (
CKDYN_MSG_*).void CkDynMsgSetUserId(CkDynMsgBuilder *b, uint32_t user_id)Sets the header’s
user_id: the reducer id of a foreign-reducer contribution.void CkDynMsgAddNone(CkDynMsgBuilder *b)A NONE slot.
void CkDynMsgAddBool(CkDynMsgBuilder *b, int v),void CkDynMsgAddI64(CkDynMsgBuilder *b, int64_t v),void CkDynMsgAddU64(CkDynMsgBuilder *b, uint64_t v),void CkDynMsgAddF64(CkDynMsgBuilder *b, double v)Scalar slots, stored inline in the slot.
void CkDynMsgAddBytes(CkDynMsgBuilder *b, const void *p, uint32_t len, uint16_t tag)A BYTES slot; the bytes are copied now.
void CkDynMsgAddBuffer(CkDynMsgBuilder *b, const CkDynBufDesc *d)A BUFFER slot; the shape is copied now, the contents at
Finish, sod->ptrstays valid untilFinishorAbandon. In thereducehook’s result builder the contents are copied now.void CkDynMsgAddFuture(CkDynMsgBuilder *b, int pe, uint32_t id)A FUTURE slot.
void CkDynMsgAddCallback(CkDynMsgBuilder *b, const CkDynCallback *cb)A CALLBACK slot.
void CkDynMsgAddSection(CkDynMsgBuilder *b, const CkDynSectionID *sid)A SECTION slot.
void CkDynMsgAddProxy(CkDynMsgBuilder *b, const CkDynTarget *t)A PROXY slot.
void *CkDynMsgReserveTail(CkDynMsgBuilder *b, uint32_t len)Reserves the tail; the binding writes
lenbytes at the returned pointer.CkDynMsg *CkDynMsgFinish(CkDynMsgBuilder *b, int ep)Allocates the message and ends the builder.
epgoes in the header; constructor messages need the class’sfirst_ep.CkDynMcastMsg *CkDynMsgFinishMcast(CkDynMsgBuilder *b, int ep)The same payload as a section message, for
CkDynSendSection.void CkDynMsgAbandon(CkDynMsgBuilder *b)Drops a builder; runs the
donefunctions of its zero-copy slots.void *CkDynMsgPriorityPtr(CkDynMsg *m)The reserved priority bits of a finished message, to fill in.
void CkDynMsgSetQueueing(CkDynMsg *m, int queueing)Sets the queueing strategy (a
CKDYN_QUEUEING_*value).void CkDynMsgFree(CkDynMsg *m)Frees a finished message that will not be sent.
const void *CkDynMsgPayload(CkDynMsg *m, uint32_t *len)The payload bytes (header to tail), for a foreign-reducer contribution.
6.12.4. Reading messages
const void *CkDynViewBufferPtr(const CkDynMsgView *v, int slot, const int64_t **shape_out)The data of a BUFFER, NCPY or DEVICE slot (device memory for DEVICE);
*shape_out, if not NULL, gets thendimshape values.const void *CkDynViewBytesPtr(const CkDynMsgView *v, int slot)The bytes of a BYTES slot.
void CkDynViewCallback(const CkDynMsgView *v, int slot, CkDynCallback *out),void CkDynViewSection(const CkDynMsgView *v, int slot, CkDynSectionID *out),void CkDynViewProxy(const CkDynMsgView *v, int slot, CkDynTarget *out)Copy a CALLBACK, SECTION or PROXY slot out.
void CkDynMsgRetain(const CkDynMsgView *v)Keeps the delivered message alive after the hook.
void CkDynMsgRelease(void *msg)Drops one retain of
view->msg.
6.12.5. Sends
See Messages. Each takes ownership of the message; o may be
NULL.
void CkDynSendChare(const CkDynTarget *t, int ep, CkDynMsg *m, const CkDynSendOpts *o)To a singleton chare or main chare.
void CkDynSendGroup(int gid, int pe, int ep, CkDynMsg *m, const CkDynSendOpts *o)To a group or array map branch;
pe-1 broadcasts.void CkDynSendGroupMulti(int gid, int npes, const int *pes, int ep, CkDynMsg *m, const CkDynSendOpts *o)To the branches on a PE list.
void CkDynSendNodeGroup(int gid, int node, int ep, CkDynMsg *m, const CkDynSendOpts *o)To a nodegroup branch;
node-1 broadcasts.void CkDynSendArray(int aid, int ndims, const int *index, int ep, CkDynMsg *m, const CkDynSendOpts *o)To an array element;
indexNULL broadcasts.void CkDynSendSection(const CkDynSectionID *sid, int ep, CkDynMcastMsg *m, const CkDynSendOpts *o)Multicast to a ready section;
ois ignored.void CkDynSendTarget(const CkDynTarget *t, int ep, CkDynMsg *m, const CkDynSendOpts *o)To whatever object the target names.
6.12.6. Creation
See Objects. Each takes ownership of the constructor message.
void CkDynCreateChare(int chare_idx, int on_pe, CkDynMsg *ctor, CkDynTarget *out)Creates a singleton chare on
on_peorCKDYN_PE_ANY;*outis usable at once.int CkDynCreateGroup(int chare_idx, CkDynMsg *ctor)Creates a group or array map; returns its id. Constrained with
CKDYN_MSG_CONSTRAINED.int CkDynCreateNodeGroup(int chare_idx, CkDynMsg *ctor)Creates a nodegroup; returns its id.
int CkDynCreateArray(int chare_idx, CkDynMsg *ctor, const CkDynArrayOpts *o)Creates an array, full or empty; returns its id.
void CkDynArrayInsert(int aid, int ndims, const int *index, int on_pe, CkDynMsg *ctor)Inserts one element;
ctorfinished with the array class’sfirst_ep.void CkDynArrayDoneInserting(int aid)Ends insertion.
void CkDynArrayDestroyElement(int aid, int ndims, const int *index)Destroys a local element.
6.12.7. Futures and callbacks
void CkDynFutureSet(int pe, uint32_t id, CkDynMsg *value)Sends
valueto the binding future(pe, id); thefuturehook runs onpe.void CkDynCallbackSend(const CkDynCallback *cb, CkDynMsg *m)Sends a message through a callback.
int CkDynGetReductionNumber(const CkDynTarget *t)The reduction number of a local group or nodegroup branch, or array element. An array map has none: the call aborts.
6.12.8. Reductions
See Reductions.
int CkDynReducers(const CkDynReducerInfo **out)The built-in reducer table; returns its length.
int CkDynForeignReducerType(void)The reducer type for binding-defined reducers.
void CkDynContribute(const CkDynTarget *me, const CkDynRedData *d, const CkDynCallback *cb)Contributes for a local group, nodegroup or array element.
6.12.9. Sections
See Sections.
void CkDynSectionCreateArray(int aid, int nelems, int ndims, const int *indices, int bfactor, const CkDynCallback *done, CkDynSectionID *out)Creates an array section rooted on the calling PE.
void CkDynSectionCreateGroup(int gid, int npes, const int *pes, int bfactor, const CkDynCallback *done, CkDynSectionID *out)Creates a section of a dynamic group.
void CkDynSectionCombine(int nparts, const CkDynSectionID *parts, const CkDynCallback *done, CkDynSectionID *out)The union of array and combined sections.
void CkDynSectionDelete(const CkDynSectionID *sid)Deletes a section.
int CkDynSectionRedNo(const CkDynTarget *me, const CkDynSectionID *sid)The member’s section reduction count.
void CkDynContributeSection(const CkDynTarget *me, const CkDynSectionID *sid, const CkDynRedData *d, const CkDynCallback *cb)Contributes to a section reduction.
6.12.10. Checkpoint, quiescence, load balancing, tracing
See Migration and pup and Coroutines, Tracing and Load Balancing.
void CkDynCheckpoint(const char *dirname, const CkDynCallback *done)Writes a checkpoint;
donefires when written and after a restart.void CkDynStartQD(const CkDynCallback *cb)Starts quiescence detection.
void CkDynAtSync(const CkDynTarget *t)AtSyncfor a local array element created withCKDYN_MSG_USESATSYNC.void CkDynSetMigratable(const CkDynTarget *t, int migratable)Allows or forbids migration of a local array element.
void CkDynMigrateMe(const CkDynTarget *t, int to_pe)Migrates a local array element.
void CkDynLBInstrument(int on)Turns load balancer instrumentation on or off.
void CkDynSetObjLoad(const CkDynTarget *t, double secs)Sets a local array element’s load.
void CkDynCallstackPush(const CkDynTarget *t),void CkDynCallstackPop(const CkDynTarget *t)Push or pop a local object on the running-object stack. Always paired: Pop pops one entry even when the object was destroyed after Push.
void CkDynCoroResume(const CkDynTarget *t, int ep, uint64_t coro_id),void CkDynCoroSuspend(const CkDynTarget *t)Bracket one run of a coroutine: running object and trace event.
int CkDynTraceRegisterUserEvent(const char *desc, int id)Registers a user event; returns its id.
void CkDynTraceBeginUserBracketEvent(int id),void CkDynTraceEndUserBracketEvent(int id)Bracket a user event.
void CkDynTraceBeginEntry(int ep),void CkDynTraceEndEntry(int ep)Bracket a region as an execute event of
ep.
6.12.11. Zero-copy and devices
void CkDynMsgAddNcpy(CkDynMsgBuilder *b, const CkDynBufDesc *d, CkDynBufDone done, void *arg)A host buffer transferred without a copy into the message.
void CkDynMsgAddDevice(CkDynMsgBuilder *b, const CkDynDevDesc *d, CkDynBufDone done, void *arg)A device buffer transferred device to device.
void CkDynPostBuffer(const CkDynTarget *t, int ep, int slot, const CkDynBufDesc *d)Posts a host destination for a slot of the next message to a local object’s entry.
void CkDynPostDeviceBuffer(const CkDynTarget *t, int ep, int slot, const CkDynDevDesc *d)Posts a device destination.
int CkDynUnpostBuffer(const CkDynTarget *t, int ep, int slot)Withdraws an outstanding post; returns 1 if there was one.
slotof a post or unpost may carryCKDYN_POST_ARGto count arguments.CkDynDevPersist *CkDynDevPersistOpen(const CkDynDevDesc *d, const CkDynCallback *done),CkDynDevPersistClose,CkDynDevPersistHandleSize,CkDynDevPersistHandle,CkDynDevPersistGet,CkDynDevPersistPut,CkDynDevPersistModePersistent device buffers: Persistent device buffers.
void CkDynHapiAddCallback(uint64_t stream, const CkDynCallback *cb)Delivers an empty message to
cbwhen the stream reaches this point.
6.12.12. Runtime wrappers
int CkDynMyPe(void),int CkDynNumPes(void),int CkDynMyNode(void),int CkDynNumNodes(void),int CkDynMyRank(void)This PE, the PE count, this process’s node, the node count, this PE’s rank in its process.
int CkDynNodeSize(int node),int CkDynNodeFirst(int node),int CkDynNodeOf(int pe)PEs of a node, its first PE, the node of a PE.
void CkDynExit(int code)Ends the program with an exit code.
void CkDynAbort(const char *msg)Aborts with a message.
void CkDynPrintf(const char *text)Prints
textthrough the runtime (no format processing).double CkDynWallTimer(void)Wall-clock seconds.
int CkDynNumPhysicalNodes(void),int CkDynPhysicalNodeID(int pe),int CkDynPhysicalRank(int pe),int CkDynNumPesOnPhysicalNode(int node)Physical node topology.
int CkDynGetPesOnPhysicalNode(int node, int *pes, int max)Fills up to
maxPEs of a physical node; returns their count.int CkDynTopoTreeEdges(int pe, int root, const int *pes, int npes, int bfactor, int *parent, int *children, int max)pe’s parent and up tomaxchildren in the topology-aware spanning tree overpes(NULL: all PEs) rooted atroot; returns the child count.void CkDynCallAfter(void (*fn)(void *arg, double now), void *arg, double msecs)Calls
fnfrom the scheduler aftermsecsmilliseconds.int CkDynIsRestarting(void)1 in a program started with
+restart, from the start of the restart to the end of the run (so adonecallback delivered by the restart can tell itself from one delivered by a later checkpoint).int CkDynHasLB(void)1 when the build has load balancing.
6.12.13. CCS
void CkDynCcsRegister(const char *name, CkDynCcsHandler fn)Registers a CCS handler;
fn(global, name, data, len)runs for each request.int CkDynCcsIsRemoteRequest(void)1 inside a handler serving a remote request.
void CkDynCcsSendReply(int len, const void *data)Replies to the current request.
void *CkDynCcsDelayReply(void),void CkDynCcsSendDelayedReply(void *token, int len, const void *data)Defer the reply and send it later with the token.
6.13. Further Reading
doc/internals/ckdyn.mdin the Charm++ repository: how CkDyn implements each feature (registration tables, the payload layout, delivery paths, the section protocol, the zero-copy prefix, pup), with the source files.src/ck-core/ckdyn.h: the API, with comments; the authority where this manual and the header differ.tests/charm++/ckdyn/: the C++ test binding (cbinding.h,cbinding.C) and the acceptance test (ckdyntest.C), which uses every feature.The charm4py repository:
src/ckdyn_py.cpp, the Python binding, and the design documents underdocs/design/:DESIGN.md(section 3 gives the reasons for CkDyn’s choices),SPEC-3.md(zero-copy and devices),SPEC-4b.md(sections and constrained groups in the binding) andSPEC-7-device.md(persistent device buffers).