Skip to content

Array files

Models always hold arrays; saving decides where each one goes. save writes an array field over 1 MiB to its own .npy file beside the saved file, and a text in files over 1 MiB to a file of its own, each with a pointer in its place, and open reads them back (see Conventions). Any other writer of qcdata documents with arrays beside them uses the same pieces.

qcdata.array_files

Large arrays and texts saved apart from a model's file, as files beside it.

Models always hold arrays; only saving decides where an array goes. An array field whose float64 size exceeds 1 MiB is written to its own .npy file in the saved file's directory, named by the saved file's stem and the array's path in the document (output.results.hessian.npy), and the document holds a pointer in its place:

{"path": "output.results.hessian.npy", "shape": [400, 400], "dtype": "float64",
 "sha256": "..."}

sha256 is the digest of the .npy file's bytes. Opening the file reads every array back, checking its digest, shape and dtype, so a loaded model holds arrays again. Only array fields (SerializableNDArray) are saved apart; free-form values such as extras and record stay inline.

A text in files whose UTF-8 size exceeds 1 MiB is saved apart the same way, when the sink is made with text_files=True (as QCDataBaseModel.save does): as its UTF-8 bytes, named by the stem and its path with no suffix added (output.results.files.orbitals%2Emolden), with {"path", "sha256"} in its place. Binary files stay inline.

QCDataBaseModel.save and QCDataBaseModel.open use this module, and any other writer of qcdata documents with arrays beside them (a worker's wire) can do the same:

from qcdata.array_files import DIRECTORY, SINK, ArraySink

sink = ArraySink()
document = model.model_dump(mode="json", context={SINK: sink})
document, written = sink.place(document, directory, stem)  # arrays to .npy files
model = type(model).model_validate(document, context={DIRECTORY: directory})

ARRAY_FILE_BYTES module-attribute

ARRAY_FILE_BYTES = 1 << 20

An array field is saved apart when its float64 size exceeds this many bytes (1 MiB).

DIRECTORY module-attribute

DIRECTORY = 'qcdata.array_directory'

Validation context key: the directory holding the saved arrays.

POINTER_KEYS module-attribute

POINTER_KEYS = frozenset(
    {"path", "shape", "dtype", "sha256"}
)

The keys of the pointer that stands in a document for an array saved apart.

SINK module-attribute

SINK = 'qcdata.array_sink'

Serialization context key: the ArraySink collecting arrays to save apart.

TEXT_POINTER_KEYS module-attribute

TEXT_POINTER_KEYS = frozenset({'path', 'sha256'})

The keys of the pointer that stands in files for a text saved apart.

ArraySink

ArraySink(*, text_files: bool = False)

Collects the arrays a save puts in files, then writes them and their pointers.

The serializer replaces each large array (and, with text_files, each large text in files) with a mark, {mark: index}, unique to this save so that no saved value can be mistaken for one. place then swaps every mark for a pointer, naming the file by the mark's path in the document.

Attributes:

Name Type Description
text_files

Whether texts in files over 1 MiB are saved apart too.

arrays list[ndarray]

The arrays held for their own files, in the order added.

texts list[str]

The texts held for their own files, in the order added.

Parameters:

Name Type Description Default
text_files bool

Also save texts in files over 1 MiB (UTF-8) apart.

False
Source code in src/qcdata/array_files.py
87
88
89
90
91
92
93
94
95
96
97
def __init__(self, *, text_files: bool = False) -> None:
    """Start a save with no arrays.

    Args:
        text_files: Also save texts in ``files`` over 1 MiB (UTF-8) apart.
    """
    self.mark = f"qcdata.array.{secrets.token_hex(8)}"
    self.text_mark = f"{self.mark}.text"
    self.text_files = text_files
    self.arrays: list[np.ndarray] = []
    self.texts: list[str] = []

add

add(array: ndarray) -> dict[str, int]

Hold an array for its own file; return the mark that takes its place.

Source code in src/qcdata/array_files.py
 99
100
101
102
def add(self, array: np.ndarray) -> dict[str, int]:
    """Hold an array for its own file; return the mark that takes its place."""
    self.arrays.append(array)
    return {self.mark: len(self.arrays) - 1}

add_text

add_text(text: str) -> str | dict[str, int]

Hold a text over 1 MiB for its own file and return its mark; else the text.

Parameters:

Name Type Description Default
text str

The content of an entry of files.

required

Returns:

Type Description
str | dict[str, int]

The mark that takes the text's place, or the text itself when it stays

str | dict[str, int]

inline (the sink saves no texts, or it is 1 MiB or less).

Source code in src/qcdata/array_files.py
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
def add_text(self, text: str) -> str | dict[str, int]:
    """Hold a text over 1 MiB for its own file and return its mark; else the text.

    Args:
        text: The content of an entry of ``files``.

    Returns:
        The mark that takes the text's place, or the text itself when it stays
        inline (the sink saves no texts, or it is 1 MiB or less).
    """
    # Up to 4 UTF-8 bytes per character: only a text near the limit is encoded.
    if not self.text_files or len(text) * 4 <= ARRAY_FILE_BYTES:
        return text
    if len(text) <= ARRAY_FILE_BYTES and len(text.encode()) <= ARRAY_FILE_BYTES:
        return text
    self.texts.append(text)
    return {self.text_mark: len(self.texts) - 1}

place

place(
    document: Any, directory: Path, stem: str
) -> tuple[Any, list[Path]]

Write each held array to its file and put its pointer in the document.

Parameters:

Name Type Description Default
document Any

The dumped model, holding marks.

required
directory Path

Where the model's file and its array files go.

required
stem str

The model file's stem, which every array file name starts with.

required

Returns:

Type Description
Any

The document with pointers for marks, and the array files written, in

list[Path]

document order.

Source code in src/qcdata/array_files.py
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
def place(
    self, document: Any, directory: Path, stem: str
) -> tuple[Any, list[Path]]:
    """Write each held array to its file and put its pointer in the document.

    Args:
        document: The dumped model, holding marks.
        directory: Where the model's file and its array files go.
        stem: The model file's stem, which every array file name starts with.

    Returns:
        The document with pointers for marks, and the array files written, in
        document order.
    """
    written: list[Path] = []
    if not self.arrays and not self.texts:
        return document, written

    def visit(node: Any, path: tuple[str, ...]) -> Any:
        if isinstance(node, dict):
            if len(node) == 1 and self.mark in node:
                name = ".".join((stem, *map(_file_part, path), "npy"))
                written.append(directory / name)
                return _write(self.arrays[node[self.mark]], directory / name)
            if len(node) == 1 and self.text_mark in node:
                name = ".".join((stem, *map(_file_part, path)))
                written.append(directory / name)
                text = self.texts[node[self.text_mark]]
                return _write_text(text, directory / name)
            return {k: visit(v, (*path, k)) for k, v in node.items()}
        if isinstance(node, list):
            return [visit(v, (*path, str(i))) for i, v in enumerate(node)]
        return node

    return visit(document, ()), written

read_text

read_text(
    pointer: dict[str, Any], directory: Path | None
) -> str

Read the text a pointer in files names, checking it is the one saved.

Parameters:

Name Type Description Default
pointer dict[str, Any]

{"path", "sha256"}, as a save writes it.

required
directory Path | None

The directory of the saved file (the DIRECTORY context).

required

Returns:

Type Description
str

The text.

Raises:

Type Description
ValueError

If the pointer is malformed or the file is missing or changed.

Source code in src/qcdata/array_files.py
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
def read_text(pointer: dict[str, Any], directory: Path | None) -> str:
    """Read the text a pointer in ``files`` names, checking it is the one saved.

    Args:
        pointer: ``{"path", "sha256"}``, as a save writes it.
        directory: The directory of the saved file (the `DIRECTORY` context).

    Returns:
        The text.

    Raises:
        ValueError: If the pointer is malformed or the file is missing or changed.
    """
    if set(pointer) != TEXT_POINTER_KEYS:
        raise ValueError(
            "A file takes its content, or the pointer a save writes for a text in its "
            "own file (path, sha256)."
        )
    return _read_bytes(pointer, directory, "text").decode()

serialize_array

serialize_array(value: Any, info: SerializationInfo) -> Any

Dump an array as nested lists, or as a mark when a save puts it in a file.

Source code in src/qcdata/array_files.py
159
160
161
162
163
164
165
def serialize_array(value: Any, info: SerializationInfo) -> Any:
    """Dump an array as nested lists, or as a mark when a save puts it in a file."""
    array = np.asarray(value)
    sink = info.context.get(SINK) if isinstance(info.context, dict) else None
    if sink is not None and array.size * 8 > ARRAY_FILE_BYTES:
        return sink.add(array)
    return array.tolist()

validate_array

validate_array(value: Any, info: ValidationInfo) -> ndarray

Make an independent float64 array, reading a saved array file's pointer.

Source code in src/qcdata/array_files.py
168
169
170
171
172
173
def validate_array(value: Any, info: ValidationInfo) -> np.ndarray:
    """Make an independent float64 array, reading a saved array file's pointer."""
    if isinstance(value, dict):
        context = info.context if isinstance(info.context, dict) else {}
        return _read(value, context.get(DIRECTORY))
    return np.array(value, dtype=np.float64)