Array files¶
Models always hold arrays; saving decides where each one goes. save writes an array
field over 1 MiB to its own .npy file beside the saved file, and a text in files over
1 MiB to a file of its own, each with a pointer in its place, and open reads them back (see Conventions). Any other
writer of qcdata documents with arrays beside them uses the same pieces.
qcdata.array_files
¶
Large arrays and texts saved apart from a model's file, as files beside it.
Models always hold arrays; only saving decides where an array goes. An array field whose
float64 size exceeds 1 MiB is written to its own .npy file in the saved file's
directory, named by the saved file's stem and the array's path in the document
(output.results.hessian.npy), and the document holds a pointer in its place:
{"path": "output.results.hessian.npy", "shape": [400, 400], "dtype": "float64",
"sha256": "..."}
sha256 is the digest of the .npy file's bytes. Opening the file reads every
array back, checking its digest, shape and dtype, so a loaded model holds arrays again.
Only array fields (SerializableNDArray) are saved apart; free-form values such as
extras and record stay inline.
A text in files whose UTF-8 size exceeds 1 MiB is saved apart the same way, when the
sink is made with text_files=True (as QCDataBaseModel.save does): as its UTF-8
bytes, named by the stem and its path with no suffix added
(output.results.files.orbitals%2Emolden), with {"path", "sha256"} in its place.
Binary files stay inline.
QCDataBaseModel.save and QCDataBaseModel.open use this module, and any other writer
of qcdata documents with arrays beside them (a worker's wire) can do the same:
from qcdata.array_files import DIRECTORY, SINK, ArraySink
sink = ArraySink()
document = model.model_dump(mode="json", context={SINK: sink})
document, written = sink.place(document, directory, stem) # arrays to .npy files
model = type(model).model_validate(document, context={DIRECTORY: directory})
ARRAY_FILE_BYTES
module-attribute
¶
ARRAY_FILE_BYTES = 1 << 20
An array field is saved apart when its float64 size exceeds this many bytes (1 MiB).
DIRECTORY
module-attribute
¶
DIRECTORY = 'qcdata.array_directory'
Validation context key: the directory holding the saved arrays.
POINTER_KEYS
module-attribute
¶
POINTER_KEYS = frozenset(
{"path", "shape", "dtype", "sha256"}
)
The keys of the pointer that stands in a document for an array saved apart.
SINK
module-attribute
¶
SINK = 'qcdata.array_sink'
Serialization context key: the ArraySink collecting arrays to save apart.
TEXT_POINTER_KEYS
module-attribute
¶
TEXT_POINTER_KEYS = frozenset({'path', 'sha256'})
The keys of the pointer that stands in files for a text saved apart.
ArraySink
¶
ArraySink(*, text_files: bool = False)
Collects the arrays a save puts in files, then writes them and their pointers.
The serializer replaces each large array (and, with text_files, each large text
in files) with a mark, {mark: index}, unique to this save so that no saved
value can be mistaken for one. place then swaps every mark for a pointer, naming
the file by the mark's path in the document.
Attributes:
| Name | Type | Description |
|---|---|---|
text_files |
Whether texts in |
|
arrays |
list[ndarray]
|
The arrays held for their own files, in the order added. |
texts |
list[str]
|
The texts held for their own files, in the order added. |
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
text_files
|
bool
|
Also save texts in |
False
|
Source code in src/qcdata/array_files.py
87 88 89 90 91 92 93 94 95 96 97 | |
add
¶
add(array: ndarray) -> dict[str, int]
Hold an array for its own file; return the mark that takes its place.
Source code in src/qcdata/array_files.py
99 100 101 102 | |
add_text
¶
add_text(text: str) -> str | dict[str, int]
Hold a text over 1 MiB for its own file and return its mark; else the text.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
text
|
str
|
The content of an entry of |
required |
Returns:
| Type | Description |
|---|---|
str | dict[str, int]
|
The mark that takes the text's place, or the text itself when it stays |
str | dict[str, int]
|
inline (the sink saves no texts, or it is 1 MiB or less). |
Source code in src/qcdata/array_files.py
104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 | |
place
¶
place(
document: Any, directory: Path, stem: str
) -> tuple[Any, list[Path]]
Write each held array to its file and put its pointer in the document.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
document
|
Any
|
The dumped model, holding marks. |
required |
directory
|
Path
|
Where the model's file and its array files go. |
required |
stem
|
str
|
The model file's stem, which every array file name starts with. |
required |
Returns:
| Type | Description |
|---|---|
Any
|
The document with pointers for marks, and the array files written, in |
list[Path]
|
document order. |
Source code in src/qcdata/array_files.py
122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 | |
read_text
¶
read_text(
pointer: dict[str, Any], directory: Path | None
) -> str
Read the text a pointer in files names, checking it is the one saved.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pointer
|
dict[str, Any]
|
|
required |
directory
|
Path | None
|
The directory of the saved file (the |
required |
Returns:
| Type | Description |
|---|---|
str
|
The text. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the pointer is malformed or the file is missing or changed. |
Source code in src/qcdata/array_files.py
206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 | |
serialize_array
¶
serialize_array(value: Any, info: SerializationInfo) -> Any
Dump an array as nested lists, or as a mark when a save puts it in a file.
Source code in src/qcdata/array_files.py
159 160 161 162 163 164 165 | |
validate_array
¶
validate_array(value: Any, info: ValidationInfo) -> ndarray
Make an independent float64 array, reading a saved array file's pointer.
Source code in src/qcdata/array_files.py
168 169 170 171 172 173 | |