Skip to content

Enable storage of maximally compact JSON-encoded data (such as fit results) - #973

Merged
thehrh merged 2 commits into
masterfrom
enable_compactified_json
Sep 29, 2026
Merged

thehrh merged 2 commits into
masterfrom
enable_compactified_json

Conversation

@thehrh

@thehrh thehrh commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

PISA currently only exposes the indent parameter of simplejson.dumps() via utils.jsons.to_json() and therefore utils.fileio.to_file(), which has a default value of 2 for pretty (easily readable and editable) JSON files. However, the minimal size of the output JSON-encoded data is achieved by combining the arguments

  1. indent=None
  2. separators=(',', ':')

In particular, according to the above simplejson docs, no value for indent results in separators without any surrounding whitespace. Hence, PISA itself is unable to produce the most compact JSON which is functionally identical to the original (extra steps such as python -m json.tool <input.json> <output.json> --compact would be required). This is desirable for large-scale analyses with many (large) fits though.

Solution

Add an intuitive compactify parameter to utils.jsons.to_json(), which if set to True (default remains False though, so precedence is given to "prettiness") overrides any indent choice and results in calling of simplejson.dumps() with separators=(',', ':') and indent=None. I have also extended the roundtrip unit test such that it makes use of compactify and checks that file sizes are indeed reduced (where this is expected).

Impact

  • The size on disk of a typical uncompressed fit results JSON file with a fit history for $10^3$ ($10^4$) function calls decreases from approx. 600 KB $\rightarrow$ 400 KB (from 5.1 MB $\rightarrow$ 3.6 MB).
  • In the bz2-compressed case, the corresponding sizes seem to be roughly unaffected (at 50 KB) respectively decrease from 260 KB $\rightarrow$ 220 KB.

Further remarks

The pull request also removes the spurious outfile argument/second positional argument accidentally passed to json.dumps() in

content, outfile, indent=indent, cls=NumpyEncoder,
, which is currently being ignored silently (only one positional argument and **kwargs in the signature of dumps()).

…pression of JSON output on demand and extend test cases accordingly
@thehrh
thehrh merged commit 9d6b3b9 into master Sep 29, 2026
2 checks passed
@thehrh
thehrh deleted the enable_compactified_json branch September 29, 2026 10:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants