Page MenuHome

BlenLib: Add JSON Serialization/Deserialization Abstraction Layer.
ClosedPublic

Authored by Jeroen Bakker (jbakker) on Sep 17 2021, 4:47 PM.

Details

Summary

Adds an abstraction layer to switch between serialization formats.
Currently only supports JSON. The abstraction layer supports
String, Int, Array, Null, Boolean, Float and Object. This
feature is only CPP complaint.

To write from a stream, the structure can be built by creating a value
(any subclass of blender::io::serialize::Value can do, and pass it to
the serialize method of a blender::io::serialize::Formatter. The
formatter is abstract and there is one implementation for JSON
(JsonFormatter).

To read from a stream use the deserialize method of the formatter.

D12693: T91406: Asset Library Indexing uses this abstraction layer to read/write asset indexes.

Diff Detail

Repository
rB Blender

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
source/blender/blenlib/BLI_serialize.hh
86

Same code style fix needed for these other private variables 😛

  • Codestyle: Use trailing _ for private members.
Jeroen Bakker (jbakker) marked 2 inline comments as done.Oct 4 2021, 4:48 PM
  • Use std::unique_ptr when deserializing JSON.
Sybren A. Stüvel (sybren) requested changes to this revision.Oct 14 2021, 10:59 AM

This feature is only CPP complaint.

I think this should be "compliant"?

The code has hardly any documentation, which is concerning. Especially when things have such generic names like Value and eValueType, documentation is vital.

source/blender/blenlib/BLI_serialize.hh
23–24

Linguistically this doesn't make much sense. This layer doesn't sit between serialization formats. I'd say "abstraction layer for serialization formats".

24

supports

33

oeps

51

Using uint64_t is error-prone for a type named IntValue with eValueType::Int.

https://stackoverflow.com/a/13502497 suggests that JavaScript numbers roughly have a range of ±2**53, so int64_t will be sufficient.

52

Any reason to use float instead of double for JSON? This should be documented.

57

Why are std::shared_ptrs used here?

59

Something as abstractly named as "Value" really needs documentation as to what it is and what it's for.

65

Either add explicit or document why this constructor should not be explicit.

76–81

I feel that these should get some documentation as to what they return, and in which scope they're valid. Is the code expected to check the type, before calling any of these? Who owns the returned pointer?

76–81

These need documentation. What they do, but also what happens when the Value is not of the required type. Does that raise an exception? Return nulltpr? Produce invalid results?

84

Documentation, certainly about the template variables. It also needs to explain what "primitive" means; after all, eValueType is not named ePrimitiveValueType, so there must be some value types that you cannot use here.

150

be

150

key-value

156

Why is this a std::shared_ptr? Who owns the values? Who is expected to be sharing the values?

170

Document

173

Why does the value need to be mutable in order to serialize it? What modifications will this function do? Either make it const, or use a pointer so that at the call itself it's clear that the function can modify the value.

177

Document

179

Unless the full 0-255 range is needed, use int8_t instead. It'll give much more sensible erroneous results when there is some off-by-one error somewhere; invalid negative values can be tested for, but invalid wrapped-around-so-still-valid values cannot.

source/blender/blenlib/intern/serialize.cc
13

This assumes that any Value with type_ == eValueType::String is a StringValue. This tight coupling is not documented at all!

70–88

These should be extracted to their own functions.

71

Why is this necessary at all?

72

What is this notation?

112–118

Such blocks should be their own function

157

Then why are they (and this code) even here?

source/blender/blenlib/tests/BLI_serialize_test.cc
26

Also test with bigger & negative integers!

35

The type is called FloatValue but you test with a double literal. Be sure to actually test the precision by which these are saved, if you want to actually test doubles.

This revision now requires changes to proceed.Oct 14 2021, 10:59 AM
Jeroen Bakker (jbakker) marked 18 inline comments as done.
  • Code review items .

    Mostly documentation and splitting functions into smaller ones.
source/blender/blenlib/intern/serialize.cc
10–12

Didn't the documentation for this function say that there will be an assert for the type-match as well?

source/blender/blenlib/intern/serialize.cc
10–12

Will adapt the documentation. seems like I had a different implementation in my head when writing the docs :-)

Jeroen Bakker (jbakker) marked 2 inline comments as done.
  • Fix comments of the as_ *_value methods to the actual implementation.
  • Updated comment of the eValueType.
Jeroen Bakker (jbakker) marked 7 inline comments as done.Oct 15 2021, 4:43 PM
  • Added more test to the int_to_json test case.

Rebased with latest master.
#Updating D12544 : BlenLib : Add JSON Serialization / \

Deserialization Abstraction Layer.
Sybren A. Stüvel (sybren) requested changes to this revision.Oct 19 2021, 11:25 AM

Just some minor notes.

source/blender/blenlib/BLI_serialize.hh
39

Shouldn't this be \see?

57

Document here, as people start reading from the top.

98

Remove "Helper function to"

135

Remove "Helper class for"

140

Instead of "internal" I'd call it "wrapped"; "internal" to me sounds too much like it's an implementation detail, but it's also part of the public interface of the type.

178

This function should be named value() (so it's consistent with IntValue::value()) and not string_value(). Otherwise it's unclear when you see an invocation like this:

j = value.as_string_value()->string_value();

as_string_value() already returns a "string value", so it's not immediately clear what different kind of "string value" would be returned by string_value().

source/blender/blenlib/intern/serialize.cc
60

will → would

74

will → would

This revision now requires changes to proceed.Oct 19 2021, 11:25 AM
Jeroen Bakker (jbakker) updated this revision to Diff 43552.EditedOct 19 2021, 11:36 AM
Jeroen Bakker (jbakker) marked 8 inline comments as done.

Code review.

  • Fixed comments.
  • renamed string_value to value in line with other 'simple' types.
Julian Eisel (Severin) requested changes to this revision.Oct 19 2021, 1:33 PM
Julian Eisel (Severin) added inline comments.
source/blender/blenlib/BLI_serialize.hh
23–25

I would add a bit more explanation on the very basics of the design. Mention how this can be used by other formats (by implementing a custom formatter using Formatter as base class), how data is modeled (what kind of primitives are supported and how ObjectValue can be used), is the order of items preserved when writing back to disk, how is versioning handled, etc.

If it becomes more than a few lines then it can just become a link to the Wiki.

150

Better default-initialize primitive values:

T inner_value_{};

Otherwise on a default construction for example, this would be uninitialized.

153

Think this should be explicit.

191–193

Very minor suggestion: I prefer using the single line syntax for such short doxygen comments:

/** Short description, blah blah. */
250

This seems a bit odd to me. I would expect the object to store items in a map itself, and a simple lookup function on the object directly. What's the reason for not doing this, to preserve the order of items? Is it really that important to preserve the order? If so, we could also store an index somehow and sort before writing. E.g. the object's map could be stored as Map<std::string, std::pair<std::unique_ptr<Value>, int>> -- maybe that makes internals more complex than they need to be, but at least the public API makes more sense to me.

Not against the current solution, just bringing up an alternative for consideration.

This revision now requires changes to proceed.Oct 19 2021, 1:33 PM

Fix compilation error after rebasing.Resolved items from code review.-

Spelling in comments.- renamed indexes to indices.-
Improved various comments.- Code style cleanups.

#Updating D12544 : BlenLib : Add JSON Serialization / \

Deserialization Abstraction Layer.
Jeroen Bakker (jbakker) marked 4 inline comments as done.

Resolved code review items.- Better usage description.

#Updating D12544 : BlenLib : Add JSON Serialization / \

Deserialization Abstraction Layer.

Added better description in the top of the header.

#Updating D12544 : BlenLib : Add JSON Serialization / \

Deserialization Abstraction Layer.
source/blender/blenlib/BLI_serialize.hh
250

Yes I considered it. Goal was a minialistic abstration layer.
I agree that the public API is much cleaner. Main reason for the current API is to control the performance hit when it is actually being used and reduction of overhead.

Ordering of data is a more complex topic. There is no default ordering. JSON for example orders the keys in alphabetic order. Other formatters can decide a different order. For our situation alphabetic would increase the index filesizes to create a better human readable index. I have chosen to don't sort at all and let the order of the keys be based by the caller.

source/blender/blenlib/BLI_serialize.hh
250

JSON specs are also not 100% clear about duplicated keys inside a JSON file.
It should be, but you are allowed to parse or add them.

This revision is now accepted and ready to land.Oct 20 2021, 5:19 PM