Problem: serialization of R list objects to a human readable/editable file

I often use R list objects for storing configuration information within shiny apps. These are used to define multiple configuration sets and are usually saved to file. I then want to be able to edit the objects on the filesystem with a text editor.

What’s a good file format for saving R list objects to file such that they’re easily editable?

Criteria:

• has an R package I can use
• easy to read and edit the saved data

Not Criteria:

• interoperability with other software
• transmission over a network
• saving to a database

Possibilities:

• RDS - even though there’s an ascii option to saveRDS it doesn’t save anything that’s human parseable
• CSV - Never going to be able to save nested list data (with NULL values!) to a CSV without a whole lot of pre- and post- data transformation
• YAML - worth a look.
• JSON - worth a look.
• XML - hand editing XML nodes in a text file seems like a good way to shoot yourself in the foot.

Actual contenders:

• JSON - Packages: RJSONIO, jsonlite
• YAML - Package: YAML

Example R list object

colour_scheme <- list(
  plain = list(
    top         = 'black',
    bottom      = 'black',
    size        = 1,
    angle       = 45.0,
    sub_options = list(),
    border      = NULL,
    tune        = NA
  ),
  fancy = list(
    top         = 'red',
    bottom      = 'blue',
    size        = 5,
    angle       = 17.5,
    sub_options = list(pieces_of_flair = c(0, 37)),
    border      = 'cursive',
    tune        = 'adagio'
  )
)

jsonlite: JSON encoding the R list

colour_scheme_json <- jsonlite::toJSON(colour_scheme, pretty=TRUE) 
colour_scheme_json

{
  "plain": {
    "top": ["black"],
    "bottom": ["black"],
    "size": [1],
    "angle": [45],
    "sub_options": [],
    "border": {},
    "tune": [null]
  },
  "fancy": {
    "top": ["red"],
    "bottom": ["blue"],
    "size": [5],
    "angle": [17.5],
    "sub_options": {
      "pieces_of_flair": [0, 37]
    },
    "border": ["cursive"],
    "tune": ["adagio"]
  }
} 

jsonlite: Restoring the R list from JSON

When the R object is recreated from the JSON, the result is not identical to the original data.

This is because the original border = NULL in the data was represented in JSON as an empty list. Upon restoration, the R object contains border = list() instead of the required NULL

restored_json <- jsonlite::fromJSON(colour_scheme_json)
identical(colour_scheme, restored_json)

[1] FALSE

RJSONIO: JSON encoding the R list

colour_scheme_json2 <- RJSONIO::toJSON(colour_scheme, pretty=TRUE) 
cat(colour_scheme_json2)

{
    "plain" : {
        "top" : "black",
        "bottom" : "black",
        "size" : 1,
        "angle" : 45,
        "sub_options" : [],
        "border" : null,
        "tune" : null
    },
    "fancy" : {
        "top" : "red",
        "bottom" : "blue",
        "size" : 5,
        "angle" : 17.5,
        "sub_options" : {
            "pieces_of_flair" : [
                0,
                37
            ]
        },
        "border" : "cursive",
        "tune" : "adagio"
    }
}

RJSONIO: Restoring the R list from JSON

When the R object is recreated from the JSON, the result is not identical to the original data.

This is because the original tune = NA in the data was represented in JSON as a null. Upon restoration, the R object contains tune=NULL instead of the required NA

RJSONIO allows you to specify how NAs are encoded, but there does not seem to be a direct representation of NA in the JSON format.

restored_json2 <- RJSONIO::fromJSON(colour_scheme_json2)
identical(colour_scheme, restored_json2)

[1] FALSE

YAML encoding the R list

colour_scheme_yaml <- yaml::as.yaml(colour_scheme) 
cat(colour_scheme_yaml)

plain:
  top: black
  bottom: black
  size: 1.0
  angle: 45.0
  sub_options: []
  border: ~
  tune: .na
fancy:
  top: red
  bottom: blue
  size: 5.0
  angle: 17.5
  sub_options:
    pieces_of_flair:
    - 0.0e+00
    - 3.7e+01
  border: cursive
  tune: adagio

Restoring the R list from YAML

When the R object is recreated from the YAML, the result is identical to the original data.

restored_yaml <- yaml::yaml.load(colour_scheme_yaml)
identical(colour_scheme, restored_yaml)

[1] TRUE

Named Vectors and Named lists

Both jsonlite and yaml convert a named vector into a plain array i.e. the names are stripped.

RJSONIO saves a named vector as a map (meaning that names are kept) but as a side effect, restoring a map with only a single primitive datatype produces a vector and not a list.

named_vec   <-    c(a=1, b=2, c=3)
named_list  <- list(a=1, b=2, c=3)
named_list2 <- list(a=1, b='cat')

yaml::as.yaml(named_vec)     %>% cat()

- 1.0
- 2.0
- 3.0

jsonlite::toJSON(named_vec)  %>% cat()

[1,2,3]

RJSONIO::toJSON(named_vec)   %>% cat()

{
 "a":      1,
"b":      2,
"c":      3 
}

RJSONIO::toJSON(named_list)  %>% cat()

{
 "a":      1,
"b":      2,
"c":      3 
}

RJSONIO::toJSON(named_vec)   %>% RJSONIO::fromJSON() %>% class()

[1] "numeric"

RJSONIO::toJSON(named_list)  %>% RJSONIO::fromJSON() %>% class()

[1] "numeric"

RJSONIO::toJSON(named_list2) %>% RJSONIO::fromJSON() %>% class()

[1] "list"

Comparison

A table comparing the representations in YAML and JSON is shown below.

• The YAML representation when restored to an R object is identical to the original R list.
  The JSON restoration is not idential to the original data.
• The representations of NULL and NA values are “not like R” in both formats
• JSON’s quoting of all names and strings seems needlessly labour-intensive
• YAML’s whitespace usage for indentation and structure will be familiar to Python users
• YAML’s pickiness of “No TABs only spaces!” could be an issue. Need to ensure text editor always replaces TABs with spaces.

Verdict

YAML.

Reasons:

• restored object is identical to original list
• more lightweight - no quoting of strings or names. Single values are not represented as short arrays.
• NULL and NA representation are a teeny tiny bit closer to how I’d think about them in R
• Familiarity with Python means I’m OK with whitespace being signficant.

Updates

• 2018-02-03 10:00 - heyaudy gave some helpful feedback that the actual JSON output is dependent upon which package I use, and that RJSONIO handles single values and NULLs better than jsonlite