What makes good sample data?

[saulpw]

The current sample_data directory is a total pile. It's a collection of small datasets in various formats, each of which was whatever I found laying around when I was developing the particular loader, under the notion that any test data is better than no data. They've been working okay I guess, but lately I've been thinking about reexamining our whole sample data strategy, which would likely lead to an overhaul of sample_data.

So the aim of this discussion is to explore desirable characteristics of test datasets, and strategies for how to accumulate/generate/curate them. In particular I would love to hear your direct experiences with sample data, positive and negative (VisiData's or otherwise).

Some opening questions:

• Should sample data be found in the wild, hand-crafted, or generated by checked-in script?
• Where would you look for small sample datasets relevant to your field?
• Have you noticed that the hand-crafted benchmark.csv has little data stories and jokes in it?
• Do you know of other research or open-source work for the curation of test/sample datasets?

[anjakefala]

Some thoughts I had on traits of a useful sample data:

1. Representative of the standard characteristics found within a file format. For example, an hdf5 sample data that did not have both Groups and Datasets would be insufficient. It is useful to have sample jsons with [] and { [ ] } (among other) characteristics. If a dataset is typed, there should be some representatives of the core types.
2. An understanding of "legal" features of a dataset which frequently can break parsers. This is stuff like comments at the beginning of the data, missing values, in-line newlines, etc. errors.csv is an example of a dataset which hopes to capture the quirkiness of csvs. These datasets can be more representative of what data is like in the wild, VisiData is not only going to encounter picture perfect datasets. However, there is also a case for not spending time trying to support a quirk that only exists in 0.0000001% of datasets.

[geekscrapy]

This kind of the reason I didn't tread into suggesting a JSON file 😏 not sure what was required.

Anyhow, I think it'd be good to have crafted files with each data type in them (int, str, date etc. bool?). Json for example would be a good example here where it can have ints, strs, bools and null, so the dataset should include rows which have them in.

[geekscrapy]

Q: Is visidata used by people to clean data? I know I use it to flatten some json files.
The reason for the question is, how much responsibility should visidata take on ingesting duff data?

[saulpw]

Yes, cleaning data is a primary use case for vd. I think it should make a reasonable best-effort to degrade gracefully:

• load whatever data it can
• catch partial errors, exposing them to the user
• continue loading past any errors

This way, even if vd doesn't work for 90% of your data, at least you can see the 10% that it can use.

[geekscrapy]

👍
Just to maybe consider in addition to your points above, one thing that I have come across recently is:
Tests for workability of any data that makes it into the sheet
e.g. all columns should be sortable etc. (like the recent len fix, although this wasn't to do with loading).
I know this is pretty much great (but has some side cases that fall through the cracks occasionally), but I'd like to highlight it as a core point to consider and test with the sample data

[frosencrantz]

Sorry I'm getting OT here. I wonder if people use vd for cleaning data.

I've tried to create simple databases of data or make small changes to existing data, and many times I end up upset. Mostly due to my own fault of typing the wrong key and losing my work, but also vd could have been more helpful. I only learned about guard-sheet as part of responding here, which would have been very helpful.

If I had cleaning needs vs exploring/reshaping I would more seriously think about trying https://github.com/OpenRefine/OpenRefine. I don't really need to clean data in that way. OR seems to have many tools with the intention of cleaning data. It seems like it would be useful to add some of these features to VisiData or as a cleaning plugin. OR seems to have a more robust history mechanism, with the history automatically saved to a file.

Partly responding here to see if #703 could be revived as a discussion thread. I think that information would also be useful for the main question of this discussion. I don't feel like I know how people regularly share their data with vd. It would be nice to get input from more folks, and it was great reading about how both Saul and AJ share data outside of vd. It looks like the wonderful #595 was converted to a discussion.