# Data Frames As a data scientist or machine learning engineer we deal with a lot of tabular data. Whether it is csv, parquet, or line delimited json, it is useful to store your training data in data frames that we can filter, aggregate, slice and dice. To follow along with the examples below feel free to grab to grab the example data from our public [CatDogBoundingBox](https://www.oxen.ai/ox/CatDogBoundingBox) repository. ```bash oxen clone http://hub.oxen.ai/ox/CatDogBoundingBox ``` ```bash cd CatDogBoundingBox ``` ## oxen df Oxen has a convenient `df` (short for "DataFrame") command to deal with tabular data. This example data has 10,000 rows and 6 columns of bounding boxes around cats or dogs. The shape hint at the top of the output can be useful for making sure you are transforming the data correctly. ```bash oxen df annotations/data.csv ``` ``` shape: (10000, 6) ┌─────────────────────────┬───────┬────────┬────────┬────────┬────────┐ │ file ┆ label ┆ min_x ┆ min_y ┆ width ┆ height │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ str ┆ f64 ┆ f64 ┆ f64 ┆ f64 │ ╞═════════════════════════╪═══════╪════════╪════════╪════════╪════════╡ │ images/000000128154.jpg ┆ cat ┆ 0.0 ┆ 19.27 ┆ 130.79 ┆ 129.58 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000544590.jpg ┆ cat ┆ 9.75 ┆ 13.49 ┆ 214.25 ┆ 188.35 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000000581.jpg ┆ dog ┆ 49.37 ┆ 67.79 ┆ 74.29 ┆ 116.08 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000236841.jpg ┆ cat ┆ 115.21 ┆ 96.65 ┆ 93.87 ┆ 42.29 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000257301.jpg ┆ dog ┆ 84.85 ┆ 161.09 ┆ 33.1 ┆ 51.26 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000130399.jpg ┆ dog ┆ 51.63 ┆ 157.14 ┆ 53.13 ┆ 29.75 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000215471.jpg ┆ cat ┆ 126.18 ┆ 71.95 ┆ 36.19 ┆ 47.81 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000251246.jpg ┆ cat ┆ 58.23 ┆ 13.27 ┆ 90.79 ┆ 97.32 │ └─────────────────────────┴───────┴────────┴────────┴────────┴────────┘ ``` Oxen uses a powerful [DataFrame library](https://pola-rs.github.io/polars-book/user-guide/introduction.html) under the hood, and uses the [Apache Arrow](https://arrow.apache.org/) data format to provide powerful cross application functionality. A lot of time and effort can be saved by transforming the data on the command line before writing a single line of application specific code or even opening a python repl or Juptyer notebook. # Useful Commands There are many ways you might want to view, transform, and filter your data on the command line before committing to the version of the dataset. To quickly see all the options on the `df` command you can run `oxen df --help`. ## Output Data Formats The `--output` option is handy for quickly transforming data files between data formats on disk. Some formats like parquet and arrow are more efficient for data different [tasks](https://towardsdatascience.com/apache-arrow-read-dataframe-with-zero-memory-69634092b1a), but are not human readable like tsv or csv. Data format is always a trade off you'll have to decide on for your application. Oxen currently supports these file extensions: `csv`, `tsv`, `parquet`, `arrow`, `json`, `jsonl`. ```bash oxen df annotations/data.csv -o annotations/data.parquet ``` ``` shape: (10000, 6) ┌─────────────────────────┬───────┬────────┬────────┬────────┬────────┐ │ file ┆ label ┆ min_x ┆ min_y ┆ width ┆ height │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ str ┆ f64 ┆ f64 ┆ f64 ┆ f64 │ ╞═════════════════════════╪═══════╪════════╪════════╪════════╪════════╡ │ images/000000128154.jpg ┆ cat ┆ 0.0 ┆ 19.27 ┆ 130.79 ┆ 129.58 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000544590.jpg ┆ cat ┆ 9.75 ┆ 13.49 ┆ 214.25 ┆ 188.35 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000000581.jpg ┆ dog ┆ 49.37 ┆ 67.79 ┆ 74.29 ┆ 116.08 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000236841.jpg ┆ cat ┆ 115.21 ┆ 96.65 ┆ 93.87 ┆ 42.29 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000257301.jpg ┆ dog ┆ 84.85 ┆ 161.09 ┆ 33.1 ┆ 51.26 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000130399.jpg ┆ dog ┆ 51.63 ┆ 157.14 ┆ 53.13 ┆ 29.75 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000215471.jpg ┆ cat ┆ 126.18 ┆ 71.95 ┆ 36.19 ┆ 47.81 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000251246.jpg ┆ cat ┆ 58.23 ┆ 13.27 ┆ 90.79 ┆ 97.32 │ └─────────────────────────┴───────┴────────┴────────┴────────┴────────┘ Writing "annotations/data.parquet" ``` ## View Schema Sometimes a DataFrame will have many columns and the default command collapses them so they are hard to see. You can use the `--schema` flag to just display the schema of this data frame. Note this is an exclusive flag. ```bash oxen df annotations/train.csv --schema ``` ``` +--------+-------+ | column | dtype | +================+ | file | str | |--------+-------| | label | str | |--------+-------| | min_x | f64 | |--------+-------| | min_y | f64 | |--------+-------| | width | f64 | |--------+-------| | height | f64 | +--------+-------+ ``` ## Slice Say you want to take a subset of the datafile and save it in another data file. You can do this with the `--slice` option. This can be handy when creating train, test, and validation sets. The two numbers represent the start and end indices you want to slice into. ```bash oxen df annotations/data.csv --slice '0..9000' -o annotations/train.parquet ``` ``` shape: (9000, 6) ┌─────────────────────────┬───────┬────────┬────────┬────────┬────────┐ │ file ┆ label ┆ min_x ┆ min_y ┆ width ┆ height │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ str ┆ f64 ┆ f64 ┆ f64 ┆ f64 │ ╞═════════════════════════╪═══════╪════════╪════════╪════════╪════════╡ │ images/000000128154.jpg ┆ cat ┆ 0.0 ┆ 19.27 ┆ 130.79 ┆ 129.58 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000544590.jpg ┆ cat ┆ 9.75 ┆ 13.49 ┆ 214.25 ┆ 188.35 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000000581.jpg ┆ dog ┆ 49.37 ┆ 67.79 ┆ 74.29 ┆ 116.08 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000236841.jpg ┆ cat ┆ 115.21 ┆ 96.65 ┆ 93.87 ┆ 42.29 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000431980.jpg ┆ dog ┆ 98.3 ┆ 110.46 ┆ 42.69 ┆ 26.64 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000071025.jpg ┆ cat ┆ 55.33 ┆ 105.45 ┆ 160.15 ┆ 73.57 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000518015.jpg ┆ cat ┆ 43.72 ┆ 4.34 ┆ 72.98 ┆ 129.1 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000171435.jpg ┆ dog ┆ 22.86 ┆ 100.03 ┆ 125.55 ┆ 41.61 │ └─────────────────────────┴───────┴────────┴────────┴────────┴────────┘ ``` ## Randomize Often you will want to randomize data before splitting into train and test sets, or even just to peek at different data values. ```bash $ oxen df annotations/data.csv --randomize shape: (10000, 6) ┌─────────────────────────┬───────┬────────┬────────┬────────┬────────┐ │ file ┆ label ┆ min_x ┆ min_y ┆ width ┆ height │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ str ┆ f64 ┆ f64 ┆ f64 ┆ f64 │ ╞═════════════════════════╪═══════╪════════╪════════╪════════╪════════╡ │ images/000000335955.jpg ┆ dog ┆ 28.98 ┆ 88.35 ┆ 39.22 ┆ 84.05 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000082475.jpg ┆ dog ┆ 0.6 ┆ 23.08 ┆ 200.92 ┆ 198.2 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000515777.jpg ┆ dog ┆ 109.83 ┆ 124.28 ┆ 58.89 ┆ 93.94 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000176089.jpg ┆ cat ┆ 106.62 ┆ 86.23 ┆ 56.53 ┆ 54.44 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000401308.jpg ┆ dog ┆ 21.12 ┆ 0.81 ┆ 202.42 ┆ 221.75 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000105030.jpg ┆ cat ┆ 11.62 ┆ 95.38 ┆ 60.21 ┆ 120.43 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000514890.jpg ┆ dog ┆ 36.76 ┆ 99.58 ┆ 12.27 ┆ 11.18 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000519218.jpg ┆ dog ┆ 71.24 ┆ 58.51 ┆ 8.57 ┆ 22.26 │ └─────────────────────────┴───────┴────────┴────────┴────────┴────────┘ ``` ## View Specific Columns Maybe you have many columns, and only need to work with a few. You can specify column names in a comma separated list with `--columns`. ```bash $ oxen df annotations/data.csv --columns 'file,label' shape: (10000, 2) ┌─────────────────────────┬───────┐ │ file ┆ label │ │ --- ┆ --- │ │ str ┆ str │ ╞═════════════════════════╪═══════╡ │ images/000000128154.jpg ┆ cat │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤ │ images/000000544590.jpg ┆ cat │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤ │ images/000000000581.jpg ┆ dog │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤ │ images/000000236841.jpg ┆ cat │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤ │ ... ┆ ... │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤ │ images/000000257301.jpg ┆ dog │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤ │ images/000000130399.jpg ┆ dog │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤ │ images/000000215471.jpg ┆ cat │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤ │ images/000000251246.jpg ┆ cat │ └─────────────────────────┴───────┘ ``` ## Filter Rows Oxen has some powerful filter commands built into the CLI. You can quickly filter data down based on a expression involving a column name, an operation, and a row value. Supported filter operations: ==, !=, >, <, <= , >= Supported logical operations: &&, || Supported row dtypes: str, i32, i64, f32, f64 ```bash $ oxen df annotations/data.csv --filter 'label == dog && height >= 200' shape: (5356, 6) ┌─────────────────────────┬───────┬────────┬────────┬───────┬────────┐ │ file ┆ label ┆ min_x ┆ min_y ┆ width ┆ height │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ str ┆ f64 ┆ f64 ┆ f64 ┆ f64 │ ╞═════════════════════════╪═══════╪════════╪════════╪═══════╪════════╡ │ images/000000000581.jpg ┆ dog ┆ 49.37 ┆ 67.79 ┆ 74.29 ┆ 216.08 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000001360.jpg ┆ dog ┆ 101.56 ┆ 178.2 ┆ 35.22 ┆ 238.34 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000362567.jpg ┆ dog ┆ 90.96 ┆ 36.65 ┆ 86.2 ┆ 285.08 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000201969.jpg ┆ dog ┆ 167.24 ┆ 73.99 ┆ 37.0 ┆ 264.94 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000237419.jpg ┆ dog ┆ 49.64 ┆ 104.53 ┆ 31.31 ┆ 248.88 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000314708.jpg ┆ dog ┆ 47.17 ┆ 138.18 ┆ 54.72 ┆ 359.55 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000257301.jpg ┆ dog ┆ 84.85 ┆ 161.09 ┆ 33.1 ┆ 251.26 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000130399.jpg ┆ dog ┆ 51.63 ┆ 157.14 ┆ 53.13 ┆ 229.75 │ └─────────────────────────┴───────┴────────┴────────┴───────┴────────┘ ``` ## Concatenate (vstack) Maybe you have filtered down data, and want to stack the data back into a single frame. The `--vstack` option takes a variable length list of files you would like to concatenate. ```bash $ oxen df annotations/data.csv --filter 'label=dog' -o /tmp/dogs.parquet $ oxen df annotations/data.csv --filter 'label=cat' -o /tmp/cats.parquet $ oxen df /tmp/cats.parquet --vstack /tmp/dogs.parquet -o annotations/data.parquet ``` ## Take Indices Sometimes you have a specific row or set of rows of data you would like to look at. This is where the `--take` option comes in handy. ```bash $ oxen df annotations/data.csv --take '1,13,42' shape: (3, 6) ┌─────────────────────────┬───────┬───────┬───────┬────────┬────────┐ │ file ┆ label ┆ min_x ┆ min_y ┆ width ┆ height │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ str ┆ f64 ┆ f64 ┆ f64 ┆ f64 │ ╞═════════════════════════╪═══════╪═══════╪═══════╪════════╪════════╡ │ images/000000544590.jpg ┆ cat ┆ 9.75 ┆ 13.49 ┆ 214.25 ┆ 188.35 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000279829.jpg ┆ cat ┆ 30.01 ┆ 13.58 ┆ 82.51 ┆ 176.39 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000209289.jpg ┆ dog ┆ 72.75 ┆ 42.06 ┆ 111.52 ┆ 153.09 │ └─────────────────────────┴───────┴───────┴───────┴────────┴────────┘ ``` ## Add Column Your data might not match the schema of a data frame you want to combine with, in this case you may need to add a column to match the schema. You can do this and project default values with `--add-col 'col:val:dtype'` ```bash $ oxen df annotations/data.csv --add-col 'is_cute:unknown:str' shape: (10000, 7) ┌─────────────────────────┬───────┬────────┬────────┬────────┬────────┬─────────┐ │ file ┆ label ┆ min_x ┆ min_y ┆ width ┆ height ┆ is_cute │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ str ┆ f64 ┆ f64 ┆ f64 ┆ f64 ┆ str │ ╞═════════════════════════╪═══════╪════════╪════════╪════════╪════════╪═════════╡ │ images/000000128154.jpg ┆ cat ┆ 0.0 ┆ 19.27 ┆ 130.79 ┆ 129.58 ┆ unknown │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┤ │ images/000000544590.jpg ┆ cat ┆ 9.75 ┆ 13.49 ┆ 214.25 ┆ 188.35 ┆ unknown │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┤ │ images/000000000581.jpg ┆ dog ┆ 49.37 ┆ 67.79 ┆ 74.29 ┆ 116.08 ┆ unknown │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┤ │ images/000000236841.jpg ┆ cat ┆ 115.21 ┆ 96.65 ┆ 93.87 ┆ 42.29 ┆ unknown │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┤ │ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┤ │ images/000000257301.jpg ┆ dog ┆ 84.85 ┆ 161.09 ┆ 33.1 ┆ 51.26 ┆ unknown │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┤ │ images/000000130399.jpg ┆ dog ┆ 51.63 ┆ 157.14 ┆ 53.13 ┆ 29.75 ┆ unknown │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┤ │ images/000000215471.jpg ┆ cat ┆ 126.18 ┆ 71.95 ┆ 36.19 ┆ 47.81 ┆ unknown │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┤ │ images/000000251246.jpg ┆ cat ┆ 58.23 ┆ 13.27 ┆ 90.79 ┆ 97.32 ┆ unknown │ └─────────────────────────┴───────┴────────┴────────┴────────┴────────┴─────────┘ ``` ## Add Row Sometimes it can be a pain to append data to a data file without writing code to do so. The `--add-row` option makes it as easy as a comma separated list and automatically parses the data to the correct dtypes. ```bash $ oxen df annotations/data.csv --add-row 'images/my_cat.jpg,cat,0,0,0,0' shape: (10001, 6) ┌─────────────────────────┬───────┬────────┬────────┬────────┬────────┐ │ file ┆ label ┆ min_x ┆ min_y ┆ width ┆ height │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ str ┆ f64 ┆ f64 ┆ f64 ┆ f64 │ ╞═════════════════════════╪═══════╪════════╪════════╪════════╪════════╡ │ images/000000128154.jpg ┆ cat ┆ 0.0 ┆ 19.27 ┆ 130.79 ┆ 129.58 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000544590.jpg ┆ cat ┆ 9.75 ┆ 13.49 ┆ 214.25 ┆ 188.35 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000000581.jpg ┆ dog ┆ 49.37 ┆ 67.79 ┆ 74.29 ┆ 116.08 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000236841.jpg ┆ cat ┆ 115.21 ┆ 96.65 ┆ 93.87 ┆ 42.29 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000130399.jpg ┆ dog ┆ 51.63 ┆ 157.14 ┆ 53.13 ┆ 29.75 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000215471.jpg ┆ cat ┆ 126.18 ┆ 71.95 ┆ 36.19 ┆ 47.81 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000251246.jpg ┆ cat ┆ 58.23 ┆ 13.27 ┆ 90.79 ┆ 97.32 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/my_cat.jpg ┆ cat ┆ 0.0 ┆ 0.0 ┆ 0.0 ┆ 0.0 │ └─────────────────────────┴───────┴────────┴────────┴────────┴────────┘ ``` ## Aggregate Oxen DataFrame aggregations can be helpful to quickly get statistics about your data. You can save these statistics to disk and commit them to track stats about your data over time. The format for an aggregation query is similar to a lambda function. The inputs to the function are the column name(s) you want to group by. The outputs are functions you want to run over the grouped results. ``` ('col_0') -> (min('col_1'), max('col_2')) ``` This simple example aggregation query would be if you wanted to find a distribution of labels in a dataset. For example in our cats vs dogs dataset you can group by the `'label'` column, and then run the `count()` function value over all the values in the `'file'` column. ``` $ oxen df annotations/train.csv -a "('label') -> (count('file'))" shape: (2, 2) ┌───────┬───────────────┐ │ label ┆ count('file') │ │ --- ┆ --- │ │ str ┆ u32 │ ╞═══════╪═══════════════╡ │ cat ┆ 4140 │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ dog ┆ 4860 │ └───────┴───────────────┘ ``` You can specify multiple functions in the output. For example if you wanted the unique file count as well as the raw count you can add the `n_unique()` function. ``` $ oxen df annotations/train.csv -a "('label') -> (count('file'), n_unique('file'))" shape: (2, 3) ┌───────┬───────────────┬──────────────────┐ │ label ┆ count('file') ┆ n_unique('file') │ │ --- ┆ --- ┆ --- │ │ str ┆ u32 ┆ u32 │ ╞═══════╪═══════════════╪══════════════════╡ │ dog ┆ 4860 ┆ 3798 │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ cat ┆ 4140 ┆ 3525 │ └───────┴───────────────┴──────────────────┘ ``` Here is a list of supported output aggregation functions: * `list` aggregate column values into a list * `count` count the aggregated values * `n_unique` unique count of the aggregated values * `min` minimum value of the group * `max` maximum value of the group * `arg_min` index of minimum value in the group * `arg_max` index of maximum value in the group * `mean` mean value of the group * `median` median value of the group * `std` standard deviation of the group * `var` variance of the group * `first` first value of the group * `last` last value in the group * `head` first 5 values of group * `tail` last 5 values of the group ## Unique Oxen can efficiently compute all the unique values given a column name, or comma separated list of column names. ``` $ oxen df annotations/train.csv --unique "file" $ oxen df annotations/train.csv -u "file,label" ``` ## Sort Sorting can be achieved with the `sort` flag. For example you may want to find the largest bounding boxes by sorting on the height column. ``` oxen df annotations/train.csv --sort "height" shape: (9000, 6) ┌─────────────────────────┬───────┬────────┬────────┬────────┬────────┐ │ file ┆ label ┆ min_x ┆ min_y ┆ width ┆ height │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ str ┆ f64 ┆ f64 ┆ f64 ┆ f64 │ ╞═════════════════════════╪═══════╪════════╪════════╪════════╪════════╡ │ images/000000580919.jpg ┆ dog ┆ 61.28 ┆ 88.31 ┆ 2.71 ┆ 1.83 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000577310.jpg ┆ dog ┆ 132.25 ┆ 193.86 ┆ 3.28 ┆ 1.95 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000393384.jpg ┆ dog ┆ 138.85 ┆ 89.89 ┆ 1.25 ┆ 2.11 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000477398.jpg ┆ dog ┆ 185.11 ┆ 195.93 ┆ 2.51 ┆ 2.6 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000069205.jpg ┆ dog ┆ 0.0 ┆ 0.0 ┆ 224.0 ┆ 224.0 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000554737.jpg ┆ cat ┆ 0.0 ┆ 0.0 ┆ 224.0 ┆ 224.0 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000213819.jpg ┆ cat ┆ 8.32 ┆ 0.0 ┆ 207.77 ┆ 224.0 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ images/000000397212.jpg ┆ cat ┆ 0.36 ┆ 0.0 ┆ 115.5 ┆ 224.0 │ └─────────────────────────┴───────┴────────┴────────┴────────┴────────┘ ``` Sort is also useful in the context of aggregations. When aggregating up statistics they do not come back in a guaranteed order. If you want to see which files have the most labels, you can group the output if an aggregation `count()` function. ``` $ oxen df annotations/train.csv -a "('file') -> (list('label'), count('label'))" --sort "count('label')" shape: (7128, 3) ┌─────────────────────────┬───────────────────────────┬────────────────┐ │ file ┆ list('label') ┆ count('label') │ │ --- ┆ --- ┆ --- │ │ str ┆ list[str] ┆ u32 │ ╞═════════════════════════╪═══════════════════════════╪════════════════╡ │ images/000000060202.jpg ┆ ["dog"] ┆ 1 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ images/000000518156.jpg ┆ ["cat"] ┆ 1 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ images/000000347879.jpg ┆ ["cat"] ┆ 1 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ images/000000290136.jpg ┆ ["dog"] ┆ 1 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ ... ┆ ... ┆ ... │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ images/000000398285.jpg ┆ ["dog", "dog", ... "dog"] ┆ 14 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ images/000000244933.jpg ┆ ["cat", "cat", ... "cat"] ┆ 17 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ images/000000016950.jpg ┆ ["dog", "dog", ... "dog"] ┆ 19 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ images/000000315555.jpg ┆ ["dog", "dog", ... "dog"] ┆ 19 │ └─────────────────────────┴───────────────────────────┴────────────────┘ ``` ## Reverse You can also reverse the order of a data table. By default `--sort` sorts in ascending order, but can be reversed with the `--reverse` flag. ``` oxen df annotations/train.csv -a "('file') -> (count('label'))" --sort "count('label')" --reverse shape: (7128, 2) ┌─────────────────────────┬────────────────┐ │ file ┆ count('label') │ │ --- ┆ --- │ │ str ┆ u32 │ ╞═════════════════════════╪════════════════╡ │ images/000000315555.jpg ┆ 19 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ images/000000016950.jpg ┆ 19 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ images/000000244933.jpg ┆ 17 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ images/000000092869.jpg ┆ 14 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ ... ┆ ... │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ images/000000038827.jpg ┆ 1 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ images/000000470862.jpg ┆ 1 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ images/000000292101.jpg ┆ 1 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ images/000000345432.jpg ┆ 1 │ └─────────────────────────┴────────────────┘ ```