๐ ๐พ Oxen#
Oxen makes versioning your datasets as easy as versioning your code.
# The first step for creating any dataset should be `oxen init`
oxen init
oxen add images/
oxen commit -m "Adding my data"
oxen config --set-remote origin https://hub.oxen.ai/ox/CatDogBoundingBox
oxen push origin main
Clone your data faster than ever before.
# The first step in collaborating on a dataset should be `oxen clone`
oxen clone https://hub.oxen.ai/ox/CatDogBoundingBox
โ Features#
Oxen was optimized to be fast on structured and unstructured data types. Unlike traditional version control systems that are optimized for text files and code, Oxen was built from the ground up to be fast on images, video, audio, text, and more.
๐ฅ Fast (10-100x faster than existing tools)
๐ง Easy to learn (same commands as git)
๐๏ธ Index lots of files (millions of images? no problem)
๐ฅ Handles large files (images, videos, audio, text, parquet, arrow, json, models, etc)
๐ Native DataFrame processing (oxen df command for data exploration)
๐ Tracks changes over time (never worry about losing the state of your data)
๐ค Collaborate with your team (sync to an oxen-server)
๐ Better data visualization on OxenHub
Why the name Oxen?#
โOxenโ ๐ comes from the fact that the tooling will plow, maintain, and version your data like a good farmer tends to their fields ๐พ. Let Oxen take care of the grunt work of your infrastructure so you can focus on the higher-level ML problems that matter to your product.
Overview#
No need to learn a new paradigm.
The Oxen Command Line Interface (CLI) mirrors git in many ways, so if you are comfortable versioning code with git, it will be straightforward to version your datasets with Oxen.
Watch as we commit hundreds of thousands of images to an Oxen repository in a matter of seconds ๐
๐ CLI Installation#
For Mac Users
brew tap Oxen-AI/oxen
brew install oxen
For other platforms follow the installation instructions.
๐ Python Installation#
$ pip install oxenai
All the same APIs that are exposed through the command line are also available in python.
import oxen
repo = oxen.LocalRepo("path/to/repo")
repo.clone("https://hub.oxen.ai/ox/CatDogBBox")
repo.add("new-cat.png")
repo.commit("Adding a new cat")
repo.push()
๐งโ๐ป OxenHub#
The shift to Software 2.0 is happening where we are replacing lines with machine learning models and large datasets. Software is already complex, without the complexity of machine learning in the mix. We need better tooling to keep track of changes as data and models evolve over time.
With the Hub, enable your team to discover, view, collaborate, and manage your datasets.
Visit https://www.oxen.ai/register to register for early access to the Hub where you can host your datasets.
Basic Commands#
Here is a quick overview of common Oxen commands. If you are familiar with git, this should be an easy learning curve.
Setup User#
For your commit history, you will have to set up your local Oxen user name and email. This is what will show up in oxen log
or in the OxenHub dashboard for who changed what.
oxen config --name "YOUR_NAME" --email "YOUR_EMAIL"
Clone a Remote Repository#
There are a few ways that you can clone an Oxen repository, depending on the level of data transfer you want to incur. The default oxen clone
with no flags will download the latest commit from the main
branch.
oxen clone https://hub.oxen.ai/ox/CatDogBBox
To fetch the latest commit from a specific branch you can use the -b
flag.
oxen clone https://hub.oxen.ai/ox/CatDogBBox -b my-pets
Downloading all the data may still be a more expensive operation than you need. You can download the minimal metadata to still interact with the remote by using the --shallow
flag.
oxen clone https://hub.oxen.ai/ox/CatDogBBox --shallow -b my-pets
This is especially handy for appending data via the remote workspace. When downloading by using the --shallow
flag you will notice no data files in your working directory. You can still see the data on the branch on the remote with the oxen remote
subcommands.
# View the remote files
oxen remote ls
You can also download a subset by using oxen remote download
to download subsets of directories or files. This is useful if you only need the testing data and not the full training data files and directories.
oxen remote download test.csv
Lastly, if you want to clone the entire commit history locally, you can use the --all
flag. This is handy if you want to pull a full history and push to a new remote, or have a workflow where you need to quickly swap between commits locally. Often for running experiments, training, or testing, all you need is a subset of the data.
oxen clone https://hub.oxen.ai/ox/CatDogBBox --all
Initialize Local Repository#
If you do not have a remote dataset, you can initialize one locally.
Similar to git: create a new directory, navigate into it, and perform
oxen init
Stage Data#
You can stage changes that you are interested in committing with the oxen add
command and giving a full file path or directory.
oxen add images/
View Status#
To see what data is tracked, staged, or not yet added to the repository you can use the status
command.
Note: since we are dealing with large datasets with many files, status
rolls up the changes and summarizes them for you.
oxen status
On branch main -> e76dd52a4fc13a6f
Directories to be committed
added: images with added 8108 files
Files to be committed:
new file: images/000000000042.jpg
new file: images/000000000074.jpg
new file: images/000000000109.jpg
new file: images/000000000307.jpg
new file: images/000000000309.jpg
new file: images/000000000394.jpg
new file: images/000000000400.jpg
new file: images/000000000443.jpg
new file: images/000000000490.jpg
new file: images/000000000575.jpg
... and 8098 others
Untracked Directories
(use "oxen add <dir>..." to update what will be committed)
annotations/ (3 items)
You can always paginate through the changes with the -s
(skip) and -l
(limit) params on the status command. Run oxen status --help
for more info.
Commit Changes#
To commit the changes that are staged with a message you can use
oxen commit -m "Some informative commit message"
Log#
You can see the history of changes on your current branch by running:
oxen log
commit 6b958e268656b0c5
Author: Ox
Date: Fri, 21 Oct 2022 16:08:39 -0700
adding 10,000 training images
commit e76dd52a4fc13a6f
Author: Ox
Date: Fri, 21 Oct 2022 16:05:22 -0700
Initialized Repo ๐
Reverting To Commit#
If ever you want to change your working directory to a point in your commit history, you can simply supply the commit id from your history to the checkout
command.
oxen checkout COMMIT_ID
Restore Working Directory#
The restore
command comes in handy if you made some changes locally and you want to revert the changes. This can be used for example if you accidentally delete or modify or stage a file that you did not intend to.
oxen restore path/to/file.txt
Restore defaults to restoring the files to the current HEAD. For more detailed options, as well as how to unstage files refer to the restore documentation.
Removing Data#
To stage a file to be removed from the next commit, use the oxen rm
command. Removing data from a commit can be useful if you find errors or simply want to create a smaller subset of data on a separate branch for debugging or testing.
oxen rm path/to/file.txt
Note: the file must be committed in the history for this to work. If you want to remove a file that has not been committed yet, simple use your /bin/rm command.
To recursively remove a directory use the -r
flag.
oxen rm -r path/to/dir
If you accidentally staged a file that you do not want to commit, you can also use oxen rm
with the --staged
flag to unstage the file or directory.
oxen rm --staged -r path/to/dir
Once data has been committed, a version of it always lives in the .oxen/versions directory. As of right now there is no way to completely remove it from the repository history, this functionality is in our backlog for sensitive data that was accidentally committed.
Advanced Features#
Oxen has many more advanced features such as computing diffs between tabular data as well as convenient DataFrame manipulation through the oxen df command.
Feel free to skip down to the more advanced features.
Diving Deeper#
Data Frames#
Oxen is smart about what file types you are adding. For example, if you track a tabular data file (with an extension .csv
, .tsv
, .parquet
, .arrow
, .jsonl
, or .ndjson
) Oxen will index and keep track of each row of data.
oxen add annotations/train.csv
oxen commit -m "adding rows and rows of data"
Under the hood, Oxen will detect the data schema and hash every row of content. This allows us to build a content addressable DataFrame to track the changes to the rows and columns over time. To learn more about the power of indexing DataFrames check out the data point level version control documentation.
Oxen also has some handy command line tooling for Exploratory Data Analysis with DataFrames. The oxen df
command lets you easily view, modify, slice, and modify the data.
oxen df annotations/train.csv
shape: (10000, 6)
โโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโฌโโโโโโโโโฌโโโโโโโโโฌโโโโโโโโโฌโโโโโโโโโ
โ file โ label โ min_x โ min_y โ width โ height โ
โ --- โ --- โ --- โ --- โ --- โ --- โ
โ str โ str โ f64 โ f64 โ f64 โ f64 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโชโโโโโโโโชโโโโโโโโโชโโโโโโโโโชโโโโโโโโโชโโโโโโโโโก
โ images/000000128154.jpg โ cat โ 0.0 โ 19.27 โ 130.79 โ 129.58 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโผโโโโโโโโโผโโโโโโโโโผโโโโโโโโโผโโโโโโโโโค
โ images/000000544590.jpg โ cat โ 9.75 โ 13.49 โ 214.25 โ 188.35 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโผโโโโโโโโโผโโโโโโโโโผโโโโโโโโโผโโโโโโโโโค
โ images/000000000581.jpg โ dog โ 49.37 โ 67.79 โ 74.29 โ 116.08 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโผโโโโโโโโโผโโโโโโโโโผโโโโโโโโโผโโโโโโโโโค
โ images/000000236841.jpg โ cat โ 115.21 โ 96.65 โ 93.87 โ 42.29 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโผโโโโโโโโโผโโโโโโโโโผโโโโโโโโโผโโโโโโโโโค
โ ... โ ... โ ... โ ... โ ... โ ... โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโผโโโโโโโโโผโโโโโโโโโผโโโโโโโโโผโโโโโโโโโค
โ images/000000257301.jpg โ dog โ 84.85 โ 161.09 โ 33.1 โ 51.26 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโผโโโโโโโโโผโโโโโโโโโผโโโโโโโโโผโโโโโโโโโค
โ images/000000130399.jpg โ dog โ 51.63 โ 157.14 โ 53.13 โ 29.75 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโผโโโโโโโโโผโโโโโโโโโผโโโโโโโโโผโโโโโโโโโค
โ images/000000215471.jpg โ cat โ 126.18 โ 71.95 โ 36.19 โ 47.81 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโผโโโโโโโโโผโโโโโโโโโผโโโโโโโโโผโโโโโโโโโค
โ images/000000251246.jpg โ cat โ 58.23 โ 13.27 โ 90.79 โ 97.32 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโดโโโโโโโโโดโโโโโโโโโดโโโโโโโโโดโโโโโโโโโ
To learn more about what you can do with tabular data in Oxen you can reference the documentation here
Integrating Labeling Tools#
For most supervised learning projects you will have some sort of annotation or labeling workflow. There are some popular open-source tools such as Label Studio for labeling data that can integrate with an Oxen workflow.
For an example of integrating Oxen into your Label Studio workflow, check out our Oxen Annotation Documentation.
Diff#
If you want to see the differences between your file and the conflicting file, you can use the oxen diff
command.
Oxen knows how to compare text files as well as tabular data between commits. Currently, you must specify the specific path to the file you want to compare the changes.
If the file is tabular data oxen diff
will show you the rows that were added or removed.
oxen df annotations/data.csv --add-row 'images/my_cat.jpg,cat,0,0,0,0' -o annotations/data.csv
oxen diff annotations/data.csv
Added Rows
โญโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโฌโโโโโโโโฌโโโโโโโโฌโโโโโโโโฌโโโโโโโโโฎ
โ file โ label โ min_x โ min_y โ width โ height โ
โโโโโโโโโโโโโโโโโโโโโผโโโโโโโโผโโโโโโโโผโโโโโโโโผโโโโโโโโผโโโโโโโโโค
โ images/my_cat.jpg โ cat โ 0 โ 0 โ 0 โ 0 โ
โฐโโโโโโโโโโโโโโโโโโโโดโโโโโโโโดโโโโโโโโดโโโโโโโโดโโโโโโโโดโโโโโโโโโฏ
1 Rows x 6 Columns
If the tabular data schema has changed oxen diff
will flag and show you the columns that were added.
oxen df annotations/data.csv --add-col 'is_fluffy:unknown:str' -o annotations/data.csv
oxen diff annotations/data.csv
Added Cols
shape: (10001, 1)
โโโโโโโโโโโโโ
โ is_fluffy โ
โ --- โ
โ str โ
โโโโโโโโโโโโโก
โ unknown โ
โโโโโโโโโโโโโค
โ unknown โ
โโโโโโโโโโโโโค
โ unknown โ
โโโโโโโโโโโโโค
โ unknown โ
โโโโโโโโโโโโโค
โ ... โ
โโโโโโโโโโโโโค
โ unknown โ
โโโโโโโโโโโโโค
โ unknown โ
โโโโโโโโโโโโโค
โ unknown โ
โโโโโโโโโโโโโค
โ unknown โ
โโโโโโโโโโโโโ
Schema has changed
Old
+------+-------+-------+-------+-------+--------+
| file | label | min_x | min_y | width | height |
| --- | --- | --- | --- | --- | --- |
| str | str | f64 | f64 | f64 | f64 |
+------+-------+-------+-------+-------+--------+
Current
+------+-------+-------+-------+-------+--------+-----------+
| file | label | min_x | min_y | width | height | is_fluffy |
| --- | --- | --- | --- | --- | --- | --- |
| str | str | f64 | f64 | f64 | f64 | str |
+------+-------+-------+-------+-------+--------+-----------+
If the file is any other type of text data, it will simply show you the added and removed lines.
oxen diff path/to/file.txt
i
+here
am a text file that
+I am modifying
-la-dee-da
+la-doo-da
+another line
Branching#
Branches are used to augment the dataset and run experiments with different subsets, transformations, or extensions of the data. The main
branch is the default branch when you start an Oxen repository. Use different branches while you run your experiments, and when you are confident in a dataset, merge it back into the main
branch.
You can create a new branch with
oxen checkout -b branch_name
Switch back to main
oxen checkout main
and delete the branch again
oxen branch -d branch_name
If you want to make the branch available to others, make sure to push it to a remote
oxen push origin branch_name
To see all the available branches you have locally run
oxen branch -a
Pulling New Changes#
To update your local repository to the latest changes, run
oxen pull origin branch_name
Again you can specify the remote and the branch name you would like to pull
Merging the changes#
If you feel confident in your changes, you can check out the main branch again, then merge your changes in.
oxen checkout main
oxen merge branch_name
If there are conflicts, Oxen will flag them and you will need to add and commit the files again in a separate commit. Oxen currently does not add any modifications to your working file, just flags it as conflicting. If you simply want to take your version, just add and commit again.
oxen add file/with/conflict.jpg
oxen commit -m "fixing conflict"
Dealing With Merge Conflicts#
Oxen currently has three ways to deal with merge conflicts.
Take the other personโs changes
oxen checkout file/with/conflict.jpg --theirs
, then add and commit.Take the changes in your current working directory (simply have to add and commit again)
Combine tabular data
oxen checkout file/with/conflict.csv --combine
If you use the --combine
flag, oxen will concatenate the data frames and unique them based on the row values.
Content Hashing and Core Metadata Extraction#
Oxen uses some core metadata around the file to be able to version and transfer the data efficiently. Any time a file gets versioned, itโs hash is computed and the contents is put into a content addressable filesystem. These files can be found in the hidden .oxen
directory. For example a file with the hash 7f65e0e4bda0acc99c56ecacbe092141
will be stored in .oxen/versions/files/7f/65e0e4bda0acc99c56ecacbe092141/
for fast random access given the hash.
To find out more info about a file with the CLI you can use the oxen info
command. For example:
$ oxen info images/test.jpg
fa42a17de86fa628b7caa8df6ba4d774 83851 image image/jpeg jpg
If you want to know the column names of each field, used the -v flag. All fields are separated by a tab so it is easy to pipe into cut
or other unix style commands.
$ oxen info images/test.jpg -v
hash size data_type mime_type extension
fa42a17de86fa628b7caa8df6ba4d774 83851 image image/jpeg jpg
Support#
If you have any questions, comments, suggestions, or just want to get in contact with the team, feel free to email us at hello@oxen.ai
Contributing#
This repository contains the documentation and release builds. To contribute to the core code base visit Oxen-AI/Oxen