You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#### 🚧 Current Status: Incubating - JZFS is not ready for production usage. The API is still evolving and documentation is lacking.
10
2
11
-
JZFS is a version control filesystem based on Git protocol for data management and publication with a command line interface and a Python API. With JZFS, you can version control arbitrarily large data, share or consume data, record your data’s provenance, and work computationally reproducible.
**JZFS** is an open-source, cloud-native version control filesystem based on Git protocol for data management and publication with a command line interface and a Python API. With JZFS, you can version control arbitrarily large data, share or consume data, record your data’s provenance, and work computationally reproducible.
58
+
59
+
JZFS adapts principles of open-source software development and distribution to address the technical challenges of data management, data sharing, and digital provenance collection across the life cycle of digital objects.
60
+
61
+
62
+
63
+
Compared with code in software development, data tend not to be as precisely
64
+
identified because data versioning is rarely or only coarsely practiced. Scientific computation
65
+
is not reproducible enough, because data provenance, the information of how a digital file
66
+
came to be, is often incomplete and rarely automatically captured. Last but not least, in
67
+
the absence of standardized data packages, there is no uniform way to declare actionable
68
+
data dependencies and derivative relationships between inputs and outputs of a computation. JZFS aims to solve these issues by providing streamlined, transparent management
69
+
of code, data, computing environments, and their relationship.
70
+
71
+
### Current Status and Roadmap
14
72
15
73
JZFS is still in the early development stages and is considered **incubating**. There is no commitment to ongoing maintenance or development. As the project evolves, this may change in the future. Therefore, we encourage you to explore, experiment, and contribute to JZFS, but do not attempt to use it in production.
16
74
17
75
The immediate next steps for the project are to fill obvious gaps, such as implementing error handling, removing panics throughout the codebase, supporting additional data types, and writing documentation. After that, development will be based on feedback and contributions.
18
76
77
+
JZFS's long-term goal is to build data ecosystems that enable new innovations.
78
+
79
+
19
80
### Reserach Data Management
20
81
21
82
JZFS is based on Git with extend capabilities, especially with respect to managing large files.
@@ -31,7 +92,7 @@ JZFS is used to collect
31
92
all experimental metadata about the complete timeline of longitudinal and multimodal animal experiments,
32
93
including MRI, histology, electrophysiology, and behavior.
33
94
34
-
95
+
35
96
Project planning and experimental details are recorded in an in-house relational cloud-based database.
36
97
37
98
A key element for both the database and the data storage is the
@@ -43,16 +104,12 @@ specific project is organized following the YODA principles (https://handbook.da
43
104
In preparation for publication and to facilitate
44
105
data reproducibility, the experimental raw and processed data is made publicly available on GitData.AI.
45
106
46
-
47
107
JZFS is used as the central data management tool (Fig. above) and for version control: It keeps track of which
48
108
files were modified, when, and by whom, and provides the ability to restore previous states. To this end, JZFS
49
109
is agnostic of the data type and provides a unified interface for managing code and data files.
50
110
51
111
52
-
#### joint management of code, data, and their relationship
53
-
54
-
55
-
### Added value
112
+
### DataHub
56
113
Our central use case is the DataHub(Like Github, buf for Data),which essentially consists of a Git version control for data and a Git collaboration for data.
57
114
58
115
In the JZFS DataHub model, each node maintains a copy of the files and all of the history of each file.
@@ -63,9 +120,9 @@ Writes are organized on branches.
63
120
Git is designed to compute differences between versions quickly.
64
121
Generally, Git relies on human action to share and merge changes.
65
122
66
-
Fault tolerance and trustless-ness are achieved via the separation of remotes from individual nodes.
67
-
Each node in the network is maintaining it's own copy, its history and coordinating via one or many remotes.
68
-
If a remote you trust gets corrupted, you have the ability to roll back to a previous good state and switch to a new remote.
123
+
Fault tolerance and trustless-ness are achieved via the separation of remotes from individual nodes.
124
+
Each node in the network is maintaining it's own copy, its history and coordinating via one or many remotes.
125
+
If a remote you trust gets corrupted, you have the ability to roll back to a previous good state and switch to a new remote.
69
126
Even if you lose your copy you can rely on other nodes' copies to restore from.
70
127
71
128
This enables us to create transparency across internal and external data.
@@ -74,18 +131,22 @@ It forms the basis for a new way of practicing data exchange and contract design
74
131
75
132
It is crucial that data exchange works both within the company and in individually controllable data networks (data space).
76
133
134
+
### Data Space
77
135
JZFS data space consists of so-called “DataHubs” like one or many remotes in Git, which are virtual data nodes for sharing data and building data networks.
78
136
A single DataHub manages various data connections and can join together with other hubs to form a network through targeted synchronization.
79
137
Based on data contracts mapped in the network, data can be released to other participants, enabling efficient data exchange.
80
138
81
139
82
140
83
141
84
-
JZFS offers git for data technology for exchanging data in data hub and data space.
142
+
JZFS offers git for data technology for exchanging data in data hub and data space.
143
+
85
144
The added value is clear: simple, transparent data management combined with intuitive linking and sharing of data in decentralized networks – data space.
86
145
This enables secure, trustworthy data exchange across organizational boundaries.
87
146
88
-
JZFS's long-term goal is to build data ecosystems that enable new innovations.
0 commit comments