Skip to content

Commit 030fd3b

Browse files
committed
2 parents 3d469cc + dcb37ab commit 030fd3b

29 files changed

+831
-247
lines changed

.gitignore

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,11 @@
55
*/*.pyc
66
.DS_Store
77
__*__
8-
.idea
9-
datajoint.egg-info/
8+
.idea/
109
*.pyc
10+
.python-version
11+
*.egg-info/
12+
dist/
13+
build/
14+
MANIFEST
15+
.vagrant/

.travis.yml

Lines changed: 5 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1,29 +1,12 @@
11
language: python
2-
python: 3.4
3-
services: mysql
42
env:
53
- DJ_TEST_HOST="127.0.0.1" DJ_TEST_USER="root" DJ_TEST_PASSWORD="" DJ_HOST="127.0.0.1" DJ_USER="root" DJ_PASSWORD=""
6-
before_install:
7-
- sudo apt-get install -qq libatlas-dev libatlas-base-dev liblapack-dev gfortran
8-
- sudo apt-get update
9-
# You may want to periodically update this, although the conda update
10-
# conda line below will keep everything up-to-date. We do this
11-
# conditionally because it saves us some downloading if the version is
12-
# the same.
13-
- wget http://repo.continuum.io/miniconda/Miniconda3-3.4.2-Linux-x86_64.sh -O miniconda.sh;
14-
- bash miniconda.sh -b -p $HOME/miniconda
15-
- export PATH="$HOME/miniconda/bin:$PATH"
16-
- hash -r
17-
- conda update -q --yes conda
18-
- conda config --set always_yes yes --set changeps1 no
19-
# Useful for debugging any issues with conda
20-
- conda info -a
4+
python:
5+
- "3.4"
6+
services: mysql
217
install:
22-
# Replace dep1 dep2 ... with your dependencies
23-
- conda create -n test-environment python=$TRAVIS_PYTHON_VERSION numpy scipy setuptools pip
24-
- source activate test-environment
25-
- pip install nose nose-cov python-coveralls pymysql networkx matplotlib
26-
- conda info -a
8+
- pip install -r requirements.txt
9+
- pip install nose nose-cov python-coveralls
2710
# command to run tests
2811
script:
2912
- nosetests -vv --with-coverage --cover-package=datajoint

Makefile

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
all:
2+
@echo 'MakeFile for DataJoint packaging '
3+
@echo ' '
4+
@echo 'make sdist Creates source distribution '
5+
@echo 'make wheel Creates Wheel dstribution '
6+
@echo 'make pypi Package and upload to PyPI '
7+
@echo 'make pypitest Package and upload to PyPI test server'
8+
@echo 'make purge Remove all build related directories '
9+
10+
11+
sdist:
12+
python setup.py sdist >/dev/null 2>&1
13+
14+
wheel:
15+
python setup.py bdist_wheel >/dev/null 2>&1
16+
17+
pypi:purge sdist wheel
18+
twine upload dist/*
19+
20+
pypitest: purge sdist wheel
21+
twine upload -r pypitest dist/*
22+
23+
purge:
24+
rm -rf dist && rm -rf build && rm -rf datajoint.egg-info
25+
26+
27+
28+

README.md

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,27 @@
1+
[![Build Status](https://travis-ci.org/eywalker/datajoint-python.svg?branch=master)](https://travis-ci.org/eywalker/datajoint-python)
2+
[![Coverage Status](https://coveralls.io/repos/datajoint/datajoint-python/badge.svg?branch=master&service=github)](https://coveralls.io/github/datajoint/datajoint-python?branch=master)
3+
[![PyPI version](https://badge.fury.io/py/datajoint.svg)](http://badge.fury.io/py/datajoint)
4+
[![Requirements Status](https://requires.io/github/datajoint/datajoint-python/requirements.svg?branch=master)](https://requires.io/github/datajoint/datajoint-python/requirements/?branch=master)
5+
[![Documentation Status](https://readthedocs.org/projects/datajoint-python/badge/?version=latest)](https://readthedocs.org/projects/datajoint-python/?badge=latest)
6+
[![Join the chat at https://gitter.im/datajoint/datajoint-python](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/datajoint/datajoint-python?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
7+
8+
# Welcome to DataJoint for Python!
19
The Python version of DataJoint is undergoing major revamping to match the features and capabilities of its more mature MATLAB counterpart. We expect to complete the revamp within a few weeks: August -- September, 2015.
210

11+
Learn more on http://datajoint.github.io.
12+
313
DataJoint for Python is a high-level programming interface for relational databases designed to support data processing chains in science labs. DataJoint is built on the foundation of the relational data model and prescribes a consistent method for organizing, populating, and querying data.
414

515
DataJoint was initially developed in 2009 by Dimitri Yatsenko in Andreas Tolias' Lab for the distributed processing and management of large volumes of data streaming from regular experiments. Starting in 2011, DataJoint has been available as an open-source project adopted by other labs and improved through contributions from several developers.
616

7-
To install datajoint using pip just run:
8-
```pip install git+https://github.com/datajoint/datajoint-python```
17+
## Quick start guide
18+
To install datajoint using `pip` just run:
19+
20+
```
21+
pip install datajoint
22+
```
23+
24+
in your favorite terminal app.
925

26+
However, please be aware that DataJoint for Python is still undergoing major changes, and thus what's available on PyPI via `pip` is in **pre-release state**!
1027

11-
[![Join the chat at https://gitter.im/datajoint/datajoint-python](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/datajoint/datajoint-python?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

Vagrantfile

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# -*- mode: ruby -*-
2+
# vi: set ft=ruby :
3+
4+
# All Vagrant configuration is done below. The "2" in Vagrant.configure
5+
# configures the configuration version (we support older styles for
6+
# backwards compatibility). Please don't change it unless you know what
7+
# you're doing.
8+
Vagrant.configure(2) do |config|
9+
# The most common configuration options are documented and commented below.
10+
# For a complete reference, please see the online documentation at
11+
# https://docs.vagrantup.com.
12+
13+
# Every Vagrant development environment requires a box. You can search for
14+
# boxes at https://atlas.hashicorp.com/search.
15+
config.vm.box = "ubuntu/trusty64"
16+
17+
# Disable automatic box update checking. If you disable this, then
18+
# boxes will only be checked for updates when the user runs
19+
# `vagrant box outdated`. This is not recommended.
20+
# config.vm.box_check_update = false
21+
22+
# Create a forwarded port mapping which allows access to a specific port
23+
# within the machine from a port on the host machine. In the example below,
24+
# accessing "localhost:8080" will access port 80 on the guest machine.
25+
# config.vm.network "forwarded_port", guest: 80, host: 8080
26+
27+
# Create a private network, which allows host-only access to the machine
28+
# using a specific IP.
29+
config.vm.network "private_network", ip: "192.168.33.10"
30+
31+
# Create a public network, which generally matched to bridged network.
32+
# Bridged networks make the machine appear as another physical device on
33+
# your network.
34+
# config.vm.network "public_network"
35+
36+
# Share an additional folder to the guest VM. The first argument is
37+
# the path on the host to the actual folder. The second argument is
38+
# the path on the guest to mount the folder. And the optional third
39+
# argument is a set of non-required options.
40+
# config.vm.synced_folder "../data", "/vagrant_data"
41+
42+
# Provider-specific configuration so you can fine-tune various
43+
# backing providers for Vagrant. These expose provider-specific options.
44+
# Example for VirtualBox:
45+
#
46+
# config.vm.provider "virtualbox" do |vb|
47+
# # Display the VirtualBox GUI when booting the machine
48+
# vb.gui = true
49+
#
50+
# # Customize the amount of memory on the VM:
51+
# vb.memory = "1024"
52+
# end
53+
#
54+
# View the documentation for the provider you are using for more
55+
# information on available options.
56+
57+
# Define a Vagrant Push strategy for pushing to Atlas. Other push strategies
58+
# such as FTP and Heroku are also available. See the documentation at
59+
# https://docs.vagrantup.com/v2/push/atlas.html for more information.
60+
# config.push.define "atlas" do |push|
61+
# push.app = "YOUR_ATLAS_USERNAME/YOUR_APPLICATION_NAME"
62+
# end
63+
64+
# Enable provisioning with a shell script. Additional provisioners such as
65+
# Puppet, Chef, Ansible, Salt, and Docker are also available. Please see the
66+
# documentation for more information about their specific syntax and use.
67+
config.vm.provision "shell", path: "misc/provision.sh"
68+
end

datajoint/__init__.py

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,13 @@
1818
'Connection', 'Heading', 'Relation', 'FreeRelation', 'Not',
1919
'Relation', 'schema',
2020
'Manual', 'Lookup', 'Imported', 'Computed',
21-
'conn', 'DataJointError']
21+
'conn']
22+
23+
# define an object that identifies the primary key in RelationalOperand.__getitem__
24+
class PrimaryKey: pass
25+
26+
27+
key = PrimaryKey
2228

2329

2430
class DataJointError(Exception):
@@ -27,7 +33,6 @@ class DataJointError(Exception):
2733
"""
2834
pass
2935

30-
3136
# ----------- loads local configuration from file ----------------
3237
from .settings import Config, CONFIGVAR, LOCALCONFIG, logger, log_levels
3338
config = Config()

datajoint/autopopulate.py

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
"""autopopulate containing the dj.AutoPopulate class. See `dj.AutoPopulate` for more info."""
22
import abc
33
import logging
4+
import datetime
45
from .relational_operand import RelationalOperand
56
from . import DataJointError
67
from .relation import Relation, FreeRelation
@@ -68,7 +69,7 @@ def populate(self, restriction=None, suppress_errors=False, reserve_jobs=False):
6869
jobs = self.connection.jobs[self.target.database]
6970
table_name = self.target.table_name
7071
unpopulated = (self.populated_from - self.target) & restriction
71-
for key in unpopulated.project():
72+
for key in unpopulated.fetch.keys():
7273
if not reserve_jobs or jobs.reserve(table_name, key):
7374
self.connection.start_transaction()
7475
if key in self.target: # already populated
@@ -100,5 +101,8 @@ def progress(self):
100101
"""
101102
total = len(self.populated_from)
102103
remaining = len(self.populated_from - self.target)
103-
print('Completed %d of %d (%2.1f%%)' % (total - remaining, total, 100 - 100 * remaining / total)
104-
if remaining else 'Complete', flush=True)
104+
print('Completed %d of %d (%2.1f%%) %s' %
105+
(total - remaining, total, 100 - 100 * remaining / total,
106+
datetime.datetime.strftime(datetime.datetime.now(), '%Y-%m-%d %H:%M:%S')
107+
) if remaining
108+
else 'Complete', flush=True)

datajoint/connection.py

Lines changed: 12 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,9 @@
66
from contextlib import contextmanager
77
import pymysql
88
import logging
9-
from collections import defaultdict
10-
from . import DataJointError, config
11-
from .erd import ERD
9+
from . import config
10+
from . import DataJointError
11+
from .erd import ERM
1212
from .jobs import JobManager
1313

1414
logger = logging.getLogger(__name__)
@@ -53,7 +53,7 @@ class Connection:
5353
"""
5454

5555
def __init__(self, host, user, passwd, init_fun=None):
56-
self.erd = ERD()
56+
self.erm = ERM(self)
5757
if ':' in host:
5858
host, port = host.split(':')
5959
port = int(port)
@@ -62,7 +62,7 @@ def __init__(self, host, user, passwd, init_fun=None):
6262
self.conn_info = dict(host=host, port=port, user=user, passwd=passwd)
6363
self._conn = pymysql.connect(init_command=init_fun, **self.conn_info)
6464
if self.is_connected:
65-
logger.info("Connected " + user + '@' + host + ':' + str(port))
65+
logger.info("Connected {user}@{host}:{port}".format(**self.conn_info))
6666
else:
6767
raise DataJointError('Connection failed.')
6868
self._conn.autocommit(True)
@@ -76,22 +76,21 @@ def __del__(self):
7676
def __eq__(self, other):
7777
return self.conn_info == other.conn_info
7878

79+
def __repr__(self):
80+
connected = "connected" if self.is_connected else "disconnected"
81+
return "DataJoint connection ({connected}) {user}@{host}:{port}".format(
82+
connected=connected, **self.conn_info)
83+
7984
@property
8085
def is_connected(self):
8186
"""
8287
Returns true if the object is connected to the database server.
8388
"""
8489
return self._conn.ping()
8590

86-
def __repr__(self):
87-
connected = "connected" if self.is_connected else "disconnected"
88-
return "DataJoint connection ({connected}) {user}@{host}:{port}".format(
89-
connected=connected, **self.conn_info)
90-
91-
9291
def query(self, query, args=(), as_dict=False):
9392
"""
94-
Execute the specified query and return the tuple generator.
93+
Execute the specified query and return the tuple generator (cursor).
9594
9695
:param query: mysql query
9796
:param args: additional arguments for the pymysql.cursor
@@ -168,4 +167,4 @@ def transaction(self):
168167
self.cancel_transaction()
169168
raise
170169
else:
171-
self.commit_transaction()
170+
self.commit_transaction()

datajoint/declare.py

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -8,12 +8,10 @@
88

99
from . import DataJointError
1010

11-
1211
logger = logging.getLogger(__name__)
1312

1413

15-
16-
def declare(full_table_name, definition, context):
14+
def declare(full_table_name, definition, context):
1715
"""
1816
Parse declaration and create new SQL table accordingly.
1917
@@ -24,6 +22,7 @@ def declare(full_table_name, definition, context):
2422
# split definition into lines
2523
definition = re.split(r'\s*\n\s*', definition.strip())
2624

25+
# check for optional table comment
2726
table_comment = definition.pop(0)[1:].strip() if definition[0].startswith('#') else ''
2827

2928
in_key = True # parse primary keys
@@ -40,7 +39,7 @@ def declare(full_table_name, definition, context):
4039
in_key = False # start parsing dependent attributes
4140
elif line.startswith('->'):
4241
# foreign key
43-
ref = eval(line[2:], context)()
42+
ref = eval(line[2:], context)() # TODO: surround this with try...except... to give a better error message
4443
foreign_key_sql.append(
4544
'FOREIGN KEY ({primary_key})'
4645
' REFERENCES {ref} ({primary_key})'
@@ -52,7 +51,7 @@ def declare(full_table_name, definition, context):
5251
primary_key.append(name)
5352
if name not in attributes:
5453
attributes.append(name)
55-
attribute_sql.append(ref.heading[name].sql())
54+
attribute_sql.append(ref.heading[name].sql)
5655
elif re.match(r'^(unique\s+)?index[^:]*$', line, re.I): # index
5756
index_sql.append(line) # the SQL syntax is identical to DataJoint's
5857
else:
@@ -66,7 +65,7 @@ def declare(full_table_name, definition, context):
6665
# compile SQL
6766
if not primary_key:
6867
raise DataJointError('Table must have a primary key')
69-
sql = 'CREATE TABLE %s (\n ' % full_table_name
68+
sql = 'CREATE TABLE IF NOT EXISTS %s (\n ' % full_table_name
7069
sql += ',\n '.join(attribute_sql)
7170
sql += ',\n PRIMARY KEY (`' + '`,`'.join(primary_key) + '`)'
7271
if foreign_key_sql:

0 commit comments

Comments
 (0)