The above code comes from section 3.5 in this notebook for some context.
I'm looking for information on how should a Python Machine Learning project be organized. For Python usual projects there is Cookiecutter and for R ProjectTemplate.
This is my current folder structure, but I'm mixing Jupyter Notebooks with actual Python code and it does not seems very clear.
.
├── cache
├── data
├── my_module
├── logs
├── notebooks
├── scripts
├── snippets
└── tools
I work in the scripts folder and currently adding all the functions in files under my_module, but that leads to errors loading data(relative/absolute paths) and other problems.
I could not find proper best practices or good examples on this topic besides this kaggle competition solution and some Notebooks that have all the functions condensed at the start of such Notebook.
You may want to look at:
http://tshauck.github.io/Gloo/
loo's goal is to tie together a lot of the data analysis actions that happen regularly and make that processes easy. Automatically loading data into the ipython environment, running scripts, making utitlity functions available and more. These are things that have to be done often, but aren't the fun part.
It's not actively maintained but the basics are there.
We've started a cookiecutter-data-science project designed for Python data scientists that might be of interest to you, check it out here. Structure is explained here.
Would love feedback if you have it! Feel free to respond here, open PRs or file issues.
In response to your issue about re-using code by importing .py files into notebooks, the most effective way that our team has found is to append to the system path. This may make some people cringe, but it seems like the cleanest way of importing code into a notebook without lots of module boilerplate and a pip -e install.
One tip is to use the %autoreload and %aimport magics with the above. Here's an example:
# Load the "autoreload" extension
%load_ext autoreload
# always reload modules marked with "%aimport"
%autoreload 1
import os
import sys
# add the 'src' directory as one where we can import modules
src_dir = os.path.join(os.getcwd(), os.pardir, 'src')
sys.path.append(src_dir)
# import my method from the source code
%aimport preprocess.build_features
The above code comes from section 3.5 in this notebook for some context.
When you are working on a data project, there are often many files that you need to store on your computer. These files may include:
It will save you time and make your project more useable and reproducible if you carefully consider how these files are stored on your computer. Below are some best practices to consider when pulling together a project.
As you create new directories and files on your computer, consider using a carefully crafted naming convention that makes it easier for anyone to find things and also to understand what each files does or contains.
It is good practice to use file and directory that are:
-
or _
to separate words within the name to make them easy to read and parse.01-max.jpg
, 02-terry.jpg
, etc), which will result in sortable files.These guidelines not only help you to organize your directories and files, but they can also help you to implement machine readable names that can be easily queried or parsed using scientific programming or other forms of scripting.