ImageMonkey

---

ImageMonkey is a free, public open source dataset. With all the great machine learning frameworks available it's pretty easy to train pre-trained Machine Learning models with your own image dataset. However, in order to do so you need a lot of images. And that's usually the point where it get's tricky. You either have to create the training images yourself or scrape them together from various datasources. ImageMonkey aims to solve this problem, by providing a platform where users can drop their photos, tag them with a label, and put them into public domain.

---

Getting started

There are basically two ways to set up your ownImageMonkeyinstance. You can either set up everything by hand, which gives you the flexibility to choose your own linux distribution, monitoring tools and scrips or you could use ourDockerfileto spin up a newImageMonkeyinstance within just a few minutes.

Docker

Run ImageMonkey inside Docker

The docker image is for development only - doNOTuse it in production!

Manual Setup

The following section contains some notes on how to set up your own instance to host ImageMonkey yourself. This should only give you an idea how youcouldconfigure your system. Of course you are totally free in choosing a different linux distribution, tools and scripts. If you are only interested in how to compile ImageMonkey, then you can jump directly to theBuild Applicationsection

Info:Some commands are distribution (Debian 10) specific and may not work on your system.

Base System Configuration

• create a new userimagemonkeywithadduser imagemonkey
• disable root login via ssh by changing thePermitRootLoginline in/etc/ssh/sshd_configtoPermitRootLogin no)
• block all ports except port 22, 443 and 80 (on eth0) with:

#!bash

iptables -P INPUT DROP && iptables -A INPUT -i eth0 -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --dport 443 -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --dport 80 -j ACCEPT

• allow all established connections with:

#!bash

iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

• allow all loopback access with:

#!bash
iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT

• installiptables-persistentto load firewall rules at startup
• save firewall rules with:iptables-save > /etc/iptables/rules.v4
• verify that rules are loaded withiptables -L

Database

• install PostgreSQL
• edit/etc/postgresql/9.6/main/postgresql.confand setlisten_addresses = 'localhost'
• restart PostgreSQL service withservice postgresql restartto apply changes
• create database by applying schema/env/postgres/schema.sqlwithpsql -f schema.sql
• create new postgres usermonkeyby executing the following in psql:

CREATE USER monkey WITH PASSWORD 'your_password';

\connect imagemonkey
GRANT ALL PRIVILEGES ON DATABASE imagemonkey to monkey;
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO monkey;
GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO monkey;
GRANT USAGE ON SCHEMA blog TO monkey;

• test if newly created user works with:psql -d imagemonkey -U monkey -h 127.0.0.1

• populate labels withgo run populate_labels.go common.go web_secrets.go

• add donation image provider withinsert into image_provider(name) values('donation');

• buildtemporal_tableextension, as described here:https://github /arkhipov/temporal_tables

• connect to imagemonkey database and executeCREATE EXTENSION temporal_tables;

• connect to imagemonkey database and executeCREATE EXTENSION uuid-ossp;

• connect to imagemonkey database and executeCREATE EXTENSION postgis;

• applydefaults.sql

• applyindexesl.sql

• apply sql functions fromenv/functionsdirectory

• apply sql stored procedures fromenv/stored_procsdirectory

Redis

• install redis withapt-get install redis-server
• make sure that redis only listens on localhost
• change redis.conf and setmaxmemory(e.g: 500mb) and setmaxmemory-policytoallkeys-lru

Install Bimg

Windows:

• install MSYS2
• open MSYS2 terminal and install zlib and pkgconfig via pacman
• download vipslib v8.6.5 fromhttps://github /libvips/libvips/releases
• set PKG_CONFIG_PATH environment variable to the folder where vips.pc resides. e.q:PKG_CONFIG_PATH=/c/Users/Bernhard/Downloads/vips-dev-w64-all-8.6.5/vips-dev-8.6/lib/pkgconfig
• build bimg with:/c/Go/bin/go get -u gopkg.in/h2non/bimg.v1

Linux:

• seehttps://github /h2non/bimg

Install Gocv

• seehttps://github /hybridgroup/gocv

Webserver & SSL

• install nginx withapt-get install nginx
• install nginx-extras withapt-get install nginx-extras
• install letsencrypt certbot withapt-get install certbot
• add a A-Record DNS entry which points to the IP address of your instance
• runcertbot certonlyto obtain a certificate for your registered domain
• modifyconf/nginx/nginx.confand replaceimagemonkey.ioandapi.imagemonkey.iowith your own domain names, copy it to/etc/nginx/nginx.confand reload nginx withservice nginx reload

Build Application

Minimalrequired Go version: v1.11.10

• install git withapt-get install git
• install golang withapt-get install golang
• clone repository
• set GOPATH withexport GOPATH=$HOME/go
• set GOBIN withexport GOBIN=$HOME/bin
• install all dependencies withgo get -d./...
• install API application withgo install api.go api_secrets.go common.go imagedb.go
• install API application withgo install web.go web_secrets.go common.go imagedb.go

Miscellaneous

• copywordlists/en/misc.txtto/home/imagemonkey/wordlists/en/misc.txt
• create donation directories with:

mkdir -p /home/imagemonkey/donations
mkdir -p /home/imagemonkey/unverified_donations

Watchdog

• install supervisor withapt-get install supervisor
• addimagemonkeyuser to supervisor group withadduser imagemonkey supervisor
• create logging directories withmkdir -p /var/log/imagemonkey-api,mkdir -p /var/log/imagemonkey-webmkdir -p /var/log/imagemonkey-statworker,mkdir -p /var/log/imagemonkey-bot,mkdir -p /var/log/imagemonkey-blog-subscription-worker,mkdir -p /var/log/imagemonkey-data-processor,mkdir -p /var/log/imagemonkey-labelsdownloader,mkdir -p /var/log/imagemonkey-trending-labels-worker
• copyconf/supervisor/imagemonkey-api.confto/etc/supervisor/conf.d/imagemonkey-api.conf
• copyconf/supervisor/imagemonkey-web.confto/etc/supervisor/conf.d/imagemonkey-web.conf
• copyconf/supervisor/imagemonkey-statworker.confto/etc/supervisor/conf.d/imagemonkey-statworker.conf
• copyconf/supervisor/imagemonkey-blog-subscription-worker.confto/etc/supervisor/conf.d/imagemonkey-blog-subscription-worker.conf
• copyconf/supervisor/imagemonkey-bot.confto/etc/supervisor/conf.d/imagemonkey-bot.conf
• copyconf/supervisor/imagemonkey-labels-downloader.confto/etc/supervisor/conf.d/imagemonkey-labels-downloader.conf
• copyconf/supervisor/imagemonkey-trending-labels-worker.confto/etc/supervisor/conf.d/imagemonkey-trending-labels-worker.conf
• addEnvironmentFile=/etc/environmentto the service section of the systemctl supervisor config file (seehttps://stackoverflow /questions/47083582/supervisor-not-using-etc-environment)
• runsystemctl daemon-reloadandsystemctl restart supervisor
• runsupervisorctl reread && supervisorctl update && supervisorctl restart all

Datasync

on imagemonkey-playground instance

• installrsyncwithapt-get install rsync
• create a new userbackupuserwithadduser backupuser(use a strong password)
• change to userbackupuserwithsu backupuserand create a new SSH key withssh-keygen -t ed25519 -a 100
• copy SSH public key to imagemonkey instance with:ssh-copy-id -i ~/.ssh/your_generated_id.pub backupuser@imagemonkey-host
• givebackupuserpermissions to write to/home/playground/donationswith:chgrp backupuser /home/playground/donations && chmod g+rwx /home/playground/donations
• add a new cronjob for the userbackupuserwith:crontab -u backupuser -eand add the following line (runs rsync every 15min):

*/15 * * * * rsync -a [email protected]:/home/imagemonkey/donations/ /home/playground/donations/