Creative Commons (CC) Legal Tools Application.This repository contains the application that manages the license tools and public domain tools (static HTML, internationalization and localization files, etc.). It consumes and generates data in thecreativecommons/cc-legal-tools-data repository.
The Creative Commons team is committed to fostering a welcoming community. This project and all other Creative Commons open source projects are governed by ourCode of Conduct.Please report unacceptable behavior to[email protected] per ourreporting guidelines.
SeeCONTRIBUTING.md
.
This application manages 639 legal tools (636 licenses and 3 public domain tools). The current version of the licenses is 4.0 and includes 6 licenses. They are international and are designed to operate globally, ensuring they are robust, enforceable and easily adopted worldwide. Prior versions were adapted to specific jurisdictions ( "ported" ). That is why there are 636 licenses.
Broadly speaking, each legal tool consists of three layers:
deed
:a plain language summary of the legal toollegalcode
:the legal tool itselfrdf
:metadata about the legal tool in RDF/XML format
With translations of the deed and translations of the legal code, this application manages over 30,000 documents.
This project is not intended to serve the legal tools directly. Instead, a command line tool can be used to save all the rendered HTML and RDF/XML pages as files. Then those files are used as part of the CreativeCommons.org site (served as static files).
- Python 3.11specified in:
- Django 4.2 (LTS)
VisitCloning a Repositoryon how to clone a GitHub repository.
Thecreativecommons/cc-legal-tools-dataproject repository should be cloned into a directory adjacent to this one:
PARENT_DIR
├── cc-legal-tools-app (git clone of this repository)
└── cc-legal-tools-data (git clone of the cc-legal-tools-data repository)
If it is not cloned into the default location, the Django
DATA_REPOSITORY_DIR
Django configuration setting, or the
DATA_REPOSITORY_DIR
environment variable can be used to configure its
location.
Use the following instructions to start the project with Docker compose. Pleaes note that CC staff use macOS for development--please help us with documenting other operating systems if you encounter issues.
- Ensure theData Repository,above, is in place
- Install Docker Engine
- Ensure you are at the top level of the directory where you cloned this repository (where
manage.py
is) - Create Django local settings file
cp cc_legal_tools/settings/local.example.py cc_legal_tools/settings/local.py
- Build the containers
docker compose build
- Run the containers
docker compose up
- app(127.0.0.1:8005): this Django
application
- Any changes made to Python will be detected and rebuilt transparently as long as the development server is running.
- static(127.0.0.1:8006): a static web
server servingcreativecommons/cc-legal-tools-data:
docs/
- app(127.0.0.1:8005): this Django
application
- Run database migrations
docker composeexecapp./manage.py migrate
- Clear data in the database
docker composeexecapp./manage.py clear_license_data
- Load legacy HTML in the database
docker composeexecapp./manage.py load_html_files
⚠️ This section may be helpful for maintaining the project, but should NOT be used for development. Please use the Docker Compose Setup, above.
- Development Environment
- Ensure theData Repository,above, is in place
- Install dependencies
- Linux:
sudo apt-get install Python 3.11 Python 3.11-dev Python 3-pip
pip3 install pipenv
- macOS: viaHomebrew:
brew install pipenv Python @3.11
- Windows:install Pythonand then use
pip
to installpipenv
:pip install pipenv
- Linux:
- Install Python environment and modules via pipenv to create a
virtualenv
- Linux:
pipenv install --dev -- Python /usr/bin/ Python 3.11
- macOS: viaHomebrew:
pipenv install --dev -- Python /usr/local/opt/ Python @3.11/libexec/bin/ Python
- Windows:
pipenv install --dev -- Python\User\Appdata\programs\python
- Linux:
- Install pre-commit hooks
pipenv run pre-commit install
- Configure Django
- Create Django local settings file
cp cc_legal_tools/settings/local.example.py cc_legal_tools/settings/local.py
- Create project database
- Linux:
sudo createdb -E UTF-8 cc_legal_tools
- macOS:
createdb -E UTF-8 cc_legal_tools
- Windows:
createdb -E UTF-8 cc_legal_tools
- Linux:
- Load database schema
pipenv run./manage.py migrate
- Run development server (127.0.0.1:8005)
pipenv run./manage.py runserver
- Any changes made to Python will be detected and rebuilt transparently as long as the development server is running.
ℹ️ The rest of the documentation assumes Docker. If you are using a manual setup, use
pipenv run
instead ofdocker compose exec app
for the commands below.
- Python Guidelines — Creative Commons Open Source
- Black:the uncompromising Python code formatter
- Coverage.py:Code coverage measurement for Python
- Docker
- flake8:a Python tool that glues together pep8, pyflakes, mccabe, and third-party plugins to check the style and quality of some Python code.
- isort:A Python utility / library to sort imports.
- pre-commit:A framework for managing and maintaining multi-language pre-commit hooks.
Best run before every commit:
./dev/coverage.sh
- Run coverage tests and report./dev/tools.sh
- Run Python code tools (isort, black, flake8)
Run as needed:
./dev/copy_theme.sh
- Copy the portions of creativecommons/vocabulary-themeneeded for local development- Run after each new release of creativecommons/vocabulary-theme
Esoteric and dangerous:
./dev/concatenatemessages.sh
- Concatenate legacy ccEngine translations into cc-legal-tools-app- rarely used (only after source strings are updated)
./dev/resetdb.sh
- Reset Django application database data (!!DANGER!!)- usually only helpful if you're doing model/schema work
./dev/updatemessages.sh
- Run Django Management nofuzzy_makemessages with helpful options (including excluding legalcode) and compilemessages
The coverage tests and report are run as part of pre-commit and as a GitHub Action. To run it manually:
- Ensure theData Repository,above, is in place
- EnsureDocker Compose Setup,above, is complete
- Coverage test
docker composeexecapp coverage run manage.pytest--noinput --keepdb
- Coverage report
docker composeexecapp coverage report
If you encounter anerror: Error building trees
error from pre-commit when
you commit, try adding your files (git add <FILES>
) before committing them.
The following CC projects are used to achieve a consistent look and feel:
- creativecommons/vocabulary-theme:WordPress Theme implementation of the Vocabulary design system
The legal tools metadata is in a database. The metadata tracks which legal tools exist, their translations, their ports, and their characteristics like what they permit, require, and prohibit.
The metadata can be downloaded by visiting the URL path:
(currently disabled)127.0.0.1:8005
/licenses/metadata.yaml
There are two main models (Django terminology for tables) in
legal_tools/models.py
:
LegalCode
Tool
A Tool can be identified by aunit
(ex.by
,by-nc-sa
,devnations
) which
is a proxy for the complete set of permissions, requirements, and prohibitions;
aversion
(ex.4.0
,3.0)
,and an optionaljurisdiction
for ports. So we
might refer to the tool by itsidentifier"BY 3.0 AM" which would be the
3.0 version of the BY license terms as ported to the Armenia jurisdiction. For
additional information see:Legal Tools Namespace-
creativecommons/cc-legal-tools-data: CC Legal Tools Data (static HTML, language
files, etc.).
There are three places legal code text could be:
- Gettext files(
.po
and.mo
) in the creativecommons/cc-legal-tools-datarepository (legal tools with full translation support):- 4.0 Licenses
- CC0
- Django template
(
legalcode_licenses_3.0_unported.html
):- Unported 3.0 Licenses (English-only)
html
field(in theLegalCode
model):- Everything else
The text that's in gettext files can be translated via Transifex atCreative Commons localization.For additional information on the Django translation domains / Transifex resources, seeHow the license translation is implemented,below.
Documentation:
The process of getting the text into the site varies by legal tool.
Note that once the site is up and running in production, the data in the site will become the canonical source, and the process described here should not need to be repeated after that.
The implementation is the Django management commandload_html_files
,which
reads from the legacy HTML legal code files in the
creativecommons/cc-legal-tools-datarepository, and populates the
database records and translation files.
load_html_files
usesBeautifulSoup4to parse the legacy HTML legal
code:
import_zero_license_html()
for CC0 Public Domain tool- HTML is handled specifically (using tag ids and classes) to populate translation strings and to be used with specific HTML formatting when displayed via template
import_by_40_license_html()
for 4.0 License tools- HTML is handled specifically (using tag ids and classes) to populate translation strings and to be used with specific HTML formatting when displayed via a template
import_by_30_unported_license_html()
for unported 3.0 License tools (English-only)- HTML is handled specifically to be used with specific HTML formatting when displayed via a template
simple_import_license_html()
for everything else- HTML is handled generically; only the title and license body are
identified. The body is stored in the
html
field of theLegalCode
model
- HTML is handled generically; only the title and license body are
identified. The body is stored in the
This process will read the HTML files from the specified directory, populate
LegalCode
andTool
models, and create the.po
portable object Gettext
files increativecommons/cc-legal-tools-data.
- Ensure theData Repository,above, is in place
- EnsureDocker Compose Setup,above, is complete
- Clear data in the database
docker composeexecapp./manage.py clear_license_data
- Load legacy HTML in the database
docker composeexecapp./manage.py load_html_files
- Optionally (and only as appropriate):
- Commit the
.po
portable object Gettext file changes in creativecommons/cc-legal-tools-data - Translation Update Process,below
- Generate Static Files,below
- Commit the
- Beautiful Soup Documentation — Beautiful Soup 4 documentation
- Quick start guide — polib documentation
To upload/download translation files to/from Transifex, you'll need an account
there with access to these translations. Then follow theAuthentication -
Transifex API v3:to get an API token, and set
TRANSIFEX[ "API_TOKEN" ]
in your environment with its value.
Thecreativecommons/cc-legal-tools-datarepository must be cloned
next to thiscc-legal-tools-app
repository. (It can be elsewhere, then you
need to setDATA_REPOSITORY_DIR
to its location.) Be sure to clone using a
URL that starts withgit@github...
and nothttps://github...
,or you won't
be able to push to it. Also seeData Repository,above.
In production, thecheck_for_translation_updates
management command should be
run hourly. SeeCheck for Translation
Updates,below.
Also seePublishing changes to git repo, below.
Babelis used for localization information.
Documentation:
Django Translation uses two sets of Gettext Files in the creativecommons/cc-legal-tools-datarepository (theData Repository,above). See that repository for detailed information and definitions.
Documentation:
- Translation | Django documentation | Django
- Transifex API
⚠️ This functionality is currently disabled.
The hourly run ofcheck_for_translation_updates
looks to see if any of the
translation files in Transifex have newer last modification times than we know
about. It performs the following process (which can also be done manually:
- Ensure theData Repository,above, is in place
- Within thecreativecommons/cc-legal-tools-data(theData
Repository):
- Checkout or create the appropriate branch.
- For example, if a French translation file for BY 4.0 has changed, the
branch name will be
cc4-fr
.
- For example, if a French translation file for BY 4.0 has changed, the
branch name will be
- Download the updated
.po
portable object Gettext file from Transifex - Do theTranslation Update Process(below)
- This is important and easy to forget,but without it, Django will keep using the old translations
- Commit that change and push it upstream.
- Checkout or create the appropriate branch.
- Within this
cc-legal-tools-app
repository:- For each branch that has been updated,Generate Static Files(below). Use the options to update git and push the changes.
- GitPython Documentation — GitPython documentation
- Requests: HTTP for Humans™ — Requests documentation
This Django Admin command must be run any time the.po
portable object
Gettext files are created or changed.
- Ensure theData Repository,above, is in place
- EnsureDocker Compose Setup,above, is complete
- Compile translation messages (update the
.mo
machine object Gettext files)docker composeexecapp./manage.py compilemessages
Generating static files updates the static files in thedocs/
directory of
thecreativecommons/cc-legal-tools-datarepository (theData
Repository,above).
This process will write the HTML files in the cc-legal-tools-data clone
directory underdocs/
.It will not commit the changes (--nogit
) and will
not push any commits (--nopush
is implied by--nogit
).
- Ensure theData Repository,above, is in place
- EnsureDocker Compose Setup,above, is complete
- Delete the contents of the
docs/
directory and then recreate/copy the static files it should contain:docker composeexecapp./manage.py publish -v2
When the site is deployed, to enable pushing and pulling the licenses data repo
with GitHub, create an SSH deploy key for the cc-legal-tools-data repo with
write permissions, and put the private key file (not password protected)
somewhere safe (owned bywww-data
if on a server), and readable only by its
owner (0o400). Then in settings, makeTRANSLATION_REPOSITORY_DEPLOY_KEY
be
the full path to that deploy key file.
- Beautiful Soup Documentation — Beautiful Soup 4 documentation
- GitPython Documentation — GitPython documentation
LICENSE
:the code within this repository is licensed under the
Expat/MITlicense.
The text of the Creative Commons public licenses (legal code) is dedicated to the public domain under theCC0 1.0 Universal (CC0 1.0) Public Domain Dedication.
COPYING
:All the code within Vocabulary is dedicated to
the public domain under theCC0 1.0 Universal (CC0 1.0) Public Domain
Dedication.
normalize.css is licensed under the Expat/MITLicense.
Accidenz Commonsby Archetypo is licensed under theCreative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) License.
JetBrains Monois licensed under theOFL-1.1 License.
Roboto Condensedby Christian Robertson is licensed under theApache License, Version 2.0.
Source Sans Proby Paul D. Hunt is licensed under theOpen Font License.
Vocabulary Icons use icons fromFont Awesomewhich are licensed under theCreative Commons Attribution 4.0 International (CC BY 4.0) License.