Oilsis our upgrade path from bash to a better language and runtime!
(The project wasslightly renamedin March 2023, so there are still old references to "Oil". Feel free to send pull requests with corrections!)
Oils 2023 FAQ/Why Create a New Unix Shell?
It's written in Python, so the code is short and easy to change. But we automatically translate it to C++ with custom tools, to make it fast and small. The deployed executable doesn't depend on Python.
This README is at the root of thegit repo.
- Try making thedev buildof Oils with the instructions on the Contributingpage. This should take 1 to 5 minutes if you have a Linux machine.
- If it doesn't, let us know. You can post on the
#oil-dev
channel of oilshell.zulipchat,or file an issue on Github. - Feel free to grab anissue from Github. Let us know what you're thinking before you get too far.
After following the instructions on theContributingpage, you'll have a Python program that you can quickly run and change! Try it interactively:
bash$ bin/osh
osh$ name=world
osh$ echo "hello $name"
hello world
- Try running a shell script you wrote with
bin/osh myscript.sh
. - TryYSHwith
bin/ysh
.
Let us know if any of these things don't work!The continuous buildtests them at every commit.
Again, note that thedeveloper buildisvery differentfrom the release tarball. TheContributingpage describes this difference in detail.
The release tarballs are linked from thehome page.(Developer builds don't work on OS X, so use the release tarballs on OS X.)
Oils is full ofmany ideas,which may be intimidating at first.
But the bar to contribution is very low. It's basically a medium size Python program with many tests, and many programmers know how to change such programs. It's great for prototyping.
- For OSH compatibility, I often mergefailingspec
tests.You don't even
have to write code! The tests alone help. I search for related tests with
grep xtrace spec/*.test.sh
,wherextrace
is a shell feature. - You only have to make your code workin Python.Plain Python programs are easy to modify. The semi-automated translation to C++ is a separate step, although it often just works.
- You caninfluence the designofYSH.If you have an itch to scratch, be ambitious. For example, you might want to show us how to implementnonlinear pipelines.
Please feel free to pingandychu
on Zulip or Github if you'rewaitingfor
a pull request review! (or to ask questions)
Usually I can respond in 24 hours. I might be traveling, in which case I'll respond with something likeI hope to look at this by Tuesday.
I might have alsomissedyour Github message, so it doesn't hurt to ping me.
Thank you for the contributions!
TheWikihas many developer docs. Feel free to edit them. If you make a major change, let us know on Zulip!
There are also READMEs in some subdirectories, likeopy/
andmycpp/
.
If you're confused, the best thing to do is to ask on Zulip and someone should produce a pointer and/or improve the docs.
Docs forend usersare linked from eachrelease page.
Try this to show a summary of what's in the repo and their line counts:
$ metrics/source-code.sh overview
(Other functions in this file may be useful as well.)
Oils is naturally structured as a set of mutually recursive parsers and evaluators. These interpreters are specified at a high-level: with regular languages, Zephyr ASDL, and a statically-typed subset of Python.
bin/ # Main entry points like bin/osh (source in bin/oils_for_unix.py)
frontend/ # Input and le xing common to OSH and YSH
osh/ # OSH parsers and evaluators (cmd, word, bắn xpr)
ysh/ # YSH parser and evaluator
data_lang/ # Languages based on JSON
core/ # Other code shared between OSH and YSH
builtin/ # Builtin commands and functions
pyext/ # Python extension modules, e.g. libc.c
pylib/ # Borrowed from the Python standard library.
tools/ # User-facing tools, e.g. the osh2oil translator
display/ # User interface
Here are the tools that transform that high-level code to efficient code:
asdl/ # ASDL implementation, derived from CPython
pgen2/ # Parser Generator, borrowed from CPython
mycpp/ # Experimental translator from typed Python to C++.
# Depends on MyPy. See mycpp/README.md
pea/ # Perhaps a cleaner version of mycpp
opy/ # Python compiler in Python (mycpp/ will replace it)
We have native code to support both the dev build (running under CPython) and
theoils-for-unix
build (pure C++):
NINJA-config.sh # Generates build.ninja
build/ # High level build
NINJA-steps.sh
NINJA_main.py # invoked by NINJA-config.sh
NINJA_subgraph.py
oil-defs/ # Files that define our slice of CPython.
py.sh # For development builds, running CPython
cpp/ # C++ code which complements the mycpp translation
NINJA-steps.sh
NINJA_subgraph.py
mycpp/ # Runtime for the translator
NINJA-steps.sh
NINJA_subgraph.py
prebuilt/ # Prebuilt files committed to git, instead of in _gen/
Python-2.7.13/ # For the slow Python build
# Temp dirs (see below)
_bin/
_build/
_gen/
_test/
Unit tests are namedfoo_test.py
and live next tofoo.py
.
test/ # Test automation
gold/ # Gold Test cases
gold.sh
sh_spec.py # shell spec test framework
spec.sh # Types of test runner: spec, unit, gold, wild
unit.sh
wild.sh
testdata/
spec/ # Spec test cases
bin/ # tools used in many spec tests
testdata/ # scripts for specific test cases
stateful/ # Tests that use pexpect
We use a lot of automation to improve the dev process. It's largely written in shell, of course!
benchmarks/ # Benchmarks should be run on multiple machines.
metrics/ # Metrics don't change between machines (e.g. code size)
client/ # Demonstration of OSH as a headless server.
deps/ # Dev dependencies and Docker images
devtools/ # For Oils developers (not end users)
release.sh # The (large) release process.
services/ # talk to cloud services
demo/ # Demonstrations of bash/shell features. Could be
# moved to tests/ if automated.
old/ # A junk drawer.
web/ # HTML/JS/CSS for tests and tools
soil/ # Multi-cloud continuous build (e.g. sourcehut, Github)
Directories that begin with_
arenotstored ingit
.The dev tools
above create and use these dirs.
_bin/ # Native executables are put here
cxx-dbg/
_build/ # Temporary build files
_cache/ # Dev dependency tarballs
_devbuild/ # Generated Python code, etc.
_gen/ # Generated C++ code that mirrors the repo
frontend/
_release/ # Source release tarballs are put here
VERSION/ # Published at oilshell.org/release/$VERSION/
benchmarks/
doc/
metrics/
test/
spec.wwz
wild.wwz
...
web/ # Static files, copy of $REPO_ROOT/web
table/
_test/ # Unit tests, mycpp examples
tasks/
_tmp/ # Output of other test suites; temp files
spec/
wild/
raw/
www/
osh-parser/
osh-runtime/
vm-baseline/
oheap/
startup/
...
These tools are built from shell scripts insoil/
.Theoil_DEPS
dir is
"parallel" to Oils because it works better with container bind mounds.
../oil_DEPS/
re2c/ # to build the lexer
cmark/ # for building docs
spec-bin/ # shells to run spec tests against
mypy/ # MyPy repo
mycpp-venv/ # MyPy binaries deps in a VirtualEnv
py3/ # for mycpp and pea/
c Python -full/ # for bootstrapping Oils-CPython
These files make the slow "Oils Python" build, which is very different than the developer buildof Oils.
Makefile
configure
install
These files are for the C++oils-for-unix
tarball (in progress):
_build/
oils.sh
doc/ # A mix of docs
doctools/ # Tools that use lazylex/ to transform Markdown/HTML
lazylex/ # An HTML lexer which doctools/ builds upon.
README.md # This page, which is For Oils developers
LICENSE.txt # For end users
INSTALL.txt
There are README files in many subdirectories, like mycpp/README.md.
- The bloghas updates on the project status.
- Oils Home Page
- oilshell.zulipchatis for any kind of discussion
- Subscribe for updates: