Table of Contents
Introduction
The goal of Project Skara is to investigate alternative SCM and code review options for the OpenJDK source code, including options based upon Git rather than Mercurial, and including options hosted by third parties.
The technical parts of project Skara includes several server-side tools (also called "bots") aiding contributors during code reviews. The Skara technical tooling also includes several command-line utilities for interacting with Git source code hosting providers from the command-line.
Using an external Git source code hosting provider comes with several benefits, including:
- Performance
- Community
- API
Many, if not all, external Git source code hosting services available have excellent performance, not only with regards to network performance but also when it comes to availability (uptime). The largest Git source code hosting providers also offer the OpenJDK community to tap into large, existing, communities of developers and potential contributors. An additional benefit of using an external Git source code hosting provider is getting access to an API. These APIs enables programs to interact with developers on the external Git source code hosting provider. Although not impossible to achieve today by interacting with developers over email, it is considerably harder to implement programs that interpret free form text in emails compared to using a structured API.
A significant risk when using an external source code hosting provider is to become dependent on the external source code hosting provider. The version control data itself will always be independent of source code hosting provider due to the distributed architecture of Git itself. However there is a large risk in metadata such as code review comments becoming "locked in" on a particular external source code hosting provider. Mitigating this risk is a large part of Project Skara and the following work have been done so far:
- replicating all discussions in all pull requests to the OpenJDK mailing lists
- archiving all discussions in pull requests in two formats:
- mbox (for human consumption)
- json (for software consumption) (not implemented yet)
- notifications of all pushes to the corresponding *-changes@openjdk.java.net mailing lists to avoid dependence on any provider's RSS feeds
- using the OpenJDK census for user organization and privilege levels to avoid any dependencies on external Git source code hosting provider's user organization tool
setting up to the domain http://git.openjdk.java.net/to redirect to OpenJDK's current external source code hosting provider to avoid polluting JBS and mailing lists with direct links to an external Git source code hosting provider
Support for multiple external source code hosting provider has been a strict requirement for all server-side and client-side tooling to avoid the issue of having any tooling take on a dependency on a particular external source code hosting provider's API. All tooling is also required to work with the GitLab Community Edition (GitLab CE) which is an open source project.
Another large part of Project Skara has been to ensure that there are multiple workflows available when interacting with a Git external source code provider, including one that preservers as much as possible of the OpenJDK community's current workflow. The Skara tooling currently supports the following workflows:
- Mailing list + CLI based (similar to current workflow)
- Web browser + CLI based
- Desktop application
- Mobile/tablet application
- CLI only
- Text editor/IDE
All workflows can be used interchangeably and different contributors can use different workflows at the same time (even while working on the same change). The support of multiple workflows comes from the APIs provided by the external Git source control hosting provider. Examples of the work done to support multiple concurrent workflows include:
- Two-way mailing list synchronization (you can comment on pull requests via OpenJDK mailing lists)
- Automatic "RFR" emails sent to mailiing lists for newly created pull requests
- Automatically determining mailing lists to "CC" for newly created pull requests
- Automatic generation and hosting of "webrevs" (including incremental webrevs)
- Automatically adding links to pull requests in issues
- Automatically adding links to issues in pull requests
- Automatically run "jcheck" on every commit in every pull request
- Implement CLI tools to list, fetch, view, approve and integrate pull requests
- Implement backwards compatible ports of jcheck, webrev and defpath
Getting Started
The following sections will get you started with Git and the external Git source code hosting provider GitHub.
Git
Installing the Git CLI client
GNU/Linux
CentOS/Oracle Linux/RHEL
$ sudo yum install git
Debian/Ubuntu
$ sudo apt install git
Fedora
$ sudo dnf install git
macOS
There are two ways to install Git on macOS: using Homebrew or using a direct download.
Using Homebrew
- Install Homebrew: https://brew.sh/
brew install git
Using direct download
Windows
Install Git For Windows (maintained by Microsoft): https://gitforwindows.org
Initial Configuration
Git requires that you configure a username and an email. Use your full name as your username and your regular email address for the email:
$ git config --global user.name 'Your Full Name' $ git config --global user.email 'your.name@host.com'
For example my name is "Erik Duveblad" and my email is "erik.helin@oracle.com". Therefore I would run:
$ git config --global user.name 'Erik Duveblad' $ git config --global user.email 'erik.helin@oracle.com'
You will also want to setup the editor that Git will use whenever you need to enter multiple lines of input, for example when writing a commit message or doing an interactive rebase. I use vim
as my editor and therefore I would run:
$ git config --global core.editor 'vim'
Additional Clients
There are several additional clients available for Git that can be used instead of (or in combination with) the Git CLI tool. Please see the following subsections for how to interact with Git from the desktop, mobile, tablet, IDE or a text editor.
Desktop
The following desktop applications can all interact with Git repositories:
- SourceTree (macOS, Windows)
- gitg (GNU/Linux)
- Tower (macOS, Windows)
- GitKraken (GNU/Linux, macOS, Windows)
- Sublime Merge (GNU/Linux, macOS, Windows)
- TortoiseGit (Windows)
- GitFiend (GNU/Linux, macOS, Windows)
- Fork (macOS, Windows)
- Git Cola (GNU/Linux, macOS, Windows)
Mobile/Tablet
The following mobile/tablet applications can all interact with Git repositories:
- Working Copy (iOS, ipadOS)
- Pocket Git (Android)
IDE
The following integrated develop environments (IDEs) all have Git support built-in:
Text Editors
The following text editors either have Git support built-in or as part of a plugin:
Resources
There are a lot blogs posts, tutorials, tweets and material about Git on the internet. Unfortunately much of this material is outdated since Git was created back in 2005. We strongly recommend the online or e-book version of the "Pro Git" book by Scott Chacon and Ben Straub available at https://git-scm.com/book/en/v2 for learning more about Git (the print version is not up-to-date). The book is available online in HTML or can be downloaded in pdf, epub and/or mobi formats. The book is available under the CC BY-NC-SA 3.0 license and a community keep it continuously up-to-date by contributing to the book's Git repository.
For more information about a particular git
command, see the online reference or the man page for the command (git help <commmand>
)
If you want to learn more about the inner workings of Git and how it can be implemented then the online book "Write yourself a Git!" by Thibault Polge is a good resource: https://wyag.thb.lt/
GitHub
GitHub is an external Git source code hosting provider at https://github.com/.
Creating an account
To create an account and get started, follow the instructions at https://github.com/join. We strongly recommend that all OpenJDK contributors enable two-factor authentication (2FA) on GitHub. To enable 2FA for your account on GitHub, follow the instructions at https://help.github.com/en/articles/securing-your-account-with-two-factor-authentication-2fa.
Associating your GitHub account and your OpenJDK username
If you are an OpenJDK Author, Committer or Reviewer then you can associate your GitHub account with your OpenJDK username by opening an issue at https://bugs.openjdk.java.net/secure/CreateIssue.jspa?pid=11300&issuetype=1. As a title for the issue, please use "Add GitHub user <YOUR-GITHUB-USERNAME>".
This way the server-side tooling (the "bots") will recognize you on GitHub as an OpenJDK Author, Committer or Reviewer.
Resources
The GitHub Guides provides a good introduction to many of the concepts on GitHub. For more information about a particular topic, see the GitHub reference documentation. If you prefer watching a video over reading an article then there are a number of good introduction videos on GitHub Guide's YouTube channel. If you want to learn using a more hands-on approach, then we recommend trying out the GitHub Learning Lab.
Note: Skara makes some slight changes to parts of the GitHub workflow, see the Workflows section for more details
Workflow
As mentioned in the "Introduction" there are several tools a contributor can use based on their preference, but all tools realize an abstract workflow have two distinct features:
- every contributor will have a personal fork of an OpenJDK repository they want to contribute to
- every change will start out as a pull request
The following two sections will describe the concepts of a personal fork and a pull request in more details. If you are new to Git then we recommend that you read at least chapter 2 (Git Basics) and chapter 3 (Git Branching) in the Pro Git book before proceeding.
The last two sections give an overview of the abstract workflow and the services and commands that Skara's server-side tooling (bots) provide.
Personal forks
A personal fork is a copy of another repository with one major difference:
- you can use a pull request to suggest that some changes from your personal fork should be incorporated into the into the original repository the fork was created from
There are several advantages to each contributor having a personal fork of the original OpenJDK repository they want contribute to:
- Contributors can freely experiment in their personal fork without affecting the original repository
- Contributors can back-up work that is in progress by pushing it to their personal fork
- Contributors can do ad-hoc collaboration with other contributors in their personal forks
There is also one drawback of each contributor having a personal fork:
- The personal fork must from time to time be synced with the original repository
This single drawback is fortunately remedied with the help of tools that reduce the overhead of syncing a personal fork with the original repository to almost nothing.
An example of a personal fork is edvbld/jdk which is a personal fork of the openjdk/jdk repository.
The concept of a personal fork is present in almost all external Git source code hosting providers.
Pull requests
A pull request is a way to suggest that some changes from a personal fork should be incorporated into the original repository the personal fork was created from. Reviewers can comment upon and need to approve a pull request before it can be integrated. The concept of a pull request is very similar to OpenJDK's concept of "RFR" emails - both are used to suggest changes and offer a way for reviewers to provide feedback on the suggested changes.
Pull requests are most commonly created from a branch in a personal fork and are said to be targeting a branch in the original repository the fork was created from. For example Jorn has created a pull request targeting the foreign-jextract branch of the OpenJDK panama-foreign repository from the ErrorCode branch in his personal fork of the panama-foreign repository.
Reviewers can leave comments on a pull request for the author. The author can use the feedback from the reviewers to update the pull request. The pull request is updated by pushing commits to the branch in the author's personal fork that the pull request was created from.
The outcome of a pull request that has been approved by reviewers is a commit on the branch that the pull request is targeting. The pull request has then been integrated.
Overview
With the concepts of personal forks and pull requests the abstract and high-level workflow for a contributor looks like:
- Contributor creates a personal fork of the OpenJDK repository they want to contribute to
- For each change the contributor wants to suggest:
- Contributor creates a branch in their personal fork
- Contributor pushes commit(s) describing suggested change to the above created branch
- Contributor creates a pull request from the above branch in the contributor's personal fork towards a branch in the original OpenJDK repository
- Contributor updates the pull request based on feedback from reviewers (if needed)
- Reviewers approve contributor's pull request
- Contributor integrates the pull request
Note that the contributor only has to create a personal fork once, the same personal fork can be re-used for multiple pull requests. Note also that all steps in the above abstract workflow might not be needed depending on which tools the contributor choose to use to realize this workflow.
Pull Request Commands
Project Skara provides contributors and reviewers with additional pull request commands that enable additional functionality. A pull request command is a comment made to a pull request that starts with a slash ("/"), for example "/integrate", "/csr" or "/sponsor".
/integrate
Syntax: /integrate [<hash>]
The pull request command that all contributors will use is the "/integrate" command that integrates an approved pull request into a repository. This is a example where the Skara workflow differs slightly from the workflow offered by most external Git source code hosting providers - almost all external Git source code hosting providers require that a reviewer/maintainer integrates a pull request into a repository. Skara instead enables the contributor to integrate the pull request with the "/integrate" command, but the contributor can only issue the "/integrate" command once the pull request passes all pre-integration checks (e.g. jcheck).
The "/integrate" command will by default squash all commits in the pull request into one, rebase the resulting commit on top of the target branch and automatically create an appropriate commit message. The squashing of all commits in the pull request enables contributors to update a pull request by simply pushing to the branch in the contributor's personal fork the pull request was created from. The rebasing of the resulting commit enables contributors to simply merge the target branch into the source branch for the pull request whenever changes from the target branch needs to be incorporated (instead of doing complicated rebases). The automatic formatting of the commit message frees contributors from having to consider the details of the commit message format.
/sponsor
/solves
/summary
/contributor
/csr
/test
Tools
A contributor can choose between multiple different tools to achieve a workflow suited to their personal preferences. Some contributors might prefer a close integration with their IDE of choice while others might work on servers not even having a graphical environment. A contributor can also choose to combine multiple different tools to great effect. We can not in detail cover all possible setups, but the most common ones will be covered in the section "Common Setups".
The following sections provides external links for those wishing to setup e.g. desktop applications, mobile apps and/or IDEs to integrate with a Git external source code hosting provider.
Desktop Applications
The following desktop applications has integrations with one or more Git external source code hosting providers:
- SourceTree (macOS, Windows)
- Tower (macOS, Windows)
- GitKraken (GNU/Linux, macOS, Windows)
- GitHub Desktop (Arch Linux, macOS, Windows)
Mobile/Tablet
The following mobile and tablet applications integrates with one or more Git external source code hosting providers:
- GitHub for Mobile (iOS, ipadOS, Android)
- Working Copy (iOS, ipadOS)
- GitPoint (iOS, Android)
- Git Hawk (iOS)
- FastHub (Android)
IDEs
The following integrated development environments (IDEs) integrates with one or more Git external source code hosting providers:
- Eclipse (requires MyLyn connector)
- IntelliJ IDEA (builtin)
- Visual Studio (requires GitHub Extension for Visual Studio plugin)
Text Editors
The following text editors integrates with one or more Git external source code hosting providers:
- Emacs (requires magit plugin and forge plugin)
- VS Code (requires GitHub Pull Requests plugin)
- Atom (requires GitHub for Atom plugin)
CLI
The following command-line interface applications integrates with one or more Git external source code hosting providers:
- hub (FreeBSD, GNU/Linux, macOS, Windows)
- GitHub CLI (GNU/Linux, macOS, Windows)
- Skara (GNU/Linux, macOS, Windows)
Web Browser
The following web applications (web sites) can be used from a web browser:
- GitHub (Firefox, Safari, Chrome, Chromium, Edge)
Common Setups
Git CLI Client + Web Browser
Skara CLI Tools + Mailing Lists
Desktop Application + IDE
Skara CLI Tools
The Skara tooling enables a CLI driven workflow where reviews are made either via the mailing lists or in an web browser using an external Git source code hosting provider's web application. The following CLI tools are currently available as part of project Skara:
git-jcheck - a backwards compatible Git port of jcheck
git-webrev - a backwards compatible Git port of webrev
git-defpath - a backwards compatible Git port of defpath
git-fork - fork a project on an external Git source code hosting provider to your personal space and optionally clone it
- git-sync - sync the personal fork of the project with the current state of the upstream repository
git-pr - interact with pull requests for a project on an external Git source code hosting provider
git-info - show OpenJDK information about commits, e.g. issue links, authors, contributors, etc.
git-token - interact with a Git credential manager for handling personal access tokens
git-translate - translate between Mercurial and Git hashes
git-skara - learn about and update the Skara CLI tools
As mentioned in the "Introduction", all tools support multiple external Git source code hosting providers.
It is not necessary to use the Skara tools to contribute changes. Contributors that prefer to use e.g. desktop applications, web browsers and/or IDEs that integrate with applicable external Git source code hosting providers are free to do so. See the Workflow section for other clients.
Overview
The following steps has to be once (the initial setup):
- Install Git
- Configure Git
- Create a GitHub account
- Install the Skara CLI tooling
- Create a personal access token
The following steps has to be done once per repository you want to contribute to:
The following steps has to be done for each change you want to make:
- Create a local branch
- Make your change
- Create a pull request
- Interact with reviewers
- Update the pull request based on reviewer feedback
- Integrate pull request
Every once in a while you will probably want to:
- Sync your personal fork with the upstream repository
After you have used the tools for a while you will probably want to:
- Configure commonly used Skara options
- Setup useful Git aliases
Installing
To install the Skara tooling, simply clone the Skara repository and include the Skara Git configuration file:
$ git clone http://git.openjdk.java.net/skara $ git config --global include.path "$PWD/skara/skara.gitconfig"
The Skara tooling will build itself the first time you use any of the Skara commands. To check that everything works run git skara help
.
Updating
To update the Skara tooling run git skara update
. This will pull eventual updates and rebuild the tooling if necessary.
Personal Access Token
Some of the Skara tools requires a "Personal Access Token" (PAT) to authenticate against an external Git source code hosting provider's API. These tools include:
- git-fork
- git-pr
- git-token
If you do not intend to use the above tools, then there is no need to set up a PAT and you can skip this section. If you want to make use of the above tools, then please read on.
Git Credential Manager
The first step is to ensure you have a Git credential manager to store your PAT once it has been generated. See the subsections below for how to set up a Git credential manager for you operating system.
Windows
If you installed Git via https://gitforwindows.org/ then you already have a credential manager from Microsoft installed (it is bundled with "Git for Windows"). If you installed Git via some other mechanism, then install https://github.com/Microsoft/Git-Credential-Manager-for-Windows.
macOS
You already have a Git credential manager in Keychain, there is nothing to install or configure.
GNU/Linux
On GNU/Linux the recommended setup is to use libsecret and the "libsecret credential helper" in order to use GNOME Keyring as the Git credential manager. If you are using a desktop environment or distribution without support for GNOME Keyring, please see the "Manual" section.
Fedora
Fedora 29 and 30 (the only two currently supported versions of Fedora) comes with libsecret and GNOME Keyring installed by default. When you install the git package you also get the libsecret credential helper installed. To configure git to use the libsecret credential helper run:
$ git config --global credential.helper /usr/libexec/git-core/git-credential-libsecret
If you want to have a graphical utility to inspect the GNOME Keyring we recommend that you install GNOME Seahorse:
sudo dnf install seahorse
Ubuntu
Ubuntu 19.04 and 18.04.2 (LTS) (the only two currently supported versions of desktop Ubuntu) comes with libsecret and GNOME Keyring installed by default. Unfortunately even if you install the Git package you will not get a binary version of the libsecret credential helper installed (you only get the source). This means you have to compile the libsecret credential helper yourself. This is easy to do, it just requires two extra commands:
$ sudo apt install libsecret-1-dev $ sudo make --directory=/usr/share/doc/git/contrib/credential/libsecret
Once you have compiled the libsecret credential helper you must configure Git to use it:
$ git config --global credential.helper /usr/share/doc/git/contrib/credential/libsecret/git-credential-libsecret
If you want to have a graphical utility to inspect the GNOME Keyring we recommend that you install GNOME Seahorse:
$ sudo apt install seahorse
Manual
If you are using a desktop environment or distribution without support for GNOME Keyring, or if you want to use your own scheme for storing the PAT, then that is also supported. You can store non-sensitive data such as your username and the URL of the Git source code hosting provider in your ~/.gitconfig
file in the "credential" section:
[credential "https://github.com"] username = foobar
For the PAT itself, all Skara tools interacting with an external Git source code hosting provider's API supports the GIT_TOKEN
environment variable. This means that instead of storing your PAT in a Git credential manager you can store it e.g. encrypted on disk using gpg
and then set the GIT_TOKEN
environment variable to the decrypted value when using applicable Skara tools. For example:
GIT_TOKEN=$(gpg --decrypt ~/pat.gpg) git pr list
Creating a Personal Access Token
To create a a Personal Access Token on GitHub go to https://github.com/settings/tokens and and click on "Generate new token". You only need to select the "repo" scope (permission). After you have generated your token, store it in your credential manager using git token store
$ git token store https://github.com Username: <insert your Github username> Password: <insert your "Personal Access Token", not your GitHub password>
Creating a personal fork
The first step in the Skara workflow is to create a personal fork of an existing OpenJDK repository. Your personal fork is a copy of the original OpenJDK repository. Having a personal fork allows you to experiment with changes without affecting the original OpenJDK project. Having a personal fork also enables you to create pull requests targeted towards the original OpenJDK repository.
Creating a personal fork is easy. Either use the GitHub web UI or use use the Skara CLI tooling. The following command will for example create a personal fork of the jdk repository.
$ git fork https://github.com/openjdk/jdk
The above Skara CLI command git fork
will also clone your personal fork to a local repository on your computer. One common way of structuring your local repositories is by following the domainname and path convention. This is not a requirement, but has proven to be a useful way of keeping track of multiple repositories. For an example, see below:
$ tree . ├── git │ └── github.com │ ├── edvbld │ │ ├── jdk │ │ └── loom │ └── openjdk │ ├── jdk │ ├── loom │ └── valhalla └── hg └── hg.openjdk.java.net ├── code-tools │ └── jtreg └── jdk └── jdk
In the above example you can see that I have two personal forks of the openjdk/jdk and openjdk/loom repositories:
For convenience I also have local clones of three upstream OpenJDK repositories: jdk, loom and valhalla.
Creating a local branch
To be able to work on mulitple changes concurrently you will want to create a local Git branch per change. Creating a local branch in Git is very easy:
$ git checkout -b <branch-name>
If you are using Git 2.24 or newer then you can also use the more recent git switch
command which reads a bit more natural:git switch --create <branch-name>
Listing local branches
A common way of structuring your local branches is to name them after the issue they correspond to, for example JDK-8237566 (or just 8237566). After making a few changes you will start to have a couple of local branches. You can list your local branches with git branch
:
$ git branch JDK-8237566 * JDK-8149128 JDK-8077146 master
A perhaps nicer way to visualize your branches is to view them in all a commit graph with the help of git log
:
$ git log --format=oneline --graph --all
Switching branches
You can switch between local branches using:
$ git checkout <branch-name>
If you are using Git 2.24 or newer then you can also use the more recent git switch
command: git switch <branch-name>
Multiple local clones
If you are used to having one local clone of a repository for each change you are working on then that is also supported. The only difference is that you first create a local clone of your personal fork, then create a local branch in that local clone. Fortunately the Skara git fork
command is idempotent - if you already have a personal fork then you will just get that one. Therefore the following commands can be run by those preferring to have one local repository per change they are working on:
$ git fork https://github.com/openjdk/jdk JDK-8237566 Fork available at: https://github.com/edvbld/jdk Cloning https://github.com/edvbld/jdk... Cloning into 'JDK-8237566'... remote: Enumerating objects: 1019523, done. remote: Total 1019523 (delta 0), reused 0 (delta 0), pack-reused 1019523 Receiving objects: 100% (1019523/1019523), 372.46 MiB | 4.99 MiB/s, done. Resolving deltas: 100% (761166/761166), done. Updating files: 100% (68663/68663), done. Adding remote 'upstream' for https://github.com/openjdk/jdk...done $ cd JDK-8237566 $ git checkout -b JDK-8237566 Switched to a new branch 'JDK-8237566'