You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
Casper Beyer 331cd3867c Include tests when running clippy 14 hours ago
.cargo let's not force folks to not use debug info… 1 month ago
.github Include tests when running clippy 13 hours ago
.gov update accounting 14 hours ago
cargo-smart-release Note about smart-release being (too) eager to release 2 days ago
ci add ci/ dir with everything copied from dua 1 year ago
etc refactor (#222) 4 days ago
experiments Release git-hash v0.8.0, git-features v0.17.0, git-actor v0.6.0, git-object v0.15.0, git-diff v0.11.0, git-traverse v0.10.0, git-pack v0.13.0, git-odb v0.23.0, git-packetline v0.12.0, git-transport v0.13.0, git-protocol v0.12.0, git-ref v0.9.0, git-repository v0.11.0, git-commitgraph v0.6.0, gitoxide-core v0.12.0, gitoxide v0.10.0, cargo-smart-release v0.5.0, safety bump 16 crates 4 days ago
git-actor Include tests when running clippy 13 hours ago
git-commitgraph Release git-hash v0.8.0, git-features v0.17.0, git-actor v0.6.0, git-object v0.15.0, git-diff v0.11.0, git-traverse v0.10.0, git-pack v0.13.0, git-odb v0.23.0, git-packetline v0.12.0, git-transport v0.13.0, git-protocol v0.12.0, git-ref v0.9.0, git-repository v0.11.0, git-commitgraph v0.6.0, gitoxide-core v0.12.0, gitoxide v0.10.0, cargo-smart-release v0.5.0, safety bump 16 crates 4 days ago
git-config Remove stale clippy allow 14 hours ago
git-diff Release git-hash v0.8.0, git-features v0.17.0, git-actor v0.6.0, git-object v0.15.0, git-diff v0.11.0, git-traverse v0.10.0, git-pack v0.13.0, git-odb v0.23.0, git-packetline v0.12.0, git-transport v0.13.0, git-protocol v0.12.0, git-ref v0.9.0, git-repository v0.11.0, git-commitgraph v0.6.0, gitoxide-core v0.12.0, gitoxide v0.10.0, cargo-smart-release v0.5.0, safety bump 16 crates 4 days ago
git-features Release git-hash v0.8.0, git-features v0.17.0, git-actor v0.6.0, git-object v0.15.0, git-diff v0.11.0, git-traverse v0.10.0, git-pack v0.13.0, git-odb v0.23.0, git-packetline v0.12.0, git-transport v0.13.0, git-protocol v0.12.0, git-ref v0.9.0, git-repository v0.11.0, git-commitgraph v0.6.0, gitoxide-core v0.12.0, gitoxide v0.10.0, cargo-smart-release v0.5.0, safety bump 16 crates 4 days ago
git-hash Release git-hash v0.8.0, git-features v0.17.0, git-actor v0.6.0, git-object v0.15.0, git-diff v0.11.0, git-traverse v0.10.0, git-pack v0.13.0, git-odb v0.23.0, git-packetline v0.12.0, git-transport v0.13.0, git-protocol v0.12.0, git-ref v0.9.0, git-repository v0.11.0, git-commitgraph v0.6.0, gitoxide-core v0.12.0, gitoxide v0.10.0, cargo-smart-release v0.5.0, safety bump 16 crates 4 days ago
git-index remove dash in all repository links 1 year ago
git-lock Release git-hash v0.7.0, git-features v0.16.5, git-actor v0.5.3, git-config v0.1.7, git-validate v0.5.3, git-object v0.14.1, git-diff v0.10.0, git-tempfile v1.0.3, git-lock v1.0.1, git-traverse v0.9.0, git-pack v0.12.0, git-odb v0.22.0, git-packetline v0.11.0, git-url v0.3.4, git-transport v0.12.0, git-protocol v0.11.0, git-ref v0.8.0, git-repository v0.10.0, cargo-smart-release v0.4.0 1 week ago
git-object Release git-hash v0.8.0, git-features v0.17.0, git-actor v0.6.0, git-object v0.15.0, git-diff v0.11.0, git-traverse v0.10.0, git-pack v0.13.0, git-odb v0.23.0, git-packetline v0.12.0, git-transport v0.13.0, git-protocol v0.12.0, git-ref v0.9.0, git-repository v0.11.0, git-commitgraph v0.6.0, gitoxide-core v0.12.0, gitoxide v0.10.0, cargo-smart-release v0.5.0, safety bump 16 crates 4 days ago
git-odb Release git-hash v0.8.0, git-features v0.17.0, git-actor v0.6.0, git-object v0.15.0, git-diff v0.11.0, git-traverse v0.10.0, git-pack v0.13.0, git-odb v0.23.0, git-packetline v0.12.0, git-transport v0.13.0, git-protocol v0.12.0, git-ref v0.9.0, git-repository v0.11.0, git-commitgraph v0.6.0, gitoxide-core v0.12.0, gitoxide v0.10.0, cargo-smart-release v0.5.0, safety bump 16 crates 4 days ago
git-pack Release git-hash v0.8.0, git-features v0.17.0, git-actor v0.6.0, git-object v0.15.0, git-diff v0.11.0, git-traverse v0.10.0, git-pack v0.13.0, git-odb v0.23.0, git-packetline v0.12.0, git-transport v0.13.0, git-protocol v0.12.0, git-ref v0.9.0, git-repository v0.11.0, git-commitgraph v0.6.0, gitoxide-core v0.12.0, gitoxide v0.10.0, cargo-smart-release v0.5.0, safety bump 16 crates 4 days ago
git-packetline Release git-hash v0.8.0, git-features v0.17.0, git-actor v0.6.0, git-object v0.15.0, git-diff v0.11.0, git-traverse v0.10.0, git-pack v0.13.0, git-odb v0.23.0, git-packetline v0.12.0, git-transport v0.13.0, git-protocol v0.12.0, git-ref v0.9.0, git-repository v0.11.0, git-commitgraph v0.6.0, gitoxide-core v0.12.0, gitoxide v0.10.0, cargo-smart-release v0.5.0, safety bump 16 crates 4 days ago
git-protocol Release git-hash v0.8.0, git-features v0.17.0, git-actor v0.6.0, git-object v0.15.0, git-diff v0.11.0, git-traverse v0.10.0, git-pack v0.13.0, git-odb v0.23.0, git-packetline v0.12.0, git-transport v0.13.0, git-protocol v0.12.0, git-ref v0.9.0, git-repository v0.11.0, git-commitgraph v0.6.0, gitoxide-core v0.12.0, gitoxide v0.10.0, cargo-smart-release v0.5.0, safety bump 16 crates 4 days ago
git-ref Release git-hash v0.8.0, git-features v0.17.0, git-actor v0.6.0, git-object v0.15.0, git-diff v0.11.0, git-traverse v0.10.0, git-pack v0.13.0, git-odb v0.23.0, git-packetline v0.12.0, git-transport v0.13.0, git-protocol v0.12.0, git-ref v0.9.0, git-repository v0.11.0, git-commitgraph v0.6.0, gitoxide-core v0.12.0, gitoxide v0.10.0, cargo-smart-release v0.5.0, safety bump 16 crates 4 days ago
git-repository Release git-hash v0.8.0, git-features v0.17.0, git-actor v0.6.0, git-object v0.15.0, git-diff v0.11.0, git-traverse v0.10.0, git-pack v0.13.0, git-odb v0.23.0, git-packetline v0.12.0, git-transport v0.13.0, git-protocol v0.12.0, git-ref v0.9.0, git-repository v0.11.0, git-commitgraph v0.6.0, gitoxide-core v0.12.0, gitoxide v0.10.0, cargo-smart-release v0.5.0, safety bump 16 crates 4 days ago
git-tempfile Release git-hash v0.7.0, git-features v0.16.5, git-actor v0.5.3, git-config v0.1.7, git-validate v0.5.3, git-object v0.14.1, git-diff v0.10.0, git-tempfile v1.0.3, git-lock v1.0.1, git-traverse v0.9.0, git-pack v0.12.0, git-odb v0.22.0, git-packetline v0.11.0, git-url v0.3.4, git-transport v0.12.0, git-protocol v0.11.0, git-ref v0.8.0, git-repository v0.10.0, cargo-smart-release v0.4.0 1 week ago
git-transport Fix false positive clippy warning. 2 days ago
git-traverse Release git-hash v0.8.0, git-features v0.17.0, git-actor v0.6.0, git-object v0.15.0, git-diff v0.11.0, git-traverse v0.10.0, git-pack v0.13.0, git-odb v0.23.0, git-packetline v0.12.0, git-transport v0.13.0, git-protocol v0.12.0, git-ref v0.9.0, git-repository v0.11.0, git-commitgraph v0.6.0, gitoxide-core v0.12.0, gitoxide v0.10.0, cargo-smart-release v0.5.0, safety bump 16 crates 4 days ago
git-tui remove dash in all repository links 1 year ago
git-url Release git-hash v0.7.0, git-features v0.16.5, git-actor v0.5.3, git-config v0.1.7, git-validate v0.5.3, git-object v0.14.1, git-diff v0.10.0, git-tempfile v1.0.3, git-lock v1.0.1, git-traverse v0.9.0, git-pack v0.12.0, git-odb v0.22.0, git-packetline v0.11.0, git-url v0.3.4, git-transport v0.12.0, git-protocol v0.11.0, git-ref v0.8.0, git-repository v0.10.0, cargo-smart-release v0.4.0 1 week ago
git-validate Release git-hash v0.7.0, git-features v0.16.5, git-actor v0.5.3, git-config v0.1.7, git-validate v0.5.3, git-object v0.14.1, git-diff v0.10.0, git-tempfile v1.0.3, git-lock v1.0.1, git-traverse v0.9.0, git-pack v0.12.0, git-odb v0.22.0, git-packetline v0.11.0, git-url v0.3.4, git-transport v0.12.0, git-protocol v0.11.0, git-ref v0.8.0, git-repository v0.10.0, cargo-smart-release v0.4.0 1 week ago
gitoxide-core Release git-hash v0.8.0, git-features v0.17.0, git-actor v0.6.0, git-object v0.15.0, git-diff v0.11.0, git-traverse v0.10.0, git-pack v0.13.0, git-odb v0.23.0, git-packetline v0.12.0, git-transport v0.13.0, git-protocol v0.12.0, git-ref v0.9.0, git-repository v0.11.0, git-commitgraph v0.6.0, gitoxide-core v0.12.0, gitoxide v0.10.0, cargo-smart-release v0.5.0, safety bump 16 crates 4 days ago
src upgrade to clap 3 beta 5 (#222) 5 days ago
tests Release git-hash v0.8.0, git-features v0.17.0, git-actor v0.6.0, git-object v0.15.0, git-diff v0.11.0, git-traverse v0.10.0, git-pack v0.13.0, git-odb v0.23.0, git-packetline v0.12.0, git-transport v0.13.0, git-protocol v0.12.0, git-ref v0.9.0, git-repository v0.11.0, git-commitgraph v0.6.0, gitoxide-core v0.12.0, gitoxide v0.10.0, cargo-smart-release v0.5.0, safety bump 16 crates 4 days ago
.editorconfig Initial commit - based on standard project template 3 years ago
.gitignore Fix includes to assure generated fixtures aren't can't be added 6 months ago
CHANGELOG.md Release git-hash v0.8.0, git-features v0.17.0, git-actor v0.6.0, git-object v0.15.0, git-diff v0.11.0, git-traverse v0.10.0, git-pack v0.13.0, git-odb v0.23.0, git-packetline v0.12.0, git-transport v0.13.0, git-protocol v0.12.0, git-ref v0.9.0, git-repository v0.11.0, git-commitgraph v0.6.0, gitoxide-core v0.12.0, gitoxide v0.10.0, cargo-smart-release v0.5.0, safety bump 16 crates 4 days ago
COLLABORATING.md A note about the project board to help with transparency 2 months ago
CONTRIBUTING.md Fix contributions link 6 months ago
Cargo.lock First pass 2 days ago
Cargo.toml Release git-hash v0.8.0, git-features v0.17.0, git-actor v0.6.0, git-object v0.15.0, git-diff v0.11.0, git-traverse v0.10.0, git-pack v0.13.0, git-odb v0.23.0, git-packetline v0.12.0, git-transport v0.13.0, git-protocol v0.12.0, git-ref v0.9.0, git-repository v0.11.0, git-commitgraph v0.6.0, gitoxide-core v0.12.0, gitoxide v0.10.0, cargo-smart-release v0.5.0, safety bump 16 crates 4 days ago
DEVELOPMENT.md [stability #164] First sketch of stability MD… 2 months ago
LICENSE-APACHE [stability #171] Prime git-tempfile and git-lock for release 2 months ago
LICENSE-MIT [stability #171] Prime git-tempfile and git-lock for release 2 months ago
Makefile feat!: rename --skip-* flags to --no-* for consistency (#198) 4 weeks ago
README.md note about magit for a better tui, some day 14 hours ago
STABILITY.md [repository #193] Lift git-features to ST1 2 months ago
cargo-features.md feat: dynamically sized full-object speeds up diff-based object counting… (#67) 1 month ago
crate-status.md Only use short hashes for logs, without detecting ambiguity for now (#198) 3 weeks ago
deny.toml [#189] remove special handling of time from deny.toml 2 months ago
general-tasks.md Fix formatting of performance-tasks.md 2 months ago
rust-toolchain.tml [stability #171] How to handle the MSRV 2 months ago
rustfmt-nightly.toml Add `make rustfmt` target. 2 months ago
rustfmt.toml Allow for more screen space when formatting 1 year ago
tasks.md update tasks list 3 months ago

README.md

Rust Crates.io

gix is a command-line interface (CLI) to access git repositories. It's written to optimize the user-experience, and perform as good or better than the canonical implementation.

Furthermore it provides an easy and safe to use API in the form of various small crates for implementing your own tools in a breeze. Please see 'Development Status' for a listing of all crates and their capabilities.

asciicast

Development Status

gitoxide (CLI)

  • please note that all functionality comes from the gitoxide-core library, which mirrors these capabilities and itself relies on all git-* crates.
  • limit amount of threads used in operations that support it.
  • choose between 'human' and 'json' output formats
  • the gix program - convenient and for humans
    • init - initialize a new non-bare repository with a main branch
    • clone - initialize a local copy of a remote repository
    • tools
      • organize - find all git repositories and place them in directories according to their remote paths
      • find - find all git repositories in a given directory - useful for tools like skim
      • estimate-hours - estimate the time invested into a repository by evaluating commit dates.
  • the gixp program (plumbing) - lower level commands for use in automation
    • pack
      • verify
      • index verify including each object sha1 and statistics
      • explode, useful for transforming packs into loose objects for inspection or restoration
        • verify written objects (by reading them back from disk)
      • receive - receive a whole pack produced by pack-send or git-upload-pack, useful for clone like operations.
      • create - create a pack from given objects or tips of the commit graph.
      • send - create a pack and send it using the pack protocol to stdout, similar to 'git-upload-pack', for consumption by pack-receive or git-receive-pack
      • index from data - create an index file by streaming a pack file as done during clone
        • support for thin packs (as needed for fetch/pull)
    • commit-graph
      • verify - assure that a commit-graph is consistent
    • remote-ref-list
      • list all (or given) references from a remote at the given URL

Crates

Follow linked crate name for detailed status. Please note that all crates follow [semver] as well as the stability guide.

Production Grade

In Development/Pre-Release

Stress Testing

  • Verify huge packs
  • Explode a pack to disk
  • Generate and verify large commit graphs
  • Generate huge pack from a lot of loose objects

Cargo features

Many crates use feature flags to allow tuning the compiled result based on your needs. Have a look at the guide for more information.

Stability

Our stability guide helps to judge how much churn can be expected when depending on crates in this workspace.

Installation

Download a Binary Release

curl -LSfs https://raw.githubusercontent.com/Byron/gitoxide/main/ci/install.sh | \
    sh -s -- --git Byron/gitoxide --crate gix-max-termion

See the releases section for manual installation and various alternative builds that are slimmer or smaller, depending on your needs, for Linux, MacOS and Windows.

From Source via Cargo

cargo is the Rust package manager which can easily be obtained through rustup. With it, you can build your own binary effortlessly and for your particular CPU for additional performance gains.

The minimum required cargo version is the latest stable release, but may run on older stable releases as well.

# The default installation, 'max'
cargo install gitoxide

# On linux, it's a little faster to compile the termion version, which also results in slightly smaller binaries
cargo install gitoxide --no-default-features --features max-termion

# For smaller binaries and even faster build times that are traded for a less fancy CLI implementation, use `lean`
# or `lean-termion` respectively.
cargo install gitoxide --no-default-features --features lean

Usage

Once installed, there are two binaries:

  • gix
    • high level commands, porcelain, for every-day use, optimized for a pleasant user experience
  • gixp
    • low level commands, plumbing, for use in more specialized cases

Project Goals

Project goals can change over time as we learn more, and they can be challenged.

  • a pure-rust implementation of git
    • including transport, object database, references, cli and tui
    • a simple command-line interface is provided for the most common git operations, optimized for user experience. A simple-git if you so will.
    • be the go-to implementation for anyone who wants to solve problems around git, and become the alternative to GitPython in the process.
    • become the foundation for a free distributed alternative to GitHub, and maybe even GitHub itself
  • learn from the best to write the best possible idiomatic Rust
    • libgit2 is a fantastic resource to see what abstractions work, we will use them
    • use Rust's type system to make misuse impossible
  • be the best performing implementation
    • use Rust's type system to optimize for work not done without being hard to use
    • make use of parallelism from the get go
  • assure on-disk consistency
    • assure reads never interfere with concurrent writes
    • assure multiple concurrent writes don't cause trouble
  • take shortcuts, but not in quality
    • binaries may use anyhow::Error exhaustively, knowing these errors are solely user-facing.
    • libraries use light-weight custom errors implemented using quick-error or thiserror.
    • internationalization is nothing we are concerned with right now.
    • IO errors due to insufficient amount of open file handles don't always lead to operation failure
  • Cross platform support, including Windows
    • With the tools and experience available here there is no reason not to support Windows.
    • Windows is tested on CI and failures do prevent releases.

Non-Goals

Project non-goals can change over time as we learn more, and they can be challenged.

  • replicate git command functionality perfectly
    • git is git, and there is no reason to not use it. Our path is the one of simplicity to make getting started with git easy.
  • be incompatible to git
    • the on-disk format must remain compatible, and we will never contend with it.
  • use async IO everywhere
    • for the most part, git operations are heavily relying on memory mapped IO as well as CPU to decompress data, which doesn't lend itself well to async IO out of the box.
    • Use blocking as well as git-features::interrupt to bring operations into the async world and to control long running operations.
    • When connecting or streaming over TCP connections, especially when receiving on the server, async seems like a must though, but behind a feature flag.

Contributions

If what you have seen so far sparked your interest to contribute, then let us say: We are happy to have you and help you to get started.

We recommend running make tests check-size during the development process to assure CI is green before pushing.

A backlog for work ready to be picked up is available in the Project's Kanban board, which contains instructions on how to pick a task. If it's empty or you have other questions, feel free to start a discussion or reach out to @Byron privately.

For additional details, also take a look at the collaboration guide.

Roadmap

Features for 1.0

Provide a CLI to for the most basic user journey:

  • initialize a repository
  • clone a repository
  • create a commit
  • add a remote
  • push
    • create (thin) pack

Ideas for Examples

  • gix tool open-remote open the URL of the remote, possibly after applying known transformations to go from ssh to https.
  • Open up SQL for git using sqlite virtual tables. Check out gitqlite as well. What would an MVP look like? Maybe even something that could ship with gitoxide.
  • A truly awesome history rewriter which makes it easy to understand what happened while avoiding all pitfalls. Think BFG, but more awesome, if that's possible.
  • git-tui should learn a lot from fossil-scm regarding the presentation of data. Maybe this can be used for prompts. Probably magit has a lot to offer, too.
  • Can markdown be used as database so issue-trackers along with meta-data could just be markdown files which are mostly human-editable? Could user interfaces be meta-data aware and just hide the meta-data chunks which are now editable in the GUI itself? Doing this would make conflicts easier to resolve than an sqlite database.
    • A git-backend for sqlite which should allow embedding sqlite databases into git repositories, which in turn can be used for bug-trackers, wikis or other features, making for a fully distributed github like experience, maybe.

Ideas for Spin-Offs

  • A system to integrate tightly with git-lfs to allow a multi-tier architecture so that assets can be stored in git and are accessible quickly from an intranet location (for example by accessing the storage read-only over the network) while changes are pushed immediately by the server to other edge locations, like the cloud or backups. Sparse checkouts along with explorer/finder integrations make it convenient to only work on a small subset of files locally. Clones can contain all configuration somebody would need to work efficiently from their location, and authentication for the git history as well as LFS resources make the system secure. One could imagine encryption support for untrusted locations in the cloud even though more research would have to be done to make it truly secure.
  • A syncthing like client/server application. This is to demonstrate how lower-level crates can be combined into custom applications that use only part of git's technology to achieve their very own thing. Watch out for big file support, multi-device cross-syncing, the possibility for untrusted destinations using full-encryption, case-insensitive and sensitive filesystems, and extended file attributes as well as ignore files.

Shortcomings & Limitations

  • fetches using protocol V1 and stateful connections, i.e. ssh, git, file, may hang
    • This can be fixed by making response parsing.
    • Note that this does not affect cloning, which works fine.
  • lean and light and small builds don't support non-UTF-8 paths in the CLI
    • This is because they depend on argh, which does not yet support parsing OsStrings. We however believe it eventually will do so and thus don't move on to pico-args.
    • Only one level of sub-commands are supported due to a limitation of argh, which forces porcelain to limit itself as well despite using clap. We deem this acceptable for plumbing commands and think that porcelain will be high-level and smart enough to not ever require deeply nested sub-commands.
  • Packfiles use memory maps
    • Even though they are comfortable to use and fast, they squelch IO errors.
    • potential remedy: We could generalize the Pack to make it possible to work on in-memory buffers directly. That way, one would initialize a Pack by reading the whole file into memory, thus not squelching IO errors at the expense of latency as well as memory efficiency.
  • Packfiles cannot load files bigger than 2^31 or 2^32 on 32 bit systems
    • As these systems cannot address more memory than that.
    • potential remedy: implement a sliding window to map and unmap portions of the file as needed.
      • However, those who need to access big packs on these systems would rather resort to git itself, allowing our implementation to be simpler and potentially more performant.
  • Objects larger than 32 bits cannot be loaded on 32 bit systems
    • in-memory representations objects cannot handle objects greater than the amount of addressable memory.
    • This should not affect git LFS though.
  • git-url might be more restrictive than what git allows as for the most part, it uses a browser grade URL parser.
    • Thus far there is no proof for this, and as potential remedy we could certainly re-implement exactly what git does to handle its URLs.
  • local time is currently impeded by this issue but it's planned to resolve it eventually.

Credits

  • itertools (MIT Licensed)
    • We use the izip! macro in code
  • deflate2 (MIT Licensed)
    • We use various abstractions to implement decompression and compression directly on top of the rather low-level miniz_oxide crate

License

This project is licensed under either of

at your option.

Fun facts

  • Originally @Byron was really fascinated by this problem and believes that with gitoxide it will be possible to provide the fastest solution for it.
  • @Byron has been absolutely blown away by git from the first time he experienced git more than 13 years ago, and tried to implement it in various shapes and forms multiple times. Now with Rust @Byron finally feels to have found the right tool for the job!