Project license detector - a command line application and a library, written in Go. It scans the given directory for license files, normalizes and hashes them and outputs all the fuzzy matches with the list of reference texts. The returned names follow SPDX standard. Read the blog post.
Why? There are no similar projects which can be compiled into a native binary without dependencies and also support the whole SPDX license database (≈400 items). This implementation is also fast, requires little memory, and the API is easy to use.
The license texts are taken directly from license-list-data repository. The detection algorithm is not template matching; this directly implies that go-license-detector does not provide any legal guarantees. The intended area of it's usage is data mining.
go get github.com/go-enry/go-license-detector/v4/licensedb
The CLI is available for download at the release page.
LICENSE
or license.md
.D
.
the corresponding text is the single line with each unigram represented by a single rune (character).1 - D / L
where L
is the number of unigrams in the quieried license.This pipeline guarantees constant time queries, though requires some initialization to preprocess the reference licenses.
If there are not license files found:
Command line:
license-detector /path/to/project
license-detector https://github.com/go-git/go-git
Library (for a single license detection):
import (
"github.com/go-enry/go-license-detector/v4/licensedb"
"github.com/go-enry/go-license-detector/v4/licensedb/filer"
)
func main() {
licenses, err := licensedb.Detect(filer.FromDirectory("/path/to/project"))
}
Library (for a convenient data structure that can be formatted as JSON):
import (
"encoding/json"
"fmt"
"github.com/go-enry/go-license-detector/v4/licensedb"
)
func main() {
results := licensedb.Analyse("/path/to/project1", "/path/to/project2")
bytes, err := json.MarshalIndent(results, "", "\t")
if err != nil {
fmt.Printf("could not encode result to JSON: %v\n", err)
}
fmt.Println(string(bytes))
}
On the dataset of ~1000 most starred repositories on GitHub as of early February 2018 (list), 99% of the licenses are detected. The analysis of detection failures is going in FAILURES.md.
Comparison to other projects on that dataset:
Detector | Detection rate | Time to scan, sec |
---|---|---|
go-license-detector | 99% (897/902) | 13.5 |
benbalter/licensee | 75% (673/902) | 111 |
google/licenseclassifier | 76% (682/902) | 907 |
boyter/lc | 88% (797/902) | 548 |
amzn/askalono | 87% (785/902) | 165 |
LiD | 94% (847/902) | 3660 |
$ cd $(go env GOPATH)/src/github.com/go-enry/go-license-detector/v4/licensedb
$ mkdir dataset && cd dataset
$ unzip ../dataset.zip
$ # go-enry/go-license-detector
$ time license-detector * \
| grep -Pzo '\n[-0-9a-zA-Z]+\n\tno license' | grep -Pa '\tno ' | wc -l
$ # benbalter/licensee
$ time ls -1 | xargs -n1 -P4 licensee \
| grep -E "^License: Other" | wc -l
$ # google/licenseclassifier
$ time find -type f -print | xargs -n1 -P4 identify_license \
| cut -d/ -f2 | sort | uniq | wc -l
$ # boyter/lc
$ time lc . \
| grep -vE 'NOASSERTION|----|Directory' | cut -d" " -f1 | sort | uniq | wc -l
$ # amzn/askalono
$ echo '#!/bin/sh
result=$(askalono id "$1")
echo "$1
$result"' > ../askalono.wrapper
$ time find -type f -print | xargs -n1 -P4 sh ../askalono.wrapper | grep -Pzo '.*\nLicense: .*\n' askalono.txt | grep -av "License: " | cut -d/ -f 2 | sort | uniq | wc -l
$ # LiD
$ time license-identifier -I dataset -F csv -O lid
$ cat lid_*.csv | cut -d, -f1 | cut -d"'" -f 2 | grep / | cut -d/ -f2 | sort | uniq | wc -l
The SPDX licenses are included into the binary. To update them, run
# go install github.com/go-bindata/go-bindata/...
make licensedb/internal/assets/bindata.go
...are welcome, see CONTRIBUTING.md and code of conduct.
Apache 2.0, see LICENSE.md.
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。