Executes the RDF and SPARQL test suites.
npm install rdf-test-suite
This tool executes the RDF
and SPARQL test suites
with any given system.
It can either output human-readable test results to the console,
or it can output machine-readable reports in the
EARL vocabulary.
Currently, the following test suites are supported:
* SPARQL 1.0
* SPARQL 1.1
* RDF/XML
* N-Triples
* N-Quads
* Turtle
* TriG
* JSON-LD (1.0 and 1.1)
* RDFa
* Microdata to RDF
* Notation3
* RDF-star and SPARQL-star
* JSON-LD-Star
Not all RDF test suites are supported at the moment.
However, this package is fully _modular_,
so that support for different test cases can be implemented easily.
Planned support:
* SPARQL 1.1 (_other specifications_)
* RDF Schema and Semantics
* JSON-LD (_other test suites_)
Either install it globally:
``bash`
$ yarn global add rdf-test-suite
Or locally (as a dev dependency):
`bash`
$ yarn add --dev rdf-test-suite
After installing, the rdf-test-suite script will become available.
This script requires some kind of _engine_ as first argument,
and a test suite manifest URL as second argument.
The engine is a JavaScript file that can handle the test suite.
The interface of this engine depends on the manifest.
For example, the SPARQL 1.1 test suite requires an
IQueryEngine,
while the RDF/XML test suite requires a
IParser.
The following command executes the SPARQL 1.1 test suite
on the engine myengine.js:
`bash`
$ rdf-test-suite myengine.js https://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl
An example of what this myengine.js looks like
is available for the Comunica SPARQL engine.
This command will output something like this:
`
...
✔ syn-bad-pname-04 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_04) 0.293588ms
✔ syn-bad-pname-05 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_05) 0.498646ms
✔ syn-bad-pname-06 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_06) 0.513092ms
✔ syn-bad-pname-07 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_07) 0.498646ms
✔ syn-bad-pname-08 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_08) 0.293588ms
✔ syn-bad-pname-09 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_09) 0.293588ms
✔ syn-bad-pname-10 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_10) 0.516346ms
✔ syn-bad-pname-11 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_11) 0.498646ms
✔ syn-bad-pname-12 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_12) 0.516346ms
✔ syn-bad-pname-13 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_13) 0.498646ms
✔ syn-pp-in-collection (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pp_coll) 0.516346ms
✖ 105 / 268 tests succeeded!
With the -t option, you can filter tests that should be executed
based on a regex that will be matched with the test URI.
For example:
`bash`
$ rdf-test-suite myengine.js https://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
-t test_pn_bad_0
With the --skip option you can filter tests that should be skipped
based on a regex that will be matched with the test URI.
For example:
`bash`
$ rdf-test-suite myengine.js https://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
--skip test_pn_bad_0bash`
$ rdf-test-suite myengine.js https://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
--skip (test_pn_bad_0|test_pn_bad_1)
By default, the printed test results are verbose,
and print expanded views of failed tests.
You can change this to a more compact view using -o summary.
For example:
`bash`
$ rdf-test-suite myengine.js https://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
-o summary
Test results are human readable by default.
This can be changed to output machine-readable reports in the
EARL vocabulary
in the Turtle serialization.
These reports can be published to report the compliance of engines
to certain specifications, like on
W3C’s RDF Test Curation Community Group.
As EARL reports require some metadata on your engine,
you will need to provide a properties file via the -p argument.
This properties file can look something like this:
`json`
{
"applicationBugsUrl": "https://github.com/comunica/comunica/issues",
"applicationDescription": "A Comunica engine for SPARQL query evaluation over heterogeneous interfaces",
"applicationHomepageUrl": "http://comunica.linkeddatafragments.org/",
"applicationNameFull": "Comunica SPARQL",
"applicationNameNpm": "@comunica/actor-init-sparql",
"applicationUri": "https://www.npmjs.com/package/@comunica/actor-init-sparql",
"authors": [
{
"homepage": "https://www.rubensworks.net/",
"name": "Ruben Taelman",
"uri": "https://www.rubensworks.net/#me"
}
],
"licenseUri": "http://opensource.org/licenses/MIT",
"reportUri": null,
"specificationUris": [
"http://www.w3.org/TR/sparql11-query/"
],
"version": "1.2.3"
}
The properties file can point to a non-existing file,
in which case it will be auto-generated from the package.json
file in the current working directory.
For example:
`bash`
$ rdf-test-suite myengine.js https://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
-o earl -p earl-meta.json
Some test suites (like the one of SPARQL 1.1) contains
tests for multiple specifications.
If you only want to test your system for a single specification,
then you can define this with the -s parameter.
For example, the following command executes the SPARQL 1.1 test cases
that apply to the http://www.w3.org/TR/sparql11-query/
specification.
`bash`
$ rdf-test-suite myengine.js https://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
-s http://www.w3.org/TR/sparql11-query/
By default, rdf-test-suite will look up all required manifest files.-c
If this takes too much time, or if you don't have internet connection,
then you can enable HTTP caching with the argument,
so that all files only have to be looked up once.
`bash`
$ rdf-test-suite myengine.js https://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
-c path/to/cache/
_If you don't provide a caching value after the ,.rdf-test-suite-cache/
then the directory will default to ._
When there are failing tests,
rdf-test-suite will exit with code 1 instead of 0.
This can be useful in Continuous Integration tools.
If you want to disable this behaviour,
you can add the -e flag to force an exit code 0.
`bash`
$ rdf-test-suite myengine.js https://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
-e
With the optional -i option,IQueryEngine
a JSON string with arguments can be passed to the engine.
These arguments will become available in
both ,IParser
and .
`bash`
$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
-i '{ "myProperty": "myValue" }'
By default, all tests are allowed to run 3000 ms.
Using the -d option, you can change this value.
`bash`
$ rdf-test-suite myengine.js https://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
-d 5000
In cases where you are developing your own test manifests,
it may be useful to intercept certain URL lookups
and return local files instead.
In order to achieve this, the -m option can be used,URL~PATH
with a mapping defined with the following pattern: .
`
$ rdf-test-suite myengine.js https://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
-m https://w3c.github.io/rdf-tests/sparql11/data-sparql11/~/path/to/my/files/
In cases where you wish to execute a test suite that contains
rdft:Rejected test entries rdf-test-suite will skip those tests-r
by default. If you wish to execute those tests as well, you can
use the option.
`bash`
$ rdf-test-suite myengine.js https://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
-r
In cases where you wish to execute a test suite that contains
rdft:Approved test entries rdf-test-suite will, by default-a
execute those tests as well as all other tests that are not
explicitly rejected. If you wish to only execute those tests
that are explicitly approved, you can use the option.
`bash`
$ rdf-test-suite myengine.js https://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
-a
| Manifest | Specification | Interface | Entry manifest |
| -------- | ------------- | --------- | -------------- |
| SPARQL 1.0 tests | SPARQL 1.0 | IQueryEngine | https://w3c.github.io/rdf-tests/sparql/sparql10/manifest.ttl |
| SPARQL 1.1 tests | SPARQL 1.1 Query | IQueryEngine | http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl |
| SPARQL 1.1 tests | SPARQL 1.1 Update | IUpdateEngine | http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl |
| SPARQL 1.1 tests | SPARQL 1.1 Results CSV/TSV | ✖ | http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl |
| SPARQL 1.1 tests | SPARQL 1.1 Results JSON | ✖ | http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl |
| SPARQL 1.1 tests | SPARQL 1.1 Federated Query | ✖ | http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl |
| SPARQL 1.1 tests | SPARQL 1.1 Entailment | ✖ | http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl |
| SPARQL 1.1 tests | SPARQL 1.1 Service Description | ✖ | http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl |
| SPARQL 1.1 tests | SPARQL 1.1 Protocol | ✖ | http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl |
| SPARQL 1.1 tests | SPARQL 1.1 HTTP RDF Update | ✖ | http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl |
| RDF/XML Syntax Tests | RDF 1.1 XML Syntax | IParser | https://w3c.github.io/rdf-tests/rdf/rdf11/rdf-xml/manifest.ttl |
| N-Triples Tests | RDF 1.1 N-Triples | IParser | https://w3c.github.io/rdf-tests/rdf/rdf11/rdf-n-triples/manifest.ttl |
| N-Quads Tests | RDF 1.1 N-Quads | IParser | https://w3c.github.io/rdf-tests/rdf/rdf11/rdf-n-quads/manifest.ttl |
| Turtle Tests | RDF 1.1 Turtle | IParser | https://w3c.github.io/rdf-tests/rdf/rdf11/rdf-turtle/manifest.ttl |
| TriG Tests | RDF 1.1 TriG | IParser | https://w3c.github.io/rdf-tests/rdf/rdf11/rdf-trig/manifest.ttl |
| JSON-LD Test Suite | JSON-LD (1.0 and 1.1) | IParser | https://w3c.github.io/json-ld-api/tests/toRdf-manifest.jsonld |
| JSON-LD Test Suite | JSON-LD (1.0 and 1.1) | IParser | https://w3c.github.io/json-ld-api/tests/html-manifest.jsonld |
| JSON-LD Test Suite | JSON-LD (1.0 and 1.1) | ISerializer | https://w3c.github.io/json-ld-api/tests/fromRdf-manifest.jsonld |
| JSON-LD-Star Test Suite | JSON-LD-Star | IParser | https://json-ld.github.io/json-ld-star/tests/toRdf-manifest.jsonld |
| JSON-LD-Star Test Suite | JSON-LD-Star | ISerializer | https://json-ld.github.io/json-ld-star/tests/fromRdf-manifest.jsonld |
| RDFa Test Suite | RDFa 1.1 | IParser | http://rdfa.info/test-suite/test-cases/rdfa1.1/html5/manifest.ttl |
| Microdata to RDF Test Suite | Microdata to RDF | IParser | https://w3c.github.io/microdata-rdf/tests/manifest.ttl |
| Notation3 Test Suite | N3 Grammar | IParser | https://w3c.github.io/N3/tests/N3Tests/manifest-parser.ttl |
| Notation3 Test Suite | Extended N3 Grammar | IParser` | https://w3c.github.io/N3/tests/N3Tests/manifest-extended.ttl |
| Turtle-star Evaluation Tests | RDF-star and SPARQL-star | ✖ | https://w3c.github.io/rdf-star/tests/turtle/eval/manifest.jsonld |
| Turtle-star Syntax Tests | RDF-star and SPARQL-star | ✖ | https://w3c.github.io/rdf-star/tests/turtle/syntax/manifest.jsonld |
| TriG-star Evaluation Tests | RDF-star and SPARQL-star | ✖ | https://w3c.github.io/rdf-star/tests/trig/eval/manifest.jsonld |
| TriG-star Syntax Tests | RDF-star and SPARQL-star | ✖ | https://w3c.github.io/rdf-star/tests/trig/syntax/manifest.jsonld |
| N-Triples-star Syntax Tests | RDF-star and SPARQL-star | ✖ | https://w3c.github.io/rdf-star/tests/nt/syntax/manifest.jsonld |
| RDF-star Semantics tests | RDF-star and SPARQL-star | ✖ | https://w3c.github.io/rdf-star/tests/semantics/manifest.jsonld |
| SPARQL-star Syntax Tests | RDF-star and SPARQL-star | ✖ | https://w3c.github.io/rdf-star/tests/sparql/syntax/manifest.jsonld |
| SPARQL-star Evaluation Tests | RDF-star and SPARQL-star | ✖ | https://w3c.github.io/rdf-star/tests/sparql/eval/manifest.jsonld |
This code is released under the MIT license.