Run multiple postman collections along with predefined configs using single feed file. Reduces commandline arguments since reporting (allure, json, cli and html) is embeded internally
npm install newman-run
newman run
newman run -e
newman run -e
...
`
Also, if we need to have multiple reports along with this run, we need to specify those each of these command line run along with the folder in which the reports should be generated as,
`
newman run --reporters cli,html,allure --reporter-html-export htmlOutput.html --reporter-allure-export
newman run -e --reporters cli,html,allure --reporter-html-export htmlOutput.html --reporter-allure-export
newman run -e --reporters cli,html,allure --reporter-html-export htmlOutput.html --reporter-allure-export
`
So, as we can see the params that we are passing in the commandline keeps on increasing and its much difficult to consolidate and have a single report for multiple newman run. In Python or in Ruby test runs we have profiles that we can specify during a run which will have most of the static commandline params.
Also to run different collections at the same time, we need to depend on the shell's default multi run with && between multiple newman runs. Here, if we are running three newman runs as,
`
newman run && newman run && newman run
`
and if the file2.json takes long time to run than other two, then if we have any failures in or that will not be shown in the jenkins jobs or any CI tools such as bitbucket pipeline. This happens because the result of the run which completes at last takes precedence over others.
Aim
This package is aimed at resolving the above particular issue along with allure reports integration which can be easily integrated with jenkins. This allows us to run multiple collections defined in a single json file and run all the collections in one shot and have a combined reports at allure.
Supports
* Multiple collections run in single shot
* Allure reports along with newman's default CLI, HTML, HTMLEXTRA & JSON
* Jenkins Integration
* Docker Execution
Pending
* Need to handle tests with iteration data.
* Need to add multi-thread run. (To parallelize the test run Help Needed)
Installation
The easiest way to install Newman-Run is using NPM. If you have Node.js installed, it is most likely that you have NPM installed as well.
`
npm install -g newman-run
`
This installs Newman-Run globally on your system allowing you to run it from anywhere. If you want to install it locally, Just remove the -g flag.
If you want this to be added to your package.json, run with either --save or --save-dev depending on your usage.
Usage
To check for the available commandline arguments use newman-run --help
$3
Here we can see our original aim to run multiple collections in single shot.
For a simple run of all the feature files in normal mode, try
`
newman-run -c
`
Here we are going to use the feed file extensively to make our package run multiple collections at a time. Feed file is nothing but a json file where we need to specify the path of collection and environment JSON's.
#### Feed files
The feed file is the core structure for this package to function. In the feed file, one has to specify the collections and environment json files path or the url in the array format inside runs variable like,
`
{
"runs":[
{
"collection": "./collections/test_scripts.postman_collection.json"
},
{
"collection": "./collections/file_upload_collection.postman_collection.json",
"environment": "./environment/test_environment.json"
}
]
}
`
If we have only collections that need to be run as part of the test, then have collection alone. If we have collection and environment files, then we need to specify both collection and environment in json format.
When we initiate the tests, the runs array will be iterated and all the tests will be run using newman package.
Ideally we should have the postman collection link as the feed file input, this gives us the leverage of modifying the tests via postman without touching the core framework. The ideal feed file will be,
`
{
"runs":[
{
"collection": "https://www.getpostman.com/collections/57c3cfef239jeijw39d93"
},
{
"collection": "https://www.getpostman.com/collections/57c3cfef239jeijwew282",
"environment": "./environment/test_environment.json"
},
{
"collection": "https://www.getpostman.com/collections/57c3cfef239jeijwew282",
"environment": "./environment/dev_environment.json"
},
{
"collection": "https://www.getpostman.com/collections/57c3cfef239jeijwew282",
"environment": "./environment/prod_environment.json"
}
]
}
`
Once the feed file is prepared, one can directly trigger the test by,
`
newman-run -f
`
Now as you can see, you can specify N number of environment combinations with the same collection file or different collection file. All these will run in one shot and we can have a collated result view in the allure.
#### A/Synchronous mode
By default, the collections are executed asynchronously via the mechanism of Javascript's callback functions: in this way the collections are executed interleaved, all in parallel, without respecting the order in which the collections are defined in the runs.json file.
By specifying the '-s' or '--synchronous' option, the system will execute all calls in order one collection at a time. E.g. first all calls from collection 1, then all calls from collection 2, etc. always in the order of definition within the collection. As newman would do by default.
`
newman-run -s -f <./feed/
`
$3
To support the basic newman functionality, the newman-run command allows you to specify a collection to be run. You can easily export your Postman Collection as a json file from the Postman App and run it using Newman.
`
newman-run -c <./collection/
`
If you have environment file to be passed, you can conviniently do the same.
`
newman-run -c <./collection/ -e <./environment/
`
You can also specifiy both the feed file as well as collections at the same time.
`
newman-run -f <./feed/ -c <./collection/
`
This allows us to run any collection dynamically along with our set of collections.
The above will take care the reporting part and we don't need to mention about that from the command line. Along with allure reports, newman's default CLI, HTML, HTMLEXTRA and JSON reports are added which can be found at reports/ path. For sophesticated classification, reports for each collection is isolated with different name. If needed JSON and HTML files can be pushed to S3 for further processing or to have a record.
You can configure the list of reporters to use:
`
newman-run -f <./feed/ -R cli html htmlextra json allure
`
$3
In order to remove the report files that are generated in the previous run, run command of -r can be added. This will clear the reports folder and then will have only the files that are corresponding to the current run.
`
newman-run -r -f <./feed/
`
$3
We need to give the file path along with the current working directory i.e., along with ./ symbol. This ensures that the proper file path is given and its used to compute the relative path in the code.
Allure
To open the allure results,
`
allure serve reports/allure
`
To install allure follow https://docs.qameta.io/allure/ with your respective machine.
Jenkins Integration with Docker images
Get any of the linux with cypress docker image as the slaves in jenkins and use the same for executing the UI automation with this framework (Sample docker image - https://hub.docker.com/r/postman/newman/). From the jenkins bash Execute the following to get the testcases to run (Provided newman-run is added to the package.json dependencies),
`
#!/bin/bash -l
npm list
ls
cd
npm install
newman-run -f <./feed/
`
In Jenkins pipeline, try to add the following snippet to execute the tests,
`
pipeline {
agent { docker { image 'postman/newman' } }
stages {
stage('build') {
steps {
sh 'cd project/'
sh 'npm install'
sh 'newman-run -f <./feed/' # or different files
}
}
}
}
`
Sample Framework structure
If you are looking for proper framework structure up on which this package is built on you can see Postman-Newman-Framework
Built With
* Newman - Automation core framework to run Postman collections.
* Newman-Allure - For Detailed reporting.
Contributing
1. Clone the repo!
2. Create your feature branch: git checkout -b my-new-feature
3. Commit your changes: git commit -am 'Add some feature'
4. Push to the branch: git push origin my-new-feature`