A web server component for the pixl-server framework.
npm install pixl-server-webThis module is a component for use in pixl-server. It implements a simple web server with support for both HTTP and HTTPS, serving static files, and hooks for adding custom URI handlers.
- Usage
- Configuration
* port
* alt_ports
* bind_address
* htdocs_dir
* max_upload_size
* temp_dir
* static_ttl
* static_index
* server_signature
* compress_text
* regex_text
* regex_json
* response_headers
* code_response_headers
* uri_response_headers
* timeout
* request_timeout
* keep_alives
+ default
+ request
+ close
* keep_alive_timeout
* socket_prelim_timeout
* max_requests_per_connection
* gzip_opts
* enable_brotli
* brotli_opts
* default_acl
* blacklist
* whitelist
* allow_hosts
* rewrites
* redirects
* log_requests
* log_request_details
* log_body_max
* regex_log
* log_perf
* perf_threshold_ms
* perf_report
* recent_requests
* max_connections
* max_concurrent_requests
* max_queue_length
* max_queue_active
* queue_skip_uri_match
* clean_headers
* log_socket_errors
* full_uri_match
* flatten_query
* req_max_dump_enabled
* req_max_dump_dir
* req_max_dump_debounce
* public_ip_offset
* legacy_callback_support
* startup_message
* debug_ttl
* debug_bind_local
* chaos
* https
* https_port
* https_alt_ports
* https_cert_file
* https_key_file
* https_ca_file
* https_force
* https_header_detect
* https_timeout
* https_bind_address
* https_cert_poll_ms
- Custom URI Handlers
* Access Control Lists
* Internal File Redirects
* Static Directory Handlers
* Sending Responses
+ Standard Response
+ Custom Response
+ JSON Response
+ Non-Response
* args
+ args.request
+ args.response
+ args.ip
+ args.ips
+ args.query
+ args.params
- Standard HTTP POST
- JSON REST POST
- Unknown POST
+ args.files
+ args.cookies
+ args.perf
+ args.server
+ args.id
+ args.setCookie
* Request Filters
- Transaction Logging
* Request Detail Logging
* Performance Threshold Logging
+ Including Diagnostic Reports
* Including Custom Metrics
- Stats
* The Server Object
* The Stats Object
* The Listeners Object
* The Sockets Object
* The Recent Object
* The Queue Object
* Stats URI Handler
- Misc
* Determining HTTP or HTTPS
* Self-Referencing URLs
* Custom Method Handlers
* Let's Encrypt / ACME TLS Certificates
+ ACME clients
+ Point your domain at your server
+ Install Certbot
- Ubuntu / Debian
- RHEL / CentOS / Fedora
+ Option A: HTTP-01 (webroot)
- Ensure HTTP is working on port 80
- Issue a certificate using webroot
+ Configure pixl-server-web for HTTPS
+ Automatic renewal
- Check that renewal timers are installed
+ Option B: DNS-01 with DNS API (wildcards, advanced)
- Using Certbot DNS plugins
- Using acme.sh
+ Where your certificates live
+ Troubleshooting
* Request Max Dump
- License
Use npm to install the module:
``sh`
npm install pixl-server pixl-server-web
Here is a simple usage example. Note that the component's official name is WebServer, so that is what you should use for the configuration key, and for gaining access to the component via your server object.
`js
const PixlServer = require('pixl-server');
let server = new PixlServer({
__name: 'MyServer',
__version: "1.0",
config: {
"log_dir": "/var/log",
"debug_level": 9,
"WebServer": {
"port": 80,
"htdocs_dir": "/var/www/html"
}
},
components: [
require('pixl-server-web')
]
});
server.startup( function() {
// server startup complete
server.WebServer.addURIHandler( '/my/custom/uri', 'Custom Name', function(args, callback) {
// custom request handler for our URI
callback(
"200 OK",
{ 'Content-Type': "text/html" },
"Hello this is custom content!\n"
);
} );
} );
`
Notice how we are loading the pixl-server parent module, and then specifying pixl-server-web as a component:
`js`
components: [
require('pixl-server-web')
]
This example is a very simple web server configuration, which will listen on port 80 and serve static files out of /var/www/html. However, if the URI is /my/custom/uri, a custom callback function is fired and can serve up any response it wants. This is a great way to implement an API.
The configuration for this component is set by passing in a WebServer key in the config element when constructing the PixlServer object, or, if a JSON configuration file is used, a WebServer object at the outermost level of the file structure. It can contain the following keys:
This is the main port to listen on. The standard web port is 80, but note that only the root user can listen on ports below 1024.
If you would like to have the server listen on additional ports, add them here as an array. Example:
`json`
{
"port": 80,
"alt_ports": [ 3000, 8080 ]
}
Optionally specify an exact local IP address to bind the listeners to. By default this binds to all available addresses on the machine. Example:
`json`
{
"bind_address": "127.0.0.1"
}
This example would cause the server to only listen on localhost, and not any external network interface.
This is the path to the directory to serve static files out of, e.g. /var/www/html.
This is the maximum allowed upload size. If uploading files, this is a per-file limit. If submitting raw data, this is an overall POST content limit. The default is 32MB.
This is where file uploads will be stored temporarily, until they are renamed or deleted. If omitted, this defaults to the operating system's temp directory, as returned from os.tmpDir().
This is the TTL (time to live) value to pass on the Cache-Control response header. This causes static files to be cached for a number of seconds. The default is 0 seconds.
This sets the filename to look for when directories are requested. It defaults to index.html.
This is a string to send back to the client with every request, as the Server HTTP response header. This is typically used to declare the web server software being used. The default is WebServer.
This is a boolean indicating whether or not to compress text responses using zlib software compression in Node.js. The default is false. The compression format is chosen automatically based on the Accept-Encoding request header sent from the client. The supported formats are Brotli (see enable_brotli), Gzip and Deflate, chosen in that order.
You can force compression on an individual response basis, by including a X-Compress: 1 response header in your URI handler code. The web server will detect this outgoing header and force-enable compression on the data, regardless of the compress_text or regex_text settings. Note that it still honors the client Accept-Encoding header, and will only enable compression if this request header is present and contains a supported scheme.
Note: The legacy gzip_text property is still supported, and is now a shortcut for compress_text.
This is a regular expression string which is compared against the Content-Type response header. When this matches, and compress_text is enabled, this will kick in compression. It defaults to (text|javascript|json|css|html).
This is a regular expression string used to determine if the incoming POST request contains JSON. It is compared against the Content-Type request header. The default is (javascript|js|json).
This param allows you to send back additional custom HTTP headers with every response. Set the param to an object containing keys for each header, like this:
`json`
{
"response_headers": {
"X-My-Custom-Header": "12345",
"X-Another-Header": "Hello"
}
}
This property allows you to include conditional response headers, based on the HTTP response code. For example, you can instruct the web server to send back a custom header with 404 (File Not Found) responses, like this:
`json`
{
"code_response_headers": {
"404": {
"X-Message": "And don't come back!"
}
}
}
An actual useful case would be to include a Retry-After header with all 429 (Too Many Requests) responses, like this:
`json`
{
"code_response_headers": {
"429": {
"Retry-After": "10"
}
}
}
This would give a hint to clients when they receive a 429 (Too Many Requests) response from the web server, that they should wait 10 seconds before trying again.
This property allows you to include conditional response headers, based on regular expression matches on incoming request URIs. You may specify multiple patterns, and multiple headers to inject for each URI match. For example, you can instruct the web server to send back custom headers for a specific URI prefix, like this:
`json`
{
"uri_response_headers": {
"^/secret": {
"X-Message": "You found the secret area!",
"X-Foo": "Bar"
}
}
}
An actual useful case would be to include a set of CSP headers for all HTML files, and URIs that end in a slash (which typically present an HTML file). Example:
`json`
{
"uri_response_headers": {
"(\/|\\.html)$": {
"Content-Security-Policy": "default-src 'none'; script-src 'self'; script-src-elem 'self'; script-src-attr 'unsafe-inline'; style-src 'self' 'unsafe-inline'; style-src-attr 'unsafe-inline'; manifest-src 'self';img-src 'self' data: blob:; font-src 'self'; connect-src 'self' ws: wss:; media-src 'self' blob:; worker-src 'self' blob:; object-src 'none'; base-uri 'self'; form-action 'self'; frame-ancestors 'none';",
"X-Content-Type-Options": "nosniff",
"Referrer-Policy": "strict-origin-when-cross-origin",
"Permissions-Policy": "camera=(), microphone=(), geolocation=(), fullscreen=()"
}
}
}
This sets the idle socket timeout for all incoming HTTP requests, in seconds. If omitted, the Node.js default is 120 seconds. Example:
`json`
{
"timeout": 120
}
This only applies to reading from sockets when data is expected. It is an idle read timeout on the socket itself, and doesn't apply to request handlers.
This property sets an actual hard request timeout for all incoming requests. If the total combined request processing, handling and response time exceeds this value, specified in seconds, then the request is aborted and a HTTP 408 Request Timeout response is sent back to the client. This defaults to 0 (disabled). Example use:
`json`
{
"request_timeout": 300
}
Note that this includes request processing time (e.g. receiving uploaded data from a HTTP POST).
This controls the HTTP Keep-Alive behavior in the web server. There are three possible settings, which should be specified as a string:
`json`
{
"keep_alives": "default"
}
This enables Keep-Alives for all incoming connections by default, unless the client specifically requests a close connection via a Connection: close header.
`json`
{
"keep_alives": "request"
}
This disables Keep-Alives for all incoming connections by default, unless the client specifically requests a Keep-Alive connection by passing a Connection: keep-alive header.
`json`
{
"keep_alives": "close"
}
This completely disables Keep-Alives for all connections. All requests result in the socket being closed after completion, and each socket only serves one single request.
This sets the HTTP Keep-Alive idle timeout for all sockets, measured in seconds. If omitted, the Node.js default is 5 seconds. See server.keepAliveTimeout for details. Example:
`json`
{
"keep_alive_timeout": 5
}
This sets a special preliminary timeout for brand new sockets when they are first connected, measured in seconds. If an HTTP request doesn't come over the socket within this timeout (specified in seconds), then the socket is hard closed. This timeout should always be set lower than the timeout if used. This defaults to 0 (disabled). Example use:
`json`
{
"socket_prelim_timeout": 3
}
The idea here is to prevent certain DDoS-style attacks, where an attacker opens a large amount of TCP connections without sending any requests over them.
Note: Do not enable this feature if you attach a WebSocket server such as ws.
This allows you to set a maximum number of requests to allow per Keep-Alive connection. It defaults to 0 which means unlimited. If set, and the maximum is reached, a Connection: close header is returned, politely asking the client to close the connection. It does not actually hard-close the socket. Example:
`json`
{
"max_requests_per_connection": 100
}
This allows you to set various options for the automatic GZip compression in HTTP responses. Example:
`json`
{
"gzip_opts": {
"level": 6,
"memLevel": 8
}
}
Please see the Node Zlib Class Options for more details on what can be set here.
Set this to true to enable Brotli compression support. The default is false (disabled). When enabled, and the client advertises support via the Accept-Encoding request header, and compress_text is enabled, and the response Content-Type matches the regex_text pattern, Brotli will be used.
Brotli is a newer compression format written by Google, which was added to Node.js in v10.16.0. With careful tuning (see below) you can produce equivalent payload sizes to Gzip but considerably faster (i.e. less CPU), or even up to ~20% smaller sizes than Gzip but much slower (i.e. more CPU).
If enable_brotli is set to true, then you can set various options via the brotli_opts configuration property. Example:
`json`
{
"brotli_opts": {
"chunkSize": 16 * 1024,
"mode": "text",
"level": 4,
"hint": 0
}
}
See the Node Brotli Class Options for more details on what can be set here. Note that mode is a convenience shortcut for zlib.constants.BROTLI_PARAM_MODE (which can set to text, font or generic), level is a shortcut for zlib.constants.BROTLI_PARAM_QUALITY, and hint is a shortcut for zlib.constants.BROTLI_PARAM_SIZE_HINT.
This allows you to configure the default ACL, which is only used for URI handlers that register themselves as private. To customize it, specify an array of IPv4 and/or IPv6 addresses, partials or CIDR blocks. It defaults to localhost plus the IPv4 private reserved and IPv6 private reserved ranges. Example:
`json`
{
"default_acl": ["127.0.0.1", "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "::1/128", "fd00::/8", "169.254.0.0/16", "fe80::/10"]
}
See Access Control Lists below for more details.
The blacklist property allows you to specify a list of IPs or IP ranges which are blacklisted. Meaning, all requests from these IPs are immediately rejected by the web server (see details below). The format of the blacklist is the same as default_acl (see Access Control Lists). It defaults to an empty list (i.e. disabled).
To customize it, specify an array of IPv4 and/or IPv6 addresses, partials or CIDR blocks. Example:
`json`
{
"blacklist": ["17.0.0.0/8", "12.0.0.0/8"]
}
This example would reject all incoming IP addresses from Apple and AT&T (who own the 17.0.0.0/8 and 12.0.0.0/8 IPv4 blocks, respectively).
When a new incoming connection is established, the socket IP is immediately checked against the blacklist, and if matched, the socket is "hard closed". This is an early detection and rejection, before the HTTP request even comes in. In this case a HTTP response isn't sent back (as the socket is simply slammed shut). However, if you are using a load balancer or proxy, the user's true IP address might not be known until later on in the request cycle, once the HTTP headers are read in. At that point all the user's IPs are checked against the blacklist again, and if any of them match, a HTTP 403 Forbidden response is sent back to the client.
The whitelist property allows you to specify a list of IPs or IP ranges which are whitelisted. Meaning, all requests must originate from these IPs, or else they are immediately rejected by the web server (see details below). The format of the whitelist is the same as default_acl (see Access Control Lists). It defaults to an empty list (i.e. disabled).
To customize it, specify an array of IPv4 and/or IPv6 addresses, partials or CIDR blocks. Example:
`json`
{
"whitelist": ["17.0.0.0/8", "12.0.0.0/8"]
}
This example would reject all incoming IP addresses *unless they were from Apple and AT&T (who own the 17.0.0.0/8 and 12.0.0.0/8 IPv4 blocks, respectively).
When a new incoming connection is established, the socket IP is immediately checked against the whitelist, and if it doesn't match, the socket is "hard closed". This is an early detection and rejection, before the HTTP request even comes in. In this case a HTTP response isn't sent back (as the socket is simply slammed shut). However, if you are using a load balancer or proxy, the user's true IP address might not be known until later on in the request cycle, once the HTTP headers are read in. At that point all the user's IPs are checked against the whitelist again, and any of them do not match, a HTTP 403 Forbidden response is sent back to the client.
The allow_hosts property allows you to specify a limited set of hosts to allow for incoming requests. Specifically, this matches the incoming HTTP Host request header, or SNI (TLS handshake) host for HTTPS, and the value must match at least one entry in the array (case-insensitive). For example, if you are hosting your application behind a domain name, you may want to restrict incoming requests so that they must explicitly point to your domain name. Here is how to set this up:
`json`
"allow_hosts": ["mydomain.com"]
In the above example, only requests to mydomain.com would be allowed. All other domains or IP addresses in the URL would be rejected with a HTTP 403 Forbidden error (or in the case of SNI / TLS handshake the socket is simply closed). Include multiple entries in the array for things like subdomains:
`json`
"allow_hosts": ["mydomain.com", "www.mydomain.com"]
If the allow_hosts array is empty or omitted entirely, all hosts are allowed. This is the default behavior.
If you need to rewrite certain incoming URLs on-the-fly, you can define rules in the rewrites object. The basic format is as follows: keys are regular expressions matched on incoming URI paths, and the values are the substitution strings to use as replacements. Here is a simple example:
`json`
{
"rewrites": {
"^/rewrite/me": "/target/path"
}
}
This would match any incoming URI paths that start with /rewrite/me and replace that section of the path with /target/path. So for example a full URI path of /rewrite/me/please?foo=bar would rewrite to /target/path/please?foo=bar. Note that the suffix after the match was copied over, as well as the query string. Rewriting happens very early in the request cycle before any other processing occurs, including URI filters, method handers and URI handlers, so they all see the final transformed URI, and not the original.
Since URIs are matched using regular expressions, you can define capturing groups and refer to them in the target substitution string, using the standard $1, $2, $3 syntax. Example:
`json`
{
"rewrites": {
"^/rewrite/me(.*)$": "/target/path?oldpath=$1"
}
}
This example would grab everything after /rewrite/me and store it in a capture group, which is then expanded into the replacement string using the $1 macro.
For even more control over your rewrites, you can specify them using an advanced syntax. Instead of the target path string, set the value to an object containing the following:
| Property Name | Type | Description |
|---------------|------|-------------|
| url | String | The target URI replacement string. |headers
| | Object | Optionally insert custom headers into the incoming request. |last
| | Boolean | Set this to true to ensure no futher rewrites happen on the request. |
Here is an example showing an advanced configuration:
`json`
{
"rewrites": {
"^/rewrite/me": {
"url": "/target/path",
"headers": { "X-Rewritten": "Yes" },
"last": true
}
}
}
A URI may be rewritten multiple times if it matches multiple rules, which are applied in the order which they appear in your configuration. You can specify a last property to ensure that rule matching stops when the specified rule matches a request.
You can use the headers property to insert custom HTTP headers into the request. These will be accessible by downstream URI handlers, and they will also be logged if log_requests is enabled.
If you need to redirect certain incoming requests to external URLs, you can define rules in the redirects object. When matched, these will interrupt the current request and return a redirect response to the client. The basic format is as follows: keys are regular expressions matched on incoming URI paths, and the values are the fully-qualified URLs to redirect to. Here is a simple example:
`json`
{
"redirects": {
"^/redirect/me": "https://disney.com/"
}
}
This would match any incoming URI paths that start with /redirect/me and redirect the user to https://disney.com/. Redirects are matched during the URI handling portion of the request cycle, so things like requests and URI filters have already been handled. URI request handlers are not invoked if a redirect occurs.
Since URIs are matched using regular expressions, you can define capturing groups and refer to them in the target redirect URL, using the standard $1, $2, $3 syntax. Example:
`json`
{
"redirects": {
"^/github/(.*)$": "https://github.com/jhuckaby/$1"
}
}
This example would grab everything after /github/ and store it in a capture group, which is then expanded into the replacement string using the $1 macro. For example, /github/pixl-server-web would redirect to https://github.com/jhuckaby/pixl-server-web.
For even more control over your redirects, you can specify them using an advanced syntax. Instead of the target URL, set the value to an object containing the following:
| Property Name | Type | Description |
|---------------|------|-------------|
| url | String | The fully qualified URL to redirect to. |headers
| | Object | Optionally insert custom headers into the incoming request. |status
| | String | The HTTP response code and status to use, default is 302 Found. |
Here is an example showing an advanced configuration:
`json`
{
"redirects": {
"^/redirect/me": {
"url": "https://disney.com/",
"headers": { "X-Redirected": "Yes" },
"status": "301 Moved Permanently"
}
}
}
You can use the headers property to insert custom HTTP headers into the redirect response. Use the status to customize the HTTP response code and status (it defaults to 302 Found).
This boolean allows you to enable transaction logging in the web server. It defaults to false (disabled). See Transaction Logging below for details.
This boolean adds verbose detail in the transaction log. It defaults to false (disabled). See Transaction Logging below for details.
Note: This property only has effect if log_requests is enabled.
This property sets the maximum allowed request and response body length that can be logged, when log_request_details is enabled. It defaults to 32768 (32K). If the request or response body length exceeds this amount, they will not be included in the transaction log.
Note: This property only has effect if log_request_details is enabled.
If log_requests is enabled, this allows you to specify a regular expression to match against incoming request URIs. Only requests that match will be logged. It defaults to match all URIs (.+). See Transaction Logging below for details.
This boolean allows you to enable performance threshold logging. It defaults to false (disabled). See Performance Threshold Logging below for details.
If log_perf is enabled, this allows you to specify the request elapsed time threshold in milliseconds. All requests equal to or longer will be logged. It defaults to 100 milliseconds. See Performance Threshold Logging below for details.
This property allows you to include a complete or partial Node.js Diagnostic Report in your Performance Threshold Log. Specifically, you can set this to an array of report keys to include in the log data. See Including Diagnostic Reports below for details.
This integer specifies the number of recent requests to provide in the getStats() response. It defaults to 10. See Stats below for details.
This integer specifies the maximum number of concurrent connections to allow. It defaults to 0 (no limit). If specified and the amount is exceeded, new incoming connections will be denied (socket force-closed without reading any data), and an error logged for each attempt (with error code maxconns).
This integer specifies the maximum number of concurrent requests to allow. It defaults to 0 (no limit). If more than the maximum allowed requests arrive in parallel, additional requests are queued, and processed as soon as slots become available. Requests are always processed in the order they were received.
The idea here is that you can set max_connections to a much higher value, for things like load balancers pre-opening connections or clients using a pool of keep-alive connections, but then only allow your application code to process a smaller amount of requests in parallel. For example:
`json`
{
"max_connections": 2048,
"max_concurrent_requests": 64
}
This would allow up to 2,048 concurrent connections (sockets) to be open at any given time, but only allow 64 active requests to run in parallel. If more than 64 requests came in at once, the remainder would be queued up, and processed as soon as other requests completed.
The max_queue_length property is designed to work in conjunction with max_concurrent_requests. It specifies the maximum number of requests to allow in the queue, before rejecting new requests. It defaults to 0 (infinite). If the number of enqueued requests reaches this limit, then new incoming requests are immediately aborted with a HTTP 429 Too Many Requests response. An error is also logged with a 429 code in this case. Example error log entry:
`
[1587614950.774][2020-04-22 21:09:10][joe16.local][93307][WebServer][error][429][Queue is maxed out (100 pending reqs), denying request from: 127.0.0.1][{"ips":["127.0.0.1"],"uri":"/sleep?ms=500","headers":{"accept-encoding":"gzip, deflate, br","user-agent":"Overflow Test Agent 1.0","host":"localhost:3012","connection":"keep-alive"},"pending":100,"active":1024,"sockets":1175}]
The error log data column includes some additional information including the total requests pending, the number of concurrent active requests, and the number of open sockets.
The max_queue_active property is designed to work in conjunction with max_connections, max_concurrent_requests and max_queue_length. It sets an upper maximum for number of concurrent active requests in the queue (i.e. concurrent active requests), before new ones are immediately rejected with an HTTP 429 response, without actually queueing up. This defaults to 0 (disabled), which means there is no limit imposed at the queue level.
The only reason you'd ever need to set this property is to handle a request overload situation by rejecting requests out of the queue via HTTP 429, rather than blocking them at the socket level (hard close), and also not allowing them to queue up (potential lag situation). Example configuration:
`json`
{
"max_connections": 8192,
"max_concurrent_requests": 1024,
"max_queue_length": 1024,
"max_queue_active": 1024
}
The idea here is that pixl-server-web will allow up to 1,024 concurrent requests, but additional requests beyond the maximum are still accepted and responded to with a nice HTTP 429 response, rather than the alternatives (i.e. allowing requests to queue up, possibly introducing unwanted lag, or performing a hard socket close). This works as long as the total concurrent sockets do not exceed the upper limit (8,192 in this case).
With both max_queue_length and max_queue_active set to non-zero values, the first limit reached aborts the request.
The queue_skip_uri_match property is designed to work in conjunction with max_concurrent_requests. It allows you to specify a URI pattern match that will always skip over the queue and be processed immediately, regardless of limits. Using this feature you can allow things like health checks (possibly from a load balancer) to always be serviced, even during an overload situation. Example use:
`json`
{
"queue_skip_uri_match": "^/server-status"
}
This property defaults to false (disabled).
This boolean enables HTTP response header cleansing. When set to true it will strip all illegal characters from your response header values, which otherwise could cause Node.js to crash. It defaults to false. The regular expression it uses is /([\x7F-\xFF\x00-\x1F\u00FF-\uFFFF])/g.
This boolean enables logging socket related errors, specifically sockets being closed unexpectedly (i.e. client closed socket, or some network error caused socket to abort). This defaults to true, meaning these will be logged as errors. If this generates too much log noise for your production stack, you can set the configuration property to false, which will only log a level 9 debug event. Example:
`json`
{
"log_socket_errors": false
}
Example error log entry:
`
[1545121086.42][2018-12-18 00:18:06][myserver01.mycompany.com][29801][WebServer][error][socket][Socket closed unexpectedly: c43593][][][{"id":"c43593","proto":"http","port":80,"time_start":1545120267519,"num_requests":886,"bytes_in":652041,"bytes_out":1307291,"total_elapsed":818901,"url":"http://mycompany.com/example/url","ips":["1.1.1.1","2.2.2.2"]}]
When this boolean is set to true, Custom URI Handlers will match against the full incoming URI, including the query string. By default this is disabled, meaning URIs are only matched using their path. Example:
`json`
{
"full_uri_match": true
}
By default, we use the Node.js core Query String module to parse query strings. This module handles duplicate query params by converting them to arrays. For example, an incoming URI such as /something?foo=bar1&foo=bar2&name=joe would produce the following args.query object:
`json`
{
"foo": ["bar1", "bar2"],
"name": "joe"
}
However, if you set flatten_query to true in your configuration, the web server will "flatten" query string parameters, so that duplicate keys will be combined into one, with the latter prevailing. Example:
`json`
{
"foo": "bar2",
"name": "joe"
}
When this boolean is set to true, the Request Max Dump system is enabled. This will produce a JSON dump file when the web server is maxed out on requests.
When the Request Max Dump system is enabled, the req_max_dump_dir property sets the directory path where JSON dump files are dropped. The directory will be created if needed.
When the Request Max Dump system is enabled, the req_max_dump_debounce property sets how many seconds should elapse between dumps, as to not overwhelm the filesystem.
This controls how args.ip is chosen from the list of IP addresses in args.ips for each incoming request. By default, the client IP is chosen by scanning the list from left to right, and selecting the first non-private IP. However, modern wisdom suggests that alternate selection logic may be more desirable to find the true public IP.
By setting public_ip_offset to an integer value, you can select exactly which IP to select from the list. Use negative numbers to select IP address from the end (right side) of the list. Here are the recommended values:
| Offset | Description |
|--------|-------------|
| 0 | The default value. Allow the server to select the public IP automatically. |-1
| | Always select the last IP in the list (i.e. the TCP socket IP). Use this mode if your server is connected to the internet directly. |-2
| | Always select the second-to-last IP in the list. Use this mode if you have a single proxy device in front of your server (e.g. a load balancer). |-3
| | Always select the third-to-last IP in the list. Use this mode if you have two proxy devices in front of your server (e.g. a load balancer and CDN / cache). |
This adds support for legacy applications, which require JSONP callback-style API responses, as well as extremely old HTML-wrapped IFRAME API responses. It defaults to disabled. It is highly recommended that you leave this disabled for all modern applications, as it prevents a classic XSS reflection attack on your APIs:
`json`
{
"legacy_callback_support": false
}
Only enable this if you are supporting a legacy application which is hosted on a private, trusted network.
When set to true and running in debug or foreground mode (i.e. --debug or --foreground CLI flags on startup), this will emit a message to the console on startup detailing all the socket listeners, ports, and URL endpoints you can hit. Example conaole message:
`
Web Server Listeners:
Listening for HTTP on port 3020, network '::' (all)
--> http://192.168.3.25:3020/
Listening for HTTPS on port 3021, network '::' (all)
--> https://192.168.3.25:3021/
`
When set to true and running in debug mode (i.e. --debug CLI flag on startup), this will override the value of static_ttl with 0. Useful for local development, i.e. reloading your web app in the browser.
This feature defaults to false (disabled).
When set to true and running in debug mode (i.e. --debug CLI flag on startup), this will override the value of bind_address with localhost. This will keep your local development environment secure, and not exposed to the network. To override this behavior, add an --expose CLI flag or explicitly set the bind_address in your config.
This feature defaults to false (disabled).
Use the chaos feature to introduce optional and random fault injection into your web requests. Used for testing purposes, this feature can introduce a random delay on every request, and also hijack requests and inject random error responses based on probabilities you specify. Here is how to use it:
`json`
"chaos": {
"enabled": true,
"uri": ".+",
"delay": {
"min":0,
"max":2000
},
"errors": {
"503 Service Unavailable": 0.1
},
"headers": {
"Retry-After": 10
}
}
Set the chaos.enabled flag to true to enable fault injection. By default, all URIs will be affected, unless you specify a chaos.uri (regular expression) to limit the requests. Set chaos.delay.min and chaos.delay.max to the range you want to delay requests (in milliseconds). Fill the chaos.errors object the HTTP repsonse codes (and status messages) you want to see, and how often. The values are interpreted as probabilities from 0.0 (never) to 1.0 (always). In the above example, the HTTP 503 error code will be injected approximately 10% of the time. When errors are injected, you can include additional response headers in the chaos.headers object.
This boolean allows you to enable HTTPS (SSL) support in the web server. It defaults to false. Note that you must also set https_port, and possibly https_cert_file and https_key_file for this to work.
The SSL certificate files are automatically reloaded if changed on disk. This is done without a server restart.
If HTTPS mode is enabled, this is the port to listen on for secure requests. The standard HTTPS port is 443.
If you would like to have the server listen on additional HTTPS ports, add them here as an array. Example:
`json`
{
"https_port": 443,
"https_alt_ports": [ 9000, 9001 ]
}
If HTTPS mode is enabled, this should point to your SSL certificate file on disk. The certificate file typically has a .crt filename extension, or possibly cert.pem if using Let's Encrypt.
If HTTPS mode is enabled, this should point to your SSL private key file on disk. The key file typically has a .key filename extension, or possibly privkey.pem if using Let's Encrypt.
If HTTPS mode is enabled, this should point to your SSL chain file on disk. This is optional, as some SSL certificates do not provide one. If using Let's Encrypt this file will be named chain.pem.
If HTTPS mode is enabled, you can set this param to boolean true to force all requests to be HTTPS. Meaning, if someone attempts a non-secure plain HTTP request to any URI, their client will be redirected to an equivalent HTTPS URI.
Your network architecture may have a proxy server or load balancer sitting in front of the web server, and performing all HTTPS/SSL encryption for you. Usually, these devices inject some kind of HTTP request header into the back-end web server request, so you can "detect" a front-end HTTPS proxy request in your code. For example, Amazon AWS load balancers inject the following HTTP request header into all back-end requests:
`
X-Forwarded-Proto: https
The https_header_detect property allows you to define any number of header regular expression matches, that will "pseudo-enable" SSL mode in the web server. Meaning, the args.request.headers.ssl property will be set to true, and calls to server.getSelfURL() will have a https:// prefix. Here is an example configuration, which detects many commonly used headers:
`json`
{
"https_header_detect": {
"Front-End-Https": "^on$",
"X-Url-Scheme": "^https$",
"X-Forwarded-Protocol": "^https$",
"X-Forwarded-Proto": "^https$",
"X-Forwarded-Ssl": "^on$"
}
}
Note that these are matched using logical OR, so only one of them needs to match to enable SSL mode. The values are interpreted as regular expressions, in case you need to match more than one header value.
This sets the idle socket timeout for all incoming HTTPS requests. If omitted, the Node.js default is 2 minutes. Please specify your value in seconds.
Optionally specify an exact local IP address to bind the HTTPS listener to. By default this uses the value of bind_address, but you can bind them differently using this property. Example:
`json`
{
"bind_address": "127.0.0.1",
"https_bind_address": "0.0.0.0"
}
This example would cause the server to only listen on localhost for plain HTTP traffic, but listen on all network interfaces for HTTPS traffic.
The https_cert_poll_ms property allows you to customize the polling interval for monitoring the SSL cert files on disk. The value is in milliseconds, and defaults to 60000 (1 minute). This is used to poll the SSL cert files on disk to see if they changed (i.e. cert renewal). If so, they are automatically reloaded without restarting the server.
You can attach your own handler methods for intercepting and responding to certain incoming URIs. So for example, instead of the URI /api/add_user looking for a static file on disk, you can have the web server invoke your own function for handling it, and sending a custom response.
To do this, call the addURIHandler() method and pass in the URI string, a name (for logging), and a callback function:
`js`
server.WebServer.addURIHandler( '/my/custom/uri', 'Custom Name', function(args, callback) {
// custom request handler for our URI
callback(
"200 OK",
{ 'Content-Type': "text/html" },
"Hello this is custom content!\n"
);
} );
URIs must match exactly (sans the query string), and the case is sensitive. If you need to implement something more complicated, such as a regular expression match, you can pass one of these in as well. Example:
`js`
server.WebServer.addURIHandler( /^\/custom\/match\/$/i, 'Custom2', function(args, callback) {...} );
Your handler function is passed exactly two arguments. First, an args object containing all kinds of useful information about the request (see args below), and a callback function that you must call when the request is complete and you want to send a response.
If you specified a regular expression with parenthesis groups for the URI, the matches array will be included in the args object as args.matches. Using this you can extract your matched groups from the URI, for e.g. /^\/api\/(\w+)/.
Note that by default, URIs are only matched on their path portion (i.e. sans query string). To include the query string in URI matches, set the full_uri_match configuration property to true.
If you want to restrict access to certain URI handlers, you can specify an ACL which represents a list of IP address ranges to allow. To use the default ACL, simply pass true as the 3rd argument to addURIHandler(), just before your callback. This flags the URI as private. Example:
`js`
server.WebServer.addURIHandler( /^\/private/, "Private Admin Area", true, function(args, callback) {
// request allowed
callback( "200 OK", { 'Content-Type': 'text/html' }, "Access granted!
\n" );
} );
This will protect the handler using the default ACL, as specified by the default_acl configuration parameter. However, if you want to specify a custom ACL per handler, simply replace the true argument with an array of IPv4 and/or IPv6 addresses, partials or CIDR blocks. Example:
`js`
server.WebServer.addURIHandler( /^\/secret/, "Super Secret Area", ['10.0.0.0/8', 'fd00::/8'], function(args, callback) {
// request allowed
callback( "200 OK", { 'Content-Type': 'text/html' }, "Access granted!
\n" );
} );
This would only allow requests from either 10.0.0.0/8 (IPv4) or fd00::/8 (IPv6).
The ACL code scans all the IP addresses from the client, including the socket IP and any passed as part of HTTP headers (populated by load balancers, proxies, etc.). See args.ips for more details on this. All the IPs must pass the ACL test in order for the request to be allowed through to your handler.
If a request is rejected, your handler isn't even called. Instead, a standard HTTP 403 Forbidden response is sent to the client, and an error is logged.
To setup an internal file redirect, you can substitute the final callback function for a string, pointing to a fully-qualified filesystem path. The target file will be served up in place of the original URI. You can also combine this with an ACL for extra protection for private files. Example:
`js`
server.WebServer.addURIHandler( /^\/secret.txt$/, "Special Secrets", true, '/private/myapp/docs/secret.txt' );
Note that the Content-Type response header is automatically set based on the target file you are redirecting to.
If you would like to host static files in other places besides htdocs_dir, possibly with different options, then look no further than the addDirectoryHandler() method. This allows you to set up static file handling with a custom base URI, a custom base directory on disk, and apply other options as well. You can call this method as many times as you like to setup multiple static file directories. Example:
`js`
server.WebServer.addDirectoryHandler( /^\/mycustomdir/, '/var/www/custom' );
The above example would catch all incoming requests starting with /mycustomdir, and serve up static files inside of the /var/www/custom directory on disk (and possibly nested directories as well). So a URL such as http://MYSERVER/mycustomdir/foo/file1.txt would map to the file /var/www/custom/foo/file1.txt on disk.
In this case a default TTL is applied to all files via static_ttl. If you would like to customize the TTL for your custom static directory, as well as specify other options, pass in an object as the 3rd argument to addDirectoryHandler(). Example of this:
`js`
server.WebServer.addDirectoryHandler( /^\/mycustomdir/, '/var/www/custom', {
acl: true
ttl: 3600,
headers: {
'X-Custom': '12345'
}
} );
In this example the files would be restricted to client IP addresses matching the default_acl, and would be served up with a custom TTL of 3600 seconds (specifically, the Cache-Control response header would be set to public, max-age=3600). Finally, all static file responses would include the X-Custom: 12345 header. Here is a list of the available properties in the options object:
| Property Name | Type | Description |
|---------------|------|-------------|
| acl | Boolean | Optionally restrict the static files to an IP-based ACL. You can set this to Boolean true to use the default_acl, or specify an array of IPv4 and/or IPv6 addresses, partials or CIDR blocks. |ttl
| | Mixed | Optionally customize the TTL (Cache-Control header). Set this to a number to use the public, max-age=### format, or a string to specify the entire header value yourself. |headers
| | Object | Optionally include additional HTTP headers with every static response. Note that you cannot use this to override built-in headers like Content-Type, Content-Length, ETag, and others. It can only be used to insert unique headers. |
There are actually four different ways you can send an HTTP response. They are all detailed below:
The first type of response is shown above, and that is passing three arguments to the callback function. The HTTP response status line (e.g. 200 OK or 404 File Not Found), a response headers object containing key/value pairs for any custom headers you want to send back (will be combined with the default ones), and finally the content body. Example:
`js`
callback(
"200 OK",
{ 'Content-Type': "text/html" },
"Hello this is custom content!\n"
);
The content body can be a string, a Buffer object, or a readable stream.
Note that you can omit the status text and just return a code, e.g. "200", and the web server will fill in the text.
The second type of response is to send content directly to the underlying Node.js server by yourself, using args.response (see below). If you do this, you can pass true to the callback function, indicating to the web server that you "handled" the response, and it shouldn't do anything else. Example:
`js`
server.WebServer.addURIHandler( '/my/custom/uri', 'Custom Name', function(args, callback) {
// send custom raw response
let response = args.response;
response.writeHead( 200, "OK", { 'Content-Type': "text/html" } );
response.write( "Hello this is custom content!\n" );
response.end();
// indicate we are done, and have handled things ourselves
callback( true );
} );
The third way is to pass a single object to the callback function, which will be serialized to JSON and sent back as an AJAX style response to the client. Example:
`js`
server.WebServer.addURIHandler( '/my/custom/uri', 'Custom Name', function(args, callback) {
// send custom JSON response
callback( {
Code: 0,
Description: "Success",
User: { Name: "Joe", Email: "foo@bar.com" }
} );
} );
This is sent as pure JSON with the Content-Type application/json. The raw HTTP response would look something like this:
`
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 79
Content-Type: application/json
Date: Sun, 05 Apr 2015 20:58:50 GMT
Server: Test 1.0
{"Code":0,"Description":"Success","User":{"Name":"Joe","Email":"foo@bar.com"}}
`
The fourth and final type of response is a non-response, and this is achieved by passing false to the callback function. This indicates to the web server that your code did not handle the request, and it should fall back to looking up a static file on disk. Example:
`js`
server.WebServer.addURIHandler( '/my/custom/uri', 'Custom Name', function(args, callback) {
// we did not handle the request, so tell the web server to do so
callback( false );
} );
Note that there is currently no logic to fallback to other custom URI handlers. The only fallback logic, if a handler returns false, is to lookup a static file on disk.
To perform an internal file redirect from inside your URI handler code, set the internalFile property of the args object to your destination filesystem path, then pass false to the callback:
`js`
server.WebServer.addURIHandler( '/intredir', "Internal Redirect", true, function(args, callback) {
// perform internal redirect to custom file
args.internalFile = '/private/myapp/docs/secret.txt';
callback(false);
} );
Your URI handler function is passed an args object containing the following properties:
This is a reference to the underlying Node.js server request object. From this you have access to things like:
| Property | Description |
|----------|-------------|
| request.httpVersion | The version of the HTTP protocol used in the request. |request.headers
| | An object containing all the HTTP request headers (lower-cased). | request.method
| | The HTTP method used in the request, e.g. GET, POST, etc. | request.url
| | The complete URI of the request (sans protocol and hostname). | request.socket
| | A reference to the underlying socket connection for the request. |
For more detailed documentation on the request object, see Node's http.IncomingMessage.
This is a reference to the underlying Node.js server response object. From this you have access to things like:
| Property / Method() | Description |
|----------|-------------|
| response.writeHead() | This writes the HTTP status code, message and headers to the socket. |response.setTimeout()
| | This sets a timeout on the response. |response.statusCode
| | This sets the HTTP status code, e.g. 200, 404, etc. |response.statusMessage
| | This sets the HTTP status message, e.g. OK, File Not Found, etc. |response.setHeader()
| | This sets a single header key / value pair in the response. |response.write()
| | This writes a chunk of data to the socket. |response.end()
| | This indicates that the response has been completely sent. |
For more detailed documentation on the response object, see Node's http.ServerResponse.
This will be set to the user's remote IP address. Generally, it will be set to the first public IP address if multiple addresses are provided via proxy HTTP headers and the socket.
Meaning, if the user is sitting behind one or more proxy servers, or your web server is behind a load balancer, this will attempt to locate the user's true public (non-private) IP address. If none is found, it'll just return the first IP address, honoring proxy headers before the socket (which is usually correct).
See public_ip_offset for details on customizing the behavior of this property.
If you just want the socket IP by itself, you can get it from args.request.socket.remoteAddress.
This will be set to an array of all the user's remote IP addresses, taking into account the socket IP and various HTTP headers populated by proxies and load balancers, if applicable. The header address(es) will come first, if applicable, followed by the socket IP at the end.
The following HTTP headers are scanned for IP addresses to build the args.ips array:
| Header | Syntax | Description |
|--------|--------|-------------|
| X-Forwarded-For | Comma-Separated | The de-facto standard header for identifying the originating IP address of a client connecting through an HTTP proxy or load balancer. See X-Forwarded-For. |Forwarded-For
| | Comma-Separated | Alias for X-Forwarded-For. |Forwarded
| | Custom | New standard header as defined in RFC 7239, with custom syntax. See Forwarded.X-Forwarded
| | Custom | Alias for Forwarded. |X-Client-IP
| | Single | Non-standard, used by Heroku, etc. |CF-Connecting-IP
| | Single | Non-standard, used by CloudFlare. |True-Client-IP
| | Single | Non-standard, used by Akamai, CloudFlare, etc. |X-Real-IP
| | Single | Non-standard, used by Nginx, FCGI, etc. |X-Cluster-Client-IP
| | Single | Non-standard, used by Rackspace, Riverbed, etc. |
This will be an object containing key/value pairs from the URL query string, if applicable, parsed via the Node.js core Query String module.
Duplicate query params become an array. For example, an incoming URI such as /something?foo=bar1&foo=bar2&name=joe would produce the following args.query object:
`json`
{
"foo": ["bar1", "bar2"],
"name": "joe"
}
See flatten_query if you would rather duplicate query parameters be flattened (latter prevails).
If the request was a HTTP POST, this will contain all the post parameters as key/value pairs. This will take one of three forms, depending on the request's Content-Type header:
#### Standard HTTP POST
If the request Content-Type was one of the standard application/x-www-form-urlencoded or multipart/form-data, all the key/value pairs from the post data will be parsed, and provided in the args.params object. We use the 3rd party Formidable module for this work.
#### JSON REST POST
If the request is a "pure" JSON POST, meaning the Content-Type contains json or javascript, the content body will be parsed as a single JSON string, and the result object placed into args.params.
#### Unknown POST
If the Content-Type doesn't match any of the above values, it will simply be treated as a plain binary data, and a Buffer will be placed into args.params.raw.
If the request was a HTTP POST and contained any file uploads, they will be accessible through this property. Files are saved to a temp directory and can be moved to a custom location, or loaded directly. They will be keyed by the POST parameter name, and the value will be an object containing the following properties:
| Property | Description |
|----------|-------------|
| size | The size of the uploaded file in bytes. |path
| | The path to the temp file on disk containing the file contents. |name
| | The filename of the file as provided by the client. |type
| | The mime type of the file, according to the client. |lastModifiedDate
| | A date object containing the last mod date of the file, if available. |
For more details, please see the documentation on the Formidable.File object.
All temp files are automatically deleted at the end of the request.
This is an object parsed from the incoming Cookie HTTP header, if present. The contents will be key/value pairs for each semicolon-separated cookie provided. For example, if the client sent in a session_id cookie, it could be accessed like this:
`js`
let session_id = args.cookies['session_id'];
This is a reference to a pixl-perf object, which is used internally by the web server to track performance metrics for the request. The metrics may be logged at the end of each request (see Transaction Logging below) and included in the stats (see Stats below).
This is a reference to the pixl-server object which handled the request.
This is an internal ID string used by the server to track and log individual requests.
A utility function used to serialize cookies into the proper format, and set or append them to the Set-Cookie response header. It accepts a name, a value, and an optional set of options. Example use:
`js`
args.setCookie( 'session', 'ABDEF01234567890', { path: '/', maxAge: 86400, secure: true, httpOnly: true, sameSite: 'Lax' } );
Filters allow you to preprocess a request, before any handlers get their hands on it. They can pass data through, manipulate it, or even interrupt and abort requests. Filters are attached to particular URIs or URI patterns, and multiple may be applied to one request, depending on your rules. They can be asynchronous, and can also pass data between one another if desired.
You can attach your own filter methods for intercepting and responding to certain incoming URIs. So for example, let's say we want to filter the URI /api/add_user before the handler gets it, and inject some custom data. To do this, call the addURIFilter() method and pass in the URI string, a name (for logging), and a callback function:
`js`
server.WebServer.addURIFilter( /.+/, "My Filter", function(args, callback) {
// add a nugget into request query
args.query.filter_nugget = 42;
// add a custom response header too
args.response.setHeader('X-Filtered', "4242");
callback(false); // passthru
} );
So here we are injecting filter_nugget into the args.query object, which is preserved and passed down to other filters and handlers. Also, we are adding a X-Filtered header to the response (whoever ends up sending it). Finally, we call the callback function passing false, which means to pass the request through to other filters and/or handlers (see below for more on this).
URI strings must match exactly (sans the query string), and the case is sensitive. If you need to match something more complicated, such as a regular expression, you can pass one of these in place of the URI string. Example:
`js`
server.WebServer.addURIFilter( /^\/custom\/match\/$/i, 'Custom2', function(args, callback) {...} );
Your filter handler function is passed exactly two arguments. First, an args` object containing all kinds of useful information about the request (see args above), and a callback function that you must invoke when the filter is complete, and you want to either allow the request