Google Cloud Bucket ·    

__Google Cloud Bucket__ is a node.js package to manage Google Cloud Buckets and its objects. All read APIs uses an exponential backoff retry strategy in cases of errors. By default, that strategy times out after 10 seconds, but it can be configured.

Table of Contents

> * Install
> * Getting started
> - Prerequisite
> - Basics
> - Configuring your bucket or your file (CORS, Public, Static Website)
> - Zipping files
> - Four ways to create a client
> - User the hosting identity
> - Using a service-account.json
> - Using explicit credentials
> - Using environment variables
> - Extra Precautions To Make Robust Queries
> - Using An External OAuth2 Token
> - Performance Tips
> * Full API Doc
> - Storage API
> - Bucket API
> - BucketObject API
> - Objects
> * Annexes
> - Bucket Name Restrictions
> - List Of All Google Cloud Platform Locations
> * About Neap
> * License

Install

``
npm i google-cloud-bucket --save
`

Getting started

Prerequisite

Before using this package, you must first:

1. Have a Google Cloud Account.
2. Have a Service Account set up with the following three roles:
- roles/storage.objectCreator
- roles/storage.objectAdmin (only if you want to update access to object or create/delete buckets)
- roles/storage.admin (only if you want to update access to an entire bucket)
3. Get the JSON keys file for that Service Account above, or set up an environment with the appropriate service identity (more about this in the 4 ways to create a client section).
4. If you're using a service account JSON key file, then save that JSON key into a service-account.json file (make sure it is located under a path that is accessible to your app), or save the following properties to either manually set up the client or set up environment variables:
- project_id
- client_email
- private_key

Basics

`js
const { join } = require('path')
const { client } = require('google-cloud-bucket')

// This is one of the many options to create a storage client. To see all the different
// way to create a client, please refer to the '4 ways to create a client' section.
const storage = client.new({
jsonKeyFile: join(__dirname, './service-account.json')
})

// LISTING ALL THE BUCKETS IN THAT PROJECT
storage.list().then(console.log)

const someObject = {
firstname: 'Nicolas',
lastname: 'Dao',
company: 'Neap Pty Ltd',
city: 'Sydney'
}

// CREATING A BUCKET (This method will fail if your bucket name is not globally unique. You also need to the role 'roles/storage.objectAdmin')
storage.bucket('your-globally-unique-bucket-name').create()
.then(data => console.log(data))

// CREATING A BUCKET IN SPECIFIC LOCATION (default is US. A detailed list of all the locations can be found in the Annexes of this document)
storage.bucket('your-globally-unique-bucket-name').create({ location: 'australia-southeast1' })
.then(data => console.log(data))

// DELETING A BUCKET
storage.bucket('your-globally-unique-bucket-name').delete({ force:true })
.then(data => console.log(data))

// GET A BUCKET'S SETUP DATA
storage.bucket('your-globally-unique-bucket-name').get()
.then(data => console.log(data))

// ADDING AN OBJECT
storage.insert(someObject, 'your-bucket/a-path/filename.json') // insert an object into a bucket 'a-path/filename.json' does not need to exist
.then(() => storage.get('your-bucket/a-path/filename.json')) // retrieve that new object
.then(res => console.log(JSON.stringify(res, null, ' ')))

// ADDING A HTML PAGE WITH PUBLIC ACCESS (warning: Your service account must have the 'roles/storage.objectAdmin' role)
const html =

Hello Giiiiirls

storage.insert(html, 'your-bucket/a-path/index.html')

// UPLOADING AN IMAGE (we assume we have access to an image as a buffer variable called 'imgBuffer')
storage.insert(imgBuffer, 'your-bucket/a-path/image.jpg')

// GETTING BACK THE OBJECT
storage.get('your-bucket/a-path/filename.json').then(obj => console.log(obj))

// GETTING THE HTML BACK
storage.get('your-bucket/a-path/index.html').then(htmlString => console.log(htmlString))

// GETTING BACK THE IMAGE
// USE CASE 1 - Loading the entire buffer in memory
storage.get('your-bucket/a-path/image.jpg').then(imgBuffer => console.log(imgBuffer))

// USE CASE 2 - Loading the image on your filesystem
storage.get('your-bucket/a-path/image.jpg', { dst: 'some-path/image.jpg' })
.then(() => console.log(Image successfully downloaded.))

// USE CASE 3 - Piping the image buffer into a custom stream reader
const { Writable } = require('stream')
const customReader = new Writable({
write(chunk, encoding, callback) {
console.log('Hello chunk of image')
callback()
}
})
storage.get('your-bucket/a-path/image.jpg', { streamReader: customReader })
.then(() => console.log(Image successfully downloaded.))

// TESTING IF A FILE OR A BUCKET EXISTS
storage.exists('your-bucket/a-path/image.jpg')
.then(fileExists => fileExists ? console.log('File exists.') : console.log('File does not exist.'))

// LISTING ALL THE FILES METADATA WITH A FILEPATH THAT STARTS WITH SPECIFIC NAME
storage.list('your-bucket/a-path/')
.then(files => console.log(files))
`

__Bucket API__

The examples above demonstrate how to insert and query any storage. We've also included a variant of those APIs that are more focused on the bucket:

`js
// THIS API:
storage.insert(someObject, 'your-bucket/a-path/filename.json')
// CAN BE REWRITTEN AS FOLLOW:
storage.bucket('your-bucket').object('a-path/filename.json').insert(someObject)

// THIS API:
storage.get('your-bucket/a-path/filename.json').then(obj => console.log(obj))
// CAN BE REWRITTEN AS FOLLOW:
storage.bucket('your-bucket').object('a-path/filename.json').get().then(obj => console.log(obj))

// THIS API:
storage.exists('your-bucket/a-path/image.jpg')
.then(fileExists => fileExists ? console.log('File exists.') : console.log('File does not exist.'))
// CAN BE REWRITTEN AS FOLLOW:
storage.bucket('your-bucket').object('a-path/image.jpg').exists()
.then(fileExists => fileExists ? console.log('File exists.') : console.log('File does not exist.'))

// THIS API:
storage.list('your-bucket/a-path/')
.then(files => console.log(files))
// CAN BE REWRITTEN AS FOLLOW:
storage.bucket('your-bucket').object('a-path/').list()
.then(files => console.log(files))
`

Configuring your bucket or your file

$3

This allows to make any files publicly readable by anybody on the web. That's usefull if you want to host a website, or publish data (e.g., RSS feed).

Once your bucket is publicly readable, everyone can access it at this url: https://storage.googleapis.com/your-bucket/some-path/index.html

> WARNING: If that bucket hosts files that hsould be accessible cross domain (e.g., an RSS feed), don't forget to also set up CORS (next section Configuring CORS On a Bucket).

`js
const bucket = storage.bucket('your-bucket')

// TEST WHETHER A BUCKET IS PUBLIC OR NOT
bucket.isPublic().then(isPublic => isPublic ? console.log(Bucket '${bucket.name}' is public) : console.log(Bucket '${bucket.name}' is not public))

// MAKING A BUCKET PUBLICLY READABLE (warning: Your service account must have the 'roles/storage.admin' role)
// Once a bucket is public, all content added to it (even when omitting the 'public' flag) is public
bucket.addPublicAccess()
.then(({ publicUri }) => console.log(Your web page is publicly available at: ${publicUri}))

// REMOVING THE PUBLICLY READABLE ACCESS FROM A BUCKET (warning: Your service account must have the 'roles/storage.admin' role)
bucket.removePublicAccess()

// MAKING AN EXISTING OBJECT PUBLICLY READABLE (warning: Your service account must have the 'roles/storage.objectAdmin' role)
bucket.object('a-path/private.html').addPublicAccess()
.then(({ publicUri }) => console.log(Your web page is publicly available at: ${publicUri}))

// REMOVING THE PUBLICLY READABLE ACCESS FROM A FILE (warning: Your service account must have the 'roles/storage.objectAdmin' role)
bucket.object('a-path/private.html').removePublicAccess()
`

$3

It is also possible to make a single file publicly readable in a single command when the file is created:

`js
storage.insert(html, 'your-bucket/a-path/index.html', { public: true })
.then(({ publicUri }) => console.log(Your web page is publicly available at: ${publicUri}))
`

Once your file is publicly readable, everyone can access it at this url: https://storage.googleapis.com/your-bucket/a-path/index.html

> WARNING: If that bucket hosts files that hsould be accessible cross domain (e.g., an RSS feed), don't forget to also set up CORS (next section Configuring CORS On a Bucket).




$3

It is also possible to set a file's content encoding in a single command when the file is created:

`js
storage.insert(html, 'your-bucket/a-path/index.html', { contentEncoding: 'gzip' })
.then(({ publicUri }) => console.log(Your gzipped file is available at: ${publicUri}))
`

$3

If your files are publicly readable on the web, they might not be accessible when referenced from other websites. To enable other websites to access your files, you will have to configure CORS on your bucket:

`js
// CONFIGURE CORS ON A BUCKET (warning: Your service account must have the 'roles/storage.admin' role)
bucket.cors.setup({
origin: ['*'],
method: ['GET', 'OPTIONS', 'HEAD', 'POST'],
responseHeader: ['Authorization', 'Origin', 'X-Requested-With', 'Content-Type', 'Accept'],
maxAgeSeconds: 3600
})
.then(() => console.log(CORS successfully set up on your bucket.))
`

If you want to check if CORS has already been set up on a bucket:

`js
bucket.cors.exists().then(yes => yes
? console.log(CORS already set up on bucket '${bucket.name}'.)
: console.log(CORS not set up yet on bucket '${bucket.name}'.))
`

You can also check if a specific CORS config exists:

`js
bucket.cors.exists({
origin: ['*'],
method: ['GET', 'OPTIONS', 'HEAD', 'POST'],
responseHeader: ['Authorization', 'Origin', 'X-Requested-With', 'Content-Type', 'Accept'],
maxAgeSeconds: 3600
}).then(yes => yes
? console.log(CORS already set up on bucket '${bucket.name}'.)
: console.log(CORS not set up yet on bucket '${bucket.name}'.))
`

To remove CORS from a bucket:

`js
bucket.cors.disable().then(() => console.log(CORS successfully disabled on bucket '${bucket.name}'.))
`

$3

To achieve this you need to setup 5 things:

1. You need to setup the service account that you've been using to manage your bucket (defined in your service-account.json) as a domain owner. To achieve that, the first step is to prove your ownership using https://search.google.com/search-console/welcome. When that's done, open the __settings__ and select __User and permissions__. There, you'll be able to add a new owner, which will allow you to add the email of your service account.
2. Create a bucket with a name matching your domain (e.g., www.your-domain-name.com)
3. Make that bucket public. Refer to section Publicly Readable Config above.
4. Add a new CNAME record in your DNS similar to this:

| Type | Name | Value |
|-------|------|----------------------------|
| CNAME | www | c.storage.googleapis.com |

5. Configure the bucket so that each index.html and the 404.html page are the default pages (otherwise, you'll have to explicitly enter http://www.your-domain-name.com/index.html to reach your website instead of simply entering http://www.your-domain-name.com):

`js
bucket.website.setup({
mainPageSuffix: 'index.html',
notFoundPage: '404.html'
}).then(console.log)
`


Zipping files

`js
const bucket = storage.bucket('your-bucket-name')

bucket.object('some-folder-path').zip({
to: {
local: 'some-path-on-your-local-machine',
bucket: {
name: 'another-existing-bucket-name', // Optional (default: Source bucket. In our example, that source bucket is 'your-bucket-name')
path: 'some-folder-path.zip' // Optional (default: 'archive.zip'). If specified, must have the '.zip' extension.
}
},
ignore:[/\.png$/, /\.jpg$/, /\.html$/] // Optional. Array of strings or regex
})
.then(({ count, data }) => {
console.log(${count} files have been zipped)
if (data)
// 'data' is null if the 'options.to' is defined
console.log(The zip file's size is: ${data.length/1024} KB)
})
`

__Extra Options__

You can also track the various steps of the zipping process with the optional on object:

`js
const bucket = storage.bucket('your-bucket-name')

bucket.object('some-folder-path').zip({
to: {
local: 'some-path-on-your-local-machine',
bucket: {
name: 'another-existing-bucket-name', // Optional (default: Source bucket. In our example, that source bucket is 'your-bucket-name')
path: 'some-folder-path.zip' // Optional (default: 'archive.zip'). If specified, must have the '.zip' extension.
}
},
on:{
'files-listed': (files) => {
console.log(Total number of files to be zipped: ${files.count})
console.log(Raw size: ${(files.size/1024/1024).toFixed(1)} MB)
// 'files.data' is an array of all the files' details
},
'file-received': ({ file, size }) => {
console.log(File ${file} (byte size: ${size}) is being zipped)
},
'finished': ({ size }) => {
console.log(Zip process completed. The zip file's size is ${size} bytes)
},
'saved': () => {
console.log('The zipped file has been saved')
},
'error': err => {
console.log(${err.message}\n${err.stack})
}
}
})
.then(({ count, data }) => {
console.log(${count} files have been zipped)
if (data)
// 'data' is null if the 'options.to' is defined
console.log(The zip file's size is: ${data.length/1024} KB)
})
`

Four ways to create a client

This library supports four different ways to create a client. The first method is the recommended way:
1. User the hosting identity
2. Using a service-account.json
3. Using explicit credentials
4. Using environment variables

$3

`js
const storage = client.new()
`

In this case, the package fetches the credentials automatically. It will try three different techniques to get those data, and if none of them work, an error is thrown. Those techniques are:
1. If the code is hosted on GCP (e.g., Cloud Compute, App Engine, Cloud Function or Cloud Run) then the credentials are extracted from the service account associated with the GCP service.
2. If the GOOGLE_APPLICATION_CREDENTIALS environment variable exists, its value is supposed to be the path to a service account JSON key file on the hosting machine.
3. If the ~/.config/gcloud/application_default_credentials.json file exists, then the credentials it contains are used (more about setting that file up below).

When developing on your local environment, use either #2 or #3. #3 is equivalent to being invited by the SysAdmin to the project and granted specific privileges. To set up ~/.config/gcloud/application_default_credentials.json, follow those steps:

- Make sure you have a Google account that can access both the GCP project and the resources you need on GCP.
- Install the GCloud CLI on your environment.
- Execute the following commands:
`
gcloud auth login
gcloud config set project
gcloud auth application-default login
`
The first command logs you in. The second command sets the as your default project. Finally, the third command creates a new ~/.config/gcloud/application_default_credentials.json file with the credentials you need for the project.

$3

We assume that you have created a Service Account in your Google Cloud Account (using IAM) and that you've downloaded a service-account.json (the name of the file does not matter as long as it is a valid json file). The first way to create a client is to provide the path to that service-account.json as shown in the following example:

`js
const storage = client.new({
jsonKeyFile: join(__dirname, './service-account.json')
})
`

$3

This method is similar to the previous one. You should have dowloaded a service-account.json, but instead of providing its path, you provide some of its details explicitly:

`js
const storage = client.new({
credentials: {
project_id: 'your-project-id',
client_email:'something-1234@your-project-id.iam.gserviceaccount.com',
private_key: '-----BEGIN PRIVATE KEY-----\n123456789-----END PRIVATE KEY-----\n'
}
})
`

$3

`js
const storage = client.new()
`

The above will only work if all the following environment variables are set:
- GOOGLE_CLOUD_BUCKET_PROJECT_ID or GOOGLE_CLOUD_PROJECT_ID
- GOOGLE_CLOUD_BUCKET_CLIENT_EMAIL or GOOGLE_CLOUD_CLIENT_EMAIL
- GOOGLE_CLOUD_BUCKET_PRIVATE_KEY or GOOGLE_CLOUD_PRIVATE_KEY

> WARNING: If you're using NPM's dotenv, wrap your PRIVATE_KEY between double-quotes, otherwise some characters are escaped which corrupts the key.

Refer to the next section to see how to pass an OAuth2 token.

Extra Precautions To Make Robust Queries

$3

Networks errors (e.g. socket hang up, connect ECONNREFUSED) are a fact of life. To deal with those undeterministic errors, this library uses a simple exponential back off retry strategy, which will reprocess your read or write request for 10 seconds by default. You can increase that retry period as follow:

`js
// Retry timeout for CHECKING FILE EXISTS
storage.exists('your-bucket/a-path/image.jpg', { timeout: 30000 }) // 30 seconds retry period timeout

// Retry timeout for INSERTS
storage.insert(someObject, 'your-bucket/a-path/filename.json', { timeout: 30000 }) // 30 seconds retry period timeout

// Retry timeout for QUERIES
storage.get('your-bucket/a-path/filename.json', { timeout: 30000 }) // 30 seconds retry period timeout
`

Using An External OAuth2 Token

If you've used the 3rd method to create a client (i.e. 3. Using a ProjectId), then all the method you use require an explicit OAuth2 token:

`js
storage.list({ token }).then(console.log)
`

All method accept a last optional argument object.

Performance Tips

The Google Cloud Storage API supports partial response. This allows to only return specific fields rather than all of them, which can improve performances if you're querying a lot of objects. The only method that currently supports partial response is the the list API.

`js
storage.bucket('your-bucket').object('a-folder/').list({ fields:['name'] })
`

The above example only returns the name field. The full list of supported fields is detailed under [bucketObject.list([options])](#bucketobjectlistoptions-promisearrayobject) section.

Full API Doc

Storage API

> To create the storage object, please refer to the previous section 3 ways to create a client.

This object allows to perform most read/write operations. It uses a string representing the path to where the objects or folders are. If using path is the stragegy you decide to employ to manage objects, then the Storage API is the way to go. If, on the contrary, you need to reason based on a specific bucket or a specific object, then it is recommended to use the Bucket API or the BucketObject API.

$3

storage.get('your-bucket-name/path-to-file/your-file.css').then(console.log)

This writes a CSS content to the output.

Gets an object located under the bucket's filePath path.
* filePath
* options