Import and manage Google Cloud infrastructure data in Lucidscale

Supported Google Cloud resources and lines

For a comprehensive list of the supported resources and lines from Google Cloud that are available in Lucidscale refer to our Supported Google Cloud resources and lines article.

Watch: Import Google Cloud infrastructure

Import Google Cloud infrastructure

You can import Google Cloud Infrastructure using one of the two methods: 

• Trusted access: provide a Google Cloud Service Account with the proper policies. This service account can be set with the following permission scopes:
  • Import via project-level access to import one or more individual projects
  • Import via organizational-level access to import all projects within a Google Cloud Organization
• File upload: execute a script in your Google Cloud Command Line Interface (CLI), which gathers the data locally as a JSON file, then upload the generated file.
  

Check out the following table for specifics on each method and to determine which is best for your use case.

*Resource type 1 is all mapped resources listed in the Supported resource section. Resource type 2 is Cloud Inventory: a broad variety of unmapped resources you can place manually in your models.
**Automatic data source refreshes are available for users to set on a recurring schedule (daily, weekly, monthly, yearly) via the data hub. Manual refreshes are performed with a single click through the data hub as well.

With trusted access imports, Lucid saves these credentials so you can refresh your data or start new imports without re-entering your information. This storage capability allows Lucid to perform the automatic data refreshes. Credentials can be deleted via the Data Hub credentials tab.

If you are working with a very large environment, use the instructions found in the Project configuration section of this article to help reduce the amount of data you are importing or reduce the number of API calls Lucidscale makes to Google Cloud. 

To render Google Cloud infrastructure in a Lucidscale document:

1. Click + New on the Lucid Home Page.
2. Hover your cursor over "Lucidscale".
3. Select Google Cloud document.
4. Choose to create a document with a previously imported Google Cloud project or click Open Data Hub to import a new project.
5. Click + New Project and choose an import method.
   
   • After choosing an import method and completing the import, the project will be listed as a data source in the "Google Cloud” section in the Data Hub. You can also find it listed as an “imported” Google Cloud project in the modal after clicking + New > Lucidscale > Google Cloud document.
6. Click the three-dot menu next to the account you want to create a document from.
7. From the menu, select Create new document.
8. Review details and select views to only add necessary resources.
   
9. Select Create Google Cloud document.

The Lucidscale document will generate in the background of your Lucid account. We will notify you once it has been rendered through the status modal that appears in the bottom-right of the page.

You can open the document from the status modal that renders directly after creating a new document. The document (and previously created documents) will also be stored and accessible in the "Documents" section of your Lucid Home Page after you generate it.

Import projects and create documents via Service Account (Recommended)

To import Google Cloud infrastructure via service account and render a Lucidscale document, follow these steps:

1. On the Lucid Home Page, click + New.
2. Hover your cursor over Lucidscale.
3. Select Google Cloud document.
4. Click + New Project.
5. Select Google Cloud Service Account (Recommended).
6. Click Create a new Service Account.
7. In Lucid, select between the two options to assign a role to the Service Account. Click the following link that corresponds with the selection you made in Lucid:
   • Use a read-only role for all resources (Recommended)
   • Create a custom role with minimum permissions

Step 2: Assign a role to the Service Account using a read-only role for all resources (Recommended)

1. In the next modal under “Step 1: Create a new Service Account”, click Create Service Account to open a new tab and log in to your Google Cloud account. Leave the Lucid browser tab open so you can return to it as needed. 
2. Ensure the “Use a read-only role for all resources (Recommended)” option is selected.
3. In your Google Cloud console, click Create Service Account.
4. Add the Service Account Details and click Create and Continue.
5. Click the “Select a role” dropdown menu.
6. Select Basic.
7. Click Viewer.
8. Click Done.
9. Skip to Step 3: Generate a JSON key and add the service account to Lucidscale.

Step 2: Assign a role to the Service Account by creating a custom role with minimum permissions

1. In your Lucid tab, select the “Create a custom role with minimum permissions” option.
   
2. Click Download luciscale.yaml to download the yaml file to your computer.
3. Select the Organization Level or Project Level tab depending on which level of data you want to visualize within Lucidscale.
   • In this example, we will be using the Project Level option.
4. Copy the resulting line of code displayed.
5. Navigate to your Google Cloud console and sign in.
6. Click on Activate Cloud Shell.
7. Click the three-dot icon above the Cloud Shell terminal and select Upload.
8. Browse to the previously downloaded yaml file and upload it.
9. Paste the copied line of code (from step 4 above) into the terminal and replace the following values within the line of code:
   • role-id = the value that you want to use for your role ID
   • project-id = your Google Cloud project ID
   • yaml-file-path = path to the .yaml file
   • The end result should look something like this (using fake values):
     • gcloud iam roles create lucidscalerole --project=test-project-id --file=lucidscale.yaml
   • If you are using the Organizational Level line of code, the example would look something like this (also using fake values):
     • gcloud iam roles create lucidscalerole --organization=test-organization-id --file=lucidscale.yaml
10. Click Authorize.
11. In the terminal, you will be given the prompt “Are you sure you want to make this change (Y/N)?”. Type in “Y” and click Enter.
12. Continue on to Step 3: Generate a JSON key and add the Service Account to Lucidscale below.

Step 3: Generate a JSON key and add the Service Account to Lucidscale

1. In your Google Cloud console, select the Service Account you created in step 1.8.
2. Click the “Keys” tab.
3. Select Add Key and click Create new key.
4. Choose JSON as the “key type”.
5. Click Create.
6. A JSON file will be downloaded to your device.
7. Navigate back to your open Lucid browser tab.
8. Select Choose file.
9. Locate and select to upload the JSON file you downloaded in step 6.
10. Click Next.

Step 4: Add projects and create documents

1. Select the project(s) you want to import to the Data Hub and use to generate a Lucidscale document.
   
2. Click Next.
3. Select resources to import.
   
4. Select Import Google Cloud projects.

You can open the document from the status modal that renders directly after creating a new document. The document (and previously created documents) will also be stored and accessible in the "Documents" section of your Lucid Home Page after you generate it.

Project configuration using a Service Account

When importing a Google Cloud project to the Lucid Data Hub using the service account method, you can omit chosen resource types or metadata from the import. This grants you the ability to visualize only resource types you’re interested in and in some cases can increase the speed of the import. Deselected resource types and metadata will not be imported or rendered in the diagram.

Import projects and create documents via Google Cloud CLI

Install Python if you don't have it already. Instructions can be found here. Ensure that you check the option “Add Python to PATH” in the installer. You can verify that Python was properly installed by trying the commands “python” or “python 3” in a terminal (MAC OS) or Command Prompt (Windows).

1. On the Lucid Home Page, click + New.
2. Hover your cursor over Lucidscale.
3. Select Google Cloud document.
4. Choose to create a document from an existing Google Cloud project or click Open Data Hub to import and render a new project. 
5. Click + New Project.
6. Select Google Cloud CLI Script.
7. Open your terminal or command prompt, and run the command: Pip3 install oauth2client google-api-python-client
8. Initialize the Gcloud SDK by running the command: gcloud init
9. Authenticate the Gcloud SDK by running the command: gcloud auth application-default login
10. In the “New Google Cloud Project” modal in Lucid, click Download Script. 
    • A file named “gcpcliscript.py” will be downloaded.
      
       
11. Browse to the downloaded location of the script, and run the command: python3 gcpcliscript.py --projects [project_id [project_ids...]] [-c] [-o <output file>]
    • project_ids: Project ids, can be found on the Google Cloud console under project list
    • -c: outputs the count.json file which contains the count of compute resources in the specified Google Cloud projects (optional)
    • -o: name of the output file
    • e.g. python3 gcpcliscript.py --projects bubbly-xxxxxx-233220 summer-alliance-254420 gcpdev-272652
12. Return to the “New Google Cloud Project” modal in Lucid and click Choose File.
13. Select and upload the gcp.json generated from the script.
14. Click Next.
15. Edit the name of the project and click Next.
    • After the import completes, the project will be listed as an Imported Google Cloud project in the modal after clicking + New > Lucidscale > Google Cloud document.
    • The project will also be displayed in the Data Hub > Google Cloud: Imported Projects section.
16. Select Create new document.
17. Review the details and add Filters to exclude unnecessary resources.
18. Select Create Google Cloud document.

The Lucidscale document will generate in the background of your Lucid account. We will notify you once it has been rendered through the status modal that appears in the bottom-right of the page.

You can open the document from the status modal that renders directly after creating a new document. The document (and previously created documents) will also be stored and accessible in the "Documents" section of your Lucid Home Page after you generate it.

Use the Data Hub to manage imported Google Cloud Infrastructure data and credentials

Imported projects

After importing the Google Cloud project data, it will be listed as a data source with the project name you specified in the "Google Cloud: Imported Projects" section of the Lucid Data Hub. From this section you can also click + New Project to import a new project to the Data Hub. The Google Cloud projects you have imported will be listed individually in the Data Hub.

Projects listed in this section are data sources that can be used to create new Lucidscale documents. Data sources (“Projects”) contain the Google Cloud project metadata used to create Lucidscale documents. The “Last updated” field indicates when your data source was last refreshed to account for any updates in your connected Google Cloud project.

The Cloud Inventory of each project can be seen on the right-hand side after selecting a project from the list. This inventory allows you to see a full count of each type of resource that has been imported before creating a document. You can also download a CSV with the list of resources and count directly from the Data Hub by clicking Download CSV.

The inventory lists all resources that can be accessed by the role permissions assigned to your Service Account. If you elect to “Use a read-only role for all resources”, all resource types in your project which are accessible by Google Cloud's “Viewer” role will be rendered in the inventory. If you create a custom role, only the resource types specified in your assigned role permissions will be rendered in the inventory. 

Select the three-dot menu rendered on a project to see up to six potential options: Create new document, Update project from Google Cloud, Schedule update, Edit resources, Rename Project, and Delete Project.

Create new document will start rendering the Google Cloud infrastructure associated with the project as a diagram in a new document's workspace. The document will be generated in the background of your Lucid Home Page and you’ll be notified once it has been rendered through the status modal that appears in the bottom right of the page.

Update Project from Google Cloud and Schedule update allows you to manually or automatically update the Google Cloud project data you have imported to the Data Hub to reflect any changes in the connected Google Cloud project. New documents created from the project after the update will reflect any changes in the connected Google Cloud project. 

Edit resources allows you to edit the resources included in each dataset. Selecting this option will open up a modal in which you can add or remove any resources from existing datasets. When you’ve finished with the selection, click Save and update Project. This will ensure that the dataset refreshes and documents referencing that dataset are updated to the latest information.
 

Rename Project allows you to change the name of the selected dataset.

Delete Project removes the imported project from the Lucid Data Hub and any documents created from that project would lose the ability to update data from Google Cloud. You can import the same Google Cloud project more than once — each creates an independent data source. Deleting an existing project is not required before re-importing it.

Saved credentials

The Saved Credentials section in "Data Hub > Google Cloud" will list the Google Cloud service accounts you can use to import Google Cloud projects to the Data Hub. From this section you can click + New Service account to add a new set of credentials that can be used to import a Google Cloud project.  

You can also share the service accounts you have created with other users. If you click the three-dot menu rendered on a service account, you will see two potential options: Share credential and Delete credential.

Share credential opens a modal where you can share a service account you created with another user on your Lucid account by specifying their email address. Granting them access to the service account will allow them to import the associated Google Cloud project to the Data Hub and create Lucidscale documents. After a credential has been shared with another user, you can revoke their access to the credential by clicking the X next to their email address in the same modal.

Delete credential removes a service account from the Data Hub. After deleting a service account, all projects that were previously imported with that credential along with documents created from those projects would lose the ability to update data from Google Cloud.  

FAQ

How does Lucidscale get access to our Google Cloud environment? What level of access does Lucidscale have?
The recommended method from Lucidscale and Google Cloud is through a Google Cloud Service Account. We provide a policy which gives us “describe and list” access to your environment so we can get an inventory of resources.

We also have a method using a python script you can run through the Command Line Interface (CLI) which generates a JSON file. Users can then upload this JSON file to Lucidscale and not populate any credential information into Lucidscale. Your imported JSON will be accessible in the Lucid Data Hub to use for creating Lucidscale documents.

What APIs does Lucidscale access?
We primarily use describe and list API endpoints.

Can you choose which metadata (the fields and values shown in the Data panel) is imported? If not, what metadata is imported?
No, you can't currently choose what metadata is imported. However, we are looking to create some ability to do that in the future. All metadata from a describe API call is imported.

Give feedback on this article