v2.42.0 appears to break custom scopes for impersonated credentials in Vertex AI Pipelines

[uros-r]

The Problem

Upgrading from 2.41.1 to 2.42.0+ causes 403: Request had insufficient authentication scopes when accessing Google Sheets API from Vertex AI Pipelines. Same code, same service account, same IAM permissions.

Error:

google.api_core.exceptions.Forbidden: 403 POST https://sheets.googleapis.com/v4/spreadsheets/[ID]/values:batchUpdate:
Request had insufficient authentication scopes.

Context:

• Oct 28, 2024: v2.42.0 released (includes PR fix: Read scopes from ADC json for impersoanted cred #1820: "fix: Read scopes from ADC json for impersonated cred")
• Nov 1, 2024: Docker build installed 2.41.1 → pipeline succeeded
• Nov 2, 2024: Docker build installed 2.42.1 → 403 scope error

Only known change between builds was google-auth version (unpinned as google-auth>=2.0.0).

In addition:

• SA has Editor permission on the target Google Sheet
• SA able to successfully authenticate to other GCP APIs
• IAM audit logs show no permission changes between working and broken builds

Environment

Platform: Vertex AI Pipelines (KFP v2.14.5), Python 3.12 in Docker container

Authentication: Impersonated service account credentials via Vertex AI runtime (ADC from GCE metadata service)

Code:

from google.auth import default

credentials, _ = default(scopes=[
    'https://www.googleapis.com/auth/drive.readonly',
    'https://www.googleapis.com/auth/spreadsheets'
])

Works in 2.41.1, fails for us in 2.42.

Workaround

google-auth>=2.0.0,<2.42.0

Questions

• Is this now intended behaviour?
• If so, what's the recommended approach for Vertex AI Pipelines needing custom API scopes?

[uros-r]

After further investigation, it appears the failures might also relate to Vertex AI Pipelines job startup. API calls seem to only fail when made at the beginning of the job, on a 'cold start'. Subsequently they appear to work regardless of google-auth version.

Closing for now as no longer sure this is a 2.42 issue. Will reopen if necessary.