Questions tagged [google-cloud-storage]

Google Cloud Storage is a RESTful service for storing and accessing your data on Google's infrastructure. The service combines the performance and scalability of Google's cloud with advanced security and sharing capabilities.

0
votes
0answers
8 views

write to Google Cloud Storage using spark to absolute path

I am trying to write a spark dataframe into google cloud storage. This dataframe has got some updates so I need a partition strategy. SO I need to write it into exact file in GCS. i have Created a ...
0
votes
1answer
17 views

ERROR: build step 0 “gcr.io/cloud-builders/docker” failed: exit status 1

I am trying to deploy my first app in google cloud bucket by using bitbucket pipeline, but I am getting the following error in google cloud console. ERROR: build step 0 "gcr.io/cloud-builders/docker" ...
1
vote
1answer
28 views

Apache Flink: How to sink stream to Google Cloud Storage File System

I was trying yo write Some Data Streams into a file in Google Cloud Storage File System as following (Using Flink 1.8 and Scala 2.11): data.addSink(new BucketingSink[(String, Int)]("gs://url-try/try....
0
votes
1answer
23 views

How to upload a file to GCS using javascript?

I am trying to upload a file to GCS using javascript. I found this answer which shows how to upload and see the progress of that file. I also followed the official documentation All of the above ...
0
votes
1answer
23 views

Recommendation for Google Cloud Storage for application

I have developed an application using Cloud ML Engine that recognizes images from uploaded images. I want to test the application and allow specific people to upload images for the next 24 hours. Not ...
0
votes
1answer
15 views

Google Cloud Storage user-centric authentication

My aim to make Google Cloud Storage files accessible directly from clients with an access token (without Gmail, Facebook etc.). I want an architecture like that: While creating a Storage file, the ...
0
votes
0answers
30 views

Google Cloud Storage upload fails with ApiError

I have a cloud function that uploads csv files to cloud storage bucket. After some time, the api starts failing with the following error. This seems to have failed on HTTP call after retries, but ...
0
votes
0answers
19 views

Failed to upload full image to Firebase storage in web using php

I am using laravel and trying to upload image to Firebase storage. When I upload image to Firebase storage, my image with given name is stored in Firebase storage but the size of image is 24 bytes. ...
0
votes
1answer
18 views

Getting spider on Scrapy Cloud to store files on Google Cloud Storage using GCSFilesStore and getting ImportError

Deploying a spider to Scraping Cloud. It gathers download links for files and should save those files in a Google Cloud bucket. It works when running locally. But when deploying to Scraping Hub it ...
1
vote
1answer
29 views

How to upload file to google cloud bucket using signedUrl

I am developing an Angular App, to display content of a google cloud storage bucket. For the back, I am using google cloud functions in nodeJS As they mentioned in the documentation to upload a file, ...
0
votes
0answers
23 views

Angular (Ionic) upload file to Google Cloud Storage bucket

I'm looking for a code sample of uploading a file from Angular 7 (Ionic 4) to a GCP storage bucket. Here's the doc page, but there's no JavaScript or TypeScript sample: https://cloud.google.com/...
0
votes
0answers
14 views

AWS EMR With Flink application for Apache_beam runner, java.nio.channels.ClosedByInterruptException error for reading files from GCS Ask Question

I have an EMR cluster with flink application on it. Running apache-beam job using flink runner. Cluster started reading and works fine for 30 to 40 min. Then suddenly channel closes and interrupted....
2
votes
0answers
39 views

Issues with Dynamic Destinations in Dataflow

I have a Dataflow job that reads data from pubsub and based on the time and filename writes the contents to GCS where the folder path is based on the YYYY/MM/DD. This allows files to be generated in ...
0
votes
0answers
31 views

Retry limit exceeded when uploading file

Context: I'm working on code which uses a read stream to download a file from an SFTP server and upload it to GCS via a writeStream, using Nodejs v10.15.3. Due to an error in the SFTP library I'm ...
0
votes
1answer
20 views

How to correctly use `gsutil -q stat` in scripts?

I am creating a KSH script to check whether a subdirectory is exist on GCS bucket. I am writing the script like this: #!/bin/ksh set -e set -o pipefail gsutil -q stat ${DESTINATION_PATH}/ ...
0
votes
1answer
33 views

How do I delete an image with cloud functions with download url?

I want to delete an image. All I have is the download url. In flutter i am able to get file path from download url and use that path to delete the file in cloud storage. Is that possible to get the ...
1
vote
2answers
56 views

How to enable real time object access analytics in google cloud storage?

I am trying to set up real time access logs for objects stored in a google cloud storage bucket. The access logs that google specifies here are generated every hour and won't work. I would like ...
-2
votes
0answers
33 views

How to deploy a vue.js web site on Google Cloud Storage?

I download a vue.js template and work fine on local (http://localhost:8080), and I can compile and generate /dist file, but not working on Google Cloud Storage. Only show a blank page. This is my ...
1
vote
1answer
59 views

Google Cloud Storage: CORS settings doesn't work for signed URLs

The response of PUT request with signed URL doesn't contain header Access-Control-Allow-Origin. import os from datetime import timedelta import requests from google.cloud import storage os.environ['...
1
vote
2answers
31 views

Cloud fuction storage trigger on bucket of other project

Not able to create cloud function trigger on bucket which is under other project. Deploying function (may take a while - up to 2 minutes)...failed. ERROR: (gcloud.functions.deploy) ...
0
votes
1answer
28 views

Performing datastore backup to cloud storage

I am attempting to create a scheduled backup of datastore via my Python Flask application (Python 3) to cloud storage. I am comfortable with the scheduling aspect of it however am having difficulty ...
0
votes
1answer
26 views

Does gsutil support copying files from Azure Storage?

gsutil supports copying files from S3 to GCS like mentioned in this blog. Does gsutil tool also supports copying files from Azure?
0
votes
0answers
25 views

GCP CDN singed URL “ServiceException: 409 Bucket XXXXX already exists”

I followed this link to config CDN signed URLs. But get error after: gsutil iam ch \ serviceAccount:service-[PROJECT_NUM]@cloud-cdn-fill.iam.gserviceaccount.com:objectViewer \ gs://[BUCKET] Here ...
0
votes
1answer
27 views

gcloud init cli command :ERROR: gcloud crashed (ValueError): the query contains a null character

I am trying to initialize my gcloud settings for a project. But when I run the gcloud init command, it gives error gcloud crashed. It was previously working but all of sudden today this command ...
0
votes
1answer
36 views

How to open an XML file with python client for google cloud storage

Im a total noob and have no other way of solving this so I need to do it with Python. I need to parse an xml-file. This works fine when the file is stored locally. But, I need to be able to open the ...
0
votes
0answers
15 views

Accessing GCS with Hadoop client from outside of Cloud

I want to access Google Cloud Storage via Hadoop client. I want to use it on machine outside of Google Cloud. I followed instructions from here. I created service account and generated key file. I ...
0
votes
1answer
51 views

Google Cloud Storage URL Signing

I have a signer that works great for uploading and downloading from GCS. Great! The signer looks like this: def sign_url(client, bucket, key, opts \\ []) do verb = opts[:verb] || "GET" ...
0
votes
1answer
37 views

Storing image references in Firestore

I'm creating a blog from scratch. I know firestore isn't the place to store images, but cloud storage is. I've learned you only need a reference to the images you store in cloud storage to retrieve ...
0
votes
1answer
40 views

Load Avro file to GCS with nested record using customized column name

I was trying to load an Avro file with nested record. One of the record was having a union of schema. When loaded to BigQuery, it created a very long name like com_mycompany_data_nestedClassname_value ...
1
vote
1answer
35 views

Unable to unzip very big files from/to google buckets when mounted with gcsfuse

On Google Cloud I have a linux Compute Engine and a bucket. I have mounted the bucket as a drive to the CE using gcsfuse - as recommended by Google - and from time to time I have had a big 7zip ...
0
votes
0answers
31 views

Storage Transfer Service To backup Google Firebase Storage

I have a mobile application that stores data in Google firebase database and firebase Storage service. To prevent data loss, I would like to backup the data stored on both services. Google recently ...
0
votes
2answers
40 views

How to mark files in a GCP storage bucket as Read Only for “everyone”?

We have a storage bucket which has sensitive data stored in it, in the form of JSON files. For a PCI compliance, we have to make these files read only. Versioning will be enabled on the bucket so ...
1
vote
0answers
36 views

How to fix domain DNS configurations when working with bucket based load balancer on GCP?

I have set up a static website using GCP Bucket Storage and loaded SSL certificate using HTTPS Load Balancer. My DNS record is set to: CNAME pointing to to bucket with www, (c.storage.googleapis.com) ...
0
votes
1answer
93 views

Cleaning data in CSV files using dataflow

I am trying to read a CSV (with header) file from GCS which has about 150 columns and then 1. Set column data for particular columns 2. Update NaN with Null values for all columns 3. Write the csv ...
1
vote
1answer
46 views

Loading data into BigQuery from Cloud Storage using Cloud Function (Alternative to functions?)

I'm currently loading data from a csv file in Cloud Storage to BigQuery using a Cloud Function. However, Cloud Functions have a timeout limit and I expect these files to get big enough where it will ...
0
votes
1answer
34 views

Firebase cloud function storage trigger first thumbnail urls are fine then the next ones are all the same thumbnails urls as the first

I am trying to upload an image to firebase and then produce 2 thumbnails. I am able to do this with no problems. My current road block is when I write the urls to the realtime database, I am always ...
0
votes
0answers
44 views

Write an image to Google Storage throwing API CallNotFoundExcpetion

I'm working on trying to store images in Google Storage and found some test code via one of their forums. I've added the various dependencies and was looking to step through the code however I've ...
0
votes
1answer
63 views

How to fix “AttributeError: 'module' object has no attribute 'storage'” error in python2.7

In apache beam pipeline, I am taking input from cloud storage and trying to write it in biqguery table. But during the execution of pipeline getting this error. "AttributeError: 'module' object has no ...
0
votes
3answers
53 views

How to upload an image to Firebase? Is it possible?

I'm tring to upload an image to firebase cloud function. I think that I have read the whole of post about this and I can't get a solution. I have used "formidable.IncomingForm()", "multer" and "...
0
votes
0answers
31 views

How to upload images on React to Google Cloud Storage with Ruby as backend

I'm attempting to upload images from ReactJS to Google Cloud Storage with Ruby as my backend. I have attempted to send these images through an API I made with Rails, and then upload them directly into ...
0
votes
1answer
33 views

Check if blob exists

I'm trying to build a DAG that first checks if a given path/blob in the Google Cloud Storage exists. The blob is the one containing some RAW data and is not the one mounted on the Composer workers. ...
0
votes
2answers
28 views

Google Cloud SDK Python Client: How to list files inside of a Cloud Storage bucket?

Trying to use Python to get and iterate through all of the files inside of a Cloud Storage bucket I own. I'm using the official library, google-cloud-storage. Using gsutil, I can run commands like ...
0
votes
1answer
41 views

How do I find the owner of a Google Cloud Storage object

My App Engine is run by a service account. It writes files to a bucket. How can I see the service account as owner of the file objects?
0
votes
1answer
29 views

Google Cloud Storage Ajax Request Not Working

I have 2 files on google cloud storage: a.wav and b.txt I want to play a.opus in my browser, which is working fine. And I want to read b.txt Google Docs tell that if the user is authenticated in the ...
2
votes
3answers
80 views

can't upload file to google cloud storage in production

I'm following this documentation to upload files to GCS. Setting Up Authentication for Server to Server Production Applications It works in local but in production i get this error: Post https://...
0
votes
1answer
41 views

How to get serve url without AppEngine for Google Cloud Storage?

I have Google Cloud Storage set up, is there is a way to get serve url without Google App Engine? I don't want to use Google App Engine, I want to use Spring Framework instead. Is there is a library ...
1
vote
1answer
36 views

Getting status of Google Cloud transfer Job

I am using node.js utility of google cloud transfer api to transfer data from s3 bucket to my Google Cloud bucket using #1 storagetransfer.transferJobs.create(request, function (err, response) And ...
1
vote
1answer
40 views

How can I make a Google Cloud Storage object publicly visible while uploading it?

I have an application which is uploading objects to Google Cloud Storage using signed URLs and I'd like to know if it's possible to make the object public during the sign/upload step. I know it's ...
1
vote
1answer
32 views

Should I use Cloud Storage or Firestore for loading static file which is required frequently

I am having a situation that everyday there will be around 1000 rows of data generated from postgresql db. These data will be served for frontend, which will be frequently call. Actually for each ...
0
votes
0answers
57 views

How can I use the GOOGLE_APPLICATION_CREDENTIALS environment variable to sign a Cloud Storage URL?

How can I use the implicit GOOGLE_APPLICATION_CREDENTIALS environment variable provided to a Cloud Function to sign a Cloud Storage URL? The following works locally when a ...

http://mssss.yulina-kosm.ru