Upload Files to Google Drive From Ubuntu

Uploading files to Google Drive directly from the Terminal (using Curlicue)

An alternative method of pushing data from a computer to the cloud.

In many cases, it may be hard to send data from a new automobile to another. Examples of this include HPC facilities which are hidden backside a login portal but do not allow ssh tunnelling or elementary headless machines which only have a few cadre programs installed. In this article, we wait at using cURL (a command-line programme for transferring data) to push button a zipped file (containing log files) onto our google bulldoze business relationship for further analysis.

Installation

Most machines will come up with ringlet installed (try typing which coil). If this is not the case we tin install information technology with

            sudo apt install coil # Linux Debian/Ubuntu

            brew install curl     # Mac

Now we have it installed, we can look at creating the credentials needed to send files.

Create your project credentials

Every bit we are allowing access to our google bulldoze, we desire to be able to manage this. This is washed by creating a project with user-divers permissions to act equally the proxy betwixt our users (in this case united states on a different automobile) and our business relationship. We start past going to the post-obit folio (link below) and creating a new project.

Afterwards this has been done we select the Credentials tab (on the left) and "create credentials" from the summit.

When asked for the app type, we select Tv and other.

Finally, this generates a client id and a client seacret .

This is your username and countersign then copy it somewhere secure.

Now nosotros need to verify the device

To practice this nosotros ssh into the automobile we wish to upload from and run the following command:

            curl -d "client_id=<client_id>&scope=              https://www.googleapis.com/auth/drive.file              " https://oauth2.googleapis.com/device/code

Doing so nosotros become a response in the post-obit format

            {"device_code": "<long cord>",
"user_code": "xxx-xxx-thirty",
"expires_in": 1800,
"interval": 5,
"verification_url": "https://www.google.com/device"}

Here we demand to visit the URL (https://www.google.com/device) and provide the user lawmaking to consummate our verification. We now continue to select our google account and grant the relevant permissions. When doing this make certain to note down the device code for the next footstep.

Get Bearer code

When we commencement uploading, this is the code we shall need to use to identify our account. We become it by using the following:

            roll -d client_id=<customer id>              -d client_secret=<client undercover>              -d device_code=<device code>              -d grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Adevice_code https://accounts.google.com/o/oauth2/token

The client id and hugger-mugger are saved from the get-go step, and the device code in the previous section. The output should exist in the format:

            {
"access_token": ".....",
"expires_in": 3599,
"refresh_token": "....",
"scope": "https://www.googleapis.com/auth/drive.file",
"token_type": "Bearer"
}

Write down the access_token as information technology will be needed in the upload stage.

Upload files

The control we apply to upload files is given below

            curlicue -10 POST -L \
              -H "Authorization: Bearer              <enter                            access token here              >" \
              -F "metadata={name :'<our.naught>'};blazon=awarding/json;charset=UTF-eight" \
              -F "file=@<our.null>;type=awarding/aught" \
              "https://world wide web.googleapis.com/upload/drive/v3/files?uploadType=multipart"

Here you may demand to enable the app API before existence allowed to upload data. The link to do this is given in the error message if this is the case.

Here multipart files are expected to only be a couple of MB in size. However if you are looking at moving larger files resumable may exist better suited (run into https://developers.google.com/bulldoze/api/v3/manage-uploads )

Wrap it all up in a script

Now nosotros know our commands work we tin can create an executable script to do all the work for us. Hither we can provide a group of files, it zips them upward and then sends them to google drive.

We start past creating a new file with nano curlgoogle; and enter the following code — remember to add your own personal auth token! Python 2.7 has been chosen as this is however the default python version on older systems, all the same the script beneath should also run for python 3.

Information technology should require no new dependencies provided coil already exists on the system.

            #!/usr/bin/python            '''
A quick python script to automate curlicue->googledrive interfacing
This should require nil more than than the system python version and curl. Written for python2.vii (with three in listen).            Dan Ellis 2020
'''            import bone,sys,json            if sys.version[0]=='3':
              raw_input = lambda(10): input(x)            ##############################
#Owner information goes here!#
##############################
name = 'curldata'
client_id= '<enter your client id>'
client_secret='<enter your client undercover>'            ##############################                          
cmd1 = json.loads(bone.popen('curlicue -d "client_id=%due south&scope=https://www.googleapis.com/auth/drive.file" https://oauth2.googleapis.com/device/code'%client_id).read())            str(raw_input('\north Enter %(user_code)s\n\n at %(verification_url)southward \n\n So hit Enter to continue.'%cmd1))            str(raw_input('(twice)'))            cmd2 = json.loads(bone.popen(('curl -d client_id=%s -d client_secret=%s -d device_code=%due south -d grant_type=urn~~3Aietf~~3Aparams~~3Aoauth~~3Agrant-type~~3Adevice_code https://accounts.google.com/o/oauth2/token'%(client_id,client_secret,cmd1['device_code'])).replace('~~','%')).read())
print(cmd2)            # zip files
cmd3 = os.popen('naught -r %s.cipher %s'%(proper noun,' '.join(sys.argv[1:]))).read
impress(cmd3)            cmd4 = os.popen('''
curl -X Postal service -L \
              -H "Authorization: Bearer %s" \
              -F "metadata={proper name :\'%due south\'};blazon=application/json;charset=UTF-8" \
              -F "file=@%south.zip;type=application/zip" \
              "https://world wide web.googleapis.com/upload/drive/v3/files?uploadType=multipart"
              '''%(cmd2["access_token"],name,name)).read()            print(cmd4)
impress('terminate')

Nosotros then brand information technology executable chmod a+x curlgoogle allowing united states of america to use information technology in an executable manner:

                          ./curlgoogle              file1 file2.txt file3.jpg etc...

Conclusions

And at that place we have it, an easy way to ship multiple log files from a headless automobile to a google bulldoze repository, which tin can be accessed by multiple people for analysis.

If you are in need of more information, stack overflow answers by Tanaike and HAKS (amid others) were particularly helpful in creating this mail.

hallidayfebt1938.blogspot.com

Source: https://towardsdatascience.com/uploading-files-to-google-drive-directly-from-the-terminal-using-curl-2b89db28bb06

Halliday Febt1938