Copy Directory Tree from Colab to MyDrive don't work or work very slow

How do I copy or move huge Directories files from CoLab drive to MyDrive on google? Because the regular code !cp doesn't work properly or work extremally slow.

I am trying to move a large folders from the colab drive to me google drive so it will be stored there. The data is in a 2.9G file (with 80,000 images). The download of the zip file from kagel take few second Then I using the following code:

from google.colab import drive
drive.mount('/content/drive',force_remount=True)
DS_dir=r"/content/drive/MyDrive/Colab Notebooks/AgroML/input"
#Unzip file
with ZipFile('/content/archive.zip','r') as Zipobj: Zipobj.extractall('/content/drive/MyDrive/archive')

|If I am trying to unzip the file directly to MyDrive? it is starting but never end. And after 30 minute or so? execution's is terminated. without extracting all the files

So i unziped

with ZipFile('/content/archive.zip','r') as Zipobj: Zipobj.extractall('/content/archive') 

And it did work fines after 75 second or so. And on my ColabPro drive I have a directory tree with 80,000 files. And I am trying to copy the files from the colab drive to MyDrive with the following code

%cp -av "/content/archive" "/content/drive/MyDrive/Colab Notebooks/download" #>copylog.txt

which runs for ever till it crash, it will copy about 20% of the files. (just 17000 files had been copied) So I had moved the code to another Google Drive and Colab account (with much less data) and it did work. It Toke 9 minutes but it copy the entire tree of 3G and 80000 files.

My questions are

  1. Why I cant work directly to MyDrive and is there a different between The colab drive and MyDrive from a program? why Can't I unzip directly to MyDrive
  2. Is there a lib or utile that allow my to move/copy directory tree on the cloud it self when I tried the shutil for example? it toke much more time.
  3. How come I can download 3G from Kagel in few seconds abut can't unzip it to Mydrive directly and How and why I am unable to copy the directory tree on one colab/Gdrive account and able to do on another account? (both are ProColab and payed Gdrive acounts) The Gdrive that I am succeeding has about 20 Gig and and the other one has 450 Gig with much much more files.
    This is not a "onetime" problem and manual solution's will not help me. thanks Adi
Reset to default

Know someone who can answer? Share a link to this question via email, Twitter, or Facebook.

Your Answer

Sign up or log in

Sign up using Google Sign up using Facebook Sign up using Email and Password

Post as a guest

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

You Might Also Like