How do I copy or move huge Directories files from CoLab drive to MyDrive on google? Because the regular code !cp doesn't work properly or work extremally slow.
I am trying to move a large folders from the colab drive to me google drive so it will be stored there. The data is in a 2.9G file (with 80,000 images). The download of the zip file from kagel take few second Then I using the following code:
from google.colab import drive
drive.mount('/content/drive',force_remount=True)
DS_dir=r"/content/drive/MyDrive/Colab Notebooks/AgroML/input"
#Unzip file
with ZipFile('/content/archive.zip','r') as Zipobj: Zipobj.extractall('/content/drive/MyDrive/archive')|If I am trying to unzip the file directly to MyDrive? it is starting but never end. And after 30 minute or so? execution's is terminated. without extracting all the files
So i unziped
with ZipFile('/content/archive.zip','r') as Zipobj: Zipobj.extractall('/content/archive') And it did work fines after 75 second or so. And on my ColabPro drive I have a directory tree with 80,000 files. And I am trying to copy the files from the colab drive to MyDrive with the following code
%cp -av "/content/archive" "/content/drive/MyDrive/Colab Notebooks/download" #>copylog.txtwhich runs for ever till it crash, it will copy about 20% of the files. (just 17000 files had been copied) So I had moved the code to another Google Drive and Colab account (with much less data) and it did work. It Toke 9 minutes but it copy the entire tree of 3G and 80000 files.
My questions are
- Why I cant work directly to MyDrive and is there a different between The colab drive and MyDrive from a program? why Can't I unzip directly to MyDrive
- Is there a lib or utile that allow my to move/copy directory tree on the cloud it self when I tried the shutil for example? it toke much more time.
- How come I can download 3G from Kagel in few seconds abut can't unzip it to Mydrive directly and How and why I am unable to copy the directory tree on one colab/Gdrive account and able to do on another account? (both are ProColab and payed Gdrive acounts)
The Gdrive that I am succeeding has about 20 Gig and and the other one has 450 Gig with much much more files.
This is not a "onetime" problem and manual solution's will not help me. thanks Adi