Import ends incomplete

[axelweichert]

My OneDrive Size is about 866 GB, which i want to migrate to my managed NextCloud installation. First it starts importing my files, creating all follders in the dedictaed subdirectory. But after a time, where about 6 - 10 GB has been imported, the import ends and the notification tells me, that 1 files has been imported.

Now, when i restart the import process, it first tells me, that the import process will start soon und then starts the process.

Is there any know issue or anything i can do?

Kind regards,

Axel

[axelweichert]

I just have look to my Logs:

"message":"OneDrive API error : Client error: GET https://graph.microsoft.com/v1.0/me/contacts resulted in a 429 Too Many Requests

Error Login in the App:

===================================================
"time":"2021-02-11T07:26:14+00:00","remoteAddr":"","user":"--","app":"integration_onedrive","method":"","url":"--","message":"OneDrive API error : Client error: GET https://graph.microsoft.com/v1.0/me/drive/root://Instagram:/children resulted in a 400 Bad Request
"time":"2021-02-11T07:26:14+00:00","remoteAddr":"","user":"--","app":"integration_onedrive","method":"","url":"--","message":"OneDrive API error : Client error: GET https://graph.microsoft.com/v1.0/me/drive/root://Kunden:/children resulted in a 400 Bad Request

Failed to get number of contacts: "Client error: GET https:\/\/graph.microsoft.com\/v1.0\/me\/contactfolders resulted in a 429 Too Many Requests response:\n{\r\n "error": {\r\n "code": "ApplicationThrottled",\r\n "message": "Application is over its Request limit.",\r\n "inn (truncated...)\n"

What can i do?

[julien-nc]

Thanks for the issue.

Error Login in the App:

What do you mean?

About the 400 Bad Request error, I removed the extra slash that was in the request URL. Let's see if it changes something on your side.

About the 429 Too Many Requests, the number of request is now a lot reduced. Do you have many contact directories in your microsoft account?

So, v0.0.9 is out. Could you try it and tell me if things are better?

[axelweichert]

Now, after the update to 0.0.8 (0.0.9 was not available in the AppStore) it shows teh following:
184 files imported (0 B) (0%)
Onedrive import process will begin soon

But nothing starts importing. Before that, an amount of about 100 GB has been imported successfully with some restarts of the migration process. Now, nothing seems to go further than

184 files imported (0 B) (0%)
Onedrive import process will begin soon

I have neither stored an contacts or calendars in my private OneDrive.

[julien-nc]

v0.0.9 will appear soon (if it's not already there).

Could you check (just in case) that you don't have any contacts/calendars in your microsoft account?

Even if you didn't launch any contact import, the simple fact to count them could have been the reason for reaching the API request number limit with every releases before v0.0.9.

If you really don't have any this means the API request limit is reached while importing files. Could you check what pops in nextcloud.log when you import the drive?

I guess your server uses the "cron" method for Nextcloud background jobs. Right?
Keep in mind the drive import process will start next time Nextcloud's cron task is executed. If you want to speed things up and launch the import directly, you can execute the cron.php script.
Assuming that you're using bash on your server, that your webserver user is www-data and that your Nextcloud is installed in /var/www/nextcloud:

sudo -u www-data php -f /var/www/nextcloud/cron.php

Another important thing to know: the import process stops after having downloaded at least 500 MB and schedules another job. The following job is launched when the next cron task is executed.

Btw, do you have enough free space in your Nextcloud to get ~1 TB?

[axelweichert]

Version 0.0.9 is not there at the moment. I will check out later if it is available.

I have only some holidays in there, but no contacts. I never used them.

Then with 0.0.0.9 it should eleminate these 429 requests, right?

My server uses the cron method with schuled cronjobs every 5 min.

Actual nextcloud log output.
#----##----##----##----##----##----##----##----##----##----##----#
Feb 12 17:08:34 storage nextcloud-nx******[994504]: {"reqId":"bFKMJc03k2WWvcwmewJy","level":2,"time":"2021-02-12T16:08:34+00:00","remoteAddr":"93.227.90.71","user":"a***.******t","app":"integration_onedrive","method":"GET","url":"/apps/in
tegration_onedrive/contact-number","message":"OneDrive API error : Client error: GET https://graph.microsoft.com/v1.0/me/contacts resulted in a 429 Too Many Requests response:\n{\r\n "error": {\r\n "code": "ApplicationThrottl
ed",\r\n "message": "Application is over its Request limit.",\r\n "inn (truncated...)\n","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36","versio
n":"20.0.4.0"}
#----##----##----##----##----##----##----##----##----##----##----#

Client error: GET https://graph.microsoft.com/v1.0/me/contacts resulted in a 429 Too Many Requests

Application is over its Request limit.

I have moved to a managed Nextcloud Installation on Hetzner StorageShare with a limited admin access - a pitty, that i did not know before. But i have about 10 TB of space there, so evough fpr importing 866 GB.

I did not know that the import process is separated in 500 MB jobs and it depends on the scheduled cron jobs.

But, great work you have done, creating this app.

Edit: Just upgraded to 0.0.9. I will write back shortly

[julien-nc]

Well I think the drive import is the cause of reaching the limit and getting 429 response codes. I have to rewrite this in a way that produces less requests. For the moment, the entire directory structure exploration is done on each job instead of catching up where last job stopped. Already imported files are ignored but the exploration itself generates a lot of requests...

So v0.0.9 will most likely not be much better in your case. Now that, thanks to your feedback, I'm more aware of what happens, I can start improving the drive import process.

Thanks for your kind words.

[axelweichert]

Ok that seems to be the reason. Because i hev stored there many - and i mean really many - photos and documents in there, which i wanted to move from OneDrive to my NextCloud. When the initial process of sturcture exploration starts i makes sense, that there a to many requests.

When you something new to test for me, it would be nice if you get in touch with me.

You are welcome.

[julien-nc]

One last question (for today 😁). When you get a 429 response, does the import really stop (the "import onedrive files" button becomes visible again in user settings)? Or is is just freezing (progress is displayed and does not move)?

I will probably solve this mess by saving a list of known unfinished directories at the end of the import job so that the next job starts from there and only explore those directories.

To be continued. And yes I'll ping you when there is something new to try. Thanks.

[axelweichert]

With 0.0.7 and some manual restarts of the process, which ended with the message "1 file imported". Until this i have imported about 100 GB of data form my OneDrive and the whole diretory structure has been created.

Now with 0.0.9 and with 0.0.7 the process freezes with the message:

184 files imported (0 B) (0%)
Onedrive import process will begin soon

Thanks a lot. Happy to hear from you back soon,.

[julien-nc]

Hey, I published a nightly build (v0.0.10-1-nightly) in which each job resumes the import where the previous one stopped. Could you try it?

If you're not comfortable with nightly builds, let's wait for next release.

[axelweichert]

Sure i can try, but i can only see the normal - not-nightly releases, because the hoster of my managed NextCloud allows only stable releases. Any option to publish it as stable? At the moment i can only see 0.0.9