Skip to content

A single interrupted batch ends the files import for good, while the UI keeps showing "importing" #138

Description

@giggio

How to use GitHub

  • Please use the 👍 reaction to show that you are affected by the same issue.
  • Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
  • Subscribe to receive notifications on status change and new comments.

Steps to reproduce

  1. Connect a OneDrive account and start a files import large enough to need many batches.
  2. While a batch is running, interrupt the PHP process — restart the cron service, reboot, deploy, kill a wedged download, or let it be OOM-killed. In my case a routine service restart during a system upgrade was enough: the unit stops cron.php with SIGTERM while the import job is running.
  3. Check the job list:
$ occ background-job:list -c 'OCA\Onedrive\BackgroundJob\ImportOnedriveJob'
(no output)

Nothing ever resumes, and the Data migration page still shows the import as in progress.

Expected behaviour

An interrupted batch is retried on a later cron run and the import continues on its own. If it truly cannot continue, the UI should stop reporting an import in progress, so the user can restart it from the interface.

Actual behaviour

The import stops permanently and silently. The UI keeps claiming it is running, so there is no indication anything is wrong, and the user cannot restart it from the interface either.

The cause is a gap between two points in the job lifecycle:

  • QueuedJob::start() (server) removes the job's oc_jobs row before run() executes.
  • OnedriveStorageAPIService::importOnedriveJob() re-adds the job with $this->jobList->add(ImportOnedriveJob::class, ['user_id' => $userId]) only at the end of a successful batch.

Any exit between those two points — signal, fatal, OOM — leaves no job row at all, while the importing_onedrive user value is still '1'.

Recovery is not possible from the UI, because startImportOnedrive() returns early while the flag is set:

$alreadyImporting = $this->config->getUserValue($userId, Application::APP_ID, 'importing_onedrive', '0') === '1';
if ($alreadyImporting) {
    return ['targetPath' => $targetPath];
}

so the import button silently does nothing. The user has to clear the flag out of band first, e.g.

occ user:setting <uid> integration_onedrive importing_onedrive 0

and only then start the import again.

The app already anticipates stale runs — importOnedriveJob() checks onedrive_import_running against Application::IMPORT_JOB_TIMEOUT — but that guard only helps if a job still exists to run it, which after an interruption it does not.

Secondary, same code path: startImportOnedrive() resets imported_size and last_onedrive_import_timestamp and deletes import_tree, but does not reset nb_imported_files. After a restart the file counter carries on from its old value while the byte counter restarts at zero, so the UI shows something like "2195 files imported (609.2 MiB)" when only 2 of those files were downloaded in the current run. Both counters should be reset together.

Note on restarting: because import_tree is deleted, a restart re-walks the whole drive over the Graph API. Files already present are correctly skipped by getFile() (nodeExists() → return 0), so nothing is re-downloaded, but on a large drive the re-listing is slow and costs a request per folder.

Suggested fixes

  1. Re-add the job on failure — a finally, or catch-and-requeue — so an interrupted batch resumes on the next cron run.
  2. Or add a small TimedJob that re-queues ImportOnedriveJob whenever importing_onedrive === '1' and no job is queued for that user. This also covers the case where the process had to be killed.
  3. Reset nb_imported_files alongside imported_size in startImportOnedrive().

Server configuration

Web server: Nginx (behind Traefik)

Database: PostgreSQL 17.10

PHP version: 8.4.23

Nextcloud version: 34.0.2

integration_onedrive version: 3.5.2

List of activated apps
Enabled:
  - activity: 7.0.0
  - app_api: 34.0.0
  - appstore: 1.0.0
  - bruteforcesettings: 7.0.0
  - calendar: 6.5.2
  - circles: 34.0.0
  - cloud_federation_api: 1.18.0
  - comments: 1.24.0
  - contacts: 8.7.5
  - contactsinteraction: 1.15.0
  - dashboard: 7.14.0
  - dav: 1.39.0
  - deck: 1.18.3
  - federatedfilesharing: 1.24.0
  - federation: 1.24.0
  - files: 2.6.0
  - files_downloadlimit: 5.2.0
  - files_lock: 34.0.1
  - files_pdfviewer: 7.0.0-dev.0
  - files_reminders: 1.7.0
  - files_sharing: 1.26.0
  - files_trashbin: 1.24.0
  - files_versions: 1.27.0
  - firstrunwizard: 7.0.0-dev.0
  - forms: 5.3.5
  - groupfolders: 22.0.6
  - integration_giphy: 2.2.1
  - integration_onedrive: 3.5.2
  - logreader: 7.0.0
  - lookup_server_connector: 1.22.0
  - mail: 5.10.10
  - maps: 1.7.1
  - music: 3.1.1
  - nextcloud_announcements: 6.0.0
  - notes: 6.0.1
  - notifications: 7.0.0-dev.1
  - notify_push: 1.3.3
  - oauth2: 1.22.0
  - office: 1.0.0
  - password_policy: 6.0.0-dev.0
  - photos: 7.0.0
  - privacy: 6.0.0-dev.1
  - profile: 1.3.0
  - provisioning_api: 1.24.0
  - recommendations: 7.0.0
  - related_resources: 5.0.0-dev.0
  - serverinfo: 6.0.0
  - settings: 1.17.0
  - sharebymail: 1.24.0
  - spreed: 24.0.3
  - support: 6.0.0
  - survey_client: 6.0.0-dev.0
  - systemtags: 1.24.0
  - tasks: 0.18.1
  - text: 8.0.0
  - theming: 2.9.0
  - twofactor_backupcodes: 1.23.0
  - twofactor_totp: 16.0.0
  - twofactor_webauthn: 2.7.0
  - updatenotification: 1.24.0
  - user_status: 1.14.0
  - viewer: 7.0.0-dev.0
  - weather_status: 1.14.0
  - webhook_listeners: 1.6.0
  - workflowengine: 2.16.0
Disabled:
  - admin_audit: 1.24.0
  - encryption: 2.22.0
  - files_external: 1.26.0
  - onlyoffice: 10.1.2 (installed 10.1.2)
  - suspicious_login: 12.0.0-dev.0
  - twofactor_nextcloud_notification: 8.0.0
  - user_ldap: 1.25.0
Nextcloud configuration ```json { "system": { "apps_paths": [ { "path": "\/nix\/store\/pppzqxmrmg6rjx387c2sp1cmqzmd00gx-nextcloud-34.0.2-with-apps\/apps", "url": "\/apps", "writable": false }, { "path": "\/nix\/store\/pppzqxmrmg6rjx387c2sp1cmqzmd00gx-nextcloud-34.0.2-with-apps\/nix-apps", "url": "\/nix-apps", "writable": false }, { "path": "\/nix\/store\/pppzqxmrmg6rjx387c2sp1cmqzmd00gx-nextcloud-34.0.2-with-apps\/store-apps", "url": "\/store-apps", "writable": true } ], "memcache.local": "\\OC\\Memcache\\APCu", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbuser": "***REMOVED SENSITIVE VALUE***", "dbtype": "pgsql", "datadirectory": "***REMOVED SENSITIVE VALUE***", "default_certificates_bundle_path": "\/nix\/store\/30wqfg7d7nxlqmlnxad14s2r775bq5mw-nss-cacert-3.125\/etc\/ssl\/certs\/ca-bundle.crt", "default_phone_region": "BR", "enabledPreviewProviders": [ "OC\\Preview\\PDF", "OC\\Preview\\SVG", "OC\\Preview\\BMP", "OC\\Preview\\GIF", "OC\\Preview\\JPEG", "OC\\Preview\\Krita", "OC\\Preview\\MarkDown", "OC\\Preview\\MP3", "OC\\Preview\\OpenDocument", "OC\\Preview\\PNG", "OC\\Preview\\TXT", "OC\\Preview\\XBitmap", "OC\\Preview\\HEIC" ], "forwarded_for_headers": [ "HTTP_X_FORWARDED_FOR", "HTTP_CF_CONNECTING_IP" ], "integrity.check.disabled": true, "log_type": "file", "loglevel": 1, "mail_domain": "***REMOVED SENSITIVE VALUE***", "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_send_plaintext_only": false, "mail_sendmailmode": "smtp", "mail_smtpauth": true, "mail_smtpdebug": false, "mail_smtphost": "***REMOVED SENSITIVE VALUE***", "mail_smtpmode": "smtp", "mail_smtpname": "***REMOVED SENSITIVE VALUE***", "mail_smtpport": 465, "mail_smtpsecure": "ssl", "mail_smtpstreamoptions": [], "mail_smtptimeout": 10, "mail_template_class": "\\OC\\Mail\\EMailTemplate", "maintenance_window_start": 5, "maxUploadSize": "512M", "memcache.distributed": "\\OC\\Memcache\\Redis", "memcache.locking": "\\OC\\Memcache\\Redis", "overwrite.cli.url": "REDACTED", "overwriteprotocol": "", "profile.enabled": false, "redis": { "host": "***REMOVED SENSITIVE VALUE***", "port": 0 }, "skeletondirectory": "", "trusted_proxies": "***REMOVED SENSITIVE VALUE***", "upgrade.disable-web": true, "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "mail_smtppassword": "***REMOVED SENSITIVE VALUE***", "version": "34.0.2.1", "dbtableprefix": "oc_", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "twofactor_enforced": "true", "twofactor_enforced_groups": [], "twofactor_enforced_excluded_groups": [], "maintenance": false, "app_install_overwrite": [], "appstoreenabled": true, "allow_local_remote_servers": true, "logfile": "\/var\/log\/nextcloud\/nextcloud.log", "trusted_domains": [ "REDACTED", "REDACTED" ], "PASS": "***REMOVED SENSITIVE VALUE***" } } ```

Browser

N/A — server-side background job behaviour. The only UI symptom is the Data migration page permanently showing an import in progress, which follows from the server state described above.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions