OVH Community

Welcome to your community space. Ask questions, search for information, post content, and interact with other OVH Community members.

Hubic - backups fail from synology nas

#1

Hello team,
I have been using hubic with my Synology NAS for almost a year now. It is the 3rd time that after setting up my backups, they run smoothly for several months and then they just start failing.
I did an investigation with Synology and the outcome is that the cause of the issue may be on hubic.
The particular entry in my logs that indicate a failure to upload a file look like this:
#10: {“error_class”:“swiftclient.exceptions.ClientException”,“error_code”:503,“error_message”:“Object PUT failed”,“success”:false}
2018-06-22T02:00:04+01:00 bud img_worker: openstack_error.cpp:70 send_file:594 failed, {“error_class”:“swiftclient.exceptions.ClientException”,“error_code”:503,“error_message”:“Object PUT failed”,“success”:false}
2018-06-22T02:00:04+01:00 bud img_worker: (15277) file_transfer.cpp:787 failed to sendfile to [bud-vol2-bak.hbk/Pool/0/4/782.bucket.20] from [/volume1/@img_bkp_cache/hubic_swift_default_.quOhmB/bud-vol2-bak.hbk/Pool/0/4/782.bucket], data_hot=[0], error = [1]
2018-06-22T02:00:04+01:00 bud img_worker: (15277) file_transfer.cpp:621 failed to sendfile: trg_relative path: [Pool/0/4/782.bucket], err: [1]

How can you please help me troubleshoot and fix the issues I am having with my account?
Thank you in advance,
kind regards.

#2

Hi,

Same issue for me : stopped in July, all backups for more than a year were fine until end of June.

As there anything to do with Hubic going end of life ?

#3

Same for me :frowning:

#4

me too

Synology says: In the meantime, I suggest you contact Hubic Technical Support because in the NAS logs I found these error messages could be related to a problem on the Hubic side:
07-04T15:56:58+02:00 DISKSTATION synoscgi_SYNO.CloudSync_1_get_property[8107]: [ERROR] dscs-hubic-proto.cpp(159): Failed to get user info (401)(expired)
2018-07-04T15:56:58+02:00 DISKSTATION synoscgi_SYNO.CloudSync_1_get_property[8107]: [ERROR] dscs-hubic-transport.cpp(60): Failed to get user info (expired)
2018-07-04T15:56:58+02:00 DISKSTATION synoscgi_SYNO.CloudSync_1_get_property[8107]: [ERROR] dscs-client-protocol.cpp(274): Failed to get account info [type: 6] ret: -110
2018-07-04T15:56:58+02:00 DISKSTATION synoscgi_SYNO.CloudSync_1_get_property[8107]: cloudsync.cpp:1726 Failed to get connection info '1’
2018-07-04T15:56:58+02:00 DISKSTATION synoscgi_SYNO.CloudSync_1_get_property[8106]: [ERROR] dscs-hubic-proto.cpp(159): Failed to get user info (401)(expired)
2018-07-04T15:56:58+02:00 DISKSTATION synoscgi_SYNO.CloudSync_1_get_property[8106]: [ERROR] dscs-hubic-transport.cpp(60): Failed to get user info (expired)
2018-07-04T15:56:58+02:00 DISKSTATION synoscgi_SYNO.CloudSync_1_get_property[8106]: [ERROR] dscs-client-protocol.cpp(274): Failed to get account info [type: 6] ret: -110
2018-07-04T15:56:58+02:00 DISKSTATION synoscgi_SYNO.CloudSync_1_get_property[8106]: cloudsync.cpp:1726 Failed to get connection info ‘1’

(I removed the Cloud Sync app from connected apps and I re-identified)

I also see in the logs:
Jul 19 17:24:07 [ERROR] dscs-openstack-client-handler.cpp(559): Failed to create large file (/Brol.ts)(

Service Unavailable

The server is currently unavailable. Please try again at a later time

what to do!? Help @ hubic does not answer …

#5

Pareil pour moi…

Plus aucune backup ne s’effectue via mon NAS Synology depuis fin juin (alors que tout fonctionnait depuis des années). Aucune nouvelle backup ne s’effectue non plus.

Les CGU indiquent une assistance technique sur https://forums.hubic.com mais ce forum n’est plus fonctionnel. L’adresse email help@hubic.com est également spécifié mais aucune réponse à mes mails.

Mon compte hubiC a été automatiquement renouvelé en avril pour une durée d’un an, et la résiliation via l’espace client ne permet pas le remboursement alors que le service n’est plus assuré. Lamentable !

#6

Hello all,
Exactly the same problem for me.
Found this kind of logs on /var/log with SSH access on my NAS.
@OVH TEAM : could you please have a look because this issue is blocking for the backup of my SYNOLOGY ?
Thanks.
LL

#7

me too
In addition, even with new test tasks selecting few files, the tasks also fail

Antonio

#8

Does OVH already have a solution for this? Backups keep failing (at the end of the backup job) with same error. Even a new backup job with little data to backup fails at the end with same error.

<#NASNAME#> img_worker: transfer_openstack.cpp:560 need retry #1: {“error_class”:“swiftclient.exceptions.ClientException”,“error_code”:503,“error_message”:“Object PUT failed”,“success”:false}

after 10 retries the job fails…

This I cut from the logs on my NAS.

2018-08-23T02:18:45+02:00 <> img_backup: [31930]img_backup.cpp:2103 Cloud Backup Task has been started: task_ID: 26
2018-08-23T02:20:17+02:00 <> img_backup: [31930]img_backup.cpp:1562 Action: [local backup], Repo Path: [/volume1/@img_bkp_cache/hubic_swift_ServerCPN_.OzkG6D], LinkKey: [<>0011321945EB_26], Cloud backup: [1], Target ID: [SyncRob.hbk], task ID: [26]
2018-08-23T02:20:17+02:00 <> img_backup: [31930]img_backup.cpp:1564 app config Path: [/volume1/@tmp/BKP_APP_0DfdW5]
2018-08-23T02:20:17+02:00 <> img_backup: [31930]img_backup.cpp:1566 data Path: [/SYNCROB]
2018-08-23T02:21:57+02:00 <> img_backup: (31930) backup_controller.cpp:1249 (31930)cloud estimate size app [0], share [9313447936]
2018-08-23T02:22:28+02:00 <> img_backup: (31930) backup_controller.cpp:3064 (31930)[BkpCtrl] All workers flush done, continue:(4)
2018-08-23T02:23:10+02:00 <> kernel: [4379657.764899] FAT-fs (synoboot2): fat_set_state: set=1 process=ActiveBackup-Of pid=17071
2018-08-23T02:23:10+02:00 <> kernel: [4379657.777911] FAT-fs (synoboot2): fat_set_state: set=0 process=ActiveBackup-Of pid=17071
2018-08-23T02:23:10+02:00 <> kernel: [4379657.894842] FAT-fs (synoboot2): fat_set_state: set=1 process=ActiveBackup-Of pid=17071
2018-08-23T02:23:10+02:00 <> kernel: [4379657.907723] FAT-fs (synoboot2): fat_set_state: set=0 process=ActiveBackup-Of pid=17071
2018-08-23T02:24:12+02:00 <> img_worker: transfer_openstack.cpp:560 need retry #1: {“error_class”:“swiftclient.exceptions.ClientException”,“error_code”:503,“error_message”:“Object PUT failed”,“success”:false}
2018-08-23T02:24:54+02:00 <> img_worker: transfer_openstack.cpp:560 need retry #2: {“error_class”:“swiftclient.exceptions.ClientException”,“error_code”:503,“error_message”:“Object PUT failed”,“success”:false}
2018-08-23T02:25:39+02:00 <> img_worker: transfer_openstack.cpp:560 need retry #3: {“error_class”:“swiftclient.exceptions.ClientException”,“error_code”:503,“error_message”:“Object PUT failed”,“success”:false}
2018-08-23T02:26:26+02:00 <> img_worker: transfer_openstack.cpp:560 need retry #4: {“error_class”:“swiftclient.exceptions.ClientException”,“error_code”:503,“error_message”:“Object PUT failed”,“success”:false}
2018-08-23T02:27:13+02:00 <> img_worker: transfer_openstack.cpp:560 need retry #5: {“error_class”:“swiftclient.exceptions.ClientException”,“error_code”:503,“error_message”:“Object PUT failed”,“success”:false}
2018-08-23T02:28:11+02:00 <> img_worker: transfer_openstack.cpp:560 need retry #6: {“error_class”:“swiftclient.exceptions.ClientException”,“error_code”:503,“error_message”:“Object PUT failed”,“success”:false}
2018-08-23T02:30:20+02:00 <> img_worker: transfer_openstack.cpp:560 need retry #1: {“error_class”:“swiftclient.exceptions.ClientException”,“error_code”:503,“error_message”:“Object PUT failed”,“success”:false}
2018-08-23T02:31:22+02:00 <> img_worker: transfer_openstack.cpp:560 need retry #2: {“error_class”:“swiftclient.exceptions.ClientException”,“error_code”:503,“error_message”:“Object PUT failed”,“success”:false}
2018-08-23T02:32:27+02:00 <> img_worker: transfer_openstack.cpp:560 need retry #3: {“error_class”:“swiftclient.exceptions.ClientException”,“error_code”:503,“error_message”:“Object PUT failed”,“success”:false}
2018-08-23T02:33:30+02:00 <> img_worker: transfer_openstack.cpp:560 need retry #4: {“error_class”:“swiftclient.exceptions.ClientException”,“error_code”:503,“error_message”:“Object PUT failed”,“success”:false}
2018-08-23T02:34:37+02:00 <> img_worker: transfer_openstack.cpp:560 need retry #5: {“error_class”:“swiftclient.exceptions.ClientException”,“error_code”:503,“error_message”:“Object PUT failed”,“success”:false}
2018-08-23T02:35:58+02:00 <> img_worker: transfer_openstack.cpp:560 need retry #6: {“error_class”:“swiftclient.exceptions.ClientException”,“error_code”:503,“error_message”:“Object PUT failed”,“success”:false}
2018-08-23T02:37:28+02:00 <> img_worker: transfer_openstack.cpp:560 need retry #7: {“error_class”:“swiftclient.exceptions.ClientException”,“error_code”:503,“error_message”:“Object PUT failed”,“success”:false}
2018-08-23T02:39:34+02:00 <> img_worker: transfer_openstack.cpp:560 need retry #8: {“error_class”:“swiftclient.exceptions.ClientException”,“error_code”:503,“error_message”:“Object PUT failed”,“success”:false}
2018-08-23T02:42:43+02:00 <> img_worker: transfer_openstack.cpp:560 need retry #9: {“error_class”:“swiftclient.exceptions.ClientException”,“error_code”:503,“error_message”:“Object PUT failed”,“success”:false}
2018-08-23T02:48:04+02:00 <> img_worker: transfer_openstack.cpp:560 need retry #10: {“error_class”:“swiftclient.exceptions.ClientException”,“error_code”:503,“error_message”:“Object PUT failed”,“success”:false}
2018-08-23T02:53:26+02:00 <> img_worker: openstack_error.cpp:70 send_file:594 failed, {“error_class”:“swiftclient.exceptions.ClientException”,“error_code”:503,“error_message”:“Object PUT failed”,“success”:false}
2018-08-23T02:53:26+02:00 <> img_worker: (31977) file_transfer.cpp:787 failed to sendfile to [SyncRob.hbk/Config/virtual_file.index/2.idx.180] from [/volume1/@img_bkp_cache/hubic_swift_ServerCPN.OzkG6D/SyncRob.hbk/Config/virtual_file.index/2.idx], data_hot=[1], error = [1]
2018-08-23T02:53:26+02:00 <> img_worker: (31977) file_transfer.cpp:621 failed to sendfile: trg_relative path: [Config/virtual_file.index/2.idx], err: [1]
2018-08-23T02:53:26+02:00 <> img_worker: (31977) cloud_upload_controller.cpp:2014 (31977)failed to upload file: [/volume1/@img_bkp_cache/hubic_swift_ServerCPN_.OzkG6D/SyncRob.hbk/Config/virtual_file.index/2.idx] --> [SyncRob.hbk/Config/virtual_file.index/2.idx], blRemoveAfterUpload: [0], dataClass: [1]
2018-08-23T02:53:26+02:00 <> img_worker: (31977) cloud_upload_controller.cpp:1779 (31977)failed to do the job
2018-08-23T02:53:27+02:00 <> img_backup: (31930) backup_controller.cpp:1197 (31930)Failed to [CMD_END]: response:[ST_UNKNOWN], resume_status: [NOT_RESUMABLE]
2018-08-23T02:53:27+02:00 <> img_backup: (31930) backup_controller.cpp:1330 (31930)failed to handle packet from cloud uploader
2018-08-23T02:53:27+02:00 <> img_backup: (31930) backup_controller.cpp:1353 (31930)failed to handle worker response
2018-08-23T02:53:27+02:00 <> img_backup: (31930) backup_controller.cpp:3198 (31930)INFO: abnormal terminate: blWorkerAlive[0], blWaitCloudUploaderEndResponse[0], blWaitBackupEnd:[0]
2018-08-23T02:53:27+02:00 <> img_worker: (31977) client_base.cpp:155 (31977)[BkpCtrl] Got INT signal ==> just cancel
2018-08-23T02:53:27+02:00 <> img_worker: (31977) cloud_upload_controller.cpp:1072 (31977)cloud uploader is finished and is waiting for die, skip doing anything: blNotify:[0], blResponseEnd:[1], blPostCloud:[0]
2018-08-23T02:53:27+02:00 <> img_worker: (31977) cloud_upload_controller.cpp:1072 (31977)cloud uploader is finished and is waiting for die, skip doing anything: blNotify:[0], blResponseEnd:[0], blPostCloud:[1]
2018-08-23T02:53:27+02:00 <> img_backup: (31930) service_wrapper.cpp:80 (31930)failed to start backup service
2018-08-23T02:53:27+02:00 <> img_backup: [31930]img_backup.cpp:1657 Failed to start backup task errno:1
2018-08-23T02:53:27+02:00 <> img_backup: [31930]img_backup.cpp:2163 failed to start task
2018-08-23T02:53:27+02:00 <> img_backup: (31930) [err] backup_progress.cpp:431 Backup task [hubiC SynCRob] completes with result [3]. Time spent: [2081 sec].
2018-08-23T02:53:27+02:00 <> img_backup: (31930) [err] backup_progress.cpp:446 Total Size(Bytes):[9854918036], Modified Size(Bytes):[9854918036], Total Directory:[2268], Modified Directory:[2268], Total File:[18240], Modified File:[18240],
2018-08-23T02:53:28+02:00 <> img_backup: [31930]img_backup.cpp:1913 Storage Statistics: TargetSize:[9143096], LastBackupTargetSize:[9143096], SourceSize:[9095180], TotalFile:[18240], ModifyFile:[0], NewFile:[0], UnchangeFile:[18240], RemoveFile:[0], RenameFile:[0], RenameLogicSize:[0], CopyFile:[0], CopyLogicSize:[0], CopyMissFile:[0], CopyMissFileLogicSize:[0]