Cloudsync / S3 and temporary files with versioning

CloudSync is a cloud drive synchronization application for fast and secure data synchronization between your TNAS and cloud drives.
Locked
User avatar
fabry68
Posts: 18
Joined: 20 Jul 2023, 15:17

Cloudsync / S3 and temporary files with versioning

Post by fabry68 »

Hello.
I am using cloudsync to backup my nas to a s3 bucket (with some small modification on config file and sqlite3 database, I can change from Amazon to Ceph s3 endpoint).

I have two problems: cloudsyncd uploads all the files with a temporary filename (.cloudsyncd_downloading_ORIGINALFILENAME), after completed renames it with the original filename.

But with S3 there are 2 big problems:
- as I know (but I can be wrong) S3 protocol doesn't have a "rename" function, but it copies the old file with a new file (using the new filename) and then deletes the old one. But, if the file is very big (I have 30/40Gbyte files), this is a long operation, and cloudysncd returned an error (due to timeout).

- as I know (but, again, i can be wrong), if I enable bucket versioning (and I NEED to do it to be safe against ransomware), every temporary files will be not deleted, and the used space will be double (no problem if I have enought space because with lifecycle will be deleted after some days). I can't find any setting to exclude some pattern (ex: .cloudsyncd_*) from versioning.

So, the best choice could be to not use a temporary filename for s3 sync (better if we can choose from sync settings).

Anyone with my same problem ?
Top
User avatar
fabry68
Posts: 18
Joined: 20 Jul 2023, 15:17

Re: Cloudsync / S3 and temporary files with versioning

Post by fabry68 »

The problema with "rename" is with size of about 3.5Gbyte or more.
Top
User avatar
fabry68
Posts: 18
Joined: 20 Jul 2023, 15:17

Re: Cloudsync / S3 and temporary files with versioning

Post by fabry68 »

This is what happens... this log is of my ceph S3 cluster, so "server side".

Fist, file is uploaded normally (with rclone) and his name is the temporary file:

212.66.111.241 - - [16/Jan/2024:09:15:48 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=96&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:15:48 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=95&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:15:57 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=97&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:15:59 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=98&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:15:59 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=99&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:06 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=100&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:11 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=101&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:11 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=102&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:16 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=103&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:21 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=105&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:22 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=104&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:26 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=106&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:32 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=107&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:35 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=109&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:35 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=108&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:36 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=110&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:36 +0100] "POST /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 200 466 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:36 +0100] "HEAD /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4 HTTP/1.1" 200 0 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:36 +0100] "GET / HTTP/1.1" 200 1899 "-" "rclone/"

Then, rclone want to rename it, but rename is copy and delete the old one...
never ends to copy, even if I can see the file with s3browser, and stops only if I delete the temporary file with s3browser.




212.66.111.241 - - [16/Jan/2024:09:16:36 +0100] "GET /terramaster-fabry?delimiter=%2F&max-keys=1000&prefix= HTTP/1.1" 200 333 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:36 +0100] "GET /terramaster-fabry?delimiter=%2F&max-keys=1000&prefix=Foto%2F HTTP/1.1" 200 1477 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:36 +0100] "GET /terramaster-fabry?delimiter=%2F&max-keys=1000&prefix=Foto%2F2019%2F HTTP/1.1" 200 1850 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:36 +0100] "GET /terramaster-fabry?delimiter=%2F&max-keys=1000&prefix=Foto%2F2019%2F2019-08-17-BoscoDelleFavole%2F HTTP/1.1" 200 734 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:36 +0100] "GET /terramaster-fabry?delimiter=%2F&max-keys=1000&prefix=Foto%2F2019%2F2019-08-17-BoscoDelleFavole%2FActionCam%2F HTTP/1.1" 200 759 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:36 +0100] "GET /terramaster-fabry?delimiter=%2F&max-keys=1000&prefix=Foto%2F2019%2F2019-08-17-BoscoDelleFavole%2FActionCam%2FDCIM%2F HTTP/1.1" 200 778 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:36 +0100] "GET /terramaster-fabry?delimiter=%2F&max-keys=1000&prefix=Foto%2F2019%2F2019-08-17-BoscoDelleFavole%2FActionCam%2FDCIM%2F100MEDIA%2F HTTP/1.1" 200 24104 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:36 +0100] "HEAD /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/YDXJ0447.MP4 HTTP/1.1" 404 0 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:36 +0100] "HEAD /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/ HTTP/1.1" 200 0 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:17:06 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/YDXJ0447.MP4 HTTP/1.1" 499 0 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:17:37 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/YDXJ0447.MP4 HTTP/1.1" 499 0 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:18:07 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/YDXJ0447.MP4 HTTP/1.1" 499 0 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:18:37 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/YDXJ0447.MP4 HTTP/1.1" 499 0 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:19:07 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/YDXJ0447.MP4 HTTP/1.1" 499 0 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:19:38 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/YDXJ0447.MP4 HTTP/1.1" 499 0 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:20:10 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/YDXJ0447.MP4 HTTP/1.1" 499 0 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:20:42 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/YDXJ0447.MP4 HTTP/1.1" 499 0 "-" "rclone/"
Top
User avatar
TMtina
TM Support
Posts: 245
Joined: 01 Nov 2022, 18:00

Re: Cloudsync / S3 and temporary files with versioning

Post by TMtina »

That's a good question, thank you. Remote temporary files are unnecessary, and we will remove them in the next version.
To contact our team, please send email to following addresses, remember to replace (at) with @:

Technical team: support(at)terra-master.com(for technical support)
Service team: service(at)terra-master.com(for purchasing, return, replacement, RMA service)
Top
User avatar
fabry68
Posts: 18
Joined: 20 Jul 2023, 15:17

Re: Cloudsync / S3 and temporary files with versioning

Post by fabry68 »

Thank you very much.
PS: as I can see, s3 will show the uploaded files only after a complete upload
I will wait for the new version so I can save my datas.
Top
User avatar
fabry68
Posts: 18
Joined: 20 Jul 2023, 15:17

Re: Cloudsync / S3 and temporary files with versioning

Post by fabry68 »

I can confirm that whith today's upgrade all is working as expected. No temporary files, and no problem with >50Gb size files.
Thank you again.
Top
Locked

Return to “CloudSync”