Cloudsync / S3 and temporary files with versioning

CloudSync is a cloud drive synchronization application for fast and secure data synchronization between your TNAS and cloud drives. It is a very practical and efficient one-stop cloud drive disaster recovery solution.
Post Reply
User avatar
fabry68
Posts: 18
Joined: 20 Jul 2023, 15:17

Cloudsync / S3 and temporary files with versioning

Post by fabry68 »

Hello.
I am using cloudsync to backup my nas to a s3 bucket (with some small modification on config file and sqlite3 database, I can change from Amazon to Ceph s3 endpoint).

I have two problems: cloudsyncd uploads all the files with a temporary filename (.cloudsyncd_downloading_ORIGINALFILENAME), after completed renames it with the original filename.

But with S3 there are 2 big problems:
- as I know (but I can be wrong) S3 protocol doesn't have a "rename" function, but it copies the old file with a new file (using the new filename) and then deletes the old one. But, if the file is very big (I have 30/40Gbyte files), this is a long operation, and cloudysncd returned an error (due to timeout).

- as I know (but, again, i can be wrong), if I enable bucket versioning (and I NEED to do it to be safe against ransomware), every temporary files will be not deleted, and the used space will be double (no problem if I have enought space because with lifecycle will be deleted after some days). I can't find any setting to exclude some pattern (ex: .cloudsyncd_*) from versioning.

So, the best choice could be to not use a temporary filename for s3 sync (better if we can choose from sync settings).

Anyone with my same problem ?
User avatar
fabry68
Posts: 18
Joined: 20 Jul 2023, 15:17

Re: Cloudsync / S3 and temporary files with versioning

Post by fabry68 »

The problema with "rename" is with size of about 3.5Gbyte or more.
User avatar
fabry68
Posts: 18
Joined: 20 Jul 2023, 15:17

Re: Cloudsync / S3 and temporary files with versioning

Post by fabry68 »

This is what happens... this log is of my ceph S3 cluster, so "server side".

Fist, file is uploaded normally (with rclone) and his name is the temporary file:

212.66.111.241 - - [16/Jan/2024:09:15:48 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=96&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:15:48 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=95&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:15:57 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=97&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:15:59 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=98&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:15:59 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=99&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:06 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=100&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:11 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=101&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:11 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=102&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:16 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=103&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:21 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=105&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:22 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=104&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:26 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=106&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:32 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=107&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:35 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=109&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:35 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=108&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:36 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?partNumber=110&uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 100 249 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:36 +0100] "POST /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4?uploadId=2~H_ZWC64srq_1ikDoxZVBviMoCynl9X5 HTTP/1.1" 200 466 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:36 +0100] "HEAD /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/.cloudsyncd_downloading_YDXJ0447.MP4 HTTP/1.1" 200 0 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:36 +0100] "GET / HTTP/1.1" 200 1899 "-" "rclone/"

Then, rclone want to rename it, but rename is copy and delete the old one...
never ends to copy, even if I can see the file with s3browser, and stops only if I delete the temporary file with s3browser.




212.66.111.241 - - [16/Jan/2024:09:16:36 +0100] "GET /terramaster-fabry?delimiter=%2F&max-keys=1000&prefix= HTTP/1.1" 200 333 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:36 +0100] "GET /terramaster-fabry?delimiter=%2F&max-keys=1000&prefix=Foto%2F HTTP/1.1" 200 1477 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:36 +0100] "GET /terramaster-fabry?delimiter=%2F&max-keys=1000&prefix=Foto%2F2019%2F HTTP/1.1" 200 1850 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:36 +0100] "GET /terramaster-fabry?delimiter=%2F&max-keys=1000&prefix=Foto%2F2019%2F2019-08-17-BoscoDelleFavole%2F HTTP/1.1" 200 734 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:36 +0100] "GET /terramaster-fabry?delimiter=%2F&max-keys=1000&prefix=Foto%2F2019%2F2019-08-17-BoscoDelleFavole%2FActionCam%2F HTTP/1.1" 200 759 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:36 +0100] "GET /terramaster-fabry?delimiter=%2F&max-keys=1000&prefix=Foto%2F2019%2F2019-08-17-BoscoDelleFavole%2FActionCam%2FDCIM%2F HTTP/1.1" 200 778 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:36 +0100] "GET /terramaster-fabry?delimiter=%2F&max-keys=1000&prefix=Foto%2F2019%2F2019-08-17-BoscoDelleFavole%2FActionCam%2FDCIM%2F100MEDIA%2F HTTP/1.1" 200 24104 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:36 +0100] "HEAD /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/YDXJ0447.MP4 HTTP/1.1" 404 0 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:16:36 +0100] "HEAD /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/ HTTP/1.1" 200 0 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:17:06 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/YDXJ0447.MP4 HTTP/1.1" 499 0 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:17:37 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/YDXJ0447.MP4 HTTP/1.1" 499 0 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:18:07 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/YDXJ0447.MP4 HTTP/1.1" 499 0 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:18:37 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/YDXJ0447.MP4 HTTP/1.1" 499 0 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:19:07 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/YDXJ0447.MP4 HTTP/1.1" 499 0 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:19:38 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/YDXJ0447.MP4 HTTP/1.1" 499 0 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:20:10 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/YDXJ0447.MP4 HTTP/1.1" 499 0 "-" "rclone/"
212.66.111.241 - - [16/Jan/2024:09:20:42 +0100] "PUT /terramaster-fabry/Foto/2019/2019-08-17-BoscoDelleFavole/ActionCam/DCIM/100MEDIA/YDXJ0447.MP4 HTTP/1.1" 499 0 "-" "rclone/"
User avatar
TMtina
TerraMaster Team
Posts: 106
Joined: 01 Nov 2022, 18:00

Re: Cloudsync / S3 and temporary files with versioning

Post by TMtina »

That's a good question, thank you. Remote temporary files are unnecessary, and we will remove them in the next version.
User avatar
fabry68
Posts: 18
Joined: 20 Jul 2023, 15:17

Re: Cloudsync / S3 and temporary files with versioning

Post by fabry68 »

Thank you very much.
PS: as I can see, s3 will show the uploaded files only after a complete upload
I will wait for the new version so I can save my datas.
User avatar
fabry68
Posts: 18
Joined: 20 Jul 2023, 15:17

Re: Cloudsync / S3 and temporary files with versioning

Post by fabry68 »

I can confirm that whith today's upgrade all is working as expected. No temporary files, and no problem with >50Gb size files.
Thank you again.
Post Reply

Return to “CloudSync”