Skip to main content ↑ ↓
Topic: On weird fact of dd detail (Read 359 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

On weird fact of dd detail

Apologize if it'd be too meticulous getting to know few Linux works but cant help

How is dd utility copying in some HW / SW condition at least here, copied 3.5 GB onto Usb disk with OS give freed 7 GB memory, its process speed decreases by a significant amount from

start ~70 MB/s to ~ 41 MB/s in a minute
then to ~27 MB/s in another minute till to ~13.7 MB/s after 4 minutes from start
then remain so till the end, all complete in ~6 min

how can we set or beef it up to do all more efficiently to drop just by slight number ?

Re: On weird fact of dd detail

Reply #1 –
That's because of caching, unfortunately there are many such levels, one in your case being dirty/writeback memory (which you can limit to maybe get a more constant speed, but it does that for the whole system which is detrimental overall), and among the others being at controller level (where there is nothing you can do).
Paradoxically I found that a fast usb3.0 drive that does writes at more than 50 MB/s has a much more constant apparent write speed than slow low end ones.
To get your real speed you can always consult iotop, section "Actual DISK WRITE".

Re: On weird fact of dd detail

Reply #2 –
Will likely adding following lines

vm.dirty_background_ratio = 5
vm.dirty_ratio = 10

into /etc/sysctl.conf

work as found, done by a 'Ubuntu guy' ?

Re: On weird fact of dd detail

Reply #3 –
Flash media (SSDs included) are equipped with a micro-controller/processor and a certain amount of cache memory. The controller, well, controls the read/write operations, reallocates failing cells etc; it uses the internal cache to hold data while they're written on the flash memory.
So, in the beginning, the reported speed is high but as the operation progresses and the buffer is saturated, it drops to the real medium's capability. This is called sustained write speed and the true measurement of the disk's speed (flash media aren't actually 'disks', but you get my meaning).
For small files the writes appear very fast; for continuous large file writes only the top-silicon drives live up to their advertised speeds.
 

Re: On weird fact of dd detail

Reply #4 –
I have a PTP camera, to transfer files by USB cable it needs the package gvfs-gphoto2. You plug it in and turn it on and it gets mounted, then I can open that dir in caja, copy all the pictures and movies, then open another directory and paste them in. It gives a time estimation of several hours which increases as the transfer proceeds, the progress bar barely moves, but that's completely wrong and bears no relation whatsoever to reality, actually it's very quick to complete transfer, not noticeably different to physically removing the SD card and plugging it in directly.
You also have to be careful with cp, sometimes if a file is already in RAM it might return the cursor very fast but then you should run sync and that might take a lot longer because the data has only been cached ready to write to an external drive but not actually written to it yet.