Fixing snapmerge...

classic Classic list List threaded Threaded
34 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Fixing snapmerge...

JakeSFR
Ok, some of my proposed changes are already incorporated into Woof (thanks for Mavrothal).

So what we gonna do with the others, especially the 'break vs. continue' case?
IMO leaving the 'break' as-is would be very bad - in the past I lost lots of files because of this, until I tracked down this issue.

For me it's obvious:
- 'break' - user loses broken file + all subsequently scheduled files
- 'continue' - user loses only broken file(s).
But maybe I'm missing something..?

I've been also thinking about that warning msg (Mavrothal's suggestion).
Maybe just a terse information that there were detected "some" problems within the layered filesystem and "pfix=fsck of a savefile is highly recommended" would be enough..?
(speaking of which - what about re-enabling fsck for encrypted savefile in init? Nothing bad happened so far, since when I've done it...)

BTW, I just encountered i/o error again, and again it was because of some whiteout files in pup_ro1:
# snapmergepuppy
Merging /initrd/pup_rw onto /initrd/pup_ro1...
stat: cannot stat '/root/.pup_event/drive_sr1/AppInfo.xml': Input/output error
stat: cannot stat '/root/.pup_event/drive_sr1/AppRun': Input/output error
stat: cannot stat '/root/.pup_event/drive_sda2/AppInfo.xml': Input/output error
stat: cannot stat '/root/.pup_event/drive_sda2/AppRun': Input/output error
# 
# ls -A /initrd/pup_ro1/root/.pup_event/drive_sda2 /initrd/pup_ro1/root/.pup_event/drive_sr1
/initrd/pup_ro1/root/.pup_event/drive_sda2:
.wh..wh..opq

/initrd/pup_ro1/root/.pup_event/drive_sr1:
.wh..wh..opq
# 
I'm not an expert in aufs stuff, so don't really know if such files are supposed to be right there and what exactly can be done about it.
I fixed that i/o error by deleting both dirs (deleting ..wh..wh..opq only was not enough) from pup_ro1 and running snapmerge again.
Oh, didn't try fsck though, maybe it would do the job as well (will try another time)...

(ref. http://www.murga-linux.com/puppy/viewtopic.php?p=744444#744444)
___________

As for '/dev/snd/' I really don't see the point in not excluding it in a section that handles files, since in previous section (directories) it is already excluded and therefore /dev/snd/ dir doesn't exist in pup_ro1, so nothing is being copied into it anyways.

(ref. http://www.murga-linux.com/puppy/viewtopic.php?p=744234#744234)
___________

And the last thing:
cp -a --attributes-only -f "$N" "$BASE/$N"
Does anyone found a better way to accomplish that?

'rsync -a -u source dest' "seems" to work better with file's attributes than cp, but wouldn't it be an overkill..?

(ref. http://www.murga-linux.com/puppy/viewtopic.php?p=743726#743726)
___________

PS. Boolean's post on the snapmerge subject, to collection:
http://www.murga-linux.com/puppy/viewtopic.php?t=91054

Opinions and ideas are welcomed! :)

Greetings!
Reply | Threaded
Open this post in threaded view
|

Re: Fixing snapmerge...

01micko
Administrator
Issues 1 and 2 look straight forward, you know what to do..

Issue 3, have you tested to see the performance penalty of rsync? I would try it and see.
Reply | Threaded
Open this post in threaded view
|

Re: Fixing snapmerge...

JakeSFR
Well, rsync takes slightly longer with single, large files, but snapmerge'ing 10,000 short files using rsync is drastically slower than cp pair.

cp (x2, the second with --attributes-only'):
# time snapmergepuppy
Merging /initrd/pup_rw onto /initrd/pup_ro1...

real	1m16.493s
user	0m3.893s
sys	0m12.413s

rsync:
# time snapmergepuppy
Merging /initrd/pup_rw onto /initrd/pup_ro1...

real	8m42.507s
user	0m7.403s
sys	0m19.773s
Maybe it's better to stick with cp...
Thanks, I will pull request those fixes soon (along with the '--attributes' mod, if it's ok with you?).
___________

By accident I found another weird behavior of aufs, dunno if it's related to snapmerge (and how can be fixed), seems to be rather "standalone":
If I create a new directory and a file inside, e.g. /root/test/somefile and run snapmerge, all is ok - dir & file end up in pup_ro1.
Then, if I delete 'somefile' from 'test' dir, there should be created /initrd/pup_rw/root/test/.wh.somefile, but it's not!
So running snapmerge again won't delete the file from pup_ro1 and it will reappear on the next boot.
But then (after reboot) deleting 'somefile' creates .wh.somefile in pup_rw, as it should.
Well, at least in this case we're not losing any files, so it's not critical. ;)

Greetings!
Reply | Threaded
Open this post in threaded view
|

Re: Fixing snapmerge...

Karl Godt
Administrator
In reply to this post by JakeSFR
I too think that your efforts should go into woof, regardless of hidden bugs .

I don't use PUPMODE 13 much , since I personally had a few disconnects/reconnects sometimes, except am using it for testing .

snapmergepuppy,
in my opinion it should
unmount /initrd/pup_ro1 when finished and mount it again
OR
at least mount -o remount,ro /initrd/pup_ro1
it, so the name ro1 says what it is : read-only while idle .


For the fsck cases of any filesystems and encryption I am too for adding them to /init .
Without widespread user input because of not available therein, it is somewhat uninteresting to work on it.

Reply | Threaded
Open this post in threaded view
|

Re: Fixing snapmerge...

JakeSFR
Thanks Karl!

Yeah, those fixes are in 'testing' now (except for init yet).

As for un/re-mounting that would be a good idea, unfortunately there's another quirk with aufs:
http://murga-linux.com/puppy/viewtopic.php?p=649992#649992
Just checked and it's still valid in my case, only this time I'm getting "mount: /initrd/pup_ro1 is busy", when trying to remount,ro /dev/loop1 (or /initrd/pup_ro1, it doesn't matter...), after doing snapmerge.
sync & dropping caches also didn't help...

Greetings!
Reply | Threaded
Open this post in threaded view
|

Re: Fixing snapmerge...

JakeSFR
I have noticed something suspicious and I'm not sure is it supposed to act like this..?
It's not direcly related to snapmerge, but PUPMODE=13 (so close enough ;).

In PUPMODE=13 /initrd/pup_rw is mounted with "realtime" option.
Now, let's take a file ("/root/testfile"), which is also already saved in pup_ro1 (and Puppy is rebooted).
Any attempt to change that file's ATIME to the past (e.g. touch -a -d "2000-01-01 12:12:12" /root/testfile) doesn't work and after a while the file's ATIME resets to the current system time.

Works fine with MTIME or when the file isn't saved into pup_ro1 yet, or saved, but Pup wasn't rebooted, or when the access time is set to the future.
Also, works as supposed to, after remounting pup_rw with "noatime" option:
mount -o remount,noatime /initrd/pup_rw

I'm rather an ignorant of aufs, so:
Anyone knows why it's been set up like that and what are possible consequences of changing it to "noatime" in the long run?

___________


A couple of posts above, I wrote:
By accident I found another weird behavior of aufs, dunno if it's related to snapmerge (and how can be fixed), seems to be rather "standalone":
If I create a new directory and a file inside, e.g. /root/test/somefile and run snapmerge, all is ok - dir & file end up in pup_ro1.
Then, if I delete 'somefile' from 'test' dir, there should be created /initrd/pup_rw/root/test/.wh.somefile, but it's not!
So running snapmerge again won't delete the file from pup_ro1 and it will reappear on the next boot.
But then (after reboot) deleting 'somefile' creates .wh.somefile in pup_rw, as it should.
Well, at least in this case we're not losing any files, so it's not critical. ;)
I think I found the cure for that...
installpkgs.sh contains the following lines:
busybox mount -t aufs -o remount,udba=notify unionfs / #remount aufs with best evaluation mode.
[some stuff...]  
busybox mount -t aufs -o remount,udba=reval unionfs / #remount with faster evaluation mode.

I've added both of them to snapmergepuppy (1st at to top, 2nd at the end) and yes! - I can't recreate the problem with missing .wh files anymore. :)
Naturally, it was a shot in the dark - I've no idea why it works.

Anyway, I'll be running Slacko with these both fixes for a few days and see what happens...

Greetings!
Reply | Threaded
Open this post in threaded view
|

Re: Fixing snapmerge...

mavrothal
could you post a patch of the modification in installpkg.sh?
Reply | Threaded
Open this post in threaded view
|

Re: Fixing snapmerge...

JakeSFR
I didn't do anything with installpkg.sh; I only mentioned it as a source of those two lines.
Anyway, here's the diff of snapmerge:

--- snapmergepuppy_old	2014-03-04 18:22:19.000000000 +0100
+++ snapmergepuppy_new	2014-03-04 20:15:59.462875368 +0100
@@ -32,6 +32,14 @@
 
 [ "`whoami`" != "root" ] && exec sudo -A ${0} ${@} #110505
 
+# SFR: fix for .wh files not being created (in some cases) in pup_rw
+if [ "`lsmod | grep "^aufs" `" != "" ]; then
+  # Remount aufs with best evaluation mode.
+  busybox mount -t aufs -o remount,udba=notify unionfs /
+  # There's more than one 'exit' in the code, so let's use 'trap' to remount aufs back to defaults
+  trap 'busybox mount -t aufs -o remount,udba=reval unionfs /' EXIT
+fi
+
 export TEXTDOMAIN=snapmergepuppy
 export OUTPUT_CHARSET=UTF-8
  

So far, so good - running on ext2 encrypted slackosave and also briefly tested in VBox on ext3/4 savefiles, and all is fine.
I'll make a commit to Woof shortly. But with only this fix, I'll wait with the "noatime" as I'm not even sure if it's a real problem.

Greetings!
Reply | Threaded
Open this post in threaded view
|

Re: Fixing snapmerge...

mavrothal
OK. Thx
Looks OK to me.
Why not make it conditional only for PUPMODE 3, 7 and 13?
It should not run in other modes anyway but you never know what puppy users may come up with ;)
Reply | Threaded
Open this post in threaded view
|

Re: Fixing snapmerge...

JakeSFR
Why not make it conditional only for PUPMODE 3, 7 and 13?
True, I have thought exactly the same thing.
Ok, done:
https://github.com/puppylinux-woof-CE/woof-CE/pull/415

Greetings!
Reply | Threaded
Open this post in threaded view
|

Re: Fixing snapmerge...

JakeSFR
In order to address the following issue:
http://www.murga-linux.com/puppy/viewtopic.php?p=768245#768245
the only idea I have is to use 'rm -rf' before copying a symlink:

 if [ -L "$N" -o "$NEXTPASSTHRU" = "" ];then
  #link, or first run of snapmergepuppy. no '-u' (update) option.
  [ -L "$N" ] && [ -d "$BASE/$N" ] && rm -rf "$BASE/$N" # SFR: in case if folder has been replaced with a symlink (cp won't overwrite a dir with a symlink)
  cp -a -f "$N" "$BASE/$N"
 else
  cp -a -u -f "$N" "$BASE/$N" #v3.02
  cp -a --attributes-only -f "$N" "$BASE/$N" # SFR: needed if only permissions/ownership have changed
 fi
But first I'd like to ensure that there's really no way to force cp to overwrite an existing dir with a symlink. Ideas?

Greetings!
Reply | Threaded
Open this post in threaded view
|

Re: Fixing snapmerge...

mavrothal
rm -rf is a bit drastic.
You may want to also verify that the file/directory the symlink is pointing to, exists (by [ -e "$N" ] I guess).
Reply | Threaded
Open this post in threaded view
|

Re: Fixing snapmerge...

JakeSFR
Yep, I'm not enthusiastic about 'rm -rf' either, but it seems to be the only way...
Interesting idea with '-e $N', however I tend to respect broken links, as they may lead to some absent, but only at given time, resources.
I think it's quite possible that a user can move a dir outside of the savefile and link it back, then unmounts the resource where dir has been moved and saves the session expecting the changes to be preserved.
I'm still trying to perceive possible dangers of this approach, but so far can't see any...

___________

From other news: since when I have installed Slacko-5.7 (a month ago), I encountered kernel panics - 2 times while shutting down the system, immediately after pressing ENTER (I have "save-on-demand" in rc.shutdown) and one time while saving via 'Save' icon.
I suspect the newly added 'remount aufs' stuff might have triggered this somehow.
It's really extremely rare case - I save the session very often, even 40-50 times a day or even more often while testing snapmerge itself - nevertheless it happend.

Anyway, I also discovered, that it's good enough to remount aufs back to defaults immediately:

# SFR: fix for .wh files not being created (in some cases) in pup_rw
if [ "`lsmod | grep "^aufs" `" != "" ]; then
  # Remount aufs with best evaluation mode.
  busybox mount -t aufs -o remount,udba=notify unionfs /
  # and remount it back immediately (this will do, no need to keep udba=notify while copying files)
  busybox mount -t aufs -o remount,udba=reval unionfs /
fi

so maybe this will also prevent those panics..?
I'm running this configuration now and in the worst case, if the panic happen again (it should reveal itself within one month or so), I'll just withdraw that stuff completely to see if it was the real culprit.
Reappearing files are better than kernel panics, after all...
___________

And yet another bug.
When dirs are being created in pup_ro1, the 'stat' command should retrieve numeric, not literal UID/GID, because in case if a dir has some weird ownership (non-existing user/group), the given ID appears as, e.g. [443] and chown won't work with such (because of []).
This will do the trick:

 OWNERSHIP="`stat --format=%u:%g "$N"`"
 chown $OWNERSHIP "$BASE/$N"

Files are being already treated this way ('cp -a' preserves _anything_), so, for consistency, dirs also should be.

Greetings!
Reply | Threaded
Open this post in threaded view
|

Re: Fixing snapmerge...

JakeSFR
Ok, pull request has been added.
https://github.com/puppylinux-woof-CE/woof-CE/pull/448

In the meantime I've been reworking the section that handles files and gained ~16% speed boost.
Still under testing, but if anyone would like to have a look.

EDIT: Hmm, something's wrong with attachements here..? - can't unpack downloaded file (archive error).
So here's base64 encoded snapmergepuppy_test.tgz instead:

H4sIAFenQlMAA+1af3fTyJLl3/GnKJTwbDOWZZswA5mTeRuIgexkkpwkLMzCJMhSy9ZEloRaiuMh
vM/+bnW35B9JYJiFs7vnuM+ByFJ3qbv6VtWtasnYTcciG4q0SNPpWS5kfudrtw7ajw8fqr9oS397
P2xsdO50ew96nYedXu/H3p1O98EPve4d6nxO8NdohczdjOhOliSfXPnnnv8/bWt3nUEYO64c1dZ6
2CbaE1KKjJ4f7lEUeiL2BF30qDHK83TTcSaTSTuQQTvJho56LMO4vBLSiYZp1B7l46hZW3viZtmU
fnGLCOJ4nEJYFMbFZdtLxrU1z40i4VOQJWOS7oVGoE9u7FPmteWoyP1kErcoT9RjysdpIClyp5CH
e6nIxm4s4pyCCNMnmSeZOxTt2lqR+m4Oyd0HdCzSnHhdbfKSdGqzREriaNrGqEKOMHsKJflJLCiM
ITMfinxORG9jXkQmxskF7tbtok5JmodJTJORiOdkB2EkJAQ8osTTwzZJeplAp6TIyWEYOViJHod1
YQbovkFxcmG6ZyKM8yzxC6hevYmlm9eh68VGu9Ol7uOxO0X/R/Tkl00MntC8Oh3JmwqFnokLaMgn
38XUY2rM6ZnHjELW2rR5i9gwWJA6Jy/n2b+mrIhjLGCThpnriaCISFyGOY97TeUGKtmdx0rg66OX
+/u7+8+J/Qx6JEXk0wDLrESf4U1xLmL/zK/Pj1QIcC5gqn6YtSnNxEWYFDKaGu1ijqzfEisyxXxa
NMAt7grBkjyW7A70WszM1Su6PXJE7jk7u8cnRwdnx4f9p8ctbEPsjiGWp3Z5edmWgWypH/yCdg9Q
xBiMn8CH8X299dUwzGSSJEH28OFDPZR/lUP1sAeY3xDDLilIMmpPRu2zM6zuLEnd94UAQsJc8KJY
cDliYbUyGQvWh2TsOr64wL64lV7jJGfd8jt9tU6AuXzZOMmENiU1toiBrUA6bqHm1oVX7vWIzSMZ
uNkmpYmU4SCCDYZj8ScbS5ol+D2mSZiP6MnBwcnTg/1nu8/bWjLPGFDwYGIQbERuQKTr+5j7qbKC
IE/9Waeyz2NI8MNgSvU8g1nXl3s86nWMFEaDMqWqBzXcNHWxAzlwgTkwcgdsOiJzsf0QJ2RTien2
el2auoEtU+U8oBe+3+30Oj/QTlIM3WiTvEi40E2q9cyQZcQAW0L4RbpJe9v7z7eetsiNZELeyI2H
gre9dKhGINCVCY/9FGt38/o+M17McAUb7qD+Jen7mYz/FONw7I42Dcxgpth5ifUKN2tprYsgCL2Q
3wSr9SgJAHVR2QJvfegkJLLMSF3YYu3bsJZIIYlXFcFDKMvMkm7H7j6ut7R5e4kvFIriRAs34jaU
pbK5Zfl1eXrApE3o+xBI1puIR46MfSXhYechSTinBMNl4SdmSGwzXKhAWFITx949hvhsEsabalV/
aNXoeTW0vtQ29zpdvKjsOu+G26/3DrZ3+juVG2YD626gf4+Onx1tErDF/oXlQ9vfE8JCLi5z1WkD
81SdgCtMEVEInjocwyQuxFi5mi36V/eHexS4MkewEpfCK9h712pvyHo3GSXuOHxn0d0tsnhpFv1O
//iH6qbXbW/T+ofOR/z3Hx/JaKZWE5dKMyf91yc7B79u7+5vyQXyVnY4eHly+PLk7OmL7aPj/snW
y5Nn9qNa7WBvR+HVWuc/VtlZY5iWob9L9+/nwMb5/fvwKYiQcz7Fc7EVpQOQ2gMwKBnHsgZHA9W5
AwYpFK6iKHzSAOuEJcEa62Ec5nXejRCBFc6Hf2fuGJ6n3a61tSsGAfCdw5eHxyfbJ/3y5rx/JuW3
a8fb/9U/3D462bLeCW8EzcVkrfM43LfoiqB3sgPqko2o3aq/s3h+D9rY5QBuM1f+rIXwrxbGy7oW
2nWIbumwwtCesl6GPNd97MTh9vHxyYujl1uWhc21EaPgkRwwkmEMuKSulPkoK/QGL/afConZICAe
HmP2qcTcjl+8xNa+2t+y4sRSWFGLstZVL17PMBMp1efoUd3gyGBoJkGJL+Pt5wTeGH4XJc9E6Ykf
bp+82LLY120qtrHpFDJzqovZrdfd7tGP/MCqvdrhlU583obj/e1DjFeb7yuqkk2smufTOj+hqyvt
a7q1GtgEZr7OW3q2t/1b/wgz+kkzkDVfBCFHQ8CoQgs92T7uV6LXP8wGfgTsIymWejjGxVm1IAQ8
6BDu2lX2KqEyRtOvBzt97O17ekB2snjnx2t3wDp/5+l/IK3rV9DokMoejbJv8671k1niT/SxNk6K
OE8TMD+y32O1PENrXoy6w1yVkap6C39RBKauvNIcoTDBQjMB5rqlRTagMBXVPBe8vam4r9oCo+13
kUQYLvFhnXLosmiGCK3/GtEaHQk1GxXdtDMYMLkTF25UuMqOIAnulcB25HSQXOrZk22GQH+ZFtEq
/IG7hcly+DeUhBz1EpUUmBdhvQPXO4fHBc8KsRwE+4ZyUhOEfBhziwNTLHQ0PRdYwLxg0CqQE+bU
rBCloOZfnlzGy5qbGyNG78+v8MMsUMMXRgRY8JbBUbDToa2v2SDv1Yvdkz5c/fFXl107fnZ8yEiU
W+sNDQCowe7TksG86diPf//eIgehwHOUkiRdSY4qUB9GnPfgjwtqXosuCKbkIr0ETEBSgCQCQ6aZ
CcjvFJDDmOEKp+zqTUHfuT5IZgJ5FiWuDxDHpC5+3up2mjXsCuDSJlsPe9sgOxNDcUn19n3nLczi
bfvNqfP7/TrmOU1BZehtE1CXAAwceF2iz1vHcX6SprPj1OmqpnGDCaFXRvs1P4EfgaN9N8AUFYez
1vctuDba4bsgeAs32YU92bdADSwmd9qglGf1ABZkxYI0w1e4G7lSk+iSDKq7zPIXHGZby90xcpcE
1qhGJX0sIcuGBMYIk9FstL5MR+ugltk53jWMmVMyf/NUSMjV+8EJwWh05sAxkIx3Nkubl8RuQntp
Xheb8f/4ZZSNofwA7/vAtvXRWf+ws//Rot7PzDGduICAtUmIPXVjHadZjArdSooin6wOX0QiRzyH
ojKVQrHw8TkuyE6viV/7Y5F8s+dk5gDMTpnhmUwU2FB0nHUyTQquKtTz8lXag2n/qyCPdwmPs++W
3nLkgv+kJXarnJidthTdH6kiSGJ8OOlEp5z7mkpCSqen6yWjpRySnwyiBK6TFT2vXGrwCwYa3lwe
aWp150nhjZbU4Sxv4oL6zR7xFrEjdIDNGwasGTqJMAXqrJav6zqWcrAWv3uGYjjZCsd/P5WiG+Ba
PmPDqULaX8DYp0G2gtKXQGm2B/Mavg1Hs95/B0RewprGbo0EzF6RFV6a8kt1rhQxd5I5MpOUSyxB
0KJIqASAXC8vEPrLREctqkIKr8mBo1ceF1tm6h8IcBJUgGPdTEOtSvs3q1ZJXo6YpuZgtp45rw6b
gIg3UiOY8R3y0tar8F0jjlGkMW9zKDrkSWp6ijt7c3c0+r8zFHKQJeec+0zHEXJAqSscvkAGrmc+
UPmfx6VicEkkTfDkEwPRHFBkYeOWwpA0/ZWIur1XZ7KF9oZTsmXrUDO7zXb0uFtxhDtVpwHAd86X
vOuquFsqtA8FaqWa/buGMOUb1kz5YnFn6LOI43d9C7a3s3vUf3pycLTb/wZ8b1a9IV280ajXpUNm
Xaq0CGiZEk6L/ig4d3bPxawMKYsBVyKBxZsLPIucTBMvTjByN4zo+94N/Kteph/2BbPP+uk4zq9O
Nf+5OmX8XZ3Kqbw6RcZ9dVoW1q9ONWm7MuTt6pRnUP4NfP03zaW+kKOxuYjxyFQUb6N7pkh+rdhd
JuP0GqRUCFrMe51M8BlTPhm3JuNMcCUEFyl7BHgY7R45JpUZtopMnGRrNveGMaqqCpMxv1BlU/ll
XhaNwrw2bzIzV7QWkirhICrmCInYj4uQdcypkjpVwU0fa1yosFFDU29dzfIdiasoDELhw269ESeF
s3eQDWIdwJXGntiy1EsPXu33j45f7CK3f4dVY6dtVSLLt+4Vm/eGFR1GzgR9rVfdF2Ze2vjt7/lW
dvZsd+/bWJiuqLv0vggRNEfIXzfnyuY0ycIctJd3xVTEWnCaAZPiquQuBW5yqqMLcHmWFAMdLrS+
ZoWzmeDas6N+X1U73vkB2U+oWxnVpKwyXCEaMspj9TAHjiTVqT6rnm2o6hnV31UszBSjabmMTCZ+
APFFpIp7g2kupLaKOruAOpaFYIgkTTtfBezFOqYdJ6qqfE36J4aILINvAieY1XCh9ad8cPZMncjB
LamzJOPX62zq9TbNnanMXJlyd/ogj64dhehK++xog2452TAPPnGcQQunGeVhxvVS+me8sfK4i5Vu
SKSqwm22ZMEbL95sexl7L86gyxqSrt2DlDKw8Lc6AAq5sGF2MsyliAIM2QvHYa5rPbbhtYpbxWIS
MVVpvI1ViQkDNEUH35wjEBNVIWvwYZ0q45gil8vj8UMWormc0te+w2bb3MtO4d3wZP6WCS7mlsr9
1S2sIry08UPEvO65ITfXCN4u1QhuHnDadhCZ2vevcKGjk77mBeorRCl9gQ02z5C+66s//ezCXJYh
DL/a98/mHugINrsO/Nk1ItnsB7Z27sdorH/o5ehrhllWxHMvVGCtnpWvf1viZ71++8q5Wz4qxoMY
QVyacUgm8nV1NYPVgpA0A4hAu+5Junf4Nla8/1aTXg7EykHv/ne/jMimOFrdtgnh4bu1nf6Tl8//
7wbsKg8tzy+UiMDvlDUTTsNUoZftzxzggT/PearqnNVURqrivjoC2beWyROCgyZJS3X9WYb0ORHj
5M8witwrtX9/JAN5xclHeCGuPNcbCZZ7k1gukVLjDd3VKch+paW7OgPhG83FOpj2QfI8TBU5UYlU
VayFo8hBQa/nHEtJi1IYJxzKnwj2NW42vYvIg8wxT1JzssM7wZoWMbKCEUFDY/jbQAUvJBf6yInh
dXZ41H+2+/qjcsNVoqb2cb3RKAH4fa+z8ajTaTbJHua0XobfsmKOBEt3gKp6nc4vJCMuZetTVKK/
EK3p74frzcq98jEWXp1MMCtc8fq5GjASmIs644KrFpn26NePissDhvJoWGWljMcafbk+FqfISDb0
71PM4Do1mEv6OOfTxzwzffKMzLVdTq3ZvK4gdVqASA09mMr0JOFzQj4mMVBeW7votXtdsgtNxGTJ
xHjtFVkbuxHwHITDIhO+8RgK7hWnVVZQZqAz7quTdWMYNvu2+WPD0sQqJ8FQbxE2Q59nwrXzxBfV
1ubzEHWw2dDfNTWrr4lo7mWl8/JvnOVCnaOsE4Sx0gxPOkgiH+kDF64H7KOAWkCbozszAbc0S2p4
qTFbUIyMeS/oAJeeljpymQBdbVcrCFOZm1Xv59tDRrX5ZnTxpQJInxDPJNhunmch6ImQNle0vlyg
1haXQ/isMlBfoKkP2MBumEU6iEUiA5hTXRwyJUxdtbr5mIGWv4QYFMPSMG/5IOKmg4lKDKkzCuWi
9aLKcogqq1Q4uPlpaUfli81cbNp1DkjpQC5VMhPli7C4b5PJ1eZ8yPJhfK0mp7HHR83W+qudb3A4
V5FoLpCAcMAkXAqK2GObQ1gK+XuRnEuQdanjuMrP2W1ovIzlkJl9Tf0641+NJn0ovStYgznlV94g
TixG6Y10pHQTROZzD8Szw73t37bqm526uj/3DZTNzoupxYMO2V6UwK75QNQO4chomJ/bfuhGydCe
uBl/Okf2YAjwIRLxhzF4f1cX39YUmYrCgYMxeogDOWdf8y3Gwqk8G1c3YSsfawvftVjr5qsXq9Y/
OuKvA14eb3XMFwWfST6rTwyuRxrI+vWYpTfMV0FUf7V9xKq/W3sZu4NoVt3l6rjJJH8D+Muiuink
c7bZrjfViqqttta1fEu/ycy6q46a9cTlJ1LgWVydJcI3zdiqZnyia+H8n8p+jblqUsVfDurk/dZX
1pXLQXfOAZo1lW7zUcbY8InyhBfqUGXxRgrXsBVI77zJVGQUDkfqwMNLxmOVl7WtL9CIYvTr1d2v
bsyf+3561VZt1VZt1VZt1VZt1VZt1VZt1VZt1VZt1VZt1VZt1VZt1VZt1VZt1VZt1VZt1Vbtf7v9
G+VJEdgAUAAA

Greetings!
Reply | Threaded
Open this post in threaded view
|

Re: Fixing snapmerge...

mavrothal
Here is a fast test (with no data to merge really) in my super slow XO-1

# cp snapmergepuppy_test /usr/sbin/snapmergepuppy
# snapmergepuppy
Merging /initrd/pup_rw onto /initrd/pup_ro1...
# time snapmergepuppy
Merging /initrd/pup_rw onto /initrd/pup_ro1...

real 0m37.667s
user 0m1.070s
sys 0m2.520s
# time /initrd/pup_ro2/usr/sbin/snapmergepuppy
Merging /initrd/pup_rw onto /initrd/pup_ro1...

real 0m47.048s
user 0m1.000s
sys 0m3.030s


See how it holds on the log run a bit.
Reply | Threaded
Open this post in threaded view
|

Re: Fixing snapmerge...

jamesbond3142
In reply to this post by JakeSFR
A very long time ago (in dog years), I was looking into the same issue. I ended up writing a replacement, I went through 19 revisions. This http://murga-linux.com/puppy/viewtopic.php?p=499860#499860 links to final public revision (s19). You can look at that for ideas. This script however supports only "aufs" and not any other kind of unionfs.

What the script does basically are:
a) clear old files in pup_save for newly create whiteouts
b) clear whiteouts in pup_save for newly create files
c) rsync all changes

After s19, the script was made Fatdog-specific, and there were 3 more revisions after that - basically to remove dependency on "lsof". But it straight forward to port it back to Puppy for someone keen enough to do it. If anyone is interested I can post a copy of the latest version here.

cheers!

On Mon, 7 Apr 2014 06:28:02 -0700 (PDT)
"JakeSFR [via woof-CE]" <[hidden email]> wrote:

>
>
> Ok, pull request has been added.
> https://github.com/puppylinux-woof-CE/woof-CE/pull/448
>
> In the meantime I've been reworking the section that handles files and
> gained ~16% speed boost.
> Still under testing, but if anyone would like to have a look...
> snapmergepuppy_test.tgz
> <http://woof-ce.26403.n7.nabble.com/file/n452/snapmergepuppy_test.tgz>  
>
Reply | Threaded
Open this post in threaded view
|

Re: Fixing snapmerge...

JakeSFR
Thanks for testing, Mavrothal!

Thanks Jamesbond - first thing I noticed is that it's just enough to use:
busybox mount -t aufs -o remount,udba=reval unionfs /
alone, in order to force reval of layers and prevent reappearance of files in some cases.

Also, renice -n -20 $$ gave me another 7% (my usual test with 10,000 short files):
# time ./snapmergepuppy2 
Merging /initrd/pup_rw onto /initrd/pup_ro1...

real	1m16.135s
user	0m3.030s
sys	0m7.777s
# 
# 
# time ./snapmergepuppy2 
25243 (process ID) old priority 0, new priority -20
Merging /initrd/pup_rw onto /initrd/pup_ro1...

real	1m10.917s
user	0m2.873s
sys	0m7.410s
# 
# echo '100-(70.917*100)/76.135' | bc
7
# 

Greetings!
Reply | Threaded
Open this post in threaded view
|

Re: Fixing snapmerge...

mavrothal
In reply to this post by jamesbond3142
I remember that. It was pretty fast (thus I had it in XOpup ;)
But I thought all this time that BK had integrated these changes.
Now that I look at it, there almost nothing common... :-o
A copy of the latest would be handy.
Reply | Threaded
Open this post in threaded view
|

Re: Fixing snapmerge...

mavrothal
In reply to this post by JakeSFR
`renice -n -20' will basically freeze any slow machine while merging.
Given that merging kicks in at regular intervals, it is not very desirable.
It would be good though as an option at the shutdown merge.
Reply | Threaded
Open this post in threaded view
|

Re: Fixing snapmerge...

jamesbond3142
In reply to this post by mavrothal
On Tue, 8 Apr 2014 04:32:17 -0700 (PDT)
"mavrothal [via woof-CE]" <[hidden email]> wrote:

> I remember that. It was pretty fast (thus I had it in XOpup ;)

Glad you like it :)

> But I thought all this time that BK had integrated these changes.
> Now that I look at it, there almost nothing common... :-o

Barry wanted to keep compatibility with the original unionfs. Plus IIRC, there was another thread with similar purpose running at the same time, and he got the patches from the other thread ... ;)

> A copy of the latest would be handy.

Here it is, version 22. This is the version actively in use in Fatdog64/FatdogArm.
Note that I have disabled the "freeze/unfreeze" because I found out that certain programs don't like it and will crash if I attempt to "pause" them; so the code that is responsible for this can easily be removed (I haven't removed them).

All that needs to be changed is in snapmergepuppy, it calls fatdog-merge-layers.sh with $1 as "merge-from" (this should be pup_rw) to $2 "merge-to" layers (this should be pup_ro1, I think). Then instead of loading /etc/BOOTSTATE, one should instead load /etc/rc.d/PUPSTATE and check for PUPMODE ( if [ $(( PUPMODE & 1)) -eq 1 ]; then xxx; done ).

In Fatdog the merge-layer script is used for other purposes too (not only for snapmerge) thus my splitting of it into two files. If this is not needed in Puppy, one can merge the two files back together.

Hope this is useful.

cheers!

snapmerge-s22.tgz (4K) Download Attachment
12