You are not logged in. Please login or register.
DriverPacks.net Forum » Feedback and Support - DriverPacks Base » Copy OEM folder contents to HD before "fake setup"?
I notice that during the "fake setup" in method 2 that the driverpacks (with shortened names) are being extracted directly from the CD/DVD which slows this process considerably. Would it not be faster if the contents of the OEM folder were copied first to the HD before the fake setup / extraction starts?
I wouldn't think so, then you would have two copy processes instead of one. One to copy from CD to HDD, then another to extract.
Extracting zip files, let alone ultra-compressed 7z's, directly from DVD/CD can be much slower than first copying the compressed file to HD then uncompressing. The overhead from the horrid seek times is the wrench in the wheels I think. Copying files over in a sequential file transfer, especially large files, from disc does not take as big a hit.
Try it yourself with DriverPack Graphics A 
On my rig, it takes approximately 41 seconds for DriverPack Graphics A to decompress directly from DVD to hard drive.
Copying DriverPack Graphics A first from DVD to disc takes a mere 4 seconds, and then only 20 seconds to decompress. The time has almost been cut in HALF!!
DVD/CD is still a profoundly slow medium.
Last edited by Fragbert (2008-12-18 06:24:05)
I agree
enter the to be developed Method Three (M3) and SoFI
with huge amounts of space that are available for install media it is my belief that compression is not needed at all.
SoFI (SAD) can be called from runonce and will run against a M1 or a raw extracted folder source.
or the M3 i have been thinking about for a while
IE
extract all DriverPacks to D\
copy D\ from your install media to SystemRoot from presetup.cmd
instead of the extraction routine that is there.
I was not sure that m3 would be faster because of the overhead assosiated with the thousands of small files.
it does create a searchable ODD for the 'found new hardware wizard' as a side effect and could be used as a SAD.
this is the outline for M3...
I agree
enter the to be developed Method Three (M3) and SoFI
with huge amounts of space that are available for install media it is my belief that compression is not needed at all.SoFI (SAD) can be called from runonce and will run against a M1 or a raw extracted folder source.
or the M3 i have been thinking about for a while
IE
extract all DriverPacks to D\
copy D\ from your install media to SystemRoot from presetup.cmd
instead of the extraction routine that is there.
I was not sure that m3 would be faster because of the overhead assosiated with the thousands of small files.
it does create a searchable ODD for the 'found new hardware wizard' as a side effect and could be used as a SAD.this is the outline for M3...
Yikes that will be just as slow. Your goal is to overcome the SUCK that is DVD/CD media. The best method is to copy a few large files (driverpacks still compressed) to the HD, not a million little files. That will slow the CD/DVD to a crawl. DVD/CDs do OK with large sequential file copies, nothing else. That is why a modified method 2 will be ideal - copy the large compressed DP's first, then uncompress once on the HD.
You are right about the compression not being needed, but keeping all those millions of files as one 7z IS needed, so perhaps have the DriverPacks repacked as simple store or zip with little to no compression. That will make things quicker with there being no overhead to uncompress the DriverPacks.
less compression... hmm
that has merit. good idea!
as long as we are useing simple CDN bandwidth is not a huge concern for releases.
it may be better for the nightlies too IDK
Yes, Fragbert.
Yes!
CD/DVD transfers ARE slowed VERY much over HD transfers, and small files take MUCH longer to transfer from optical drives to hard drives. It has always been so.
Your idea of copying the (highly) compressed .7z files to HD for subsequent extraction is a REALLY GOOD IDEA.
Jeff, you are also right on to remind us that storage capacity is FAR greater than it ever was, so it makes sense to have a decompressed flat of driver files somewhere on the HD for easier/quicker access by the driver installation application.
I dunno why I just jumped right in here, but Fragbert has a really good idea that will decidedly improve the user experience and should not require y'all to do any kind of major re-architecting of the otherwise sterling Driver Packs executable.
Heck, I'm still wrestling with X64 challenges, both with XP and Vista, and all the headaches that come with downgrading a new system from Vista back to tried-and-true XP, whether X64 or X86. It's a shame you can get a great deal on a Core2Duo laptop ($599) and be stuck in the cold with only Vista X64 driver support from a MAJOR hardware vendor (hey sir, ya know what I mean? Rhymes with "hey sir".)
If it weren't for you guys, this IT Warrior Chief would be outta luck and stuck with the terribly slow and bloated OS they call "Vista".
Thanks for all your hard work. I'm hoping y'all can pull off a small miracle and give us all a truly wonderful holiday gift of your latest and greatest handiwork. You're my heroes, plain and simple.
Best wishes from balmy, rainy Atlanta, GA.
MARK STRELECKI
still thinking... so a M3 slipstream might be like this...
base unpacks our normal DriverPacks
base then repacks the users selected packs into a single large archive with minimal or no compression.
at this point do we still copy to the HDD first or does it not matter without the overhead of compression?
this is somewhat in line with what i have been thinking about for a while.
It is both viable and would not be terribly difficult to do.
fragbert would you please do some more speed tests related to these ideas and report back?
Ie take a normal set of packs and try:
multiple packs with light compression
all combined into a single pack with light compression
multiple packs no compression
all combined into a single pack no compression
and with and without precopy to HDD...
I know it's a lot of work and quite a few blanks to burn.
it may be important to note both IDE and SATA times...
I belive to get the best answer a team effort will be required.
it would save me more than a few hours work (so i can devote more time to other new features and bug fixes. 
)
If we could really cut our setup time in half i am all for it.
Even if it is only a few minutes now... only a few seconds is better 
thinking again...
M2 was optimized for space. with the intention that DriverPacks and an OS could fit on a CD.
Perhaps M3's destiny is speed... 
For example I think extraction is faster without the progress bar, so we could skip that too.
tweak as many things as possable for speed and forget about space and asthetics altogether.
M2 gets an upgrade as a byproduct.
PS i don't see any reason why m2 could not be upgraded if there are no issues that result from the change.
it would be easy enough to test we could just post a modified presetup.cmd to use for testing purposes.
We simply add a line to copy the packs to %SystemRoot%\D first
Then modify the line for extraction to reflect the %SystemRoot%\D as the source.
the packs will get extracted to their normal subfolders and then the whole thing is deleted automaticaly by the finisher
if all goes well we could easily implement that change into base / presetup.cmd.
Great post Fragbert...
edited previous
I have always made them on DVD with 3rd party packs so just for my two cents fitting on a CD hasnt been relevant for a long time.
Fastest (should/would) be extracting straight from CD/DVD from packs packed with minimal compression to HDD. Reading a sequentially large file is quick (think ripping an ISO, or verifying a burn). HD>HD extraction is going to be slower unless you have a super fast PC/HDD.
If anyone is interested I make a DVD with the added packs, create ~3.5Gb padfile, add it with UltraISO setting pad file priority to 9 (means it gets put on centre of disk, low LBA) - all the useful files are then placed on the outside of the DVD so xfer rates are about 4x faster overall. You can also keep the DP folder priority 0, and everything else 1 if you want max extract speed (this will place them on outside of disk, so you get about 22x read over 4x on inside)
Let me make it very clear - you absolutely, positively, unequivocally, unconditionally MUST copy any archive to the hard drive first before extracting (high or low compression). You DO NOT WANT to ever start extracting an archive full of millions of little files DIRECTLY off a CD. Your grandchildren will be gray by then
still thinking... so a M3 slipstream might be like this...
base then repacks the users selected packs into a single large archive with minimal or no compression.
at this point do we still copy to the HDD first or does it not matter without the overhead of compression?all combined into a single pack no compression
and with and without precopy to HDD...
Fastest (should/would) be extracting straight from CD/DVD from packs packed with minimal compression to HDD. Reading a sequentially large file is quick (think ripping an ISO, or verifying a burn).
No, just trust me on that one. If you copy the archive to disk, then you are just reading from CD (decent speed). Extracting directly off CD means you will be checking back and forth to the archive after each iddy-bitty file (horribly slow due to CD seeks). You would think that the archive would be loaded/extracted from memory but in practice it just doesn't work that way.
HD>HD extraction is going to be slower unless you have a super fast PC/HDD.
Actually, a current generation 7200rpm drive is all it takes to cut your extraction time in half, see my above extraction time test
If anyone is interested I make a DVD with the added packs, create ~3.5Gb padfile, add it with UltraISO setting pad file priority to 9 (means it gets put on centre of disk, low LBA) - all the useful files are then placed on the outside of the DVD so xfer rates are about 4x faster overall. You can also keep the DP folder priority 0, and everything else 1 if you want max extract speed (this will place them on outside of disk, so you get about 22x read over 4x on inside)
Actually I experimented with this before too - depending upon the PC, the overhead on extraction based on compression can be minor on a decent PC. And no doubt you will be adding time to copy from CD/DVD if your uncompressed file is 3.5GB vs 350MB compressed.
Basically you want these files to be put on the fastest medium (HDD) as quickly as possible (OEM setup) before they do any work. Take the CD/DVD out of the picture ASAP.
Last edited by Fragbert (2008-12-19 02:13:43)
I hope to clarify Overflow's thinking a bit.
There is NO NEED to change the compression format of the existing files. I am in total agreement with Fragbert that these files should be copied over to HD BEFORE they are expanded (like he says during OEM setup somewhere - I was thinking during the early text mode file copy phase of Windows setup), so they exist on the MUCH faster HD medium for subsequent extraction and processing.
Let's keep this as simple as we can, and leverage the most work from past efforts.
Same .7z files, just copied (early on, if possible) to HD prior to extraction.
Besides, I always hated seeing those thin little windows at 1st GUI boot saying "extracting x of y" that seemed to take SO long.....
Gosh, this next release is shaping up to be a real HUMMER, baby. SAD, "M3", better BartPE support (??), and mucho more drivers. I'm feeling a bit faint.
Can you say "butter my backside and call me a BISCUIT!"
Yeah, it's THAT kinda happy thing you got going here. I'm loving the ride, too.
Thanks again from another IT Warrior Emeritus in Atlanta, GA.
OK so useing large files with max compression (as we now do)
will effectively get the files off the ODD and onto a HDD in the most efficent manner.
Ready to then be extracted in a fraction of the time.
so all we need is this presetup.cmd to test with
If exist "%systemroot%\system32\setupold.exe" DEL /F "%systemroot%\system32\setupold.exe"
SET TAGFILE=\OEM
FOR %%i IN (C D E F G H I J K L M N O P Q R S T U V W X Y) DO IF EXIST "%%i:%TAGFILE%" SET CDDRIVE=%%i:& GOTO DPsFound
:DPsFound
Start /wait /high /separate "" XCOPY /S /Y %CDDRIVE%\OEM\. %SystemDrive%\D\
Start /wait /high /separate "" %SystemDrive%\D\bin\un7zip.exe %SystemDrive%\D\DP*.7z %SystemDrive%\
%SystemDrive%\D\bin\un7zip.exe %SystemDrive%\D\bin\*.7z %SystemDrive%\
COPY /Y %SystemDrive%\D\bin\DPsFnshr.ini %SystemDrive%\
IF EXIST %SystemDrive%\D\*.ins COPY /Y %CDDRIVE%\OEM\*.ins %SystemDrive%\
%SystemDrive%\D\bin\DevPath.exe %SystemDrive%\D
START %SystemDrive%\DSPdsblr.exe
EXITi haven't tested this but it should be close to done... im sure i snuck a typo in there somewhere.. ;P
....
We had a HUGE number of complaints when we did not have the progress bar because many people belived thier install was hung up and there was a staggering outcry for the progress bar. And i must agree it does seem to be hung up during extraction without that visual feedback.
Did a quick test via batch file, then lost this thread which I just found again!
I burned three packs to CD-RW as a test.
------------------
This is CD>HDD ~ 3 min 10 s
------------------
c:\>echo. | time | find "."
The current time is: 15:29:17.71
c:\>un7zip e:\DP_Graphics_A_wnt5_x86-32_812b1.7z c:\test c:\ut.log
c:\>un7zip e:\DP_Graphics_b_wnt5_x86-32_812b2.7z c:\test c:\ut.log
c:\>un7zip e:\DP_Graphics_c_wnt5_x86-32_812b2.7z c:\test c:\ut.log
c:\>echo. | time | find "."
The current time is: 15:32:29.84
---------------------------
Doing it CD>HDD>HDD ~ 2m 20s
---------------------------
c:\>echo. | time | find "."
The current time is: 15:37:05.23
c:\>copy /y e:\DP_Graphics_A_wnt5_x86-32_812b1.7z c:\test
1 file(s) copied.
c:\>copy /y e:\DP_Graphics_b_wnt5_x86-32_812b2.7z c:\test
1 file(s) copied.
c:\>copy /y e:\DP_Graphics_c_wnt5_x86-32_812b2.7z c:\test
1 file(s) copied.
c:\>echo. | time | find "."
The current time is: 15:38:29.28
c:\>un7zip c:\test\DP_Graphics_A_wnt5_x86-32_812b1.7z c:\test\out c:\ut.log
c:\>un7zip c:\test\DP_Graphics_b_wnt5_x86-32_812b2.7z c:\test\out c:\ut.log
c:\>un7zip c:\test\DP_Graphics_c_wnt5_x86-32_812b2.7z c:\test\out c:\ut.log
c:\>echo. | time | find "."
The current time is: 15:39:27.26
Ultimately it was faster on my PC copying then extracting by half a minute, although I have 2x 7200.10's in RAID0. So I expect this margin to decrease to almost nothing on slower HDD IO.
I will try this again on a laptop when I get home tonight.
Last edited by chud (2008-12-22 21:25:17)
Like I posted earlier, you do not need a RAID 0 array to see the speed benefits of copying to the HD first.
My test w/ DriverPack Graphics A was on a laptop 7200rpm drive, which by many standards is slow compared to a nominal desktop drive.
guys you are forgetting something very important...
you do not have 32bit drivers in PE, you are useing txtmode drivers when the copy/extraction is done.
So testing in a non Preinstalation environment is not a valid test...
the only valid feedback is from presetup.cmd in a live test (useing the above supplied presetup.cmd - or a similar one.)
tia
Doing that now
OK test results with 23 driverpacks (latest nightlies + all 3rd party packs except for HID and printers):
Traditional presetup.cmd: 5 minutes 20 seconds
Modified presetup.cmd: 3 minutes 45 seconds.
This is on a laptop drive. Not a speed demon. Not as much as I would hope, but still cuts time by a large margin
Notes: There seemed to be a long pause to extract packs no matter the size, even the tiny 3rd party stuff. Wonder what that is? It's definitely not the CD/DVD drive, that is observably no longer touched with this new test method.
Also: the extraction routine will cause low memory warnings on 512MB RAM machines or less! It still finishes unattended but perhaps something as robust as this method should have a minimum requirement of a 1GB RAM PC?
Last edited by Fragbert (2008-12-25 05:45:16)
the answer i belive is 7zip loads the file into available ram (think smartdrv) before it begins to extract.
basicaly the same idea you had, Get the file onto as fast a media as possable first, then process.
not much faster media than ram for this task. so the lag is ram buffering...
This is why i was curious as to the true performance boost.
In theory the file would be read from the ODD to RAM as a continuous stream until the default block size is hit.
I belive that is 128MB but i am not positive. it is also possable it allocates them in multiples of teh default block size.
(more ram longer to buffer) if i am correct machines with more ram will obvioulsy extract faster, if all other factors are equal.
IE a machine with 4 gig of ram may not benifit at all...
IDK that is why your input is so very imortant to me!
- Thank you for your Time!!!
useing 7zip itself instead of our 7zip wrapper may yield some answers or shed insight to features we are not fully taking advantage of.
as you can see the txt mode operation is not quite the same as a full blown windows operation.
(the drivers that will provide our best performance are not installed yet )
30% faster is quite signifigant. if i can offer that kind of improvement it is worth doing...
- Can we do even better?
- Does the amount of available RAM matter? I vote yes...
- Can we set the amount of ram that 7zip will use to do the extraction? (like a /RAM=Max switch)
- other programs can extract 7zip files too...
is there a free extractor with progress bar that is faster?
perhaps an old version of unzip
side note: I think the additional time required to defrag after this mess will be way more than 2 minutes it saved lol {kidding}
Jeff
Extracting zip files, let alone ultra-compressed 7z's, directly from DVD/CD can be much slower than first copying the compressed file to HD then uncompressing. The overhead from the horrid seek times is the wrench in the wheels I think. Copying files over in a sequential file transfer, especially large files, from disc does not take as big a hit.
Try it yourself with DriverPack Graphics A
On my rig, it takes approximately 41 seconds for DriverPack Graphics A to decompress directly from DVD to hard drive.
Copying DriverPack Graphics A first from DVD to disc takes a mere 4 seconds, and then only 20 seconds to decompress. The time has almost been cut in HALF!!DVD/CD is still a profoundly slow medium.
Now *that's* odd! The main reason for M2 was space efficiency as OverFlow points out in a later reply, but speed was also a driving factor. It's significantly faster than M1.
It does make sense that copying the file over first and then extracting is faster now, for various reasons:
- the DriverPacks are *much* bigger than when I experimented with M2 about 3-4 years ago. Which means more file access is needed and therefore the optical disk drive performance hit is much bigger
- hard drive performance has increased a lot too, which makes that initial transfer again that bit faster to reach that impressive 30% speed boost
For example I think extraction is faster without the progress bar, so we could skip that too.
tweak as many things as possable for speed and forget about space and asthetics altogether.
Having a progress bar removes the impression that the system hangs. The overhead is *very*, very, very minimal (yes, that's very^3). I'd estimate it at 2% slower decompression in the very worst case. It's probably more in the neighbourhood of 0.2%.
It's incredible that almost 5 years after I started working on this, people are still trying to find ways to do it better Awesome
Yay for innovation!
Also: the extraction routine will cause low memory warnings on 512MB RAM machines or less! It still finishes unattended but perhaps something as robust as this method should have a minimum requirement of a 1GB RAM PC?
The ammount of memory needed for 7z decompression depends on comression degree (size of dictionary)
A liittle decrease of dictionary size lead to little increase of DP size (and must speed up decompression )
It would be more reliable for low RAM machines
Last edited by magon (2008-12-25 15:22:10)
The amount of RAM needed to decompress is significantly lower than that required to compress!
A dictionary size of 64 MB requires mere 66 MB to decompress (there's a little overhead I guess) but 709 MB to compress.
The lowest-end machine I am using the DriverPacks on has 384 MB of memory in and it has yet to prompt a memory error during decompression.
512 MB of RAM should be more than enough - form my personal experiences, the minimum requirement does not lay at the 1 GB mark...
Helmi, look at last DP_Graphics !
DP_Graphics_B_wnt5_x86-32_812.7z requires ~750MB to decompress
what size of dictionary is used?!
Last edited by magon (2008-12-26 16:38:03)
I used a 1 GB dictionary size to compress all the graphics DriverPacks. I haven't read anything that says the dictionary size slows down decompression. I'll have to reread the doc.
Powered by PunBB, supported by Informer Technologies, Inc.
Currently installed 3 official extensions. Copyright © 2003–2009 PunBB.
[ Generated in 0.022 seconds, 10 queries executed ]
