Topic: Repack DriverPack's

Colleagues. I wish you would request.

At this time you compress driver packages for the best possible options. A 7-zip is very demanding of resources when unpacking DriverPack's for computers with low memory - 512 or less. Therefore, decompression takes a huge amount of time.

What is offered:
- Change the size of the dictionary to 32MB
- Archiving optimizes compression inf-file and ini-files.

As a result, we get the gain in performance when unpacking on slower computers, the maximum size optimization, and optimization software for the automatic installation of drivers, which first create a database from the contents of inf-files (DriverPack Solution, Drivers Installer Assistant and others).

The idea of this method of repacking belongs BadPointer.

How to use:
   1. Unpack the archive Dstr_Repack!.7z to folder where the driver packages.
   2. Run "repack_all_NN.bat". NN - dictionary size in megabytes. Recommeded NN=32.
   3. Repacked DriverPack's will be placed in the folder "out", without changing the original DriverPack's.

I'd like to hear your opinion on this subject.

Dstr_Repack!.7z

It's not as bad as it seems, but not as good as it could be
My Site My Forum Donate Project

Re: Repack DriverPack's

I tried your script.
For most DriverPacks, there was a minor decrease in compressed size.  Some DriverPacks had an increase in size (mostly packs with large .exe files inside).
Do you have any comparison tests with times to show how much time is saved during decompression?

Read BEFORE you post.  HWID tool   DriverPacks Tutorial   DONATE!
http://driverpacks.net/userbar/admin-1.png
Not all heroes wear capes, some wear Kevlar!

Re: Repack DriverPack's

We discussed this years ago... I don't remember where the thread is now.

Erik used to use a huge dictionary size and we had people who were not even able to extract them on machines with 128 meg of ram... at that time he reduced the size of the dictionary.

The higher dictionary size means the better compression, but it is slower and needs more memory. Memory usage for LZMA compressing is about 10 times more than dictionary size. Memory usage for LZMA decompressing is close to value of dictionary size.

the command line presented is

..\7za.exe a ..\out\%1 -mmt=off -m0=BCJ2 -m1=LZMA:d%2m:fb273 -m2=LZMA:d512k -m3=LZMA:d512k -mb0:1 -mb0s1:2 -mb0s2:3 *.ini -ir!*.inf

note -mmt=off   Seems odd.
If a machine supports hyperthreading then why disable it?
This will drastically reduce performance when creating the archive on a PC that supports hyperthreading. (like mine)

I too am curious how much time we are talking about...

Since the max dictionary size on the version of 7zip we use is 256 - I have a hard time beliving that there are really that many machines that have less than 256 meg of free ram... If someone is running on a machine with less than 256 meg of free ram then shame on them. We should not penalize everyone else with a viable system to cater to a few people whos machines aren't even valid for windows xp anyway. (less than 256 meg of free ram is not valid in my opinion). IE the base system ram should be at least 384 meg, 512 prefered.

It is my belief that this recomendation will only help machines that are not viable for XP anyway. 
A machine with less than 512 meg of RAM is going to be painfully slow, period.
I belive that xp boxes with less than 512 meg of ram are a very small percentage of the total machines running XP.
if a machine only has 256 meg of ram (total)... then load win98se on it and call it a day big_smile.

If a machine has 512 meg of ram or more on it (and the vast majority do) it is my belief this will have little or no effect at all... am i wrong?

How much does this increas the size of the (main) DriverPacks?
How much time is saved; @256mb ram, @384mb ram, @512mb ram, >512mb ram?

DP BartPE Tutorial   DP_BASE Tutorial   HWID's Tool     Read BEFORE you post    UserBars!
http://driverpacks.net/userbar/admin-1.png
The DriverPacks, the DP_Base program, and Support Forum are FREE!.

Re: Repack DriverPack's

Let's break this script into 3 parts and discuss them separately.
1. Tunning for max compression. Disabling hyperthreading support(-mmt=off) improves compression while sacrificing compression speed because using single stream allows better compression when using two independent streams.
2. Improved speed of indexing driverpacks(DriverPack Solution, Drivers Installer Assistant).
3. Suggests using a smaller dictionary.

You hope you aren't opposing the first two changes and we can discuss the third one.
The suggestion to use 32MB dictionary was based on measurement of compression ratios. Each doubling the dictionary size improves compression rate by only 1%. Changing the dictionary size doesn't affect performance providing a PC has enough memory but has a huge performance hit when there is not enough memory. Sometimes Windows fails to allocate enough memory and the operation fails completely.

OverFlow wrote:

Since the max dictionary size on the version of 7zip we use is 256 - I have a hard time beliving that there are really that many machines that have less than 256 meg of free ram... If someone is running on a machine with less than 256 meg of free ram then shame on them. We should not penalize everyone else with a viable system to cater to a few people whos machines aren't even valid for windows xp anyway. (less than 256 meg of free ram is not valid in my opinion). IE the base system ram should be at least 384 meg, 512 prefered.

It is my belief that this recomendation will only help machines that are not viable for XP anyway. 
A machine with less than 512 meg of RAM is going to be painfully slow, period.
I belive that xp boxes with less than 512 meg of ram are a very small percentage of the total machines running XP.
if a machine only has 256 meg of ram (total)... then load win98se on it and call it a day .

At workplace we still have such old PCs which are alive and kicking. Obsolete DDR and SDRAM are difficult to find and they are too expensive. Downgrading to Win98 isn't an option because it doesn't support Microsoft Office 2003, many thumb drives and printers.


OverFlow wrote:

If a machine has 512 meg of ram or more on it (and the vast majority do) it is my belief this will have little or no effect at all... am i wrong?

Smaller dictionary definitely improves speed because Windows doesn't have to swap data into pagafile so often.

P.S. Suggestion to use 32MB dictionary is only a recommendation. The main reason I made this script is improving indexing times.

Last edited by BadPointer (2010-09-04 19:41:08)

Re: Repack DriverPack's

We are always open to ideas... we don't always agree big_smile

Improved speed of indexing driverpacks(DriverPack Solution, Drivers Installer Assistant).
seems reasonable, and worth doing. this would also help other apps like WarmSnows and our own SAD platform.

how long it takes us to compress them is of little issue. We only have to do it once big_smile.

i find it hard to belive that increasing the dictionary size to 256 only increases the compression 3% (32, 64, 128, 256, = 3%?)

main questions not addressed...


How much does this increas the size of the (main) DriverPacks?
How much time is saved; @256mb ram, @384mb ram, @512mb ram, >512mb ram?

DP BartPE Tutorial   DP_BASE Tutorial   HWID's Tool     Read BEFORE you post    UserBars!
http://driverpacks.net/userbar/admin-1.png
The DriverPacks, the DP_Base program, and Support Forum are FREE!.

Re: Repack DriverPack's

OverFlow wrote:

How much does this increas the size of the (main) DriverPacks?

I wish I knew it. My home PC(AMD x64 3800+, 2.4GHz, 1GB DDR2) and work PC(Intel Pentium 4, 3.0GHz, 2GB DDR2) are too old. It takes ages to pack driverpacks and I don't think I have enough RAM to even pack them.

However I have data with the dictionary size of 32MB and 64MB.
http://www.mediafire.com/i/?q5baw5dm8833ia8

OverFlow wrote:

How much time is saved; @256mb ram, @384mb ram, @512mb ram, >512mb ram?

I can measure it the next time I get my hands on such old PCs. Unpacking these driverpacks often fails completely with Windows saying that it was unable to allocate enough virtual RAM before time runs out. I worked around this issue by unpacking driverpacks on my PC and putting an unpacked driver directory on a thumb drive.

I think someone has to run this script with different dictionary sizes and publish the table like the one I did. This way we will know which dictionary size is optimal.

Last edited by BadPointer (2010-09-05 00:45:57)

Re: Repack DriverPack's

If we make concessions that would make a better user experience for some users while minimizing the impact to the majority, I'm all for it.
But I'm hesitant to blindly make those changes without proof.
I'll do a test on my netbook.  The Atom processor is far from speedy.  I'll also do another test on Tuesday on an old machine we still have at work.
I'll report back later.

BTW, the dictionary size is only one part of the equation.  You also have the word-size and solid-block-size.  When I pack my DriverPacks I routinely use 256mb/256mb/4gb settings which seemed optimal for most users.

OverFlow wrote:

how long it takes us to compress them is of little issue. We only have to do it once big_smile.

Well NOT making use of all the cores of my quad-core is just silly in my opinion! wink

Read BEFORE you post.  HWID tool   DriverPacks Tutorial   DONATE!
http://driverpacks.net/userbar/admin-1.png
Not all heroes wear capes, some wear Kevlar!

Re: Repack DriverPack's

mr_smartepants wrote:

BTW, the dictionary size is only one part of the equation.  You also have the word-size and solid-block-size.  When I pack my DriverPacks I routinely use 256mb/256mb/4gb settings which seemed optimal for most users.

Other settings can turned for max compression and the worst thing that can happen is longer packing or unpacking times(unpacking from the middle of the archive).
The dictionary size is the only one option that may prevent unpacking altogether. This is why I made this script so that the user could pick the dictionary size and none of the other options.

mr_smartepants wrote:

Well NOT making use of all the cores of my quad-core is just silly in my opinion!

You can try packing 2-3 driverpacks concurrently. This shouldn't affect compression and you still can make use of your cores.

Re: Repack DriverPack's

Test setup:
test1: Dictionary size: 256MB, Word size: 256, Solid block size: 4GB
test2: 32MB script

Folder sizes (current 10 nightly main DriverPacks)
test1: 602 MB
test2: 618 MB (16MB/2.657% increase in size)

Time to unpack
Acer netbook (Atom CPU, 2GB RAM)
test1: 7:09
test2: 6:55 (3.263% decrease)

Gateway notebook (Pentium M 1.4ghz CPU, 1GB RAM)
test1: 6:25
test2: 5:44 (10.6% decrease)

Toshiba notebook (Celeron 2.8GHz CPU, 704MB RAM)
test1: 7:56
test2: 8:36 (8.4% INCREASE)

Toshiba notebook (Celeron 2.8GHz CPU, 192MB RAM)
test1: 1:05:07   (1 hour, 5 min, 7 sec!)
test2: 9:54 (557.75% decrease)

More tests to follow...

Read BEFORE you post.  HWID tool   DriverPacks Tutorial   DONATE!
http://driverpacks.net/userbar/admin-1.png
Not all heroes wear capes, some wear Kevlar!

Re: Repack DriverPack's

I suggest you also try repacking main DriverPacks with 256MB script. This might be different from the test1.

Re: Repack DriverPack's

Well the one thing we did not discuss was solid archive...

with a solid archive the entire pack must be extracted to get one file...

if we do not do this then apps which wish to only extract a single file will benefit greatly...

What is the negative impact of that on archive size
and do any of these apps do this (i belive they do)

DP BartPE Tutorial   DP_BASE Tutorial   HWID's Tool     Read BEFORE you post    UserBars!
http://driverpacks.net/userbar/admin-1.png
The DriverPacks, the DP_Base program, and Support Forum are FREE!.

Re: Repack DriverPack's

DriverPack Solution and Drivers Installer Assistant are capable of extracting several folders in a single pass. That means that in the worst case scenario they will have to process whole driverpack one time.

I expect splitting archive into multiple solid blocks will have similar effect on the compression rates as packing with multitreading support. This would improve extracting times when the files to be extracted are in the middle-ending of the archive. Personally I could care less about these times(it isn't as bad when PC lacks RAM) but if doing so isn't affecting compression too much, I would go for it.

BTW SamLab repacked his custom driverpacks.
He originally packed them with the 100MB dictionary.
After repacking with the 32MB dictionary most driverpacks decreased in size but some big driverpacks almost doubled in size. After repacking with the 64MB dictionary, they got smaller then they were originally packed.

The point is you don't have to use the 32MB dictionary if it greatly affects compression. Anyway it looks like the 256MB dictionary is overkill and 64MB should suffice for the big driverpacks. Also it is safe to assume that a PC with Windows Vista/7 will have more RAM available than a PC with XP.

Last edited by BadPointer (2010-09-06 20:31:29)

Re: Repack DriverPack's

I've been updating my tests a few posts above.
This test has me sold.  I'll be building all my DriverPacks using either the 32 or 64MB scripts.
Toshiba notebook (Celeron 2.8GHz CPU, 192MB RAM)
test1: 1:05:07   (1 hour, 5 min, 7 sec!)
test2: 9:54 (557.75% decrease)

Read BEFORE you post.  HWID tool   DriverPacks Tutorial   DONATE!
http://driverpacks.net/userbar/admin-1.png
Not all heroes wear capes, some wear Kevlar!

Re: Repack DriverPack's

It is possible to improve compression and speed even more by switching the compression method from LZMA to LZMA2. Just replace all "LZMA" with "LZMA2" in the script and don't forget to update the 7za.exe to v9.17.

DP_BASE uses 7-Zip v4.42 that doesn't support the LZMA2 method so you have to update it to 7-Zip v9.17, otherwise DP_BASE won't be able to work with new 3rd party driverpacks that are already coming out.

Re: Repack DriverPack's

Well whenever the new version of 7zip comes out of beta then maybe we'll consider it. tongue
But then we may have to rewrite our un7zip.exe utility.

Read BEFORE you post.  HWID tool   DriverPacks Tutorial   DONATE!
http://driverpacks.net/userbar/admin-1.png
Not all heroes wear capes, some wear Kevlar!

Re: Repack DriverPack's

Thanks for all the feedback, this is really good stuff guys!

DP BartPE Tutorial   DP_BASE Tutorial   HWID's Tool     Read BEFORE you post    UserBars!
http://driverpacks.net/userbar/admin-1.png
The DriverPacks, the DP_Base program, and Support Forum are FREE!.

Re: Repack DriverPack's

Just letting you know: 7-Zip 9.20 is out of beta.

Re: Repack DriverPack's

Yup, we know all about it.  We're busy updating DriverPacks BASE...please be patient.

Read BEFORE you post.  HWID tool   DriverPacks Tutorial   DONATE!
http://driverpacks.net/userbar/admin-1.png
Not all heroes wear capes, some wear Kevlar!

Re: Repack DriverPack's

That's ok. I posted it in case you weren't aware of it.

Re: Repack DriverPack's

bump

DP BartPE Tutorial   DP_BASE Tutorial   HWID's Tool     Read BEFORE you post    UserBars!
http://driverpacks.net/userbar/admin-1.png
The DriverPacks, the DP_Base program, and Support Forum are FREE!.

Re: Repack DriverPack's

Hi,

DriverPacks.net is really an invaluable service that has become a staple of the computer shop where I volunteer smile Partly to help out the community, here's a little something I made for myself...

As said before, decreasing word size can lead to significantly larger packages when the original package size is already large. So to further improve DriverPack Solution's performance on machines with less RAM, I rewrote much of the batch logic of repack_single.bat to re-repack until it finds the best compromise of final size vs smaller dictionary size (also changing it to use LZMA2, which should always be available), as well as various added sanity checks and error/debug stuff. You can find my rewrite below:

@echo off
setlocal EnableDelayedExpansion
REM -----------------------------------
REM http://forum.driverpacks.net/viewtopic.php?id=4789
REM
REM User Configuration
REM
REM _toobig is max size difference in BYTES (1MB=1048576B)
REM _step is multiplicative stepping (which cannot be a fraction)
REM _maxdict is duh
REM
REM Note: Always automatically creates logfile of batchfilename.log
REM -----------------------------------

set _toobig=104857600
set _step=2
set _maxdict=512

set 7zexe=

REM -----------------------------------
REM END User Configuration
REM -----------------------------------
REM -----------------------------------

IF NOT EXIST !7zexe! (                                
  IF EXIST "%ProgramFiles%\7-zip\7z.exe" (
     set 7zexe="%ProgramFiles%\7-zip\7z.exe"
  ) ELSE ( 
     IF EXIST "%ProgramFiles(x86)%\7-zip\7z.exe" (
       set 7zexe="%ProgramFiles(x86)%\7-zip\7z.exe"
       call :echolog "WARNING: Using 32-bit 7-zip on 64-bit Windows"
     ) ELSE (
       call :echolog "WARNING: 7-zip not installed and no alternative provided, assuming we have 7za.exe"
       set 7zexe=7za.exe
     )
   )
)

set _bestdictsize=0
set _bestdict=
set _starttime=%TIME%

set _dict=%2
set _file=
IF "!_dict!"=="" call "%0" "%~1" 32
set _file=%~1
IF "%_file%"=="" GOTO :usage
IF NOT EXIST "%_file%" GOTO :error
set _initialsize=%~z1
call :echolog "Start: %DATE%"
call :echolog "File: [%_file%]"
!7zexe! | findstr /I /C:"7-zip" >> "%~n0.log"
GOTO :continue

:usage
echo Usage: FILE_TO_REPACK [starting dictionary]
echo.
echo To repack multiple files, run 'repack_all_XX.bat' instead.
echo.
pause
GOTO :done

:error
call :echolog "ERROR: Cannot access [%_file%] or it does not exist."
pause
GOTO :done

:decompresserror
call :echolog "ERROR: Decompression of [%_file%] failed. Aborting."
rd "%_file%_temp" /s /q 2>Nul
IF EXIST "out\%_file%.!_bestdict!" move /y "out\%_file%.!_bestdict!" "out\%_file%"
echo y | del /q "out\%_file%.*" 2>Nul
cd /d "%~dp0"
pause
GOTO :done

:compresserror
call :echolog "ERROR: Compression of [%_file%] failed. Aborting."
rd "%_file%_temp" /s /q 2>Nul
IF EXIST "out\%_file%.!_bestdict!" move /y "out\%_file%.!_bestdict!" "out\%_file%"
echo y | del /q "out\%_file%.*" 2>Nul
cd /d "%~dp0"
pause
GOTO :done

:echolog
echo %~1
echo %~1 >> "%~n0.log"
GOTO :eof

:getsize
set _size=%~z1
GOTO :eof

:continue
rd "%_file%_temp" /s /q 2> Nul
md "%_file%_temp"
compact /C "%_file%_temp"
!7zexe! x "%_file%" -o"%_file%_temp" -r || GOTO :decompresserror
echo y | del "out\%_file%" 2>Nul
cd "%_file%_temp"
( !7zexe! a "%~dp0\out\%_file%" -mmt=off -m0=BCJ2 -m1=LZMA2:d!_dict!m:fb273 -m2=LZMA2:d512k -m3=LZMA2:d512k -mb0:1 -mb0s1:2 -mb0s2:3 *.ini -ir^^!*.inf && !7zexe! a "%~dp0\out\%_file%" -mmt=off -m0=BCJ2 -m1=LZMA2:d!_dict!m:fb273 -m2=LZMA2:d512k -m3=LZMA2:d512k -mb0:1 -mb0s1:2 -mb0s2:3 -xr^^!*.inf -x^^!*.ini ) || GOTO :compresserror
call :getsize "%~dp0\out\%_file%"
cd ..
IF !_bestdictsize! EQU 0 ( 
   set _bestdictsize=!_size!
   set _adjustedsize=!_size!
) ELSE (
   set /a "_adjustedsize=!_bestdictsize! + (%_toobig% / 2)"
)
IF !_size! LEQ !_adjustedsize! (
   echo y | del "%~dp0\out\%_file%.!_dict!" 2>Nul
   set _bestdictsize=!_size!
   set _bestdict=!_dict!
   move /Y "%~dp0\out\%_file%" "%~dp0\out\%_file%.!_dict!"
)
set /a "_size=!_size! - !_initialsize!"
IF !_size! GTR %_toobig% (
   set /a "_dict=!_dict! * %_step%"
   call :echolog "INFO: Resulting file is over size limit, so trying with bigger dictionary !_dict!."
   IF !_dict! GTR %_maxdict% (
      IF !_bestdict! EQU %2 IF !_dict! EQU %2 (
         set /a "_size=%_toobig% / 1048576"
         call :echolog "WARNING: All dictionaries of %2 and greater resulted in an increase of size by !_size!MB or more, so giving up."
      ) ELSE (
         call :echolog "INFO: Dictionary is already at max %_maxdict%, so going back to !_bestdict! size."
      )
   ) ELSE ( 
      GOTO :continue
   )
)

move /Y "%~dp0\out\%_file%.!_bestdict!" "%~dp0\out\%_file%"
rd "%_file%_temp" /s /q 2>Nul
call :echolog "Done working on [%_file%]"
call :echolog "Repacked using dictionary size of: !_bestdict!"
set /a "_size=(!_bestdictsize! - !_initialsize!) / 1048576"
call :echolog "Size changed by !_size!MB"
set _minutediff=
FOR /F "tokens=1,2 delims=:." %%G IN ("%TIME%") DO (
   set /a "_minutediff=(%%G * 60) + %%H"
)
set _hourmin=
FOR /F "tokens=1,2 delims=:." %%G IN ("!_starttime!") DO (
    set /a "_minutediff-=(%%G * 60) + %%H"
)
set /a "_hourdiff=!_minutediff! / 60"
set /a "_minutediff=!_minutediff! %% 60"
call :echolog "Total repacking time: !_hourdiff! hours and !_minutediff! minutes"
call :echolog -------------------------------------------------------------
cd /d "%~dp0"

:done

More details on how it works:
- The batch file increases the dictionary size (default: doubles it) up to a certain limit (default: 512) when the size difference on the repack is too much (default 100MB or more).

- If it reaches the maximum dictionary size without decreasing the size enough, then it'll use either the initial dictionary size or the dictionary size that created an archive of less than HALF the limit (default would be 50MB).

- The temporary directory it uses is named per-file, so if you have a nice SSD and/or RAID0 like I do, you can run multiple instances at once by manually calling repack_single.bat.

So most archives will pack fine with 32, but some will go up to 64 or 128 (mainly the video card driver pack), also depending on what version of 7z you are using (it will use your installed version of 7-zip by default, otherwise 7za.exe provided earlier). If you don't care as much about saving space as you do dictionary size, just edit the batch file to decrease either limit.

For those interested, this version can quite easily be expanded upon to use all sorts of different compression parameters, rather than just playing with the dictionary size. So we could use even better parameters for decompression on crappy machines, then compare the difference. For instance, LZMA may be more preferable on crappy machines than LZMA2 (I'm not sure), so the batch file could be changed to try LZMA first.

Edit: Small typo in batch file made 2nd parameter ignored. Also fixed an endless loop that could happen if you don't give a 2nd parameter.

Last edited by generalmx (2014-04-13 13:12:52)

Re: Repack DriverPack's

that is pretty cool big_smile

Thank you, and welcome to DriverPacks.net

DP BartPE Tutorial   DP_BASE Tutorial   HWID's Tool     Read BEFORE you post    UserBars!
http://driverpacks.net/userbar/admin-1.png
The DriverPacks, the DP_Base program, and Support Forum are FREE!.

Re: Repack DriverPack's

Remember, compressed archive size is only a small part of it.  Support for systems with restricted RAM is paramount.  DriverPacks would be useless on systems with limited RAM (256mb or less) if the dictionary size used exceeded the available RAM.
Read my test results from post#9 above and see for yourself.
http://forum.driverpacks.net/viewtopic. … 311#p40311

Read BEFORE you post.  HWID tool   DriverPacks Tutorial   DONATE!
http://driverpacks.net/userbar/admin-1.png
Not all heroes wear capes, some wear Kevlar!

Re: Repack DriverPack's

wonder what it would be with 128MB RAM?

rofl

DP BartPE Tutorial   DP_BASE Tutorial   HWID's Tool     Read BEFORE you post    UserBars!
http://driverpacks.net/userbar/admin-1.png
The DriverPacks, the DP_Base program, and Support Forum are FREE!.

Re: Repack DriverPack's

I have updated my original repack tool.
http://www.mediafire.com/download/yq9yu … repack2.7z

Changes:
1. Cat files are now placed in the same block as inf files.
2. Added bat files for LZMA2.