Hey, I'm back, and I've recently been looking into different compression tools for my own purposes. One such tool I've come across is "Precomp", which is preprocessor that can make a bit-to-bit identical recompression of compressed streams *inside* of files, which I think would get Driverpacks considerable savings as they're full of already compressed data. I think the most memory-intensive part of precomp is the compression, not the decompression. Use of precomp would require explicit support in the program, as it creates a "PCF" file. How you do it is put all desired files in a NON-compressed, non-solid archive, put the archive through precomp to make a PCF file, then make the archive as normal. I'll be testing it out with the Driverpacks to see if I can keep it at least below 256MB of memory required, if not less.
2 2014-04-13 10:18:32
Re: Repack DriverPack's (30 replies, posted in Software)
Hi,
DriverPacks.net is really an invaluable service that has become a staple of the computer shop where I volunteer 
Partly to help out the community, here's a little something I made for myself...
As said before, decreasing word size can lead to significantly larger packages when the original package size is already large. So to further improve DriverPack Solution's performance on machines with less RAM, I rewrote much of the batch logic of repack_single.bat to re-repack until it finds the best compromise of final size vs smaller dictionary size (also changing it to use LZMA2, which should always be available), as well as various added sanity checks and error/debug stuff. You can find my rewrite below:
@echo off
setlocal EnableDelayedExpansion
REM -----------------------------------
REM http://forum.driverpacks.net/viewtopic.php?id=4789
REM
REM User Configuration
REM
REM _toobig is max size difference in BYTES (1MB=1048576B)
REM _step is multiplicative stepping (which cannot be a fraction)
REM _maxdict is duh
REM
REM Note: Always automatically creates logfile of batchfilename.log
REM -----------------------------------
set _toobig=104857600
set _step=2
set _maxdict=512
set 7zexe=
REM -----------------------------------
REM END User Configuration
REM -----------------------------------
REM -----------------------------------
IF NOT EXIST !7zexe! (
IF EXIST "%ProgramFiles%\7-zip\7z.exe" (
set 7zexe="%ProgramFiles%\7-zip\7z.exe"
) ELSE (
IF EXIST "%ProgramFiles(x86)%\7-zip\7z.exe" (
set 7zexe="%ProgramFiles(x86)%\7-zip\7z.exe"
call :echolog "WARNING: Using 32-bit 7-zip on 64-bit Windows"
) ELSE (
call :echolog "WARNING: 7-zip not installed and no alternative provided, assuming we have 7za.exe"
set 7zexe=7za.exe
)
)
)
set _bestdictsize=0
set _bestdict=
set _starttime=%TIME%
set _dict=%2
set _file=
IF "!_dict!"=="" call "%0" "%~1" 32
set _file=%~1
IF "%_file%"=="" GOTO :usage
IF NOT EXIST "%_file%" GOTO :error
set _initialsize=%~z1
call :echolog "Start: %DATE%"
call :echolog "File: [%_file%]"
!7zexe! | findstr /I /C:"7-zip" >> "%~n0.log"
GOTO :continue
:usage
echo Usage: FILE_TO_REPACK [starting dictionary]
echo.
echo To repack multiple files, run 'repack_all_XX.bat' instead.
echo.
pause
GOTO :done
:error
call :echolog "ERROR: Cannot access [%_file%] or it does not exist."
pause
GOTO :done
:decompresserror
call :echolog "ERROR: Decompression of [%_file%] failed. Aborting."
rd "%_file%_temp" /s /q 2>Nul
IF EXIST "out\%_file%.!_bestdict!" move /y "out\%_file%.!_bestdict!" "out\%_file%"
echo y | del /q "out\%_file%.*" 2>Nul
cd /d "%~dp0"
pause
GOTO :done
:compresserror
call :echolog "ERROR: Compression of [%_file%] failed. Aborting."
rd "%_file%_temp" /s /q 2>Nul
IF EXIST "out\%_file%.!_bestdict!" move /y "out\%_file%.!_bestdict!" "out\%_file%"
echo y | del /q "out\%_file%.*" 2>Nul
cd /d "%~dp0"
pause
GOTO :done
:echolog
echo %~1
echo %~1 >> "%~n0.log"
GOTO :eof
:getsize
set _size=%~z1
GOTO :eof
:continue
rd "%_file%_temp" /s /q 2> Nul
md "%_file%_temp"
compact /C "%_file%_temp"
!7zexe! x "%_file%" -o"%_file%_temp" -r || GOTO :decompresserror
echo y | del "out\%_file%" 2>Nul
cd "%_file%_temp"
( !7zexe! a "%~dp0\out\%_file%" -mmt=off -m0=BCJ2 -m1=LZMA2:d!_dict!m:fb273 -m2=LZMA2:d512k -m3=LZMA2:d512k -mb0:1 -mb0s1:2 -mb0s2:3 *.ini -ir^^!*.inf && !7zexe! a "%~dp0\out\%_file%" -mmt=off -m0=BCJ2 -m1=LZMA2:d!_dict!m:fb273 -m2=LZMA2:d512k -m3=LZMA2:d512k -mb0:1 -mb0s1:2 -mb0s2:3 -xr^^!*.inf -x^^!*.ini ) || GOTO :compresserror
call :getsize "%~dp0\out\%_file%"
cd ..
IF !_bestdictsize! EQU 0 (
set _bestdictsize=!_size!
set _adjustedsize=!_size!
) ELSE (
set /a "_adjustedsize=!_bestdictsize! + (%_toobig% / 2)"
)
IF !_size! LEQ !_adjustedsize! (
echo y | del "%~dp0\out\%_file%.!_dict!" 2>Nul
set _bestdictsize=!_size!
set _bestdict=!_dict!
move /Y "%~dp0\out\%_file%" "%~dp0\out\%_file%.!_dict!"
)
set /a "_size=!_size! - !_initialsize!"
IF !_size! GTR %_toobig% (
set /a "_dict=!_dict! * %_step%"
call :echolog "INFO: Resulting file is over size limit, so trying with bigger dictionary !_dict!."
IF !_dict! GTR %_maxdict% (
IF !_bestdict! EQU %2 IF !_dict! EQU %2 (
set /a "_size=%_toobig% / 1048576"
call :echolog "WARNING: All dictionaries of %2 and greater resulted in an increase of size by !_size!MB or more, so giving up."
) ELSE (
call :echolog "INFO: Dictionary is already at max %_maxdict%, so going back to !_bestdict! size."
)
) ELSE (
GOTO :continue
)
)
move /Y "%~dp0\out\%_file%.!_bestdict!" "%~dp0\out\%_file%"
rd "%_file%_temp" /s /q 2>Nul
call :echolog "Done working on [%_file%]"
call :echolog "Repacked using dictionary size of: !_bestdict!"
set /a "_size=(!_bestdictsize! - !_initialsize!) / 1048576"
call :echolog "Size changed by !_size!MB"
set _minutediff=
FOR /F "tokens=1,2 delims=:." %%G IN ("%TIME%") DO (
set /a "_minutediff=(%%G * 60) + %%H"
)
set _hourmin=
FOR /F "tokens=1,2 delims=:." %%G IN ("!_starttime!") DO (
set /a "_minutediff-=(%%G * 60) + %%H"
)
set /a "_hourdiff=!_minutediff! / 60"
set /a "_minutediff=!_minutediff! %% 60"
call :echolog "Total repacking time: !_hourdiff! hours and !_minutediff! minutes"
call :echolog -------------------------------------------------------------
cd /d "%~dp0"
:doneMore details on how it works:
- The batch file increases the dictionary size (default: doubles it) up to a certain limit (default: 512) when the size difference on the repack is too much (default 100MB or more).
- If it reaches the maximum dictionary size without decreasing the size enough, then it'll use either the initial dictionary size or the dictionary size that created an archive of less than HALF the limit (default would be 50MB).
- The temporary directory it uses is named per-file, so if you have a nice SSD and/or RAID0 like I do, you can run multiple instances at once by manually calling repack_single.bat.
So most archives will pack fine with 32, but some will go up to 64 or 128 (mainly the video card driver pack), also depending on what version of 7z you are using (it will use your installed version of 7-zip by default, otherwise 7za.exe provided earlier). If you don't care as much about saving space as you do dictionary size, just edit the batch file to decrease either limit.
For those interested, this version can quite easily be expanded upon to use all sorts of different compression parameters, rather than just playing with the dictionary size. So we could use even better parameters for decompression on crappy machines, then compare the difference. For instance, LZMA may be more preferable on crappy machines than LZMA2 (I'm not sure), so the batch file could be changed to try LZMA first.
Edit: Small typo in batch file made 2nd parameter ignored. Also fixed an endless loop that could happen if you don't give a 2nd parameter.