Create random file
N4p0l&0n
I wish create a large random file (for testing copy in powershell
between servers on WAN) i'm using a get-random function but it's very
slow. A file can have a size of 1-5 Gb
Do you have ideas for accelerate this program?
Thanks and happy new year.
My tiny program
function GenChaine([int]$n)
# G�n�ration d'une chaine al�atoire de n caract�res
{
$caracteres = "a|z|e|r|t|y|u|i|o|p|q|s|d|f|g|h|j|k|l|m|w|x|c|v|b|n"
$caracteres += "A|Z|E|R|T|Y|U|I|O|P|Q|S|D|F|G|H|J|K|L|M|W|X|C|V|B|N"
$caracteres += "0|1|2|3|4|5|6|7|8|9"
$chaine = ""
$liste = $caracteres.split("|")
for ($i=1;$i -le $n;$i++)
{
#$chaine += ( $liste | Get-Random)
$chaine += "A"
}
return $chaine
}
$blocs = (1024*1024*3)
if ( Test-Path -Path $Nomfichier ) {Remove-Item -Path $NomFichier
-Force}
for ($i=1; $i -le $blocs; $i++)
{
$a = GenChaine(1022)
Add-Content -Path $NomFichier -Value $a
}
Marco Shaw [MVP]
Do the contents have to be randomized also or can it be anything? Are you
using PowerShell v1 or v2?
Marco
"N4p0l&0n" <n...@po.auz> wrote in message
news:OsbAYaLj...@TK2MSFTNGP02.phx.gbl...
N4p0l&0n
Le 03/01/2010 22:00, Marco Shaw [MVP] a �crit :
Marco Shaw [MVP]
From Bruce Payette, author of PowerShell in Action
(www.manning.com/payette2, 2nd edition coming soon):
"Write all at once (allocates 1mb buffer) ...
set-content -value (new-object byte[] 1mb) -encoding byte out.dat
or 1kb block at a time
1..1024 | % { ,(new-object byte[] 1kb) } | set-content -encoding byte
out.dat
Note the ,(...) - this is so the blocks are written as a unit instead of 1
byte at a time..."
"N4p0l&0n" <n...@po.auz> wrote in message
news:#cfj3fMj...@TK2MSFTNGP04.phx.gbl...
Joe Morris
> Le 03/01/2010 22:00, Marco Shaw [MVP] a �crit :
>>> I wish create a large random file (for testing copy in powershell
>>> between servers on WAN) i'm using a get-random function but it's very
>>> slow. A file can have a size of 1-5 Gb
[snip code fragment]
>> Do the contents have to be randomized also or can it be anything? Are
>> you using PowerShell v1 or v2?
> Randomized too and i'm using pws V2. Idealy files would be binary
My immediate reaction is that POSH (or any other scripting language) isn't a
good choice for generating huge files like that. As you've found the
overhead is quite non-trivial; that's not an issue for something like
generating a temporary folder name or selecting ten userids to receive a
user satisfaction survey, but for multiple gigabyte-sized files it's a
showstopper.
There are commercial data generators you could buy (no, I can't recommend
one since the last time I needed one was on IBM mainframes 20+ years ago --
anyone else here remember IEBDG?) but for what sounds like a one-time use
it shouldn't take too long to write one in the compiled language of your
choice...or to find one via Google.
Joe Morris
Martin Zugec
fsutil file create?
Martin
"Joe Morris" <j.c.m...@verizon.net> wrote in message
news:hhrid...@news4.newsguy.com...
N4p0l&0n
Great ! Thanks for the syntax.
Le 04/01/2010 02:26, Marco Shaw [MVP] a �crit :
N4p0l&0n
content. Why ?
It's for testing appliance compression between sites (appliances
Riverbade, expand, ipanema, ...) ...
Le 04/01/2010 22:12, Martin Zugec a �crit :
Robert Robelo
It creates a new file or overwrites an existing file, and writes up to 5120 MB of random chars -your set of chars- in one huge string through a stream writer.
If you want to write many lines instead of one huge line, use IO.StreamRiter's WriteLine Method instead, but be aware that it'll affect the function's perfomance. Showing the progress will also affect performance.
In my system it writes a 1GB in ~1.2 minutes and a 3GB file in ~2.78 mimutes without hogging resources.
The function's MegaByte parameter is limited to 5KB, i.e. up to 5GB files.
# create a 1MB file (default size)
New-BigRandomFile File1.rnd
# create a 75MB file and show its progress
New-BigRandomFile File2.rnd 75 -show
# create a 1GB file
New-BigRandomFile File3.rnd 1kb
# - - - - - - - - - - - - - #
Function New-BigRandomFile {
# Requires -Version 2.0
param(
[Parameter(Mandatory = $true, Position = 0)]
[String]$File,
[Parameter(Mandatory = $false, Position = 1)]
[ValidateRange(1, 5120)]
[UInt16]$MegaByte = 1,
[Switch]$ShowProgress
)
# get/create the file's full path
$path = if (Test-Path $File -PathType Leaf) {
Resolve-Path $File
} else {
Join-Path $pwd $File
}
$total = 1mb * $MegaByte
$strings = $bytes = 0
# create the stream writer
$sw = New-Object IO.streamWriter $path
# get a 64 element Char[]; I added the - and _ to have 64 chars
[char[]]$chars =
'azertyuiopqsdfghjklmwxcvbnAZERTYUIOPQSDFGHJKLMWXCVBN0123456789-_'
1..$MegaByte | ForEach-Object {
# get 1MB of chars from 4 256KB strings
1..4 | ForEach-Object {
# randomize all chars and...
$rndChars = $chars | Get-Random -Count $chars.Count
# ...join them in a string
$str = -join $rndChars
# repeat random string 4096 times to get a 256KB string
$str_ = $str * 4kb
# write 256KB string to file
$sw.Write($str_)
# show progress
if ($ShowProgress) {
$strings++
$bytes += $str_.Length
Write-Progress -Activity "Writing String #$strings" `
-Status "$bytes Bytes written" `
-PercentComplete ($bytes / $total * 100)
}
# release resources by clearing string variables
Clear-Variable str, str_
}
}
$sw.Close()
$sw.Dispose()
# release resources through garbage collection
[GC]::Collect()
}
--
Robert
N4p0l&0n
Le 05/01/2010 22:38, Robert Robelo a �crit :