Go is two times slower than python code?
Alex Dong
basically reads a tweeter message from a csv file, tokenize it and put
the word->id into a map called lexicon. It turned out that same logic
took 22 seconds for python 2.6 to finish whereas 57 seconds for
golang!
Wondering are there anything I've done wrong?
Here is the python code:
$ cat loader.py
lexicon = {}
pnct_ptn = re.compile(r'([\.,\\/\'"\?!=_\)\(\]\[\{\}:;]+|http://[^ ]
+)')
def tokenize(s):
s = pnct_ptn.sub(' ', s)
return [t for t in s.split() if len(t)>=3]
for line in open("result.csv").readlines():
parts = line.split(',', 3)
if len(parts) != 4: continue
msg = parts[3]
s = msg.decode('utf8','ignore').lower()
for word in tokenize(s):
if not lexicon.has_key(word):
unique_words += 1
lexicon[word] = unique_words
Here is the go code:
$ cat loader.go
package main
import (
"bufio"
"os"
"regexp"
"strings"
)
var (
pr, _ = regexp.Compile(`(http://[^ ]+|['".\\,=()*:;?!/]|-)`) //
pattern for removal
)
func tokenize(s string) []string {
ms := pr.ReplaceAllString(strings.ToLower(s), " ")
return strings.Split(ms, " ", 0)
}
func main() {
lex := make(map[string] int) // lexicon
dic := make(map[int] string) // lookup
tw := 0 // total
words
ps := false // present
r, _ := os.Open("result.csv", os.O_RDONLY, 0444)
defer r.Close()
in := bufio.NewReader(r)
for i := 0; i >= 0; i++ {
line, err := in.ReadString('\n')
if err != nil {
break
}
parts := strings.Split(line, ",", 4)
if len(parts) != 4 {
continue
}
ts := tokenize(parts[3])
for d := 0; d < len(ts); d++ {
w := ts[d]
if len(w) < 3 {
continue
}
_, ps = lex[w]
if ps == false {
lex[w] = tw
dic[tw] = w
tw ++
}
}
}
}
Cheers,
Alex
Jessta
> I'm writing a simple 'word picker' using golang and python. It
> basically reads a tweeter message from a csv file, tokenize it and put
> the word->id into a map called lexicon. It turned out that same logic
> took 22 seconds for python 2.6 to finish whereas 57 seconds for
> golang!
This has come up a few times and explanation is that
Go's garbage collector is currently slow and the regex library is
currently slow.
Go isn't stable enough to spend lots of time on optimisations because
things will likely change. Premature optimisation etc etc.
The optimisations can be done later.
- jessta
=====================
http://jessta.id.au
Rob 'Commander' Pike
I hope to see a much more efficient regular expression package for Go before long. Wrapping PCRE is another option but one I shy away from because that "(usually)" hides a scary story. See http://swtch.com/~rsc/regexp/regexp1.html for some of it. The Go code is an inefficient implementation of the computationally more dependable algorithm. An efficient implementation is a lot of work but at least the inefficient implementation won't ever blow up.
-rob
Kevin Ballard
issue as mentioned in your link? I'm not finding any relevant
information after a quick google search.
-Kevin Ballard
--
Kevin Ballard
http://kevin.sb.org
kbal...@gmail.com
Rob 'Commander' Pike
is not a regular language. The existence of that usually enough to
tell.
-rob
Kevin Ballard
be a regular expression, but patterns that do not use backreferencing
can be implemented using Thompson NFA instead of the exponential
algorithm (e.g. all regular expressions need to have an exponential
implementation in order to support backreferencing, but can specialize
for non-backreferencing patterns). However, I am unable to find out if
Oniguruma does this. At this time I'm going to assume it doesn't, but
it would still be nice to know.
-Kevin Ballard
Russ Cox
contrary to popular terminology. All regular expressions
(in the actual definition of the word) describe patterns
that can be matched in a single linear-time pass over the
input text. This is what package regexp implements.
Russ
Kevin Ballard
regular expressions. Are there any plans on implementing
backreferences in Go's Regexp package (e.g. using two implementations,
one for patterns w/backreferences and one for patterns w/o
backreferences)?
-Kevin Ballard
--
Rob 'Commander' Pike
On Mar 9, 2010, at 12:49 PM, Kevin Ballard wrote:
> Ah, you're talking about CS regular expressions, not "practical"
> regular expressions. Are there any plans on implementing
> backreferences in Go's Regexp package (e.g. using two implementations,
> one for patterns w/backreferences and one for patterns w/o
> backreferences)?
No. We hope to have a fast matcher for true regular expressions,
though.
-rob
Giuseppe Altea
Vivek Khurana
On Mar 10, 6:01 am, "Rob 'Commander' Pike" <r...@google.com> wrote:
>
> No. We hope to have a fast matcher for true regular expressions,
> though.
So should we expect a Thompson NFA based regex matcher for GO ?
regards
Vivek
Russ Cox
That's what package regexp already is.
The submatch tracking slows things a bit
but guarantees not to take exponential time.
Eventually we hope to have something more
like RE2 (http://code.google.com/p/re2/),
which picks off many important common cases
and handles them with the same asymptotic
efficiency but smaller constant factors.
Russ
Thomas Modeneis
I've been facing this issue for a while, and on my experience the best workaround is to serialize the CSV to the disk after reading it.
