Running a child for the DTM wrapper got PermissionError: [Errno 13] Permission denied

[Evaristo Caraballo]

Hi everyone,

I am not sure if this is something to post here as it seems a problem on my side but I am getting a Permission Error running this project even though keeping the location of the bin in a personal folder, not linked to any root (linux)?

I am not sure if using chmod would solve the problem but I prefer to ask before doing something else (not a programmer)?

My error seems to start at:

201         check_output(cmd, stderr=PIPE)

From where I get the exception. A writing permission?

Please advise

Regards

Evaristo

[Lev Konstantinovskiy]

Hi Evaristo,
So that I could help you better, could you please post more of the code you are running and a full text of the error?

Regards
Lev

[Evaristo Caraballo]

Hi Lev,

Following my code attached ( typical, according this link with a canonical example ):

time_seq = [50]*2 #assumming a topic every 20 messages

corpora = []
#df['EmailBody'] is a pandas Series object containing lists of tuples per cell
for x in df['EmailBody']:
    s = [t[0] for t in x]
    corpora.append(s)

class DTMcorpus(gensim.corpora.textcorpus.TextCorpus):

    def get_texts(self):
        return self.input

    def __len__(self):
        return len(self.input)

corpus = DTMcorpus(corpora)


#print(corpus.dictionary.token2id)

#setting an environment variable for the dtm
dtm_home = os.environ.get('DTM_HOME', "/my/personal/directory/bin/dtm-master")
#I selected the dtm-linux64 binary for my project
#I saved as dtm_home_address+"/bin/dtm-linux64"  
dtm_path = os.path.join(dtm_home, 'bin','dtm-linux64') if dtm_home else None

model = gensim.models.wrappers.dtmmodel.DtmModel(dtm_path, corpus, time_seq, num_topics=2, id2word=corpus.dictionary, initialize_lda=True)

All initialisations of the model seemed to occur without issues according to logs:

INFO:gensim.corpora.dictionary:adding document #0 to Dictionary(0 unique tokens: [])
INFO:gensim.corpora.dictionary:built Dictionary(2715 unique tokens: ['holding', 'freshman', 'virtuous', 'aware', 'govt']...) from 100 documents (total 6171 corpus positions)
INFO:gensim.models.wrappers.dtmmodel:serializing temporary corpus to /tmp/eb0661_train-mult.dat
INFO:gensim.corpora.bleicorpus:no word id mapping provided; initializing from corpus
INFO:gensim.corpora.bleicorpus:storing corpus in Blei's LDA-C format into /tmp/eb0661_train-mult.dat
INFO:gensim.corpora.bleicorpus:saving vocabulary of 2715 words to /tmp/eb0661_train-mult.dat.vocab
INFO:gensim.models.wrappers.dtmmodel:training DTM with args --ntopics=2 --model=dtm  --mode=fit --initialize_lda=true --corpus_prefix=/tmp/eb0661_train --outname=/tmp/eb0661_train_out --alpha=0.01 --lda_max_em_iter=10 --lda_sequence_min_iter=6  --lda_sequence_max_iter=20 --top_chain_var=0.005 --rng_seed=0 
INFO:gensim.models.wrappers.dtmmodel:Running command ['.../bin/dtm-master/bin/dtm-linux64', '--ntopics=2', '--model=dtm', '--mode=fit', '--initialize_lda=true', '--corpus_prefix=/tmp/eb0661_train', '--outname=/tmp/eb0661_train_out', '--alpha=0.01', '--lda_max_em_iter=10', '--lda_sequence_min_iter=6', '--lda_sequence_max_iter=20', '--top_chain_var=0.005', '--rng_seed=0']

After initialisation of the model, I got an error when running the command. The error I got was:

PermissionError                           Traceback (most recent call last)
<ipython-input-132-8a1f0e522247> in <module>()
     38 dtm_path = os.path.join(dtm_home, 'bin','dtm-linux64') if dtm_home else None
     39 
---> 40 model = gensim.models.wrappers.dtmmodel.DtmModel(dtm_path, corpus, time_seq, num_topics=2, id2word=corpus.dictionary, initialize_lda=True)
     41 #model = gensim.models.wrappers.dtmmodel(dtm_path, df['EmailBody'], time_seq, num_topics=5, initialize_lda=True)
     42 

.../lib/python3.4/site-packages/gensim/models/wrappers/dtmmodel.py in __init__(self, dtm_path, corpus, time_slices, mode, model, num_topics, id2word, prefix, lda_sequence_min_iter, lda_sequence_max_iter, lda_max_em_iter, alpha, top_chain_var, rng_seed, initialize_lda)
    126 
    127         if corpus is not None:
--> 128             self.train(corpus, time_slices, mode, model)
    129 
    130     def fout_liklihoods(self):

.../lib/python3.4/site-packages/gensim/models/wrappers/dtmmodel.py in train(self, corpus, time_slices, mode, model)
    199         cmd = [self.dtm_path] + arguments.split()
    200         logger.info("Running command %s" % cmd)
--> 201         check_output(cmd, stderr=PIPE)
    202 
    203         self.em_steps = np.loadtxt(self.fem_steps())

.../lib/python3.4/site-packages/gensim/utils.py in check_output(*popenargs, **kwargs)
   1140     """
   1141     try:
-> 1142         process = subprocess.Popen(stdout=subprocess.PIPE, *popenargs, **kwargs)
   1143         output, unused_err = process.communicate()
   1144         retcode = process.poll()

/usr/lib/python3.4/subprocess.py in __init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds)
    857                                 c2pread, c2pwrite,
    858                                 errread, errwrite,
--> 859                                 restore_signals, start_new_session)
    860         except:
    861             # Cleanup if the child failed starting.

/usr/lib/python3.4/subprocess.py in _execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, restore_signals, start_new_session)
   1455                             else:
   1456                                 err_msg += ': ' + repr(orig_executable)
-> 1457                     raise child_exception_type(errno_num, err_msg)
   1458                 raise child_exception_type(err_msg)
   1459 

PermissionError: [Errno 13] Permission denied

All my python3.4 installation runs from a virtenv and gensim is also installed there. It seems that the binary is looking for python 3.4 in my usr folder to run the subprocess (again, not a programmer myself)? This line above highlighted:

/usr/lib/python3.4/subprocess.py

Thanks in advance

[Evaristo Caraballo]

Sorry: Just clarifying:

NOT that all my python3.4 is as virtenv: it is THIS project that I am running from a virtenv.

[Lev Konstantinovskiy]

Hi Evaristo,

Thanks for sending the code.

gensim.DTMWrapper is just a wrapper running bash commands. It prints out the exact commands that it is trying to run. What happens when you run this command yourself? Do you have write permissions in that folder?

The bash command is in this line:

INFO:gensim.models.wrappers.dtmmodel:Running command ['.../bin/dtm-master/bin/dtm-linux64', '--ntopics=2', '--model=dtm', '--mode=fit', '--initialize_lda=true', '--corpus_prefix=/tmp/eb0661_train', '--outname=/tmp/eb0661_train_out', '--alpha=0.01', '--lda_max_em_iter=10', '--lda_sequence_min_iter=6', '--lda_sequence_max_iter=20', '--top_chain_var=0.005', '--rng_seed=0']

By the way, you are welcome to check out our new pure python DTM implementation. It is about 5 times slower but will be parallelized soon. 

Regards
Lev

[Evaristo Caraballo]

Thanks Lev,

Yes, I have permission (I then tried to open the binary with `open`: no reported issue).

I tried as you said:

What happens when you run this command yourself? Do you have write permissions in that folder?

Even after moving the folder to a more "open" location and running the code directly from my terminal I get a Permission Denied error (?).

Just to say that I copied the binary directly into my folder from its repo. Should that be an issue?

you are welcome to check out our new pure python

The python implementation seems like a reasonable option as I am currently learning and testing the methodology. I will have a look! However, any comment about the error above will be still welcome.

By the way: a LOT of broken figures in the notebook address you provided?

Regards,

Evaristo

Op dinsdag 4 oktober 2016 12:54:23 UTC+2 schreef Lev Konstantinovskiy:

[Evaristo Caraballo]

Hi Lev:

* Tried the python implementation
* Got an error (module not found)
* Found that in the __init__.py file of the models folder the ldaseqmodel.py is not required
* I added an import in the __init__.py importing the module and the class
* seems to be working as expected

In fact, there are a bunch of the modules in the models folder that are NOT included in the corresponding __init__.py

Hope this helps

Evaristo


Op dinsdag 4 oktober 2016 16:02:51 UTC+2 schreef Evaristo Caraballo:

[Lev Konstantinovskiy]

Hi Evaristo,

running the code directly from my terminal I get a Permission Denied error (?).

This means that the error is in the linux permissions and not in the Gensim wrapper. It means it is out of our hands. Do you have write permissions in the output folder and in /tmp?

Thank you for the correction - the notebook is better viewed in nbviewer. That way the figures are not broken.

[Lev Konstantinovskiy]

Hi Evaristo

import ldaseqmodel.LdaSeqModel should work even when it is missing in __init__.py. The modules in __init.py are mentioned just to save some typing.

Thanks for trying out the new DTM - would be very thankful for any feedback you have.

Regards

Lev

[Evaristo Caraballo]

import ldaseqmodel.LdaSeqModel should work even when it is missing in __init__.py

Not in my case, probably due to my setups (eg. virtenv).

Do you have write permissions in the output folder and in /tmp?

I think it has to do with that. Thanks again for your comment.

----

Thanks for your attention to this, Lev. I will let you know of any feedback about the fully-python DTM.

Regards,

Evaristo

Op dinsdag 4 oktober 2016 17:20:51 UTC+2 schreef Lev Konstantinovskiy:

[Johannes]

I have run into the same problem on macOS Sierra. Did you solve it?

[Lev Konstantinovskiy]

Hi Johannes,

What happens when you run the command directly from the terminal? The command is in Running command ['...

Regards
Lev

[Sophia Sun]

Hi!

Have you tried granting permission to the executable file? Try:

 chmod +x dtm-darwin64 

 

Here dtm-darwin64 is my executable. The permission denied error went away after I did this.

Let me know if this is helpful!