"SpinnmanIOException: IO Error: Failed to communicate with the machine"

[Saurav Pawar]

Hi,

I am trying to run this script (https://github.com/NeuromorphicProcessorProject/snn_toolbox/blob/master/examples/mnist_keras_spiNNaker.py). After calling the main function I am getting the following error. 

Can anyone please help on that ?

```

SNN toolbox WARNING: Delay (0.0) must be at least one time step (0.1). Setting delay = dt. SNN toolbox WARNING: For optimal correspondence between the original ANN and the converted SNN simulated on pyNN, the threshold should be 0.01. Current value: 1.0. Initializing spiNNaker simulator...

2022-08-11 15:14:43 INFO: Read cfg files: /home/sauravpawar/.local/lib/python3.8/site-packages/spinn_front_end_common/interface/spinnaker.cfg, /home/sauravpawar/.local/lib/python3.8/site-packages/spynnaker/pyNN/spynnaker.cfg, /home/sauravpawar/.spynnaker.cfg 2022-08-11 15:14:43 INFO: Will search these locations for binaries: /home/sauravpawar/.local/lib/python3.8/site-packages/spinn_front_end_common/common_model_binaries : /home/sauravpawar/.local/lib/python3.8/site-packages/spynnaker/pyNN/model_binaries 2022-08-11 15:14:43 WARNING: /home/sauravpawar/Desktop/spiNNaker_cnn/reports has 59 old reports that have not been closed 2022-08-11 15:14:43 WARNING: A timestep was entered that has forced sPyNNaker to automatically slow the simulation down from real time by a factor of 10. To remove this automatic behaviour, please enter a timescaleFactor value in your .spynnaker.cfg 2022-08-11 15:14:43 INFO: Setting time scale factor to 10. 2022-08-11 15:14:43 INFO: Setting machine time step to 100 micro-seconds.

['/home/sauravpawar/.local/lib/python3.8/site-packages/spinn_front_end_common/interface/spinnaker.cfg', '/home/sauravpawar/.local/lib/python3.8/site-packages/spynnaker/pyNN/spynnaker.cfg', '/home/sauravpawar/.spynnaker.cfg'] Loading data set from '.npz' files in /home/sauravpawar/temp1/1660216478.0015025. Model: "model" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_1 (InputLayer) [(None, 28, 28, 1)] 0 conv2d (Conv2D) (None, 12, 12, 16) 400 conv2d_1 (Conv2D) (None, 10, 10, 32) 4608 average_pooling2d (AverageP (None, 5, 5, 32) 0 ooling2D) conv2d_2 (Conv2D) (None, 5, 5, 8) 2304 flatten (Flatten) (None, 200) 0 dropout (Dropout) (None, 200) 0 dense (Dense) (None, 10) 2000 ================================================================= Total params: 9,312 Trainable params: 9,312 Non-trainable params: 0 _________________________________________________________________ Evaluating input model on 5 samples... Top-1 accuracy: 100.00% Top-5 accuracy: 100.00% Parsing input model... Skipping layer InputLayer. Parsing layer Conv2D. Using activation relu. Parsing layer Conv2D. Using activation relu. Parsing layer AveragePooling2D. Parsing layer Conv2D. Using activation relu. Parsing layer Flatten. Skipping layer Dropout. Parsing layer Dense. Using activation softmax. Building parsed model... Compiling parsed model... Model: "model_1" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_1 (InputLayer) [(1, 28, 28, 1)] 0 0Conv2D_12x12x16 (Conv2D) (1, 12, 12, 16) 416 1Conv2D_10x10x32 (Conv2D) (1, 10, 10, 32) 4640 2AveragePooling2D_5x5x32 (A (1, 5, 5, 32) 0 veragePooling2D) 3Conv2D_5x5x8 (Conv2D) (1, 5, 5, 8) 2312 4Flatten_200 (Flatten) (1, 200) 0 5Dense_10 (Dense) (1, 10) 2010 ================================================================= Total params: 9,378 Trainable params: 9,378 Non-trainable params: 0 _________________________________________________________________ Normalizing parameters... Using 6000 samples for normalization. INFO: Need ['0.06', '0.08', '0.00', '0.00'] GB for layer activations. May have to reduce size of data set used for normalization. Calculating activations of layer 0Conv2D_12x12x16 ... 6000/6000 [==============================] - 2s 332us/step Writing activations to disk... Scale factor: 1.99. Calculating activations of layer 1Conv2D_10x10x32 ... 6000/6000 [==============================] - 2s 403us/step Writing activations to disk... Scale factor: 6.00. Calculating activations of layer 3Conv2D_5x5x8 ... 6000/6000 [==============================] - 3s 469us/step Writing activations to disk... Scale factor: 10.84. Calculating activations of layer 5Dense_10 ... 6000/6000 [==============================] - 3s 486us/step

2022-08-11 15:15:00 WARNING: Size of the population 0Conv2D_12x12x16 rounded from 2304 to 2304. Please use int values for size

Writing activations to disk... Scale factor: 1.00. Using scale factor 1.00 for softmax layer. Evaluating parsed model on 5 samples... Top-1 accuracy: 100.00% Top-5 accuracy: 100.00% Building spiking model... Instantiating layer: 0Conv2D_12x12x16 Building layer: 0Conv2D_12x12x16 Transposing kernels. ................ Weights scaled by a factor of 99.99546021312976

2022-08-11 15:15:00 WARNING: Size of the population 1Conv2D_10x10x32 rounded from 3200 to 3200. Please use int values for size

Instantiating layer: 1Conv2D_10x10x32 Building layer: 1Conv2D_10x10x32 Transposing kernels. ................................ Weights scaled by a factor of 99.99546021312976

2022-08-11 15:15:01 WARNING: Size of the population 2AveragePooling2D_5x5x32 rounded from 800 to 800. Please use int values for size

Instantiating layer: 2AveragePooling2D_5x5x32 Building layer: 2AveragePooling2D_5x5x32 ................................

2022-08-11 15:15:01 WARNING: Size of the population 3Conv2D_5x5x8 rounded from 200 to 200. Please use int values for size

Weights scaled by a factor of 99.99546021312976 Instantiating layer: 3Conv2D_5x5x8 Building layer: 3Conv2D_5x5x8 Transposing kernels. ........

2022-08-11 15:15:01 WARNING: Size of the population 5Dense_10 rounded from 10 to 10. Please use int values for size /home/sauravpawar/.local/lib/python3.8/site-packages/snntoolbox/simulation/target_simulators/spiNNaker_target_sim.py:114: RuntimeWarning: Activation 'softmax' not implemented. Using 'relu' activation instead. warnings.warn("Activation 'softmax' not implemented. Using 'relu' " 2022-08-11 15:15:01 INFO: Increasing spike rate while recording requires a "reset unless additional_parameters "max_rate" is set 2022-08-11 15:15:01 INFO: Starting execution process 2022-08-11 15:15:01 INFO: Simulating for 500 0.1ms timesteps using a hardware timestep of 1000us 2022-08-11 15:15:01 INFO: Creating transceiver for 192.168.240.254 2022-08-11 15:15:01 INFO: Working out if machine is booted

Weights scaled by a factor of 99.99546021312976 Instantiating layer: 4Flatten_200 Building layer: 4Flatten_200 Instantiating layer: 5Dense_10 Building layer: 5Dense_10 Weights scaled by a factor of 99.99546021312976 (200, 10) Not swapping data_format of Flatten layer. Compiling spiking model... Number of operations of ANN: 1156000 Number of neurons: 6514 Number of synapses: 791888 Not saving model to /home/sauravpawar/temp1/1660216478.0015025... Starting new simulation... There was a problem with serialisation.

2022-08-11 15:15:05 INFO: Attempting to boot machine 2022-08-11 15:15:14 INFO: Attempting to boot machine 2022-08-11 15:15:23 INFO: Attempting to boot machine 2022-08-11 15:15:40 ERROR: Error when calling spinn_front_end_common.interface.interface_functions.MachineGenerator.__call__ with inputs dict_keys(['report_waiting_logs', 'hostname', 'reset_machine_on_start_up', 'bmp_details', 'downed_chips', 'downed_cores', 'downed_links', 'board_version', 'auto_detect_bmp', 'scamp_connection_data', 'boot_port_num', 'max_sdram_size', 'repair_machine', 'ignore_bad_ethernets', 'default_report_directory'])

--------------------------------------------------------------------------- SpinnmanIOException Traceback (most recent call last) Input In [5], in <cell line: 67>() 62 config.write(configfile) 64 # RUN SNN TOOLBOX # 65 ################### ---> 67 main(config_filepath) File ~/.local/lib/python3.8/site-packages/snntoolbox/bin/run.py:31, in main(filepath) 29 if filepath is not None: 30 config = update_setup(filepath) ---> 31 run_pipeline(config) 32 return 34 parser = argparse.ArgumentParser( 35 description='Run SNN toolbox to convert an analog neural network into ' 36 'a spiking neural network, and optionally simulate it.') File ~/.local/lib/python3.8/site-packages/snntoolbox/bin/utils.py:145, in run_pipeline(config, queue) 142 return snn.run(**test_set) 144 # Simulate network --> 145 results = run(spiking_model, **testset) 147 # Clean up 148 spiking_model.end_sim() File ~/.local/lib/python3.8/site-packages/snntoolbox/bin/utils.py:220, in run_parameter_sweep.<locals>.decorator.<locals>.wrapper(snn, **testset) 216 if len(param_values) > 1: 217 print("\nCurrent value of parameter to sweep: " + 218 "{} = {:.2f}\n".format(param_name, p)) --> 220 results.append(run_single(snn, **testset)) 222 # Plot and return results of parameter sweep. 223 try: File ~/.local/lib/python3.8/site-packages/snntoolbox/bin/utils.py:142, in run_pipeline.<locals>.run(snn, **test_set) 140 @run_parameter_sweep(config, queue) 141 def run(snn, **test_set): --> 142 return snn.run(**test_set) File ~/.local/lib/python3.8/site-packages/snntoolbox/simulation/utils.py:606, in AbstractSNN.run(self, x_test, y_test, dataflow, **kwargs) 603 # Main step: Run the network on a batch of samples for the duration 604 # of the simulation. 605 print("\nStarting new simulation...\n") --> 606 output_b_l_t = self.simulate(**data_batch_kwargs) 608 # Halt if model is to be serialised only. 609 if self.config.getboolean('tools', 'serialise_only'): File ~/.local/lib/python3.8/site-packages/snntoolbox/simulation/target_simulators/spiNNaker_target_sim.py:381, in SNN.simulate(self, **kwargs) 379 if self.config.getboolean('tools', 'serialise_only'): 380 sys.exit('finished after serialisation') --> 381 self.sim.run(self._duration) 382 print("\nCollecting results...") 383 output_b_l_t = self.get_recorded_vars(self.layers) File ~/.local/lib/python3.8/site-packages/spynnaker8/__init__.py:703, in run(simtime, callbacks) 701 if not globals_variables.has_simulator(): 702 raise ConfigurationException(FAILED_STATE_MSG) --> 703 return __pynn["run"](simtime, callbacks=callbacks) File ~/.local/lib/python3.8/site-packages/pyNN/common/control.py:111, in build_run.<locals>.run(simtime, callbacks) 96 def run(simtime, callbacks=None): 97 """ 98 Advance the simulation by `simtime` ms. 99 (...) 109 the initial conditions (time ``t = 0``), use the ``reset()`` function. 110 """ --> 111 return run_until(simulator.state.t + simtime, callbacks) File ~/.local/lib/python3.8/site-packages/pyNN/common/control.py:93, in build_run.<locals>.run_until(time_point, callbacks) 90 callback_events.extend((callback(simulator.state.t), callback) 91 for callback in active_callbacks) 92 else: ---> 93 simulator.state.run_until(time_point) 94 return simulator.state.t File ~/.local/lib/python3.8/site-packages/spynnaker8/spinnaker.py:119, in SpiNNaker.run_until(self, tstop) 114 """ Run the simulation until the given simulation time. 115 116 :param tstop: when to run until in milliseconds 117 """ 118 # Build data --> 119 self._run_wait(tstop - self.t) File ~/.local/lib/python3.8/site-packages/spynnaker8/spinnaker.py:150, in SpiNNaker._run_wait(self, duration_ms, sync_time) 143 def _run_wait(self, duration_ms, sync_time=0.0): 144 """ Run the simulation for a length of simulation time. 145 146 :param duration_ms: The run duration, in milliseconds 147 :type duration_ms: int or float 148 """ --> 150 super(SpiNNaker, self).run(duration_ms, sync_time) File ~/.local/lib/python3.8/site-packages/spynnaker/pyNN/abstract_spinnaker_common.py:380, in AbstractSpiNNakerCommon.run(self, run_time, sync_time) 371 if (self.config.getboolean("Reports", "reports_enabled") and 372 self.config.getboolean( 373 "Reports", "write_redundant_packet_count_report") and 374 not self._use_virtual_board and run_time is not None and 375 not self._has_ran and self._config.getboolean( 376 "Reports", "writeProvenanceData")): 377 self.extend_extra_post_run_algorithms( 378 ["RedundantPacketCountReport"]) --> 380 super().run(run_time, sync_time) 381 for projection in self._projections: 382 projection._clear_cache() File ~/.local/lib/python3.8/site-packages/spinn_front_end_common/interface/abstract_spinnaker_base.py:780, in AbstractSpinnakerBase.run(self, run_time, sync_time) 778 @overrides(SimulatorInterface.run) 779 def run(self, run_time, sync_time=0): --> 780 self._run(run_time, sync_time) File ~/.local/lib/python3.8/site-packages/spinn_front_end_common/interface/abstract_spinnaker_base.py:932, in AbstractSpinnakerBase._run(self, run_time, sync_time) 929 self._max_run_time_steps = None 931 if self._machine is None: --> 932 self._get_machine(total_run_time, n_machine_time_steps) 933 self._do_mapping(run_time, total_run_time) 935 # Check if anything has per-timestep SDRAM usage File ~/.local/lib/python3.8/site-packages/spinn_front_end_common/interface/abstract_spinnaker_base.py:1211, in AbstractSpinnakerBase._get_machine(self, total_run_time, n_machine_time_steps) 1208 # If we are using a directly connected machine, add the details to get 1209 # the machine and transceiver 1210 if self._hostname is not None: -> 1211 self._machine_by_hostname(n_machine_time_steps, total_run_time) 1213 elif self._use_virtual_board: 1214 self._machine_by_virtual(n_machine_time_steps, total_run_time) File ~/.local/lib/python3.8/site-packages/spinn_front_end_common/interface/abstract_spinnaker_base.py:1267, in AbstractSpinnakerBase._machine_by_hostname(self, n_machine_time_steps, total_run_time) 1264 outputs.append("MemoryMachine") 1265 outputs.append("MemoryTransceiver") -> 1267 executor = self._run_algorithms( 1268 inputs, algorithms, outputs, [], [], "machine_generation") 1269 self._machine = executor.get_item("MemoryMachine") 1270 self._txrx = executor.get_item("MemoryTransceiver") File ~/.local/lib/python3.8/site-packages/spinn_front_end_common/interface/abstract_spinnaker_base.py:1195, in AbstractSpinnakerBase._run_algorithms(self, inputs, algorithms, outputs, tokens, required_tokens, provenance_name, optional_algorithms) 1192 except Exception as e3: 1193 logger.warning("problem when shutting down {}".format(e3), 1194 exc_info=True) -> 1195 raise e File ~/.local/lib/python3.8/site-packages/spinn_front_end_common/interface/abstract_spinnaker_base.py:1175, in AbstractSpinnakerBase._run_algorithms(self, inputs, algorithms, outputs, tokens, required_tokens, provenance_name, optional_algorithms) 1165 executor = PACMANAlgorithmExecutor( 1166 algorithms=algorithms, optional_algorithms=optional, 1167 inputs=inputs, tokens=tokens, (...) 1171 provenance_name=provenance_name, 1172 provenance_path=self._pacman_executor_provenance_path) 1174 try: -> 1175 executor.execute_mapping() 1176 self._pacman_provenance.extract_provenance(executor) 1177 return executor File ~/.local/lib/python3.8/site-packages/pacman/executor/pacman_algorithm_executor.py:666, in PACMANAlgorithmExecutor.execute_mapping(self) 664 if self._do_direct_injection: 665 with injection_context(self._internal_type_mapping): --> 666 self.__execute_mapping() 667 else: 668 self.__execute_mapping() File ~/.local/lib/python3.8/site-packages/pacman/executor/pacman_algorithm_executor.py:682, in PACMANAlgorithmExecutor.__execute_mapping(self) 679 timer.start_timing() 681 # Execute the algorithm --> 682 results = algorithm.call(self._internal_type_mapping) 684 if self._provenance_path: 685 self._report_full_provenance(algorithm, results) File ~/.local/lib/python3.8/site-packages/pacman/executor/algorithm_classes/abstract_python_algorithm.py:77, in AbstractPythonAlgorithm.call(self, inputs) 74 method_inputs = self._get_inputs(inputs) 76 # Run the algorithm and get the results ---> 77 results = self.call_python(method_inputs) 79 if results is not None and not isinstance(results, tuple): 80 results = (results,) File ~/.local/lib/python3.8/site-packages/pacman/executor/algorithm_classes/python_class_algorithm.py:97, in PythonClassAlgorithm.call_python(self, inputs) 93 method = self._python_method 94 logger.error("Error when calling {}.{}.{} with inputs {}", 95 self._python_module, self._python_class, method, 96 inputs.keys()) ---> 97 raise e File ~/.local/lib/python3.8/site-packages/pacman/executor/algorithm_classes/python_class_algorithm.py:89, in PythonClassAlgorithm.call_python(self, inputs) 87 method = getattr(instance, self._python_method) 88 try: ---> 89 return method(**inputs) 90 except Exception as e: 91 method = "__call__" File ~/.local/lib/python3.8/site-packages/spinn_front_end_common/interface/interface_functions/machine_generator.py:143, in MachineGenerator.__call__(self, hostname, bmp_details, downed_chips, downed_cores, downed_links, board_version, auto_detect_bmp, scamp_connection_data, boot_port_num, reset_machine_on_start_up, report_waiting_logs, max_sdram_size, repair_machine, ignore_bad_ethernets, default_report_directory) 139 if board_version is None: 140 raise ConfigurationException( 141 "Please set a machine version number in the " 142 "corresponding configuration (cfg) file") --> 143 txrx.ensure_board_is_ready() 144 txrx.discover_scamp_connections() 145 return txrx.get_machine_details(), txrx File ~/.local/lib/python3.8/site-packages/spinnman/transceiver.py:981, in Transceiver.ensure_board_is_ready(self, number_of_boards, width, height, n_retries, extra_boot_values) 979 # verify that the version is the expected one for this transceiver 980 if version_info is None: --> 981 raise SpinnmanIOException( 982 "Failed to communicate with the machine") 983 if (version_info.name != _SCAMP_NAME or 984 not self.is_scamp_version_compabible( 985 version_info.version_number)): 986 raise SpinnmanIOException( 987 "The machine is currently booted with {}" 988 " {} which is incompatible with this transceiver, " 989 "required version is {} {}".format( 990 version_info.name, version_info.version_number, 991 _SCAMP_NAME, _SCAMP_VERSION)) SpinnmanIOException: IO Error: Failed to communicate with the machine

```

Thanks,

Saurav

[Edward Jones]

Hi,

This error shows there is a problem communicating with SpiNNaker. I assume that you have a board and are trying to run this script locally rather than through the HBP JupyterLab interface (https://spinn-20.cs.man.ac.uk). It might be worth trying the code through the JupyterLab first (the MNIST example is there with all the dependencies) to see expected output.

[Saurav Pawar]

Hi Edward,

I did logged in using my EBRAINS account but wasn't able to find the MNIST example. If possible can you please share the path to the example ?

Thanks,

Saurav

[Edward Jones]

It should be under ANN-to-SNN Conversion/MNIST Example.ipynb

Let me know if you can find it.

Kind regards,

Ed

[Saurav Pawar]

This is what I see once I login.

[Andrew Rowley]

Hi,

 

Note that in general the Jupyter workspaces are not shared between users.  To have shared content, you would need an EBRAINS collab on:

https://wiki.ebrains.eu/

 

Once you have this, you can then access shared content via the “drive” folder.

 

Andrew :)

 

--
You received this message because you are subscribed to the Google Groups "SpiNNaker Users Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spinnakeruser...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/spinnakerusers/3dfb0392-141a-4862-b5f1-d16a9fe79408n%40googlegroups.com.

[Saurav Pawar]

Hi Andrew,

I have an EBRAINS collab and I am also able to access the "drive" folder. Could you please tell me the path for the MNIST spinnaker notebook ? Because I searched in folders named "n-minist", "n-minist-1" and "Neuromorphic Computing in EBRAINS". Both ("n-minist" and "n-minist-1") are empty and "Neuromorphic Computing in EBRAINS" doesn't have anything related to SpiNNaker. I have also attached the screenshots for your convenience.

Kind regards,

Saurav

[Andrew Rowley]

Hi,

 

Sorry for the confusion.  I think Ed would need to add the files he has to a shared drive folder before you can access them; I am guessing this hasn’t been done so far as I can’t see any such shared folder as you say.

 

Andrew :)

 

To view this discussion on the web, visit https://groups.google.com/d/msgid/spinnakerusers/939c907b-f534-4786-89b2-a0db5983d497n%40googlegroups.com.

[Saurav Pawar]

Okay Andrew. Thank you

[Saurav Pawar]

Hi,

Wanted to follow up on this.

Also, is there a different way to access spinnaker and try out my spiking CNN on it ?

Best,

Saurav

[Andrew Rowley]

Hi,

 

I am unsure what you are after here.  It is possible to access SpiNNaker remotely through Jupyter and through the batch submission system.  If you have already got access to Jupyter, that is likely the best and easiest way to proceed.

 

Andrew :)

 

To view this discussion on the web, visit https://groups.google.com/d/msgid/spinnakerusers/fec330b8-591d-4008-a8f1-a5e2ac8dae61n%40googlegroups.com.

[Andrew Gait]

Hi Saurav, we answered your question about putting a spiking CNN onto SpiNNaker last week:

I think the code can deal with all that.  Note that there is a p.PoolDenseConnector() to use as a connector and p.PoolDense() to use as a synapse type as well.

 

For the connectors, there is some information in the connectors themselves as to their use:

https://github.com/SpiNNakerManchester/sPyNNaker/blob/master/spynnaker/pyNN/models/neural_projections/connectors/convolution_connector.py#L56-L116

https://github.com/SpiNNakerManchester/sPyNNaker/blob/master/spynnaker/pyNN/models/neural_projections/connectors/pool_dense_connector.py#L54-L90

Andy

----------------------------------------------------
Dr. Andrew Gait (he/him),

Research Software Engineer,

APT Group,
Department of Computer Science,
The University of Manchester,
Oxford Road, Manchester M13 9PL

email: andre...@manchester.ac.uk
web: http://personalpages.manchester.ac.uk/staff/Andrew.Gait/

I do not expect a reply to this email outside of normal working hours

---

From: spinnak...@googlegroups.com <spinnak...@googlegroups.com> on behalf of Saurav Pawar <saurav...@gmail.com>
Sent: 27 September 2022 09:32

To view this discussion on the web, visit https://groups.google.com/d/msgid/spinnakerusers/fec330b8-591d-4008-a8f1-a5e2ac8dae61n%40googlegroups.com.