All trails are getting failed

nni Environment:

nni version: 0.9
nni mode(local|pai|remote): local
OS: windows
python version: 3.6
is conda or virtualenv used?: no
is running in docker?: no

I'm just starting with NNI, I'm following the document step by step.
And I see all my trails are getting failed
The exact command I ran and it's output:
C:\Users\User>nnictl create --config nni\examples\trials\mnist\config_windows.yml --port 8082
INFO: expand searchSpacePath: search_space.json to C:\Users\User\nni\examples\trials\mnist\search_space.json
INFO: expand codeDir: . to C:\Users\User\nni\examples\trials\mnist\.
INFO: Starting restful server...
INFO: Successfully started Restful server!
INFO: Setting local config...
INFO: Successfully set local config!
INFO: Starting experiment...
INFO: Successfully started experiment!
-----------------------------------------------------------------------
The experiment id is JAqxevyD
The Web UI urls are: http://<IP_ADDRESS>:8082   http://<IP_ADDRESS>:8082   http://<IP_ADDRESS>:8082   http://<IP_ADDRESS>:8082   http://<IP_ADDRESS>:8082   http://<IP_ADDRESS>:8082
-----------------------------------------------------------------------

You can use these commands to get more information about the experiment
-----------------------------------------------------------------------
         commands                       description
1. nnictl experiment show        show the information of experiments
2. nnictl trial ls               list all of trial jobs
3. nnictl top                    monitor the status of running experiments
4. nnictl log stderr             show stderr log content
5. nnictl log stdout             show stdout log content
6. nnictl stop                   stop an experiment
7. nnictl trial kill             kill a trial job by id
8. nnictl --help                 get help information about nnictl
-----------------------------------------------------------------------

I have attached the image of the WebUI which shows the trails has failed

Moreover, I keep getting a pop-up :
----------------------------------------------------------------
#!/usr/bin/env bash
# Copyright (C) 2014, Alexey Pavlov
#<EMAIL_ADDRESS># This file is part of Minimal SYStem version 2.
#   https://sourceforge.net/p/msys2/wiki/MSYS2%20installation/
# File: cmd

"$COMSPEC" "$@"
--------------------------------------------------------------------

how to disable it?
And where can I see logs? why are my trails failing?
Please let me know if I'm missing anything.
Any leads will be appreciated.
Thanks

Hello @anandh02, you could use "nnictl experiment show JAqxevyD" to check the path of log.
You could check the stderr file in a trial folder and you could post the log of "dispatcher.log" and "nni_manager.log"
Thank you.

Hi @xuehui1991, there is no message in stdrerr.
nnimanager logs:

[7/25/2019, 1:29:51 AM] INFO [ 'Datastore initialization done' ]
[7/25/2019, 1:29:51 AM] INFO [ 'Rest server listening on: http://<IP_ADDRESS>:8082' ]
[7/25/2019, 1:29:51 AM] INFO [ 'RestServer start' ]
[7/25/2019, 1:29:51 AM] INFO [ 'Construct local machine training service.' ]
[7/25/2019, 1:29:51 AM] INFO [ 'RestServer base port is 8082' ]
[7/25/2019, 1:29:52 AM] INFO [ 'NNIManager setClusterMetadata, key: trial_config, value: {"command":"python mnist.py","codeDir":"C:\\Users\\User\\nni\\examples\\trials\\mnist\\.","gpuNum":0}' ]
[7/25/2019, 1:29:52 AM] INFO [ 'required GPU number is 0' ]
[7/25/2019, 1:29:53 AM] INFO [ 'Starting experiment: JAqxevyD' ]
[7/25/2019, 1:29:53 AM] INFO [ 'Change NNIManager status from: INITIALIZED to: RUNNING' ]
[7/25/2019, 1:29:53 AM] INFO [ 'Add event listeners' ]
[7/25/2019, 1:29:53 AM] INFO [ 'Run local machine training service.' ]
[7/25/2019, 1:29:54 AM] INFO [ 'NNIManager received command from dispatcher: ID, ' ]
[7/25/2019, 1:29:54 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7430019065482085, "conv_size": 7, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}' ]
[7/25/2019, 1:29:58 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\"parameter_id\": 0, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.7430019065482085, \"conv_size\": 7, \"hidden_size\": 124, \"batch_size\": 1, \"learning_rate\": 0.1}, \"parameter_index\": 0}","index":0}}' ]
[7/25/2019, 1:30:08 AM] INFO [ 'Trial job JLCwt status changed from WAITING to FAILED' ]
[7/25/2019, 1:30:08 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7055245865212583, "conv_size": 7, "hidden_size": 512, "batch_size": 8, "learning_rate": 0.1}, "parameter_index": 0}' ]
[7/25/2019, 1:30:13 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\"parameter_id\": 1, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.7055245865212583, \"conv_size\": 7, \"hidden_size\": 512, \"batch_size\": 8, \"learning_rate\": 0.1}, \"parameter_index\": 0}","index":0}}' ]
[7/25/2019, 1:30:19 AM] INFO [ 'Trial job lRgva status changed from WAITING to FAILED' ]
[7/25/2019, 1:30:19 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 2, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7104704507552245, "conv_size": 7, "hidden_size": 124, "batch_size": 16, "learning_rate": 0.1}, "parameter_index": 0}' ]
[7/25/2019, 1:30:24 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\"parameter_id\": 2, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.7104704507552245, \"conv_size\": 7, \"hidden_size\": 124, \"batch_size\": 16, \"learning_rate\": 0.1}, \"parameter_index\": 0}","index":0}}' ]
[7/25/2019, 1:30:30 AM] INFO [ 'Trial job j5v2w status changed from WAITING to RUNNING' ]
[7/25/2019, 1:30:36 AM] INFO [ 'Trial job j5v2w status changed from RUNNING to FAILED' ]
[7/25/2019, 1:30:36 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 3, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6830578122412669, "conv_size": 7, "hidden_size": 512, "batch_size": 32, "learning_rate": 0.1}, "parameter_index": 0}' ]
[7/25/2019, 1:30:41 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\"parameter_id\": 3, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.6830578122412669, \"conv_size\": 7, \"hidden_size\": 512, \"batch_size\": 32, \"learning_rate\": 0.1}, \"parameter_index\": 0}","index":0}}' ]
[7/25/2019, 1:30:51 AM] INFO [ 'Trial job hyXdP status changed from WAITING to FAILED' ]
[7/25/2019, 1:30:51 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 4, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.8794833854527577, "conv_size": 3, "hidden_size": 1024, "batch_size": 32, "learning_rate": 0.01}, "parameter_index": 0}' ]
[7/25/2019, 1:30:56 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\"parameter_id\": 4, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.8794833854527577, \"conv_size\": 3, \"hidden_size\": 1024, \"batch_size\": 32, \"learning_rate\": 0.01}, \"parameter_index\": 0}","index":0}}' ]
[7/25/2019, 1:31:07 AM] INFO [ 'Trial job aJP2F status changed from WAITING to FAILED' ]
[7/25/2019, 1:31:07 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 5, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6163793636385104, "conv_size": 5, "hidden_size": 512, "batch_size": 16, "learning_rate": 0.0001}, "parameter_index": 0}' ]
[7/25/2019, 1:31:12 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\"parameter_id\": 5, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.6163793636385104, \"conv_size\": 5, \"hidden_size\": 512, \"batch_size\": 16, \"learning_rate\": 0.0001}, \"parameter_index\": 0}","index":0}}' ]
[7/25/2019, 1:31:18 AM] INFO [ 'Trial job KhFdA status changed from WAITING to FAILED' ]
[7/25/2019, 1:31:18 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 6, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7082566300398873, "conv_size": 5, "hidden_size": 1024, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}' ]
[7/25/2019, 1:31:23 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\"parameter_id\": 6, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.7082566300398873, \"conv_size\": 5, \"hidden_size\": 1024, \"batch_size\": 1, \"learning_rate\": 0.1}, \"parameter_index\": 0}","index":0}}' ]
[7/25/2019, 1:31:29 AM] INFO [ 'Trial job QHR8N status changed from WAITING to FAILED' ]
[7/25/2019, 1:31:29 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 7, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.8616380623467225, "conv_size": 5, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.001}, "parameter_index": 0}' ]
[7/25/2019, 1:31:34 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\"parameter_id\": 7, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.8616380623467225, \"conv_size\": 5, \"hidden_size\": 124, \"batch_size\": 1, \"learning_rate\": 0.001}, \"parameter_index\": 0}","index":0}}' ]
[7/25/2019, 1:31:45 AM] INFO [ 'Trial job i6is6 status changed from WAITING to FAILED' ]
[7/25/2019, 1:31:45 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 8, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.8879214396569852, "conv_size": 3, "hidden_size": 512, "batch_size": 4, "learning_rate": 0.001}, "parameter_index": 0}' ]
[7/25/2019, 1:31:50 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\"parameter_id\": 8, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.8879214396569852, \"conv_size\": 3, \"hidden_size\": 512, \"batch_size\": 4, \"learning_rate\": 0.001}, \"parameter_index\": 0}","index":0}}' ]
[7/25/2019, 1:32:00 AM] INFO [ 'Trial job InrzT status changed from WAITING to FAILED' ]
[7/25/2019, 1:32:01 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 9, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.64958081561231, "conv_size": 3, "hidden_size": 1024, "batch_size": 8, "learning_rate": 0.0001}, "parameter_index": 0}' ]
[7/25/2019, 1:32:06 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\"parameter_id\": 9, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.64958081561231, \"conv_size\": 3, \"hidden_size\": 1024, \"batch_size\": 8, \"learning_rate\": 0.0001}, \"parameter_index\": 0}","index":0}}' ]
[7/25/2019, 1:32:11 AM] INFO [ 'Trial job glM5m status changed from WAITING to FAILED' ]
[7/25/2019, 1:32:11 AM] INFO [ 'Change NNIManager status from: RUNNING to: NO_MORE_TRIAL' ]
[7/25/2019, 1:32:11 AM] INFO [ 'Change NNIManager status from: NO_MORE_TRIAL to: DONE' ]
[7/25/2019, 1:32:11 AM] INFO [ 'Experiment done.' ]
[7/25/2019, 1:32:11 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 10, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.53239802026354, "conv_size": 3, "hidden_size": 512, "batch_size": 16, "learning_rate": 0.001}, "parameter_index": 0}' ]
dispatcher logs:
[07/25/2019, 01:29:54 AM] INFO (hyperopt.utils/MainThread) Failed to load dill, try installing dill via "pip install dill" for enhanced pickling support.
[07/25/2019, 01:29:54 AM] INFO (hyperopt.fmin/MainThread) Failed to load dill, try installing dill via "pip install dill" for enhanced pickling support.
[07/25/2019, 01:29:54 AM] INFO (nni.msg_dispatcher_base/MainThread) Start dispatcher
[07/25/2019, 01:29:54 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.005075 seconds
[07/25/2019, 01:29:54 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/25/2019, 01:30:08 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.001997 seconds
[07/25/2019, 01:30:08 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/25/2019, 01:30:19 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.005175 seconds
[07/25/2019, 01:30:19 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/25/2019, 01:30:36 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.003342 seconds
[07/25/2019, 01:30:36 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/25/2019, 01:30:51 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.004096 seconds
[07/25/2019, 01:30:51 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/25/2019, 01:31:07 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.001040 seconds
[07/25/2019, 01:31:07 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/25/2019, 01:31:18 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.002023 seconds
[07/25/2019, 01:31:18 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/25/2019, 01:31:29 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.005700 seconds
[07/25/2019, 01:31:29 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/25/2019, 01:31:45 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.005522 seconds
[07/25/2019, 01:31:45 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/25/2019, 01:32:01 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.004645 seconds
[07/25/2019, 01:32:01 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/25/2019, 01:32:11 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.005195 seconds
[07/25/2019, 01:32:11 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials

Pls check if tensorflow is installed. If yes, check stderr in experiments/JAqxevyD/trials/jLCwt/stderr

From the "nni_manager" log, it seems has some error in trial code.

@suiguoxin Yes, TensorFlow is installed and stderr file is empty

@xuehui1991 any suggestions? what could be the issue?

@ult yuge zhang

Can you please give me all your contents in experiments/JAqxevyD/trials/jLCwt/?
Make sure you have the correct python selected and didn't mess around with Python 2 or Python 3 or whatever virtualenv you are using, which I assume you are not.
Also, try running the command after setting your working directory to C:\Users\User\nni\examples\trials\mnist\; that is, do nnictl create --config config_windows.yml straightly, instead of running it in C:\Users\User.
BTW, put your log in ```something``` (backticks) so that we can see clearly.
If it still doesn't work, feel free to ask.

Hello @ultmaster
I'm using python 3 on local env.
I tried changing the working directory to \mnsit\ still it's not working.
Inside experiments/JAqxevyD/trials/jLCwt/:
parameter.cfg file contains :
 {"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7430019065482085, "conv_size": 7, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}
and run.ps1 file contains:
$env:NNI_PLATFORM="local"
$env:NNI_SYS_DIR="C:\Users\User\nni\experiments\JAqxevyD\trials\JLCwt"
$env:NNI_TRIAL_JOB_ID="JLCwt"
$env:NNI_OUTPUT_DIR="C:\Users\User\nni\experiments\JAqxevyD\trials\JLCwt"
$env:NNI_TRIAL_SEQ_ID="0"
$env:MULTI_PHASE="false"
$env:CUDA_VISIBLE_DEVICES="-1"
cmd /c python mnist.py 2>C:\Users\User\nni\experiments\JAqxevyD\trials\JLCwt\stderr
$NOW_DATE = [int64](([datetime]::UtcNow)-(get-date "1/1/1970")).TotalSeconds
$NOW_DATE = "$NOW_DATE" + (Get-Date -Format fff).ToString()
Write $LASTEXITCODE " " $NOW_DATE  | Out-File C:\Users\User\nni\experiments\JAqxevyD\trials\JLCwt\.nni\state -NoNewline -encoding utf8 ```

stderr file is empty

and inside experiments\JAqxevyD\trials\JLCwt\.nni\
metrics file is empty
and state file contains :<PHONE_NUMBER>036

Let me know if you wanna do a remote connection to look into the problem.
I'm sure, I'm missing something but not sure what is it.

Sorry for the delayed reply; it's Saturday here...
Are you sure it runs successfully when you directly run python mnist.py? When you are using nnictl, make sure it runs from the same console, as which you run python mnist.py previously.
Also you can try setting your working directory to where you put your mnist.py and call \the\path\to\your\run.ps1 and see what happens.
Furthermore, try creating a virtualenv and redo everything (re-install nni in virtualenv and etc.) If you don't know how to do it, Google it.
Remote connection would be great; but I'm not sure whether I can make it on time.

Hi @ultmaster No problem.
I tried running python mnist.py (commented the NNI library calls in the code)
It didn't run successfully.
C:\Users\User\nni\examples\trials\mnist>python mnist.py
WARNING:tensorflow:From mnist.py:151: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use alternatives such as official/mnist/dataset.py from tensorflow/models.
WARNING:tensorflow:From C:\Users\User\AppData\Roaming\Python\Python36\site-packages\tensorflow\contrib\learn\python\learn\datasets\mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Please write your own downloading logic.
WARNING:tensorflow:From C:\Users\User\AppData\Roaming\Python\Python36\site-packages\tensorflow\contrib\learn\python\learn\datasets\base.py:252: _internal_retry.<locals>.wrap.<locals>.wrapped_fn (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Please use urllib or similar directly.
Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.
WARNING:tensorflow:From C:\Users\User\AppData\Roaming\Python\Python36\site-packages\tensorflow\contrib\learn\python\learn\datasets\mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use tf.data to implement this functionality.
Extracting /tmp/tensorflow/mnist/input_data\train-images-idx3-ubyte.gz
Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.
WARNING:tensorflow:From C:\Users\User\AppData\Roaming\Python\Python36\site-packages\tensorflow\contrib\learn\python\learn\datasets\mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use tf.data to implement this functionality.
Extracting /tmp/tensorflow/mnist/input_data\train-labels-idx1-ubyte.gz
WARNING:tensorflow:From C:\Users\User\AppData\Roaming\Python\Python36\site-packages\tensorflow\contrib\learn\python\learn\datasets\mnist.py:110: dense_to_one_hot (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use tf.one_hot on tensors.
Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.
Extracting /tmp/tensorflow/mnist/input_data\t10k-images-idx3-ubyte.gz
Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.
Extracting /tmp/tensorflow/mnist/input_data\t10k-labels-idx1-ubyte.gz
WARNING:tensorflow:From C:\Users\User\AppData\Roaming\Python\Python36\site-packages\tensorflow\contrib\learn\python\learn\datasets\mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use alternatives such as official/mnist/dataset.py from tensorflow/models.
Mnist download data done.
WARNING:tensorflow:From mnist.py:113: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See `tf.nn.softmax_cross_entropy_with_logits_v2`.

2019-07-28 01:56:49.054102: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2019-07-28 01:56:50.084176: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1411] Found device 0 with properties:
name: GeForce GTX 1050 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.62
pciBusID: 0000:01:00.0
totalMemory: 4.00GiB freeMemory: 3.30GiB
2019-07-28 01:56:50.095127: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1490] Adding visible gpu devices: 0
2019-07-28 01:56:52.894762: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-28 01:56:52.899582: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977]      0
2019-07-28 01:56:52.905277: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0:   N
2019-07-28 01:56:52.910534: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1103] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3013 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2019-07-28 01:56:55.704300: W tensorflow/core/common_runtime/bfc_allocator.cc:215] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.59GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-07-28 01:56:55.712542: W tensorflow/core/common_runtime/bfc_allocator.cc:215] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.34GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-07-28 01:56:55.719716: W tensorflow/core/common_runtime/bfc_allocator.cc:215] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.32GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-07-28 01:56:56.143649: W tensorflow/core/common_runtime/bfc_allocator.cc:215] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.90GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-07-28 01:56:56.154531: W tensorflow/core/common_runtime/bfc_allocator.cc:215] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.42GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.

It goes out of memory.

@anandh02 Set gpuNum to 0 in config_windows.yml and try again.

@ultmaster gpuNum is 0 by default in config_windows.yml
and any configuration in config_windows.yml shouldn't even matter I guess. Since I'm running >>python mnist.py
And I even tried running in virtualenv it fails over there too.

@anandh02 Re-read your log, and I believe that the error is not really "out of memory", it said that "The caller indicates that this is not a failure". So are you sure about the reason it fails? I mean, it would fail, since it couldn't get parameters from NNI.
Did you try nnictl in your virtualenv?

@ultmaster I'm sorry, my bad.
It ran successfully both in the local system and in virtualenv without NNI but through NNI it fails.

@anandh02 Okay, I just need to confirm everything here.

Of all the files listed in the screenshot above, you don't have trial.log and your stderr is empty. Am I right?
Can you try to run run.ps1 directly to see if everything is working fine?
BTW, please set --debug when you run nnictl and provide me with nnimanager.log and dispatcher.log one more time. thx

@ultmaster Yes, I don't have trial.log and my stderr is empty.
nnimanager.logs
[7/29/2019, 12:47:21 AM] DEBUG [ 'Database directory: C:\\Users\\User\\nni\\experiments\\n5ssDtas\\db' ]
[7/29/2019, 12:47:21 AM] INFO [ 'Datastore initialization done' ]
[7/29/2019, 12:47:21 AM] INFO [ 'Rest server listening on: http://<IP_ADDRESS>:9056' ]
[7/29/2019, 12:47:21 AM] INFO [ 'RestServer start' ]
[7/29/2019, 12:47:21 AM] INFO [ 'Construct local machine training service.' ]
[7/29/2019, 12:47:21 AM] INFO [ 'RestServer base port is 9056' ]
[7/29/2019, 12:47:21 AM] DEBUG [ 'GET: /check-status: body:\n{}' ]
[7/29/2019, 12:47:22 AM] DEBUG [ 'PUT: /experiment/cluster-metadata: body:\n{\n    "trial_config": {\n        "command": "python mnist.py",\n        "codeDir": "C:\\\\Users\\\\User\\\\nni\\\\examples\\\\trials\\\\mnist\\\\.",\n        "gpuNum": 0\n    }\n}' ]
[7/29/2019, 12:47:22 AM] INFO [ 'NNIManager setClusterMetadata, key: trial_config, value: {"command":"python mnist.py","codeDir":"C:\\\\Users\\\\User\\\\nni\\\\examples\\\\trials\\\\mnist\\\\.","gpuNum":0}' ]
[7/29/2019, 12:47:22 AM] INFO [ 'required GPU number is 0' ]
[7/29/2019, 12:47:23 AM] DEBUG [ 'POST: /experiment: body:\n{\n    "authorName": "default",\n    "experimentName": "example_mnist",\n    "trialConcurrency": 1,\n    "maxExecDuration": 3600,\n    "maxTrialNum": 10,\n    "searchSpace": "{\\"dropout_rate\\": {\\"_type\\": \\"uniform\\", \\"_value\\": [0.5, 0.9]}, \\"conv_size\\": {\\"_type\\": \\"choice\\", \\"_value\\": [2, 3, 5, 7]}, \\"hidden_size\\": {\\"_type\\": \\"choice\\", \\"_value\\": [124, 512, 1024]}, \\"batch_size\\": {\\"_type\\": \\"choice\\", \\"_value\\": [1, 4, 8, 16, 32]}, \\"learning_rate\\": {\\"_type\\": \\"choice\\", \\"_value\\": [0.0001, 0.001, 0.01, 0.1]}}",\n    "trainingServicePlatform": "local",\n    "tuner": {\n        "builtinTunerName": "TPE",\n        "classArgs": {\n            "optimize_mode": "maximize"\n        },\n        "className": "TPE"\n    },\n    "versionCheck": false,\n    "clusterMetaData": [\n        {\n            "key": "codeDir",\n            "value": "C:\\\\Users\\\\User\\\\nni\\\\examples\\\\trials\\\\mnist\\\\."\n        },\n        {\n            "key": "command",\n            "value": "python mnist.py"\n        }\n    ]\n}' ]
[7/29/2019, 12:47:23 AM] INFO [ 'Starting experiment: n5ssDtas' ]
[7/29/2019, 12:47:23 AM] DEBUG [ 'Setup tuner...' ]
[7/29/2019, 12:47:23 AM] DEBUG [ 'dispatcher command: python -m nni --tuner_class_name TPE --tuner_args {"optimize_mode":"maximize"}' ]
[7/29/2019, 12:47:23 AM] INFO [ 'Change NNIManager status from: INITIALIZED to: RUNNING' ]
[7/29/2019, 12:47:23 AM] INFO [ 'Add event listeners' ]
[7/29/2019, 12:47:23 AM] DEBUG [ 'ipcInterface command type: [IN], content:[{"dropout_rate": {"_type": "uniform", "_value": [0.5, 0.9]}, "conv_size": {"_type": "choice", "_value": [2, 3, 5, 7]}, "hidden_size": {"_type": "choice", "_value": [124, 512, 1024]}, "batch_size": {"_type": "choice", "_value": [1, 4, 8, 16, 32]}, "learning_rate": {"_type": "choice", "_value": [0.0001, 0.001, 0.01, 0.1]}}]' ]
[7/29/2019, 12:47:23 AM] INFO [ 'Run local machine training service.' ]
[7/29/2019, 12:47:23 AM] DEBUG [ 'Send tuner command: INITIALIZE: {"dropout_rate": {"_type": "uniform", "_value": [0.5, 0.9]}, "conv_size": {"_type": "choice", "_value": [2, 3, 5, 7]}, "hidden_size": {"_type": "choice", "_value": [124, 512, 1024]}, "batch_size": {"_type": "choice", "_value": [1, 4, 8, 16, 32]}, "learning_rate": {"_type": "choice", "_value": [0.0001, 0.001, 0.01, 0.1]}}' ]
[7/29/2019, 12:47:23 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:47:24 AM] INFO [ 'NNIManager received command from dispatcher: ID, ' ]
[7/29/2019, 12:47:24 AM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 12:47:24 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.5040585530428365, "conv_size": 5, "hidden_size": 512, "batch_size": 8, "learning_rate": 0.001}, "parameter_index": 0}' ]
[7/29/2019, 12:47:28 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:47:28 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 0, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.5040585530428365, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 8, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 12:47:28 AM] DEBUG [ 'submitTrialJob: return: {"id":"iz1GC","status":"WAITING","submitTime":1564382848578,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\iz1GC","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 0, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.5040585530428365, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 8, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\iz1GC","sequenceId":0,"gpuIndices":[]} ' ]
[7/29/2019, 12:47:28 AM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.5040585530428365, "conv_size": 5, "hidden_size": 512, "batch_size": 8, "learning_rate": 0.001}, "parameter_index": 0}, jobDetail: {"id":"iz1GC","status":"WAITING","submitTime":1564382848578,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\iz1GC","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 0, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.5040585530428365, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 8, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\iz1GC","sequenceId":0,"gpuIndices":[]}' ]
[7/29/2019, 12:47:33 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:47:38 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:47:39 AM] DEBUG [ 'trailJob status update: iz1GC, FAILED' ]
[7/29/2019, 12:47:39 AM] INFO [ 'Trial job iz1GC status changed from WAITING to FAILED' ]
[7/29/2019, 12:47:39 AM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"iz1GC","status":"FAILED","submitTime":1564382848578,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\iz1GC","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 0, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.5040585530428365, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 8, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\iz1GC","sequenceId":0,"gpuIndices":[],"startTime":1564382855361,"pid":16016,"endTime":1564382859224}' ]
[7/29/2019, 12:47:39 AM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"iz1GC","event":"FAILED","hyper_params":"{\\"parameter_id\\": 0, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.5040585530428365, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 8, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 12:47:39 AM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 12:47:39 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.8993949196832431, "conv_size": 2, "hidden_size": 124, "batch_size": 8, "learning_rate": 0.1}, "parameter_index": 0}' ]
[7/29/2019, 12:47:43 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:47:44 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 1, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.8993949196832431, \\"conv_size\\": 2, \\"hidden_size\\": 124, \\"batch_size\\": 8, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 12:47:44 AM] DEBUG [ 'submitTrialJob: return: {"id":"HGgpA","status":"WAITING","submitTime":1564382864238,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\HGgpA","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 1, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.8993949196832431, \\"conv_size\\": 2, \\"hidden_size\\": 124, \\"batch_size\\": 8, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\HGgpA","sequenceId":1,"gpuIndices":[]} ' ]
[7/29/2019, 12:47:44 AM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.8993949196832431, "conv_size": 2, "hidden_size": 124, "batch_size": 8, "learning_rate": 0.1}, "parameter_index": 0}, jobDetail: {"id":"HGgpA","status":"WAITING","submitTime":1564382864238,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\HGgpA","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 1, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.8993949196832431, \\"conv_size\\": 2, \\"hidden_size\\": 124, \\"batch_size\\": 8, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\HGgpA","sequenceId":1,"gpuIndices":[]}' ]
[7/29/2019, 12:47:48 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:47:49 AM] DEBUG [ 'trailJob status update: HGgpA, FAILED' ]
[7/29/2019, 12:47:49 AM] INFO [ 'Trial job HGgpA status changed from WAITING to FAILED' ]
[7/29/2019, 12:47:49 AM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"HGgpA","status":"FAILED","submitTime":1564382864238,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\HGgpA","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 1, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.8993949196832431, \\"conv_size\\": 2, \\"hidden_size\\": 124, \\"batch_size\\": 8, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\HGgpA","sequenceId":1,"gpuIndices":[],"startTime":1564382867346,"pid":32020,"endTime":1564382869824}' ]
[7/29/2019, 12:47:49 AM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"HGgpA","event":"FAILED","hyper_params":"{\\"parameter_id\\": 1, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.8993949196832431, \\"conv_size\\": 2, \\"hidden_size\\": 124, \\"batch_size\\": 8, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 12:47:49 AM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 12:47:49 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 2, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.8654507247351698, "conv_size": 2, "hidden_size": 512, "batch_size": 16, "learning_rate": 0.1}, "parameter_index": 0}' ]
[7/29/2019, 12:47:53 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:47:54 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 2, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.8654507247351698, \\"conv_size\\": 2, \\"hidden_size\\": 512, \\"batch_size\\": 16, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 12:47:54 AM] DEBUG [ 'submitTrialJob: return: {"id":"TjohE","status":"WAITING","submitTime":1564382874839,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\TjohE","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 2, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.8654507247351698, \\"conv_size\\": 2, \\"hidden_size\\": 512, \\"batch_size\\": 16, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\TjohE","sequenceId":2,"gpuIndices":[]} ' ]
[7/29/2019, 12:47:54 AM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 2, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.8654507247351698, "conv_size": 2, "hidden_size": 512, "batch_size": 16, "learning_rate": 0.1}, "parameter_index": 0}, jobDetail: {"id":"TjohE","status":"WAITING","submitTime":1564382874839,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\TjohE","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 2, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.8654507247351698, \\"conv_size\\": 2, \\"hidden_size\\": 512, \\"batch_size\\": 16, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\TjohE","sequenceId":2,"gpuIndices":[]}' ]
[7/29/2019, 12:47:58 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:48:00 AM] DEBUG [ 'trailJob status update: TjohE, FAILED' ]
[7/29/2019, 12:48:00 AM] INFO [ 'Trial job TjohE status changed from WAITING to FAILED' ]
[7/29/2019, 12:48:00 AM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"TjohE","status":"FAILED","submitTime":1564382874839,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\TjohE","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 2, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.8654507247351698, \\"conv_size\\": 2, \\"hidden_size\\": 512, \\"batch_size\\": 16, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\TjohE","sequenceId":2,"gpuIndices":[],"startTime":1564382879241,"pid":36024,"endTime":1564382880419}' ]
[7/29/2019, 12:48:00 AM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"TjohE","event":"FAILED","hyper_params":"{\\"parameter_id\\": 2, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.8654507247351698, \\"conv_size\\": 2, \\"hidden_size\\": 512, \\"batch_size\\": 16, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 12:48:00 AM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 12:48:00 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 3, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7766423718362717, "conv_size": 5, "hidden_size": 124, "batch_size": 32, "learning_rate": 0.001}, "parameter_index": 0}' ]
[7/29/2019, 12:48:03 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:48:05 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 3, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7766423718362717, \\"conv_size\\": 5, \\"hidden_size\\": 124, \\"batch_size\\": 32, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 12:48:05 AM] DEBUG [ 'submitTrialJob: return: {"id":"knUSL","status":"WAITING","submitTime":1564382885436,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\knUSL","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 3, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7766423718362717, \\"conv_size\\": 5, \\"hidden_size\\": 124, \\"batch_size\\": 32, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\knUSL","sequenceId":3,"gpuIndices":[]} ' ]
[7/29/2019, 12:48:05 AM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 3, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7766423718362717, "conv_size": 5, "hidden_size": 124, "batch_size": 32, "learning_rate": 0.001}, "parameter_index": 0}, jobDetail: {"id":"knUSL","status":"WAITING","submitTime":1564382885436,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\knUSL","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 3, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7766423718362717, \\"conv_size\\": 5, \\"hidden_size\\": 124, \\"batch_size\\": 32, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\knUSL","sequenceId":3,"gpuIndices":[]}' ]
[7/29/2019, 12:48:08 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:48:13 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:48:16 AM] DEBUG [ 'trailJob status update: knUSL, FAILED' ]
[7/29/2019, 12:48:16 AM] INFO [ 'Trial job knUSL status changed from WAITING to FAILED' ]
[7/29/2019, 12:48:16 AM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"knUSL","status":"FAILED","submitTime":1564382885436,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\knUSL","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 3, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7766423718362717, \\"conv_size\\": 5, \\"hidden_size\\": 124, \\"batch_size\\": 32, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\knUSL","sequenceId":3,"gpuIndices":[],"startTime":1564382891203,"pid":31676,"endTime":1564382896125}' ]
[7/29/2019, 12:48:16 AM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"knUSL","event":"FAILED","hyper_params":"{\\"parameter_id\\": 3, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7766423718362717, \\"conv_size\\": 5, \\"hidden_size\\": 124, \\"batch_size\\": 32, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 12:48:16 AM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 12:48:16 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 4, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.5424572727356373, "conv_size": 5, "hidden_size": 512, "batch_size": 16, "learning_rate": 0.01}, "parameter_index": 0}' ]
[7/29/2019, 12:48:18 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:48:21 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 4, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.5424572727356373, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 16, \\"learning_rate\\": 0.01}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 12:48:21 AM] DEBUG [ 'submitTrialJob: return: {"id":"n1HXB","status":"WAITING","submitTime":1564382901141,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\n1HXB","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 4, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.5424572727356373, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 16, \\"learning_rate\\": 0.01}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\n1HXB","sequenceId":4,"gpuIndices":[]} ' ]
[7/29/2019, 12:48:21 AM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 4, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.5424572727356373, "conv_size": 5, "hidden_size": 512, "batch_size": 16, "learning_rate": 0.01}, "parameter_index": 0}, jobDetail: {"id":"n1HXB","status":"WAITING","submitTime":1564382901141,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\n1HXB","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 4, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.5424572727356373, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 16, \\"learning_rate\\": 0.01}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\n1HXB","sequenceId":4,"gpuIndices":[]}' ]
[7/29/2019, 12:48:23 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:48:26 AM] DEBUG [ 'trailJob status update: n1HXB, FAILED' ]
[7/29/2019, 12:48:26 AM] INFO [ 'Trial job n1HXB status changed from WAITING to FAILED' ]
[7/29/2019, 12:48:26 AM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"n1HXB","status":"FAILED","submitTime":1564382901141,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\n1HXB","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 4, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.5424572727356373, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 16, \\"learning_rate\\": 0.01}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\n1HXB","sequenceId":4,"gpuIndices":[],"startTime":1564382903145,"pid":34220,"endTime":1564382906612}' ]
[7/29/2019, 12:48:26 AM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"n1HXB","event":"FAILED","hyper_params":"{\\"parameter_id\\": 4, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.5424572727356373, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 16, \\"learning_rate\\": 0.01}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 12:48:26 AM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 12:48:26 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 5, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7972814466096125, "conv_size": 7, "hidden_size": 124, "batch_size": 4, "learning_rate": 0.1}, "parameter_index": 0}' ]
[7/29/2019, 12:48:28 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:48:31 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 5, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7972814466096125, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 4, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 12:48:31 AM] DEBUG [ 'submitTrialJob: return: {"id":"m0NKL","status":"WAITING","submitTime":1564382911624,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\m0NKL","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 5, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7972814466096125, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 4, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\m0NKL","sequenceId":5,"gpuIndices":[]} ' ]
[7/29/2019, 12:48:31 AM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 5, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7972814466096125, "conv_size": 7, "hidden_size": 124, "batch_size": 4, "learning_rate": 0.1}, "parameter_index": 0}, jobDetail: {"id":"m0NKL","status":"WAITING","submitTime":1564382911624,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\m0NKL","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 5, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7972814466096125, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 4, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\m0NKL","sequenceId":5,"gpuIndices":[]}' ]
[7/29/2019, 12:48:33 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:48:37 AM] DEBUG [ 'trailJob status update: m0NKL, FAILED' ]
[7/29/2019, 12:48:37 AM] INFO [ 'Trial job m0NKL status changed from WAITING to FAILED' ]
[7/29/2019, 12:48:37 AM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"m0NKL","status":"FAILED","submitTime":1564382911624,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\m0NKL","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 5, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7972814466096125, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 4, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\m0NKL","sequenceId":5,"gpuIndices":[],"startTime":1564382915126,"pid":20080,"endTime":1564382917328}' ]
[7/29/2019, 12:48:37 AM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"m0NKL","event":"FAILED","hyper_params":"{\\"parameter_id\\": 5, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7972814466096125, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 4, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 12:48:37 AM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 12:48:37 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 6, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6800245119371473, "conv_size": 3, "hidden_size": 1024, "batch_size": 16, "learning_rate": 0.1}, "parameter_index": 0}' ]
[7/29/2019, 12:48:38 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:48:42 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 6, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6800245119371473, \\"conv_size\\": 3, \\"hidden_size\\": 1024, \\"batch_size\\": 16, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 12:48:42 AM] DEBUG [ 'submitTrialJob: return: {"id":"EdmO9","status":"WAITING","submitTime":1564382922342,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\EdmO9","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 6, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6800245119371473, \\"conv_size\\": 3, \\"hidden_size\\": 1024, \\"batch_size\\": 16, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\EdmO9","sequenceId":6,"gpuIndices":[]} ' ]
[7/29/2019, 12:48:42 AM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 6, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6800245119371473, "conv_size": 3, "hidden_size": 1024, "batch_size": 16, "learning_rate": 0.1}, "parameter_index": 0}, jobDetail: {"id":"EdmO9","status":"WAITING","submitTime":1564382922342,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\EdmO9","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 6, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6800245119371473, \\"conv_size\\": 3, \\"hidden_size\\": 1024, \\"batch_size\\": 16, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\EdmO9","sequenceId":6,"gpuIndices":[]}' ]
[7/29/2019, 12:48:43 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:48:47 AM] INFO [ 'Trial job EdmO9 status changed from WAITING to RUNNING' ]
[7/29/2019, 12:48:47 AM] DEBUG [ 'storeTrialJobEvent: event: RUNNING, data: undefined, jobDetail: {"id":"EdmO9","status":"RUNNING","submitTime":1564382922342,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\EdmO9","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 6, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6800245119371473, \\"conv_size\\": 3, \\"hidden_size\\": 1024, \\"batch_size\\": 16, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\EdmO9","sequenceId":6,"gpuIndices":[],"startTime":1564382926952,"pid":27136}' ]
[7/29/2019, 12:48:48 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:48:53 AM] DEBUG [ 'trailJob status update: EdmO9, FAILED' ]
[7/29/2019, 12:48:53 AM] INFO [ 'Trial job EdmO9 status changed from RUNNING to FAILED' ]
[7/29/2019, 12:48:53 AM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"EdmO9","status":"FAILED","submitTime":1564382922342,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\EdmO9","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 6, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6800245119371473, \\"conv_size\\": 3, \\"hidden_size\\": 1024, \\"batch_size\\": 16, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\EdmO9","sequenceId":6,"gpuIndices":[],"startTime":1564382926952,"pid":27136,"endTime":1564382933578}' ]
[7/29/2019, 12:48:53 AM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"EdmO9","event":"FAILED","hyper_params":"{\\"parameter_id\\": 6, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6800245119371473, \\"conv_size\\": 3, \\"hidden_size\\": 1024, \\"batch_size\\": 16, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 12:48:53 AM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 12:48:53 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 7, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.529345593174789, "conv_size": 7, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}' ]
[7/29/2019, 12:48:53 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:48:58 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 7, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.529345593174789, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 12:48:58 AM] DEBUG [ 'submitTrialJob: return: {"id":"jNswV","status":"WAITING","submitTime":1564382938592,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\jNswV","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 7, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.529345593174789, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\jNswV","sequenceId":7,"gpuIndices":[]} ' ]
[7/29/2019, 12:48:58 AM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 7, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.529345593174789, "conv_size": 7, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}, jobDetail: {"id":"jNswV","status":"WAITING","submitTime":1564382938592,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\jNswV","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 7, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.529345593174789, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\jNswV","sequenceId":7,"gpuIndices":[]}' ]
[7/29/2019, 12:48:58 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:49:04 AM] INFO [ 'Trial job jNswV status changed from WAITING to RUNNING' ]
[7/29/2019, 12:49:04 AM] DEBUG [ 'storeTrialJobEvent: event: RUNNING, data: undefined, jobDetail: {"id":"jNswV","status":"RUNNING","submitTime":1564382938592,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\jNswV","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 7, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.529345593174789, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\jNswV","sequenceId":7,"gpuIndices":[],"startTime":1564382943513,"pid":41388}' ]
[7/29/2019, 12:49:04 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:49:09 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:49:09 AM] DEBUG [ 'trailJob status update: jNswV, FAILED' ]
[7/29/2019, 12:49:09 AM] INFO [ 'Trial job jNswV status changed from RUNNING to FAILED' ]
[7/29/2019, 12:49:09 AM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"jNswV","status":"FAILED","submitTime":1564382938592,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\jNswV","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 7, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.529345593174789, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\jNswV","sequenceId":7,"gpuIndices":[],"startTime":1564382943513,"pid":41388,"endTime":1564382949785}' ]
[7/29/2019, 12:49:09 AM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"jNswV","event":"FAILED","hyper_params":"{\\"parameter_id\\": 7, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.529345593174789, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 12:49:09 AM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 12:49:09 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 8, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7487311518261832, "conv_size": 2, "hidden_size": 1024, "batch_size": 8, "learning_rate": 0.1}, "parameter_index": 0}' ]
[7/29/2019, 12:49:14 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:49:14 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 8, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7487311518261832, \\"conv_size\\": 2, \\"hidden_size\\": 1024, \\"batch_size\\": 8, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 12:49:14 AM] DEBUG [ 'submitTrialJob: return: {"id":"s1n9d","status":"WAITING","submitTime":1564382954795,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\s1n9d","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 8, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7487311518261832, \\"conv_size\\": 2, \\"hidden_size\\": 1024, \\"batch_size\\": 8, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\s1n9d","sequenceId":8,"gpuIndices":[]} ' ]
[7/29/2019, 12:49:14 AM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 8, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7487311518261832, "conv_size": 2, "hidden_size": 1024, "batch_size": 8, "learning_rate": 0.1}, "parameter_index": 0}, jobDetail: {"id":"s1n9d","status":"WAITING","submitTime":1564382954795,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\s1n9d","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 8, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7487311518261832, \\"conv_size\\": 2, \\"hidden_size\\": 1024, \\"batch_size\\": 8, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\s1n9d","sequenceId":8,"gpuIndices":[]}' ]
[7/29/2019, 12:49:19 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:49:24 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:49:25 AM] DEBUG [ 'trailJob status update: s1n9d, FAILED' ]
[7/29/2019, 12:49:25 AM] INFO [ 'Trial job s1n9d status changed from WAITING to FAILED' ]
[7/29/2019, 12:49:25 AM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"s1n9d","status":"FAILED","submitTime":1564382954795,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\s1n9d","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 8, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7487311518261832, \\"conv_size\\": 2, \\"hidden_size\\": 1024, \\"batch_size\\": 8, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\s1n9d","sequenceId":8,"gpuIndices":[],"startTime":1564382960284,"pid":24072,"endTime":1564382965293}' ]
[7/29/2019, 12:49:25 AM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"s1n9d","event":"FAILED","hyper_params":"{\\"parameter_id\\": 8, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7487311518261832, \\"conv_size\\": 2, \\"hidden_size\\": 1024, \\"batch_size\\": 8, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 12:49:25 AM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 12:49:25 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 9, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7014019414695803, "conv_size": 5, "hidden_size": 512, "batch_size": 4, "learning_rate": 0.1}, "parameter_index": 0}' ]
[7/29/2019, 12:49:29 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:49:30 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 9, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7014019414695803, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 4, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 12:49:30 AM] DEBUG [ 'submitTrialJob: return: {"id":"XdGZ9","status":"WAITING","submitTime":1564382970310,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\XdGZ9","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 9, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7014019414695803, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 4, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\XdGZ9","sequenceId":9,"gpuIndices":[]} ' ]
[7/29/2019, 12:49:30 AM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 9, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7014019414695803, "conv_size": 5, "hidden_size": 512, "batch_size": 4, "learning_rate": 0.1}, "parameter_index": 0}, jobDetail: {"id":"XdGZ9","status":"WAITING","submitTime":1564382970310,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\XdGZ9","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 9, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7014019414695803, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 4, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\XdGZ9","sequenceId":9,"gpuIndices":[]}' ]
[7/29/2019, 12:49:34 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:49:35 AM] INFO [ 'Change NNIManager status from: RUNNING to: NO_MORE_TRIAL' ]
[7/29/2019, 12:49:39 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:49:41 AM] DEBUG [ 'trailJob status update: XdGZ9, FAILED' ]
[7/29/2019, 12:49:41 AM] INFO [ 'Trial job XdGZ9 status changed from WAITING to FAILED' ]
[7/29/2019, 12:49:41 AM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"XdGZ9","status":"FAILED","submitTime":1564382970310,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\XdGZ9","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 9, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7014019414695803, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 4, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\XdGZ9","sequenceId":9,"gpuIndices":[],"startTime":1564382977209,"pid":13720,"endTime":1564382981001}' ]
[7/29/2019, 12:49:41 AM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"XdGZ9","event":"FAILED","hyper_params":"{\\"parameter_id\\": 9, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7014019414695803, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 4, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 12:49:41 AM] INFO [ 'Change NNIManager status from: NO_MORE_TRIAL to: DONE' ]
[7/29/2019, 12:49:41 AM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 12:49:41 AM] INFO [ 'Experiment done.' ]
[7/29/2019, 12:49:41 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 10, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.5040460409287476, "conv_size": 3, "hidden_size": 1024, "batch_size": 4, "learning_rate": 0.01}, "parameter_index": 0}' ]
[7/29/2019, 12:49:44 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:49:49 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:49:54 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:49:59 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:50:04 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:50:09 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:50:14 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:50:19 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:50:24 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:50:29 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:50:34 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:50:39 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:50:44 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]

dispatcher.logs
[07/29/2019, 12:47:23 AM] DEBUG (nni.main/MainThread) START
[07/29/2019, 12:47:24 AM] INFO (hyperopt.utils/MainThread) Failed to load dill, try installing dill via "pip install dill" for enhanced pickling support.
[07/29/2019, 12:47:24 AM] INFO (hyperopt.fmin/MainThread) Failed to load dill, try installing dill via "pip install dill" for enhanced pickling support.
[07/29/2019, 12:47:24 AM] DEBUG (nni.msg_dispatcher/MainThread) Assessor is not configured
[07/29/2019, 12:47:24 AM] INFO (nni.msg_dispatcher_base/MainThread) Start dispatcher
[07/29/2019, 12:47:24 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'IN000322']
[07/29/2019, 12:47:24 AM] DEBUG (nni.protocol/MainThread) Received command, data: [{"dropout_rate": {"_type": "uniform", "_value": [0.5, 0.9]}, "conv_size": {"_type": "choice", "_value": [2, 3, 5, 7]}, "hidden_size": {"_type": "choice", "_value": [124, 512, 1024]}, "batch_size": {"_type": "choice", "_value": [1, 4, 8, 16, 32]}, "learning_rate": {"_type": "choice", "_value": [0.0001, 0.001, 0.01, 0.1]}}]
[07/29/2019, 12:47:24 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:47:24 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:47:24 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Initialize], data: [OrderedDict([('dropout_rate', OrderedDict([('_type', 'uniform'), ('_value', [0.5, 0.9])])), ('conv_size', OrderedDict([('_type', 'choice'), ('_value', [2, 3, 5, 7])])), ('hidden_size', OrderedDict([('_type', 'choice'), ('_value', [124, 512, 1024])])), ('batch_size', OrderedDict([('_type', 'choice'), ('_value', [1, 4, 8, 16, 32])])), ('learning_rate', OrderedDict([('_type', 'choice'), ('_value', [0.0001, 0.001, 0.01, 0.1])]))])]
[07/29/2019, 12:47:24 AM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'ID000000']
[07/29/2019, 12:47:24 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:47:24 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 12:47:24 AM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 12:47:24 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 12:47:24 AM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [0]
[07/29/2019, 12:47:24 AM] DEBUG (nni.tuner/Thread-1) generating param for 0
[07/29/2019, 12:47:24 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.000999 seconds
[07/29/2019, 12:47:24 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 12:47:24 AM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000203{"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.5040585530428365, "conv_size": 5, "hidden_size": 512, "batch_size": 8, "learning_rate": 0.001}, "parameter_index": 0}']
[07/29/2019, 12:47:28 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:47:28 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:47:28 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:47:33 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:47:33 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:47:33 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:47:38 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:47:38 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:47:38 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:47:39 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000282']
[07/29/2019, 12:47:39 AM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"iz1GC","event":"FAILED","hyper_params":"{\"parameter_id\": 0, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.5040585530428365, \"conv_size\": 5, \"hidden_size\": 512, \"batch_size\": 8, \"learning_rate\": 0.001}, \"parameter_index\": 0}"}]
[07/29/2019, 12:47:39 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 12:47:39 AM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 12:47:39 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 12:47:39 AM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [1]
[07/29/2019, 12:47:39 AM] DEBUG (nni.tuner/Thread-1) generating param for 1
[07/29/2019, 12:47:39 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.004124 seconds
[07/29/2019, 12:47:39 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 12:47:39 AM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'iz1GC'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.5040585530428365, "conv_size": 5, "hidden_size": 512, "batch_size": 8, "learning_rate": 0.001}, "parameter_index": 0}')])]
[07/29/2019, 12:47:39 AM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000201{"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.8993949196832431, "conv_size": 2, "hidden_size": 124, "batch_size": 8, "learning_rate": 0.1}, "parameter_index": 0}']
[07/29/2019, 12:47:43 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:47:43 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:47:43 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:47:48 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:47:48 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:47:48 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:47:49 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000280']
[07/29/2019, 12:47:49 AM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"HGgpA","event":"FAILED","hyper_params":"{\"parameter_id\": 1, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.8993949196832431, \"conv_size\": 2, \"hidden_size\": 124, \"batch_size\": 8, \"learning_rate\": 0.1}, \"parameter_index\": 0}"}]
[07/29/2019, 12:47:49 AM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'HGgpA'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.8993949196832431, "conv_size": 2, "hidden_size": 124, "batch_size": 8, "learning_rate": 0.1}, "parameter_index": 0}')])]
[07/29/2019, 12:47:49 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 12:47:49 AM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 12:47:49 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 12:47:49 AM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [2]
[07/29/2019, 12:47:49 AM] DEBUG (nni.tuner/Thread-1) generating param for 2
[07/29/2019, 12:47:49 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.006025 seconds
[07/29/2019, 12:47:49 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 12:47:49 AM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000202{"parameter_id": 2, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.8654507247351698, "conv_size": 2, "hidden_size": 512, "batch_size": 16, "learning_rate": 0.1}, "parameter_index": 0}']
[07/29/2019, 12:47:53 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:47:53 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:47:53 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:47:58 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:47:58 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:47:58 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:48:00 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000281']
[07/29/2019, 12:48:00 AM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"TjohE","event":"FAILED","hyper_params":"{\"parameter_id\": 2, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.8654507247351698, \"conv_size\": 2, \"hidden_size\": 512, \"batch_size\": 16, \"learning_rate\": 0.1}, \"parameter_index\": 0}"}]
[07/29/2019, 12:48:00 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 12:48:00 AM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 12:48:00 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 12:48:00 AM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [3]
[07/29/2019, 12:48:00 AM] DEBUG (nni.tuner/Thread-1) generating param for 3
[07/29/2019, 12:48:00 AM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'TjohE'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 2, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.8654507247351698, "conv_size": 2, "hidden_size": 512, "batch_size": 16, "learning_rate": 0.1}, "parameter_index": 0}')])]
[07/29/2019, 12:48:00 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.006348 seconds
[07/29/2019, 12:48:00 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 12:48:00 AM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000204{"parameter_id": 3, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7766423718362717, "conv_size": 5, "hidden_size": 124, "batch_size": 32, "learning_rate": 0.001}, "parameter_index": 0}']
[07/29/2019, 12:48:03 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:48:03 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:48:03 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:48:08 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:48:08 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:48:08 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:48:13 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:48:13 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:48:13 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:48:16 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000283']
[07/29/2019, 12:48:16 AM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"knUSL","event":"FAILED","hyper_params":"{\"parameter_id\": 3, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.7766423718362717, \"conv_size\": 5, \"hidden_size\": 124, \"batch_size\": 32, \"learning_rate\": 0.001}, \"parameter_index\": 0}"}]
[07/29/2019, 12:48:16 AM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'knUSL'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 3, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7766423718362717, "conv_size": 5, "hidden_size": 124, "batch_size": 32, "learning_rate": 0.001}, "parameter_index": 0}')])]
[07/29/2019, 12:48:16 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 12:48:16 AM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 12:48:16 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 12:48:16 AM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [4]
[07/29/2019, 12:48:16 AM] DEBUG (nni.tuner/Thread-1) generating param for 4
[07/29/2019, 12:48:16 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.004289 seconds
[07/29/2019, 12:48:16 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 12:48:16 AM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000203{"parameter_id": 4, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.5424572727356373, "conv_size": 5, "hidden_size": 512, "batch_size": 16, "learning_rate": 0.01}, "parameter_index": 0}']
[07/29/2019, 12:48:18 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:48:18 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:48:18 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:48:23 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:48:23 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:48:23 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:48:26 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000282']
[07/29/2019, 12:48:26 AM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"n1HXB","event":"FAILED","hyper_params":"{\"parameter_id\": 4, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.5424572727356373, \"conv_size\": 5, \"hidden_size\": 512, \"batch_size\": 16, \"learning_rate\": 0.01}, \"parameter_index\": 0}"}]
[07/29/2019, 12:48:26 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 12:48:26 AM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 12:48:26 AM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'n1HXB'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 4, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.5424572727356373, "conv_size": 5, "hidden_size": 512, "batch_size": 16, "learning_rate": 0.01}, "parameter_index": 0}')])]
[07/29/2019, 12:48:26 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 12:48:26 AM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [5]
[07/29/2019, 12:48:26 AM] DEBUG (nni.tuner/Thread-1) generating param for 5
[07/29/2019, 12:48:26 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.003028 seconds
[07/29/2019, 12:48:26 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 12:48:26 AM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000201{"parameter_id": 5, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7972814466096125, "conv_size": 7, "hidden_size": 124, "batch_size": 4, "learning_rate": 0.1}, "parameter_index": 0}']
[07/29/2019, 12:48:28 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:48:28 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:48:28 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:48:33 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:48:33 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:48:33 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:48:37 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000280']
[07/29/2019, 12:48:37 AM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"m0NKL","event":"FAILED","hyper_params":"{\"parameter_id\": 5, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.7972814466096125, \"conv_size\": 7, \"hidden_size\": 124, \"batch_size\": 4, \"learning_rate\": 0.1}, \"parameter_index\": 0}"}]
[07/29/2019, 12:48:37 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 12:48:37 AM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 12:48:37 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 12:48:37 AM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [6]
[07/29/2019, 12:48:37 AM] DEBUG (nni.tuner/Thread-1) generating param for 6
[07/29/2019, 12:48:37 AM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'm0NKL'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 5, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7972814466096125, "conv_size": 7, "hidden_size": 124, "batch_size": 4, "learning_rate": 0.1}, "parameter_index": 0}')])]
[07/29/2019, 12:48:37 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.002901 seconds
[07/29/2019, 12:48:37 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 12:48:37 AM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000203{"parameter_id": 6, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6800245119371473, "conv_size": 3, "hidden_size": 1024, "batch_size": 16, "learning_rate": 0.1}, "parameter_index": 0}']
[07/29/2019, 12:48:38 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:48:38 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:48:38 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:48:43 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:48:43 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:48:43 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:48:48 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:48:48 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:48:48 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:48:53 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000282']
[07/29/2019, 12:48:53 AM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"EdmO9","event":"FAILED","hyper_params":"{\"parameter_id\": 6, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.6800245119371473, \"conv_size\": 3, \"hidden_size\": 1024, \"batch_size\": 16, \"learning_rate\": 0.1}, \"parameter_index\": 0}"}]
[07/29/2019, 12:48:53 AM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'EdmO9'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 6, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6800245119371473, "conv_size": 3, "hidden_size": 1024, "batch_size": 16, "learning_rate": 0.1}, "parameter_index": 0}')])]
[07/29/2019, 12:48:53 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 12:48:53 AM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 12:48:53 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 12:48:53 AM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [7]
[07/29/2019, 12:48:53 AM] DEBUG (nni.tuner/Thread-1) generating param for 7
[07/29/2019, 12:48:53 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.004421 seconds
[07/29/2019, 12:48:53 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 12:48:53 AM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000200{"parameter_id": 7, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.529345593174789, "conv_size": 7, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}']
[07/29/2019, 12:48:53 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:48:53 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:48:53 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:48:58 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:48:58 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:48:58 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:49:04 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:49:04 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:49:04 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:49:09 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:49:09 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:49:09 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:49:09 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000279']
[07/29/2019, 12:49:09 AM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"jNswV","event":"FAILED","hyper_params":"{\"parameter_id\": 7, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.529345593174789, \"conv_size\": 7, \"hidden_size\": 124, \"batch_size\": 1, \"learning_rate\": 0.1}, \"parameter_index\": 0}"}]
[07/29/2019, 12:49:09 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 12:49:09 AM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'jNswV'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 7, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.529345593174789, "conv_size": 7, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}')])]
[07/29/2019, 12:49:09 AM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 12:49:09 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 12:49:09 AM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [8]
[07/29/2019, 12:49:09 AM] DEBUG (nni.tuner/Thread-1) generating param for 8
[07/29/2019, 12:49:09 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.002992 seconds
[07/29/2019, 12:49:09 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 12:49:09 AM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000202{"parameter_id": 8, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7487311518261832, "conv_size": 2, "hidden_size": 1024, "batch_size": 8, "learning_rate": 0.1}, "parameter_index": 0}']
[07/29/2019, 12:49:14 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:49:14 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:49:14 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:49:19 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:49:19 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:49:19 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:49:24 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:49:24 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:49:24 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:49:25 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000281']
[07/29/2019, 12:49:25 AM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"s1n9d","event":"FAILED","hyper_params":"{\"parameter_id\": 8, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.7487311518261832, \"conv_size\": 2, \"hidden_size\": 1024, \"batch_size\": 8, \"learning_rate\": 0.1}, \"parameter_index\": 0}"}]
[07/29/2019, 12:49:25 AM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 's1n9d'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 8, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7487311518261832, "conv_size": 2, "hidden_size": 1024, "batch_size": 8, "learning_rate": 0.1}, "parameter_index": 0}')])]
[07/29/2019, 12:49:25 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 12:49:25 AM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 12:49:25 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 12:49:25 AM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [9]
[07/29/2019, 12:49:25 AM] DEBUG (nni.tuner/Thread-1) generating param for 9
[07/29/2019, 12:49:25 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.004995 seconds
[07/29/2019, 12:49:25 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 12:49:25 AM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000201{"parameter_id": 9, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7014019414695803, "conv_size": 5, "hidden_size": 512, "batch_size": 4, "learning_rate": 0.1}, "parameter_index": 0}']
[07/29/2019, 12:49:29 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:49:29 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:49:29 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:49:34 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:49:34 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:49:34 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:49:39 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:49:39 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:49:39 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:49:41 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000280']
[07/29/2019, 12:49:41 AM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"XdGZ9","event":"FAILED","hyper_params":"{\"parameter_id\": 9, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.7014019414695803, \"conv_size\": 5, \"hidden_size\": 512, \"batch_size\": 4, \"learning_rate\": 0.1}, \"parameter_index\": 0}"}]
[07/29/2019, 12:49:41 AM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'XdGZ9'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 9, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7014019414695803, "conv_size": 5, "hidden_size": 512, "batch_size": 4, "learning_rate": 0.1}, "parameter_index": 0}')])]
[07/29/2019, 12:49:41 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 12:49:41 AM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 12:49:41 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 12:49:41 AM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [10]
[07/29/2019, 12:49:41 AM] DEBUG (nni.tuner/Thread-1) generating param for 10
[07/29/2019, 12:49:41 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.005100 seconds
[07/29/2019, 12:49:41 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 12:49:41 AM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000204{"parameter_id": 10, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.5040460409287476, "conv_size": 3, "hidden_size": 1024, "batch_size": 4, "learning_rate": 0.01}, "parameter_index": 0}']
[07/29/2019, 12:49:44 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:49:44 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:49:44 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:49:49 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:49:49 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:49:49 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:49:54 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:49:54 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:49:54 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:49:59 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:49:59 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:49:59 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:50:04 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:50:04 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:50:04 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:50:09 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:50:09 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:50:09 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:50:14 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:50:14 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:50:14 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:50:19 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:50:19 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:50:19 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:50:24 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:50:24 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:50:24 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:50:29 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:50:29 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:50:29 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:50:34 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:50:34 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:50:34 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:50:39 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:50:39 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:50:39 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:50:44 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:50:44 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:50:44 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:50:49 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:50:49 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:50:49 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:50:54 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:50:54 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:50:54 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:50:59 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:50:59 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:50:59 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:51:04 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:51:04 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:51:04 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:51:09 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:51:09 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:51:09 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:51:14 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:51:14 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:51:14 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []

And how to run run.ps1
This is what comes when I try to run  run.ps1 from cmd :
cd C:\Users\User\nni\examples\trials\mnist\.
$env:NNI_PLATFORM="local"
$env:NNI_SYS_DIR="C:\Users\User\nni\experiments\n5ssDtas\trials\iz1GC"
$env:NNI_TRIAL_JOB_ID="iz1GC"
$env:NNI_OUTPUT_DIR="C:\Users\User\nni\experiments\n5ssDtas\trials\iz1GC"
$env:NNI_TRIAL_SEQ_ID="0"
$env:MULTI_PHASE="false"
$env:CUDA_VISIBLE_DEVICES="-1"
cmd /c python mnist.py 2>C:\Users\User\nni\experiments\n5ssDtas\trials\iz1GC\stderr
$NOW_DATE = [int64](([datetime]::UtcNow)-(get-date "1/1/1970")).TotalSeconds
$NOW_DATE = "$NOW_DATE" + (Get-Date -Format fff).ToString()
Write $LASTEXITCODE " " $NOW_DATE  | Out-File C:\Users\User\nni\experiments\n5ssDtas\trials\iz1GC\.nni\state -NoNewline -encoding utf8

@anandh02 It seems that you didn't run it. Please do it in your Powershell:
.\run.ps1

Make sure your virtualenv is activated beforehand if you are using them. Then check your output in console and whatever is in C:\Users\User\nni\experiments\n5ssDtas\trials\iz1GC\stderr.

@ultmaster
By running run.ps1
This was the output
#!/usr/bin/env bash
# Copyright (C) 2014, Alexey Pavlov
#<EMAIL_ADDRESS># This file is part of Minimal SYStem version 2.
#   https://sourceforge.net/p/msys2/wiki/MSYS2%20installation/
# File: cmd

"$COMSPEC" "$@"

and the directory changed to nni\examples\trials\mnist
How come running run.ps1 will run the entire program?
Since run.ps1 is in the folder nni\experiments\uH5ettE0\trials
where  \uH5ettE0 itself is generated after the running the program through nnictl.
Let me know if you're talking about some other run.ps1 file. Since I couldn't find anyother  run.ps1 in my NNI directory.

@anandh02 If you tried to access run.ps1 with a text editor, you will find that it's a script hat does some environment settings, some cds and a python execution.
Now I cannot tell whether your python is successfully executed or not, since you didn't provide me with C:\Users\User\nni\experiments\n5ssDtas\trials\iz1GC\stderr. Please provide me with that info.
If it's convenient for you, I'm also interested in the exit code of .\run.ps1, you can get that by searching "print exit code of last command" on Google and do whatever works for you.

@ultmaster
In  run.ps1 there is no python execution command. It just sets some env variable and changes the dir and writes today's date in the file.
To my understanding, any command (nnictl command) which will be running in cmd won't get affected because of this.
Still, I attached the screenshot of the execution of run.ps1

And if you look at the previous run.ps1 code I shared. There is already a command to print $LASTEXITCODE
And stderr is still empty.
Attaching the logs of nnimanager and dispatcher with debug mode.
dispatcher:
[07/29/2019, 10:04:33 PM] DEBUG (nni.main/MainThread) START
[07/29/2019, 10:04:34 PM] INFO (hyperopt.utils/MainThread) Failed to load dill, try installing dill via "pip install dill" for enhanced pickling support.
[07/29/2019, 10:04:34 PM] INFO (hyperopt.fmin/MainThread) Failed to load dill, try installing dill via "pip install dill" for enhanced pickling support.
[07/29/2019, 10:04:34 PM] DEBUG (nni.msg_dispatcher/MainThread) Assessor is not configured
[07/29/2019, 10:04:34 PM] INFO (nni.msg_dispatcher_base/MainThread) Start dispatcher
[07/29/2019, 10:04:34 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'IN000322']
[07/29/2019, 10:04:34 PM] DEBUG (nni.protocol/MainThread) Received command, data: [{"dropout_rate": {"_type": "uniform", "_value": [0.5, 0.9]}, "conv_size": {"_type": "choice", "_value": [2, 3, 5, 7]}, "hidden_size": {"_type": "choice", "_value": [124, 512, 1024]}, "batch_size": {"_type": "choice", "_value": [1, 4, 8, 16, 32]}, "learning_rate": {"_type": "choice", "_value": [0.0001, 0.001, 0.01, 0.1]}}]
[07/29/2019, 10:04:34 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:04:34 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:04:34 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Initialize], data: [OrderedDict([('dropout_rate', OrderedDict([('_type', 'uniform'), ('_value', [0.5, 0.9])])), ('conv_size', OrderedDict([('_type', 'choice'), ('_value', [2, 3, 5, 7])])), ('hidden_size', OrderedDict([('_type', 'choice'), ('_value', [124, 512, 1024])])), ('batch_size', OrderedDict([('_type', 'choice'), ('_value', [1, 4, 8, 16, 32])])), ('learning_rate', OrderedDict([('_type', 'choice'), ('_value', [0.0001, 0.001, 0.01, 0.1])]))])]
[07/29/2019, 10:04:34 PM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'ID000000']
[07/29/2019, 10:04:34 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:04:34 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 10:04:34 PM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 10:04:34 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 10:04:34 PM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [0]
[07/29/2019, 10:04:34 PM] DEBUG (nni.tuner/Thread-1) generating param for 0
[07/29/2019, 10:04:34 PM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.003999 seconds
[07/29/2019, 10:04:34 PM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 10:04:34 PM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000204{"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.8100726687947541, "conv_size": 5, "hidden_size": 512, "batch_size": 1, "learning_rate": 0.0001}, "parameter_index": 0}']
[07/29/2019, 10:04:37 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:04:37 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:04:37 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:04:42 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:04:42 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:04:42 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:04:47 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:04:47 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:04:47 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:04:48 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000283']
[07/29/2019, 10:04:48 PM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"R1gZq","event":"FAILED","hyper_params":"{\"parameter_id\": 0, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.8100726687947541, \"conv_size\": 5, \"hidden_size\": 512, \"batch_size\": 1, \"learning_rate\": 0.0001}, \"parameter_index\": 0}"}]
[07/29/2019, 10:04:48 PM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'R1gZq'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.8100726687947541, "conv_size": 5, "hidden_size": 512, "batch_size": 1, "learning_rate": 0.0001}, "parameter_index": 0}')])]
[07/29/2019, 10:04:48 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 10:04:48 PM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 10:04:48 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 10:04:48 PM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [1]
[07/29/2019, 10:04:48 PM] DEBUG (nni.tuner/Thread-1) generating param for 1
[07/29/2019, 10:04:48 PM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.006125 seconds
[07/29/2019, 10:04:48 PM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 10:04:48 PM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000202{"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.846059812817086, "conv_size": 2, "hidden_size": 124, "batch_size": 4, "learning_rate": 0.001}, "parameter_index": 0}']
[07/29/2019, 10:04:52 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:04:52 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:04:52 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:04:57 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:04:57 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:04:57 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:05:02 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:05:02 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:05:02 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:05:03 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000281']
[07/29/2019, 10:05:03 PM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"sZ8Br","event":"FAILED","hyper_params":"{\"parameter_id\": 1, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.846059812817086, \"conv_size\": 2, \"hidden_size\": 124, \"batch_size\": 4, \"learning_rate\": 0.001}, \"parameter_index\": 0}"}]
[07/29/2019, 10:05:03 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 10:05:03 PM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 10:05:03 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 10:05:03 PM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [2]
[07/29/2019, 10:05:03 PM] DEBUG (nni.tuner/Thread-1) generating param for 2
[07/29/2019, 10:05:03 PM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'sZ8Br'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.846059812817086, "conv_size": 2, "hidden_size": 124, "batch_size": 4, "learning_rate": 0.001}, "parameter_index": 0}')])]
[07/29/2019, 10:05:03 PM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.010331 seconds
[07/29/2019, 10:05:03 PM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 10:05:03 PM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000202{"parameter_id": 2, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7355670839028239, "conv_size": 5, "hidden_size": 124, "batch_size": 8, "learning_rate": 0.01}, "parameter_index": 0}']
[07/29/2019, 10:05:07 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:05:07 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:05:07 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:05:12 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:05:12 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:05:12 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:05:14 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000281']
[07/29/2019, 10:05:14 PM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"WRRgp","event":"FAILED","hyper_params":"{\"parameter_id\": 2, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.7355670839028239, \"conv_size\": 5, \"hidden_size\": 124, \"batch_size\": 8, \"learning_rate\": 0.01}, \"parameter_index\": 0}"}]
[07/29/2019, 10:05:14 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 10:05:14 PM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 10:05:14 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 10:05:14 PM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [3]
[07/29/2019, 10:05:14 PM] DEBUG (nni.tuner/Thread-1) generating param for 3
[07/29/2019, 10:05:14 PM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.004090 seconds
[07/29/2019, 10:05:14 PM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 10:05:14 PM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000202{"parameter_id": 3, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6466162689528365, "conv_size": 5, "hidden_size": 124, "batch_size": 4, "learning_rate": 0.01}, "parameter_index": 0}']
[07/29/2019, 10:05:14 PM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'WRRgp'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 2, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7355670839028239, "conv_size": 5, "hidden_size": 124, "batch_size": 8, "learning_rate": 0.01}, "parameter_index": 0}')])]
[07/29/2019, 10:05:17 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:05:17 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:05:17 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:05:22 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:05:22 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:05:22 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:05:25 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000281']
[07/29/2019, 10:05:25 PM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"aKOrH","event":"FAILED","hyper_params":"{\"parameter_id\": 3, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.6466162689528365, \"conv_size\": 5, \"hidden_size\": 124, \"batch_size\": 4, \"learning_rate\": 0.01}, \"parameter_index\": 0}"}]
[07/29/2019, 10:05:25 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 10:05:25 PM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 10:05:25 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 10:05:25 PM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [4]
[07/29/2019, 10:05:25 PM] DEBUG (nni.tuner/Thread-1) generating param for 4
[07/29/2019, 10:05:25 PM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.004069 seconds
[07/29/2019, 10:05:25 PM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'aKOrH'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 3, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6466162689528365, "conv_size": 5, "hidden_size": 124, "batch_size": 4, "learning_rate": 0.01}, "parameter_index": 0}')])]
[07/29/2019, 10:05:25 PM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 10:05:25 PM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000201{"parameter_id": 4, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7149598193386633, "conv_size": 7, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}']
[07/29/2019, 10:05:27 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:05:27 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:05:27 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:05:32 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:05:32 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:05:32 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:05:37 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:05:37 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:05:37 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:05:40 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000280']
[07/29/2019, 10:05:40 PM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"LRc4Z","event":"FAILED","hyper_params":"{\"parameter_id\": 4, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.7149598193386633, \"conv_size\": 7, \"hidden_size\": 124, \"batch_size\": 1, \"learning_rate\": 0.1}, \"parameter_index\": 0}"}]
[07/29/2019, 10:05:40 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 10:05:40 PM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 10:05:40 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 10:05:40 PM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [5]
[07/29/2019, 10:05:40 PM] DEBUG (nni.tuner/Thread-1) generating param for 5
[07/29/2019, 10:05:40 PM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.005306 seconds
[07/29/2019, 10:05:40 PM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 10:05:40 PM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000201{"parameter_id": 5, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6649355552943245, "conv_size": 2, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}']
[07/29/2019, 10:05:40 PM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'LRc4Z'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 4, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7149598193386633, "conv_size": 7, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}')])]
[07/29/2019, 10:05:42 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:05:42 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:05:42 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:05:47 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:05:47 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:05:47 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:05:52 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:05:52 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:05:52 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:05:56 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000280']
[07/29/2019, 10:05:56 PM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"kDoG9","event":"FAILED","hyper_params":"{\"parameter_id\": 5, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.6649355552943245, \"conv_size\": 2, \"hidden_size\": 124, \"batch_size\": 1, \"learning_rate\": 0.1}, \"parameter_index\": 0}"}]
[07/29/2019, 10:05:56 PM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'kDoG9'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 5, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6649355552943245, "conv_size": 2, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}')])]
[07/29/2019, 10:05:56 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 10:05:56 PM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 10:05:56 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 10:05:56 PM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [6]
[07/29/2019, 10:05:56 PM] DEBUG (nni.tuner/Thread-1) generating param for 6
[07/29/2019, 10:05:56 PM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.002995 seconds
[07/29/2019, 10:05:56 PM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 10:05:56 PM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000201{"parameter_id": 6, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6894522330563883, "conv_size": 7, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}']
[07/29/2019, 10:05:57 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:05:57 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:05:57 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:06:02 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:06:02 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:06:02 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:06:07 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000280']
[07/29/2019, 10:06:07 PM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"EOV8K","event":"FAILED","hyper_params":"{\"parameter_id\": 6, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.6894522330563883, \"conv_size\": 7, \"hidden_size\": 124, \"batch_size\": 1, \"learning_rate\": 0.1}, \"parameter_index\": 0}"}]
[07/29/2019, 10:06:07 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 10:06:07 PM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 10:06:07 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 10:06:07 PM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [7]
[07/29/2019, 10:06:07 PM] DEBUG (nni.tuner/Thread-1) generating param for 7
[07/29/2019, 10:06:07 PM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.004056 seconds
[07/29/2019, 10:06:07 PM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 10:06:07 PM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000202{"parameter_id": 7, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.5621371754023703, "conv_size": 7, "hidden_size": 512, "batch_size": 32, "learning_rate": 0.1}, "parameter_index": 0}']