Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OutOfRangeError (see above for traceback): RandomShuffleQueue '_2_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 1, current size 0) #5

Open
ghost opened this issue Mar 29, 2019 · 10 comments

Comments

@ghost
Copy link

ghost commented Mar 29, 2019

Hi !

At first, sorry for that I uploaded this problem as 'requests' at noon, I didn't notice the mistake when I wrote it.

I want to run this code, train it and see the results, but when I start training , the code will automatically quit after this:
OutOfRangeError (see above for traceback): RandomShuffleQueue '_2_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 1, current size 0)
[[Node: shuffle_batch = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]
How could this happen and how to do?

@ambarishgurjar
Copy link

Even I got the same error

python -m cyclegan.main --to_train=1 --log_dir=cyclegan/output/cyclegan/exp_01 --config_filename=cyclegan/configs/exp_01.json
WARNING:tensorflow:From /home/deependra/GANresearch/cyclegan/main.py:93: get_or_create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.get_or_create_global_step
Model/d_A/c1/Conv/weights:0
Model/d_A/c1/Conv/biases:0
Model/d_A/c2/Conv/weights:0
Model/d_A/c2/Conv/biases:0
Model/d_A/c2/instance_norm/scale:0
Model/d_A/c2/instance_norm/offset:0
Model/d_A/c3/Conv/weights:0
Model/d_A/c3/Conv/biases:0
Model/d_A/c3/instance_norm/scale:0
Model/d_A/c3/instance_norm/offset:0
Model/d_A/c4/Conv/weights:0
Model/d_A/c4/Conv/biases:0
Model/d_A/c4/instance_norm/scale:0
Model/d_A/c4/instance_norm/offset:0
Model/d_A/c5/Conv/weights:0
Model/d_A/c5/Conv/biases:0
Model/d_B/c1/Conv/weights:0
Model/d_B/c1/Conv/biases:0
Model/d_B/c2/Conv/weights:0
Model/d_B/c2/Conv/biases:0
Model/d_B/c2/instance_norm/scale:0
Model/d_B/c2/instance_norm/offset:0
Model/d_B/c3/Conv/weights:0
Model/d_B/c3/Conv/biases:0
Model/d_B/c3/instance_norm/scale:0
Model/d_B/c3/instance_norm/offset:0
Model/d_B/c4/Conv/weights:0
Model/d_B/c4/Conv/biases:0
Model/d_B/c4/instance_norm/scale:0
Model/d_B/c4/instance_norm/offset:0
Model/d_B/c5/Conv/weights:0
Model/d_B/c5/Conv/biases:0
Model/g_A/c1/Conv/weights:0
Model/g_A/c1/Conv/biases:0
Model/g_A/c1/instance_norm/scale:0
Model/g_A/c1/instance_norm/offset:0
Model/g_A/c2/Conv/weights:0
Model/g_A/c2/Conv/biases:0
Model/g_A/c2/instance_norm/scale:0
Model/g_A/c2/instance_norm/offset:0
Model/g_A/c3/Conv/weights:0
Model/g_A/c3/Conv/biases:0
Model/g_A/c3/instance_norm/scale:0
Model/g_A/c3/instance_norm/offset:0
Model/g_A/r1/c1/Conv/weights:0
Model/g_A/r1/c1/Conv/biases:0
Model/g_A/r1/c1/instance_norm/scale:0
Model/g_A/r1/c1/instance_norm/offset:0
Model/g_A/r1/c2/Conv/weights:0
Model/g_A/r1/c2/Conv/biases:0
Model/g_A/r1/c2/instance_norm/scale:0
Model/g_A/r1/c2/instance_norm/offset:0
Model/g_A/r2/c1/Conv/weights:0
Model/g_A/r2/c1/Conv/biases:0
Model/g_A/r2/c1/instance_norm/scale:0
Model/g_A/r2/c1/instance_norm/offset:0
Model/g_A/r2/c2/Conv/weights:0
Model/g_A/r2/c2/Conv/biases:0
Model/g_A/r2/c2/instance_norm/scale:0
Model/g_A/r2/c2/instance_norm/offset:0
Model/g_A/r3/c1/Conv/weights:0
Model/g_A/r3/c1/Conv/biases:0
Model/g_A/r3/c1/instance_norm/scale:0
Model/g_A/r3/c1/instance_norm/offset:0
Model/g_A/r3/c2/Conv/weights:0
Model/g_A/r3/c2/Conv/biases:0
Model/g_A/r3/c2/instance_norm/scale:0
Model/g_A/r3/c2/instance_norm/offset:0
Model/g_A/r4/c1/Conv/weights:0
Model/g_A/r4/c1/Conv/biases:0
Model/g_A/r4/c1/instance_norm/scale:0
Model/g_A/r4/c1/instance_norm/offset:0
Model/g_A/r4/c2/Conv/weights:0
Model/g_A/r4/c2/Conv/biases:0
Model/g_A/r4/c2/instance_norm/scale:0
Model/g_A/r4/c2/instance_norm/offset:0
Model/g_A/r5/c1/Conv/weights:0
Model/g_A/r5/c1/Conv/biases:0
Model/g_A/r5/c1/instance_norm/scale:0
Model/g_A/r5/c1/instance_norm/offset:0
Model/g_A/r5/c2/Conv/weights:0
Model/g_A/r5/c2/Conv/biases:0
Model/g_A/r5/c2/instance_norm/scale:0
Model/g_A/r5/c2/instance_norm/offset:0
Model/g_A/r6/c1/Conv/weights:0
Model/g_A/r6/c1/Conv/biases:0
Model/g_A/r6/c1/instance_norm/scale:0
Model/g_A/r6/c1/instance_norm/offset:0
Model/g_A/r6/c2/Conv/weights:0
Model/g_A/r6/c2/Conv/biases:0
Model/g_A/r6/c2/instance_norm/scale:0
Model/g_A/r6/c2/instance_norm/offset:0
Model/g_A/r7/c1/Conv/weights:0
Model/g_A/r7/c1/Conv/biases:0
Model/g_A/r7/c1/instance_norm/scale:0
Model/g_A/r7/c1/instance_norm/offset:0
Model/g_A/r7/c2/Conv/weights:0
Model/g_A/r7/c2/Conv/biases:0
Model/g_A/r7/c2/instance_norm/scale:0
Model/g_A/r7/c2/instance_norm/offset:0
Model/g_A/r8/c1/Conv/weights:0
Model/g_A/r8/c1/Conv/biases:0
Model/g_A/r8/c1/instance_norm/scale:0
Model/g_A/r8/c1/instance_norm/offset:0
Model/g_A/r8/c2/Conv/weights:0
Model/g_A/r8/c2/Conv/biases:0
Model/g_A/r8/c2/instance_norm/scale:0
Model/g_A/r8/c2/instance_norm/offset:0
Model/g_A/r9/c1/Conv/weights:0
Model/g_A/r9/c1/Conv/biases:0
Model/g_A/r9/c1/instance_norm/scale:0
Model/g_A/r9/c1/instance_norm/offset:0
Model/g_A/r9/c2/Conv/weights:0
Model/g_A/r9/c2/Conv/biases:0
Model/g_A/r9/c2/instance_norm/scale:0
Model/g_A/r9/c2/instance_norm/offset:0
Model/g_A/c4/Conv2d_transpose/weights:0
Model/g_A/c4/Conv2d_transpose/biases:0
Model/g_A/c4/instance_norm/scale:0
Model/g_A/c4/instance_norm/offset:0
Model/g_A/c5/Conv2d_transpose/weights:0
Model/g_A/c5/Conv2d_transpose/biases:0
Model/g_A/c5/instance_norm/scale:0
Model/g_A/c5/instance_norm/offset:0
Model/g_A/c6/Conv/weights:0
Model/g_A/c6/Conv/biases:0
Model/g_B/c1/Conv/weights:0
Model/g_B/c1/Conv/biases:0
Model/g_B/c1/instance_norm/scale:0
Model/g_B/c1/instance_norm/offset:0
Model/g_B/c2/Conv/weights:0
Model/g_B/c2/Conv/biases:0
Model/g_B/c2/instance_norm/scale:0
Model/g_B/c2/instance_norm/offset:0
Model/g_B/c3/Conv/weights:0
Model/g_B/c3/Conv/biases:0
Model/g_B/c3/instance_norm/scale:0
Model/g_B/c3/instance_norm/offset:0
Model/g_B/r1/c1/Conv/weights:0
Model/g_B/r1/c1/Conv/biases:0
Model/g_B/r1/c1/instance_norm/scale:0
Model/g_B/r1/c1/instance_norm/offset:0
Model/g_B/r1/c2/Conv/weights:0
Model/g_B/r1/c2/Conv/biases:0
Model/g_B/r1/c2/instance_norm/scale:0
Model/g_B/r1/c2/instance_norm/offset:0
Model/g_B/r2/c1/Conv/weights:0
Model/g_B/r2/c1/Conv/biases:0
Model/g_B/r2/c1/instance_norm/scale:0
Model/g_B/r2/c1/instance_norm/offset:0
Model/g_B/r2/c2/Conv/weights:0
Model/g_B/r2/c2/Conv/biases:0
Model/g_B/r2/c2/instance_norm/scale:0
Model/g_B/r2/c2/instance_norm/offset:0
Model/g_B/r3/c1/Conv/weights:0
Model/g_B/r3/c1/Conv/biases:0
Model/g_B/r3/c1/instance_norm/scale:0
Model/g_B/r3/c1/instance_norm/offset:0
Model/g_B/r3/c2/Conv/weights:0
Model/g_B/r3/c2/Conv/biases:0
Model/g_B/r3/c2/instance_norm/scale:0
Model/g_B/r3/c2/instance_norm/offset:0
Model/g_B/r4/c1/Conv/weights:0
Model/g_B/r4/c1/Conv/biases:0
Model/g_B/r4/c1/instance_norm/scale:0
Model/g_B/r4/c1/instance_norm/offset:0
Model/g_B/r4/c2/Conv/weights:0
Model/g_B/r4/c2/Conv/biases:0
Model/g_B/r4/c2/instance_norm/scale:0
Model/g_B/r4/c2/instance_norm/offset:0
Model/g_B/r5/c1/Conv/weights:0
Model/g_B/r5/c1/Conv/biases:0
Model/g_B/r5/c1/instance_norm/scale:0
Model/g_B/r5/c1/instance_norm/offset:0
Model/g_B/r5/c2/Conv/weights:0
Model/g_B/r5/c2/Conv/biases:0
Model/g_B/r5/c2/instance_norm/scale:0
Model/g_B/r5/c2/instance_norm/offset:0
Model/g_B/r6/c1/Conv/weights:0
Model/g_B/r6/c1/Conv/biases:0
Model/g_B/r6/c1/instance_norm/scale:0
Model/g_B/r6/c1/instance_norm/offset:0
Model/g_B/r6/c2/Conv/weights:0
Model/g_B/r6/c2/Conv/biases:0
Model/g_B/r6/c2/instance_norm/scale:0
Model/g_B/r6/c2/instance_norm/offset:0
Model/g_B/r7/c1/Conv/weights:0
Model/g_B/r7/c1/Conv/biases:0
Model/g_B/r7/c1/instance_norm/scale:0
Model/g_B/r7/c1/instance_norm/offset:0
Model/g_B/r7/c2/Conv/weights:0
Model/g_B/r7/c2/Conv/biases:0
Model/g_B/r7/c2/instance_norm/scale:0
Model/g_B/r7/c2/instance_norm/offset:0
Model/g_B/r8/c1/Conv/weights:0
Model/g_B/r8/c1/Conv/biases:0
Model/g_B/r8/c1/instance_norm/scale:0
Model/g_B/r8/c1/instance_norm/offset:0
Model/g_B/r8/c2/Conv/weights:0
Model/g_B/r8/c2/Conv/biases:0
Model/g_B/r8/c2/instance_norm/scale:0
Model/g_B/r8/c2/instance_norm/offset:0
Model/g_B/r9/c1/Conv/weights:0
Model/g_B/r9/c1/Conv/biases:0
Model/g_B/r9/c1/instance_norm/scale:0
Model/g_B/r9/c1/instance_norm/offset:0
Model/g_B/r9/c2/Conv/weights:0
Model/g_B/r9/c2/Conv/biases:0
Model/g_B/r9/c2/instance_norm/scale:0
Model/g_B/r9/c2/instance_norm/offset:0
Model/g_B/c4/Conv2d_transpose/weights:0
Model/g_B/c4/Conv2d_transpose/biases:0
Model/g_B/c4/instance_norm/scale:0
Model/g_B/c4/instance_norm/offset:0
Model/g_B/c5/Conv2d_transpose/weights:0
Model/g_B/c5/Conv2d_transpose/biases:0
Model/g_B/c5/instance_norm/scale:0
Model/g_B/c5/instance_norm/offset:0
Model/g_B/c6/Conv/weights:0
Model/g_B/c6/Conv/biases:0
2019-03-30 14:58:39.544914: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2
('In the epoch ', 0)
Saving image 0/20
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/deependra/GANresearch/cyclegan/main.py", line 458, in
main()
File "/home/deependra/.local/lib/python2.7/site-packages/click/core.py", line 764, in call
return self.main(*args, **kwargs)
File "/home/deependra/.local/lib/python2.7/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/home/deependra/.local/lib/python2.7/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/deependra/.local/lib/python2.7/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/home/deependra/GANresearch/cyclegan/main.py", line 452, in main
cyclegan_model.train()
File "/home/deependra/GANresearch/cyclegan/main.py", line 293, in train
self.save_images(sess, epoch)
File "/home/deependra/GANresearch/cyclegan/main.py", line 198, in save_images
inputs = sess.run(self.inputs)
File "/home/deependra/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 895, in run
run_metadata_ptr)
File "/home/deependra/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1128, in _run
feed_dict_tensor, options, run_metadata)
File "/home/deependra/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1344, in _do_run
options, run_metadata)
File "/home/deependra/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1363, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: RandomShuffleQueue '_1_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 1, current size 0)
[[Node: shuffle_batch = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]

Caused by op u'shuffle_batch', defined at:
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/deependra/GANresearch/cyclegan/main.py", line 458, in
main()
File "/home/deependra/.local/lib/python2.7/site-packages/click/core.py", line 764, in call
return self.main(*args, **kwargs)
File "/home/deependra/.local/lib/python2.7/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/home/deependra/.local/lib/python2.7/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/deependra/.local/lib/python2.7/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/home/deependra/GANresearch/cyclegan/main.py", line 452, in main
cyclegan_model.train()
File "/home/deependra/GANresearch/cyclegan/main.py", line 249, in train
True, self._do_flipping)
File "cyclegan/data_loader.py", line 80, in load_data
[inputs['image_i'], inputs['image_j']], 1, 5000, 100)
File "/home/deependra/.local/lib/python2.7/site-packages/tensorflow/python/training/input.py", line 1287, in shuffle_batch
name=name)
File "/home/deependra/.local/lib/python2.7/site-packages/tensorflow/python/training/input.py", line 838, in _shuffle_batch
dequeued = queue.dequeue_many(batch_size, name=name)
File "/home/deependra/.local/lib/python2.7/site-packages/tensorflow/python/ops/data_flow_ops.py", line 475, in dequeue_many
self._queue_ref, n=n, component_types=self._dtypes, name=name)
File "/home/deependra/.local/lib/python2.7/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 2445, in _queue_dequeue_many_v2
component_types=component_types, timeout_ms=timeout_ms, name=name)
File "/home/deependra/.local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/deependra/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3160, in create_op
op_def=op_def)
File "/home/deependra/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1625, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

OutOfRangeError (see above for traceback): RandomShuffleQueue '_1_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 1, current size 0)
[[Node: shuffle_batch = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]

@ghost
Copy link
Author

ghost commented Mar 31, 2019

@ambarishgurjar Same to you. Have you deal with it?

@ambarishgurjar
Copy link

It was really frustrating, I tried downgrading the tensorflow version and it still didn't work.

Finally I downloaded another implementation of the same project

https://github.com/architrathore/CycleGAN/

It is the same algorithm but slightly simplified.

You may encounter some problems in this one too
For that you will need to change size() to size at lines 99 and 104 I think if you get some error of int not being a callable object. And secondly you will need to initialize both global and local variables before the TF session.run(init). Lastly you might need to replace the download_datasets.sh file by the file in this repository. It will then work.

If you need the exact changes in code then I will give it to you once I go to my office on Monday. All my data is stored in the work PC

@ghost
Copy link
Author

ghost commented Mar 31, 2019

@ambarishgurjar Thanks a lot !!

@ghost ghost mentioned this issue Apr 8, 2019
@FTC55
Copy link

FTC55 commented Jun 2, 2019

I'm having the same type of error even without shuffling. Problem is i really need this specific implementation because I'm trying to run an implementation that adds attention capabilities to the GAN and is based off of this one. I was hoping to solve with a downgrade, but from what I gain it doesn't work. How far did you rollback? I was thinking to try tf 0.8.

@FTC55
Copy link

FTC55 commented Jun 3, 2019

I solved the issue. "create_cyclegan_dataset.py" creates .csv files that end with this sequence of characters to separate lines : "\r\r\n" and that causes errors during parsing that lead to files not being loaded at all thus making the queues empty. If this sequence of characters is replaced with a simple "\n" character in any text editor, the csv is parsed correctly and everything is back to working order.

@heleibin
Copy link

@ambarishgurjar Hi,I have met the same issue as you
OutOfRangeError (see above for traceback): RandomShuffleQueue '_1_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 1, current size 0)
[[node shuffle_batch (defined at E:\Python exercise\generalizaton\domain\CycleGAN-1-master\data_loader.py:76) ]]

Have you figured it out ?I am really confused,thanks a lot

@FTC55
Copy link

FTC55 commented Dec 24, 2019

Hey @heleibin have you tried removing the characters at the end of the line? It worked for me as i said up there in my last comment.

@heleibin
Copy link

@FTC55 ,Hey, thanks for your answer,but i had tried as you said,but it didnot work.
I then tried another version tensorflow implementation of CycleGAN,it worked perfectly!

Thanks for your advice!

@ambarishgurjar
Copy link

ambarishgurjar commented Dec 25, 2019 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants