InvalidArgumentError when using of tensorflow.data.Dataset.from_generator

Question

I'm trying to generate my own image dataset to measure inference performance using Tensorflow Dataset API on single GPU

resolutions = [
    (2048, 1080)
]

def generate_image(size, channels):
    image_value = random.random()
    image_shape = [1, size[1], size[0], channels]
    return tf.constant(
        value=image_value,
        shape=image_shape,
        dtype=tf.float32)

def generate_single_input(size):
    source = generate_image(size, 3)
    target = generate_image(size, 3)
    return source, target

def input_generator_fn():
    for res in resolutions:
        for i in range(10):
            yield generate_single_input(res)


def benchmark():
    ...
    ds = tf.data.Dataset.from_generator(
        generator=input_generator_fn,
        output_types=(tf.float32, tf.float32),
        output_shapes=(tf.TensorShape([1, 1080, 2048, 3]),
                       tf.TensorShape([1, 1080, 2048, 3])))
    iterator = ds.make_one_shot_iterator()
    next_record = iterator.get_next()

    inputs = next_record[0]
    outputs = next_record[1]

    predictions = {
        'input_images': inputs
        'output_images': outputs
    }
    session = tf.Session()
    with session:
        tf.global_variables_initializer()
        for res in resolutions:
           for i in range(10):
               session.run(predictions)
               .....

But I'm observing the following exception after running:

2018-04-06 13:38:44.050448: W tensorflow/core/framework/op_kernel.cc:1198] Invalid argument: ValueError: setting an array element with a sequence.

2018-04-06 13:38:44.050581: W tensorflow/core/framework/op_kernel.cc:1198]   Invalid argument: ValueError: setting an array element with a sequence.
     [[Node: PyFunc = PyFunc[Tin=[DT_INT64], Tout=[DT_FLOAT, DT_FLOAT], token="pyfunc_1"](arg0)]]

Traceback (most recent call last):
File "tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1350, in _do_call
    return fn(*args)

File "tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1329, in _run_fn
    status, run_metadata)

File "tensorflow/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in __exit__
    c_api.TF_GetCode(self.status.status))
    tensorflow.python.framework.errors_impl.InvalidArgumentError: ValueError: setting an array element with a sequence.
     [[Node: PyFunc = PyFunc[Tin=[DT_INT64], Tout=[DT_FLOAT, DT_FLOAT], token="pyfunc_1"](arg0)]]
     [[Node: IteratorGetNext = IteratorGetNext[output_shapes=[[1,1080,2048,3], [1,1080,2048,3]], output_types=[DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](OneShotIterator)]]

Run into the same problem while using this .from_generator(), The reason is unknown to me. Does anyone have possible solution to this? — Li haonan
– Li haonan, Commented Aug 24, 2018 at 7:13

youwei · Accepted Answer · 2019-06-18 04:38:24Z

In short, the reason is that from_generator can flatten a NumPy array but not a Tensor.

Here is a shorter code that will reproduce the error:

import tensorflow as tf
import numpy as np

print(tf.__version__)
def g():
  img = tf.random_uniform([3])
  # img = np.random.rand(3)
  # img = tf.convert_to_tensor(img)
  yield img

dataset = tf.data.Dataset.from_generator(g, tf.float64, tf.TensorShape([3]))
iterator = dataset.make_one_shot_iterator()
next_iterator = iterator.get_next()

sess = tf.Session()
sess.run(next_iterator)

The error message in version 1.14 is very helpful. (The exact line of code will change due to different versions, but I have checked 1.12 and 1.13 I am using the cause is the same.)

InvalidArgumentError: TypeError: `generator` yielded an element that could not be converted to the expected type. The expected type was float64, but the yielded element was Tensor("random_uniform:0", shape=(3,), dtype=float32).
Traceback (most recent call last):

  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/dataset_ops.py", line 530, in generator_py_func
    ret, dtype=dtype.as_numpy_dtype))

  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 169, in _convert
    result = np.asarray(value, dtype=dtype, order="C")

  File "/usr/local/lib/python3.6/dist-packages/numpy/core/numeric.py", line 538, in asarray
    return array(a, dtype, copy=False, order=order)

ValueError: setting an array element with a sequence.

When the generated element is a Tensor, from_generator will flatten it to output_types. The convert function does not work.

To solve this problem, just dont use from_generator when your generator generates a tensor. You can use from_tensors or from_tensor_slices.

img = tf.random_uniform([3])

dataset = tf.data.Dataset.from_tensors(img).repeat()
iterator = dataset.make_initializable_iterator()
next_iterator = iterator.get_next()

sess = tf.Session()
sess.run(iterator.initializer)
sess.run(next_iterator)

ludauter · Accepted Answer · 2018-04-20 21:43:52Z

0

Did you figure this out?

I was running into the exact same type of issue and my issue was a dimension mismatch in between the generator and what I was feeding into output_shapes.

Also looking at you code I believe you have to be feeding valid data in, such as numpy arrays, not TensorFlow constants.

answered Apr 20, 2018 at 21:43

ludauter

216 bronze badges

Collectives™ on Stack Overflow

InvalidArgumentError when using of tensorflow.data.Dataset.from_generator

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related