2,081 questions
6
votes
0
answers
108
views
partially decode, stream and filter big data with tensorflow_datasets (tfds)
I have two issues (Note that this code is generated in google colab):
Issue 1 I want to stream the droid dataset, which is almost 2TB big. I want to only use data which matches my filter conditions. ...
0
votes
1
answer
90
views
multi output forcast shape prob. with keras.preprocessing.timeseries_dataset_from_array
The model was fuctioinnal with only one output, now I try to use multi output forcasting one timestep, but got this error:
Epoch 1/10
2025-11-04 16:20:09.341769: W tensorflow/core/framework/op_kernel....
- 453
0
votes
1
answer
126
views
SSL:CERTIFICATE_VERIFY_FAILED for TensorFlow dataset loading
Issue
I have previously loaded this dataset in Google Colab without error, but I now encounter an error.
Code
!pip install --upgrade 'tensorflow_data_validation[visualization]<2'
import tensorflow ...
- 23
1
vote
3
answers
74
views
Tensorflow dataset running out of data durin training
I'm creating a dataset from a csv file containing path-names to image locations and mask-locations (xtr and ytr). Initially, the csv contains approx. 1000 elements. Each image is loaded from cloud ...
- 1,597
0
votes
1
answer
72
views
Dataset from generator in tensorflow error GetNextFromShard
I have a train step for a Vertex AI pipeline that uses a template code for setting up distributed training. When fitting any model I get a message in the first epoch indicating the following message:
...
0
votes
0
answers
99
views
Visualize Filters of CNN using Tensor Flow
I'm working on Visualizing Filters of a CNN using TensorFlow - Guided Project in Ccoursera.
I encountered the error
Invalid shape (1, 96, 96, 3) for image data
when trying to call the plot_image ...
- 11
0
votes
1
answer
40
views
tensorflow dataset loop endless will model.predict
I use this code to setup my dataset for trainning and predict:
train_dataset = train_dataset.batch(train_batch_sz)
train_dataset = train_dataset.repeat().prefetch(5)
test_dataset_sim = test_dataset
...
- 453
0
votes
1
answer
57
views
InvalidArgumentError with tf.data.Dataset.from_generator() at random points during training
I want to use a python generator with a tf Dataset to extract and preprocess sequences from a (rather large) numpy array, but I keep getting this error at random points during the first epoch of ...
- 1
1
vote
2
answers
121
views
How to load small sample of tensorflow_dataset data
I'm trying to play around with a neural network using the "lm1b" dataset in my computer before training a larger model in a cloud machine.
What I'm struggling with is that every time I try ...
- 481
0
votes
1
answer
31
views
Stream data to a model form tensorflow datasets without having it to download locally
I was trying to load the dataset from tensorflow datasets without having it to download locally and process it to a pipeline in batches so i can feed it into my model. I'm not sure if I'm doing it ...
- 9
1
vote
1
answer
111
views
Tensorflow dataset splitted sizing parameter problem: Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Pretty new with data generator and dataset from tensorflow. I struggle with sizing batch, epochs and step... I can't figure the good set up to remove error "Local rendezvous is aborting with ...
- 453
-1
votes
1
answer
49
views
Convolutional Neural Network is getting unexpectedly high validation accuracy, what could be causing this?
I am building/optimizing a CNN for classifcation of cars from this dataset.
My baseline model is getting suprisingly high accuracy from a very simple model architecture, I am concerned there is some ...
0
votes
0
answers
44
views
Using 'from_tensor_slices' causes the model to stop learning
I am trying to train a simple CNN on custom image data. While training with raw numpy data, the model learns as expected. However when I convert the same raw numpy data to dataset using tf.data....
- 13.9k
-2
votes
1
answer
51
views
Improving Accuracy [closed]
I am working on testing accuracy and performance using deep learning models on a complex dataset but I have reached a good accuracy but I need to improve it so any suggestions other than what I did(...
- 5
1
vote
1
answer
33
views
Give a list of image path pairs to load using tf.data.Dataset.map but it seems to only read 2
I am learning how to use Tensorflow with a piece of old code provided at a workshop a couple of years back (i.e. it should've been tested and works). However it didn't work, and my investigations led ...
- 13
0
votes
0
answers
22
views
Strange repetition in loss curves (Tensorflow)
I am training on a custom dataset (of ~200,000 images) that has been constructed to sample from three different datasets at equal frequencies, the main construction steps are:
sample_dict = {"...
0
votes
1
answer
74
views
VAalue_Error: as_list() is not defined on an unknown TensorShape when using dataset generator
There are a number of previous answers to this error but none relate to my issue.
I am creating a tensorflow dataset using a generator and for the examples sake am trying fit a very simple API model.
...
- 11
0
votes
0
answers
40
views
Problem with RAM when creating a DataFrame with a large number of columns from a TensorFlow Dataset
I work with large amounts of data which I process using TensorFlow Dataset (TFDS) and save to pandas.DataFrame. My goal is to convert the data from one format to another for further analysis. But when ...
- 1
-1
votes
1
answer
128
views
Input image is not compatible with tensorflow model input shape [closed]
I was building a model and I wanted to test its performance, thus I imported a local file and load it and try to predict its label with the following code:
from tensorflow.preprocessing import image
# ...
- 197
0
votes
0
answers
31
views
Can TFRecordDataSetm start reading elements from a random spot in the file?
I have an application with 200GB data and I can't load it in RAM. Based on the data/problem, it will be extremely beneficial to have TFRecord files "sorted" in a certain way that makes them ...
0
votes
1
answer
68
views
Batch items in Tensorflow Dataset that contain varying first dimension, but constant second dimension
I have a Tensorflow dataset that contains tensors of rank 2. Each tensor was read from a file and has varying first dimension and constant second dimension. For example, the shapes of each item can be ...
- 11
1
vote
1
answer
148
views
Tensorflow fit won't take custom dataset, but throws error: ValueError: Cannot take the length of shape with unknown rank
I'm trying to train a model previously working fine until recently. The fit function throws the error below:
<ipython-input-20-01755a6ded38> in <cell line: 1>()
----> 1 model.fit(
...
- 25
1
vote
0
answers
43
views
Multi-Input network for image classification using TensorflowDataset gets none values not supported error
I have a dataset of tfrecords and mapped them to a Tensorflow dataset. The dataset structure has a dict structure as follows:
{ "image" : ImageDataTensor, "additional_features" : ...
- 23
1
vote
0
answers
85
views
Conda package conflicts while exporting Tensorflow trained model into ONNX
I used these instructions to build up a conda env and train tensorflow model:
conda create --name tf210 python=3.8
conda activate tf210
conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0
pip ...
- 11
0
votes
1
answer
81
views
tensorflow from_tensor_slices gives "args_0:0"
I'm trying to create a tensorflow dataset using from_tensor_slices but the string I'm passing is getting messed up within the map function:
def test(x):
print(x)
return x
filenames=['a','b','...
- 1,009
-3
votes
1
answer
70
views
keras dataset import issue [closed]
I am trying to use the keras fashion_mnist dataset but for some reason I cannot seem to load it. I am using an anaconda environment. I have tried creating new environments, redownloading tensorflow, ...
- 13
0
votes
0
answers
62
views
How can I create sequences (windows) from data in a TFRecord?
I have around 1800 short videos. Each video is around 30 seconds. I trained a VAE to encode each frame into a latent vector (of size 200).
Using this VAE, then I created a TFrecord with an entry for ...
- 1,638
0
votes
0
answers
169
views
i keep getting this error : TypeError: Expected any non-tensor type, but got a tensor instead
i am trying to train mobilevit using this bee dataset from tensorflow and i keep getting this error
TypeError: Expected any non-tensor type, but got a tensor instead.
this is the code i wrote, ...
1
vote
0
answers
74
views
Tensorflow Performance Drop When Training Sequential Model Using `tf.data.Dataset.from_generator`
I'm training a Sequential model in a binary classification problem. My dataset is in HDF format and consists of many files, which are often too large to fit in memory. To handle this, I tried using a ...
- 11
0
votes
1
answer
264
views
I can't use Flower's Partitioning Object for Local Data(ChosenPartitione)
I'm trying to open a local dataset that I generated in image format, but Visual studio is already displaying an error message for the line of code below:
from flwr_datasets.partitioner import ...
2
votes
0
answers
44
views
Time Series Data: Sequence classification with tensorflow - ValueError:
I am currently in the process of using an existing network architecture (TSMixer: An All-MLP Architecture for Time Series Forecasting) for a sequence classification.
The existing structure of my data: ...
- 21
0
votes
0
answers
65
views
TensorFlow image_dataset_from_directory not mapping labels correctly
I am using the following code to read *.jpg files from the
"labeled_data_dir"
Labels are age of person.
*
*training_dataset= tf.keras.utils.image_dataset_from_directory(
labeled_data_dir,...
- 119
1
vote
0
answers
47
views
How to convert image list and mask list to TensorFlow dataset?
I'm having trouble converting image and mask data into Dataset that can be used in keras.Model.
I have a list of 100 images with a shape of (100, 128, 128, 3)) and I have a list of 100 ...
- 11
1
vote
0
answers
183
views
Cardinality issue with graph execution in TensorFlow
I was using a TFRecord and couldn't get past a cardinality issue to get it to work with graph execution. I switched back to a regular dataset loading method and now I have another cardinality issue.
...
- 61
0
votes
0
answers
236
views
model.fit is not working "RuntimeError: `tf.data.Dataset` only supports Python-style iteration in eager mode or within tf.function."
I want to code a simple machine learning on JupyterLab but got error as Title
I have collect the data as x are lamda, eps, c (the shape of data is 102010 rows × 3 columns) and y is sen (the shape of ...
- 1
0
votes
0
answers
251
views
Python) tf.keras.utils.audio_dataset_from_directory does not return correct labels
I am trying to study the code from [a tensorflow tutorial on audio recognition].(https://www.tensorflow.org/tutorials/audio/simple_audio) But it seems the tf.keras.utils.audio_dataset_from_directory() ...
- 1
0
votes
0
answers
281
views
Tensorflow returning TypeError for no reason
I am trying to work with tensorflow_datasets library, the dataset I am working on is STL-10, but it doesn't matter much, because no matter what dataset I use, the same error is returned. I am using ...
- 71
1
vote
0
answers
60
views
Solution for high ram consumption for loading large NPZ file?
So, I am working with video dataset where I have preprocessed my training data into npz files which approximately consumes about 20 GB size. After that when I try to load the .npz file in below way:
...
- 35
0
votes
0
answers
141
views
How can I tackle Attribute Error when I import tensorflow_datasets?
I have been trying to import the tensorflow_datasets module but it repeatedly shows an attribute error.
I have tried clearing the cache, reinstalling the package again and also restarting the device ...
2
votes
1
answer
244
views
Issue Accessing .bin Files in React Native App
I'm encountering an issue with accessing .bin files in my React Native app. I followed the process of saving my TensorFlow model in .h5 format, converting it to JSON and binary files, and then moving ...
- 21
0
votes
0
answers
163
views
Why is KerasCV augmentation layer destructively editing image labels?
I'm currently approximately following Gitesh Chawda's guide for YOLOV8 from here
Everything is going as expected until the image augmentation stage, at which point the images are correctly augmented, ...
0
votes
1
answer
288
views
How to load individual .npz files as samples into a Tensorflow Dataset
I am having a problem with loading the data that make up the dataset. My previous (working) approach was to use a pandas DataFrame, but for larger datasets, the training process gets killed, as the ...
- 156
-1
votes
1
answer
113
views
Food101 Tensorflow Dataset Requested split "validation" does not exist
I'm trying to use food101 dataset but while it has meta folder of train and test it can only see train data.
(train_data, test_data), ds_info = tfds.load(name="food101",
...
0
votes
1
answer
27
views
tf.data generates training results that do not improve
I am trying to use tf.data to augment a dataset that I have. The dataset is arranged locally in my computer like this:
datasets/fruits/{class_name}/*jpg
{class_name} include 7 different kinds of ...
2
votes
0
answers
90
views
"Correct" way of preparing and feeding multiple pandas dataframes containing multivariate time series into a LSTM in tensorflow? (data pipeline)
I am pretty new with working with Tensorflow and in coding in general. So I'm sorry if this question seems trivial or was answered somewhere else in a way that I maybe didn't understand as a solution ...
- 21
0
votes
1
answer
136
views
How to convert Object Detection Dataset into Tensorflow Dataset
I'm trying to do an object detection model on tensorflow with keras but I've been having so difficulty. I automated the task of finding the bounding boxes of my training images(training dataset is of ...
0
votes
1
answer
98
views
create an efficient dataset to feed a keras siamese model
I need help to build a generator for a Siamese model in Keras and Tensorflow. I succeeded in creating a python generator but it resulted too slow for my purposes. My idea was to use Tensorflow ...
- 11
0
votes
0
answers
21
views
TFDS extract file name
In tfds, we do
def _generate_examples(self, file_paths):
for file_path in file_paths:
yield file_path, {
'image': image,
'label': label
}
I am already ...
- 2,742
0
votes
1
answer
124
views
tf.data.dataset next(iter()) yields the same value
I am trying to build a datapipeline for inference on some video files. The videos are alot in number hence, I am using the the tf.data.dataset pipeline with a from_generator method to create ...
- 51
0
votes
1
answer
181
views
Training Loss for Classification Model Isn't Decreasing [closed]
I'm currently building a video classification model for engagement detection but I'm having some trouble training it. The model takes in two tensors as inputs: a 10x48x48x1 tensor which holds a stack ...
- 21