tensorboard - tensorflow.python.framework.errors_impl.NotFoundError:

Question

I just installed my new environment based on tf.__version__ = 1.14.

Here is how I configure my tensorboard:

tensorboard = \
    tf.keras.callbacks.TensorBoard(log_dir='./logs', 
                                   batch_size=TRAIN_BATCH_SIZE, 
                                   write_graph=True, 
                                   write_grads=True, 
                                   write_images=True)

Note that I have a folder called "logs" in the same directory where I run my code. But this folder is empty.

And here is how I call it from my fit function.

history = model.fit(x=train_x,
                    y=train_y,
                    batch_size=BATCH_SIZE,
                    epochs=5,
                    verbose=2,
                    callbacks=[tensorboard],
                    validation_data=(dev_x, dev_y),
                    shuffle=True,
                    class_weight=class_weight,
                    steps_per_epoch=None, 
                    validation_steps=None)

And when I run the code, I get the following error:

2019-07-02 13:04:26.341623: E tensorflow/core/platform/default/device_tracer.cc:68] CUPTI error: CUPTI could not be loaded or symbol could not be found.
Traceback (most recent call last):
  File "C:\Users\sinthes\Desktop\AI_Project\BigBang\src\train_fit.py", line 286, in <module>
    validation_steps=None) #devset_steps_per_epoch)                       
  File "C:\Users\sinthes\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\keras\engine\training.py", line 780, in fit
    steps_name='steps_per_epoch')
  File "C:\Users\sinthes\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py", line 374, in model_iteration
    callbacks._call_batch_hook(mode, 'end', batch_index, batch_logs)
  File "C:\Users\sinthes\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\keras\callbacks.py", line 248, in _call_batch_hook
    batch_hook(batch, logs)
  File "C:\Users\sinthes\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\keras\callbacks.py", line 531, in on_train_batch_end
    self.on_batch_end(batch, logs=logs)
  File "C:\Users\sinthes\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\keras\callbacks_v1.py", line 362, in on_batch_end
    profiler.save(self.log_dir, profiler.stop())
  File "C:\Users\sinthes\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\eager\profiler.py", line 144, in save
    gfile.MakeDirs(plugin_dir)
  File "C:\Users\sinthes\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 438, in recursive_create_dir
    recursive_create_dir_v2(dirname)
  File "C:\Users\sinthes\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 453, in recursive_create_dir_v2
    pywrap_tensorflow.RecursivelyCreateDir(compat.as_bytes(path))
tensorflow.python.framework.errors_impl.NotFoundError: Failed to create a directory: ./logs\plugins\profile\2019-07-02_13-04-26; No such file or directory
[Finished in 10.7s with exit code 1]
[shell_cmd: python -u "C:\Users\sinthes\Desktop\AI_Project\BigBang\src\train_fit.py"]
[dir: C:\Users\sinthes\Desktop\AI_Project\BigBang\src]
[path: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\libnvvp;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\Microsoft VS Code\bin;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Users\sinthes\AppData\Local\Programs\Python\Python37;C:\Users\sinthes\AppData\Local\Programs\Python\Python37\Scripts;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Users\sinthes\AppData\Local\Microsoft\WindowsApps;]

It was working in my previous environment but it is not working anymore after installing my new environment with tf 1.14.

I just don't know how to fix it. Any ideas?

I had faced this problem and have solved it: stackoverflow.com/a/60448427/9349225 — Victoria
– Victoria, Commented Feb 28, 2020 at 9:25

edn · Accepted Answer · 2019-07-02 11:33:39Z

4

Finally solved.

It looks like it is a Windows specific bug in Tensorflow. Defining the log directory as following is solving the problem..

import datetime
log_dir = os.path.join(
    "logs",
    "fit",
    datetime.datetime.now().strftime("%Y%m%d-%H%M%S"),
)

For more information, see: https://github.com/tensorflow/tensorflow/issues/26021

answered Jul 2, 2019 at 11:33

edn

2,2016 gold badges36 silver badges68 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

tensorboard - tensorflow.python.framework.errors_impl.NotFoundError:

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related