Oct. 31, 2018, midnight

CoLab TPUs One Month Later

By : Eric A. Scuccimarra

After having used both CoLab GPUs and TPUs for almost a month I must significantly revise my previous opinion. Even for a Keras model not written or optimized for TPUs, with some minimal configuration changes TPUs perform much faster - minimum of twice the speed. In addition to making sure that all operations are TPU compatible, the only major configuration change required is increasing the batch size by 8. At first I was playing around with the batch size, but I realized that this was unnecessary. TPUs have 8 shards, so you simply multiple the GPU batch size by 8 and that should be a good baseline. 

The model I am currently training on a TPU and a GPU simultaneously is training 3-4x faster on the TPU than on the GPU and the code is exactly the same. I have this block of code:

use_tpu = True
# if we are using the tpu copy the keras model to a new var and assign the tpu model to model
if use_tpu:
    TPU_WORKER = 'grpc://' + os.environ['COLAB_TPU_ADDR']
    
    # create network and compiler
    tpu_model = tf.contrib.tpu.keras_to_tpu_model(
        model, strategy = tf.contrib.tpu.TPUDistributionStrategy(
            tf.contrib.cluster_resolver.TPUClusterResolver(TPU_WORKER)))
    
    BATCH_SIZE = BATCH_SIZE * 8

The model is created with Keras and the only change I make is setting use_tpu to True on the TPU instance. 

One other thing I thought I would mention is that CoLab creates separate instances for GPU, TPU and CPU, so you can run multiple notebooks without sharing RAM or processor if you give each one a different type.

Labels: machine_learning , tensorflow , google , google_cloud

Comments


Nov. 27, 2018, 12:49 p.m.

Привет всем! прикольный у вас сайт! Нашел топ базу кино: мультфильмы 2018 смотреть бесплатно в хорошем качестве http://kinovalenok.tv/ Тут: фэнтези смотреть бесплатно в хорошем фильмы фэнтези онлайн хорошее качество список 2019 Здесь: фильмы драмы смотреть хорошем качестве http://kinovalenok.tv/drama/ список 2019 Тут: Документальные фильмы онлайн смотреть http://kinovalenok.tv/dokumentalnyy/ список 2019 Здесь: http://kinovalenok.tv/detektiv/5281-dzhek-potroshitel-jack-the-ripper-1988.html Смотреть Джек-потрошитель / Jack the Ripper (1988) онлайн бесплатно Тут: http://kinovalenok.tv/news/11260-obyavlena-data-premery-filma-lara-kroft.html


Dec. 17, 2018, 6:42 p.m.

Hello, can I load the file to VM and work with TPU?


Aug. 15, 2019, 2:05 p.m.

Propecia Cheap Acheter Cialis Discretement Purchasing isotretinoin skin health medication vente viagra 40mg Cialis


Aug. 22, 2019, 8:08 a.m.

Silagra Online Uk Cialis Original Sale cialis tratamiento diario Interaction Amoxicillin With Doxicillin

Login or register to comment