Posit AI Weblog: TensorFlow 2.0 is right here

The wait is over – TensorFlow 2.0 (TF 2) is now formally right here! What does this imply for us, customers of R packages keras and/or tensorflow, which, as we all know, rely on the Python TensorFlow backend?

Earlier than we go into particulars and explanations, right here is an all-clear, for the involved person who fears their keras code may turn into out of date (it gained’t).

Don’t panic

• If you’re utilizing keras in customary methods, akin to these depicted in most code examples and tutorials seen on the internet, and issues have been working superb for you in current keras releases (>= 2.2.4.1), don’t fear. Most every thing ought to work with out main adjustments.
• If you’re utilizing an older launch of keras (< 2.2.4.1), syntactically issues ought to work superb as nicely, however it would be best to examine for adjustments in conduct/efficiency.

And now for some information and background. This publish goals to do three issues:

• Clarify the above all-clear assertion. Is it actually that easy – what precisely is happening?
• Characterize the adjustments caused by TF 2, from the perspective of the R person.
• And, maybe most curiously: Check out what’s going on, within the r-tensorflow ecosystem, round new performance associated to the arrival of TF 2.

Some background

So if all nonetheless works superb (assuming customary utilization), why a lot ado about TF 2 in Python land?

The distinction is that on the R facet, for the overwhelming majority of customers, the framework you used to do deep studying was keras. tensorflow was wanted simply often, or by no means.

Between keras and tensorflow, there was a transparent separation of obligations: keras was the frontend, relying on TensorFlow as a low-level backend, identical to the original Python Keras it was wrapping did. . In some instances, this result in individuals utilizing the phrases keras and tensorflow virtually synonymously: Perhaps they stated tensorflow, however the code they wrote was keras.

Issues have been completely different in Python land. There was authentic Python Keras, however TensorFlow had its personal layers API, and there have been plenty of third-party high-level APIs constructed on TensorFlow.
Keras, in distinction, was a separate library that simply occurred to depend on TensorFlow.

So in Python land, now we now have a giant change: With TF 2, Keras (as integrated within the TensorFlow codebase) is now the official high-level API for TensorFlow. To deliver this throughout has been a significant level of Google’s TF 2 data marketing campaign because the early levels.

As R customers, who’ve been specializing in keras on a regular basis, we’re basically much less affected. Like we stated above, syntactically most every thing stays the best way it was. So why differentiate between completely different keras variations?

When keras was written, there was authentic Python Keras, and that was the library we have been binding to. Nonetheless, Google began to include authentic Keras code into their TensorFlow codebase as a fork, to proceed improvement independently. For some time there have been two “Kerases”: Unique Keras and tf.keras. Our R keras provided to modify between implementations , the default being authentic Keras.

In keras launch 2.2.4.1, anticipating discontinuation of authentic Keras and desirous to prepare for TF 2, we switched to utilizing tf.keras because the default. Whereas to start with, the tf.keras fork and authentic Keras developed kind of in sync, the newest developments for TF 2 introduced with them greater adjustments within the tf.keras codebase, particularly as regards optimizers.
Because of this, if you’re utilizing a keras model < 2.2.4.1, upgrading to TF 2 it would be best to examine for adjustments in conduct and/or efficiency.

That’s it for some background. In sum, we’re comfortable most current code will run simply superb. However for us R customers, one thing should be altering as nicely, proper?

TF 2 in a nutshell, from an R perspective

The truth is, essentially the most evident-on-user-level change is one thing we wrote a number of posts about, greater than a yr in the past . By then, keen execution was a brand-new choice that needed to be turned on explicitly; TF 2 now makes it the default. Together with it got here customized fashions (a.okay.a. subclassed fashions, in Python land) and customized coaching, making use of tf$GradientTape. Let’s speak about what these termini consult with, and the way they’re related to R customers.

Keen Execution

In TF 1, it was all in regards to the graph you constructed when defining your mannequin. The graph, that was – and is – an Summary Syntax Tree (AST), with operations as nodes and tensors “flowing” alongside the sides. Defining a graph and working it (on precise knowledge) have been completely different steps.

In distinction, with keen execution, operations are run instantly when outlined.

Whereas it is a more-than-substantial change that should have required plenty of sources to implement, in case you use keras you gained’t discover. Simply as beforehand, the standard keras workflow of create mannequin -> compile mannequin -> prepare mannequin by no means made you consider there being two distinct phases (outline and run), now once more you don’t must do something. Despite the fact that the general execution mode is keen, Keras fashions are skilled in graph mode, to maximise efficiency. We’ll speak about how that is accomplished partly 3 when introducing the tfautograph package deal.

If keras runs in graph mode, how will you even see that keen execution is “on”? Effectively, in TF 1, whenever you ran a TensorFlow operation on a tensor , like so

that is what you noticed:

Tensor("Cumprod:0", form=(5,), dtype=int32)

To extract the precise values, you needed to create a TensorFlow Session and run the tensor, or alternatively, use keras::k_eval that did this below the hood:

[1]   1   2   6  24 120

With TF 2’s execution mode defaulting to keen, we now mechanically see the values contained within the tensor:

tf.Tensor([  1   2   6  24 120], form=(5,), dtype=int32)

In order that’s keen execution. In our final yr’s Keen-category weblog posts, it was all the time accompanied by custom models, so let’s flip there subsequent.

Customized fashions

As a keras person, in all probability you’re accustomed to the sequential and purposeful types of constructing a mannequin. Customized fashions enable for even larger flexibility than functional-style ones. Try the documentation for tips on how to create one.

Final yr’s collection on keen execution has loads of examples utilizing customized fashions, that includes not simply their flexibility, however one other necessary side as nicely: the best way they permit for modular, easily-intelligible code.

Encoder-decoder situations are a pure match. If in case you have seen, or written, “old-style” code for a Generative Adversarial Community (GAN), think about one thing like this as an alternative:

# define the generator (simplified)
generator <-
  function(name = NULL) {
    keras_model_custom(name = name, function(self) {
      
      # define layers for the generator 
      self$fc1 <- layer_dense(units = 7 * 7 * 64, use_bias = FALSE)
      self$batchnorm1 <- layer_batch_normalization()
      # more layers ...
      
      # define what should happen in the forward pass
      function(inputs, mask = NULL, training = TRUE) {
        self$fc1(inputs) %>%
          self$batchnorm1(training = training) %>%
          # call remaining layers ...
      }
    })
  }

# define the discriminator
discriminator <-
  function(name = NULL) {
    keras_model_custom(name = name, function(self) {
      
      self$conv1 <- layer_conv_2d(filters = 64, #...)
      self$leaky_relu1 <- layer_activation_leaky_relu()
      # more layers ...
    
      function(inputs, mask = NULL, training = TRUE) {
        inputs %>% self$conv1() %>%
          self$leaky_relu1() %>%
          # call remaining layers ...
      }
    })
  }

# zooming in on a single batch of a single epoch
with(tf$GradientTape() %as% gen_tape, { with(tf$GradientTape() %as% disc_tape, {
  
  # first, it is the generator's name (yep pun meant)
  generated_images <- generator(noise)
  # now the discriminator provides its verdict on the actual photographs 
  disc_real_output <- discriminator(batch, coaching = TRUE)
  # in addition to the pretend ones
  disc_generated_output <- discriminator(generated_images, coaching = TRUE)
  
  # relying on the discriminator's verdict we simply received,
  # what is the generator's loss?
  gen_loss <- generator_loss(disc_generated_output)
  # and what is the loss for the discriminator?
  disc_loss <- discriminator_loss(disc_real_output, disc_generated_output)
}) })

# now exterior the tape's context compute the respective gradients
gradients_of_generator <- gen_tape$gradient(gen_loss, generator$variables)
gradients_of_discriminator <- disc_tape$gradient(disc_loss, discriminator$variables)
 
# and apply them!
generator_optimizer$apply_gradients(
  purrr::transpose(list(gradients_of_generator, generator$variables)))
discriminator_optimizer$apply_gradients(
  purrr::transpose(list(gradients_of_discriminator, discriminator$variables)))

Once more, examine this with pre-TF 2 GAN coaching – it makes for a lot extra readable code.

As an apart, final yr’s publish collection might have created the impression that with keen execution, you have to make use of customized (GradientTape) coaching as an alternative of Keras-style match. The truth is, that was the case on the time these posts have been written. At this time, Keras-style code works simply superb with keen execution.

So now with TF 2, we’re in an optimum place. We can use customized coaching after we need to, however we don’t must if declarative match is all we’d like.

That’s it for a flashlight on what TF 2 means to R customers. We now have a look round within the r-tensorflow ecosystem to see new developments – recent-past, current and future – in areas like knowledge loading, preprocessing, and extra.

New developments within the r-tensorflow ecosystem

These are what we’ll cowl:

• tfdatasets: Over the current previous, tfdatasets pipelines have turn into the popular manner for knowledge loading and preprocessing.
• function columns and function specs: Specify your options recipes-style and have keras generate the sufficient layers for them.
• Keras preprocessing layers: Keras preprocessing pipelines integrating performance akin to knowledge augmentation (at the moment in planning).
• tfhub: Use pretrained fashions as keras layers, and/or as function columns in a keras mannequin.
• tf_function and tfautograph: Pace up coaching by working components of your code in graph mode.

tfdatasets enter pipelines

For two years now, the tfdatasets package deal has been out there to load knowledge for coaching Keras fashions in a streaming manner.

Logically, there are three steps concerned:

1. First, knowledge must be loaded from some place. This may very well be a csv file, a listing containing photographs, or different sources. On this current instance from Image segmentation with U-Net, details about file names was first saved into an R tibble, after which tensor_slices_dataset was used to create a dataset from it:

knowledge <- tibble(
  img = list.files(right here::here("data-raw/prepare"), full.names = TRUE),
  masks = list.files(right here::here("data-raw/train_masks"), full.names = TRUE)
)

knowledge <- initial_split(knowledge, prop = 0.8)

dataset <- coaching(knowledge) %>%  
  tensor_slices_dataset() 

2. As soon as we now have a dataset, we carry out any required transformations, mapping over the batch dimension. Persevering with with the instance from the U-Internet publish, right here we use capabilities from the tf.image module to (1) load photographs based on their file sort, (2) scale them to values between 0 and 1 (changing to float32 on the identical time), and (3) resize them to the specified format:

dataset <- dataset %>%
  dataset_map(~.x %>% list_modify(
    img = tf$picture$decode_jpeg(tf$io$read_file(.x$img)),
    masks = tf$picture$decode_gif(tf$io$read_file(.x$masks))[1,,,][,,1,drop=FALSE]
  )) %>% 
  dataset_map(~.x %>% list_modify(
    img = tf$picture$convert_image_dtype(.x$img, dtype = tf$float32),
    masks = tf$picture$convert_image_dtype(.x$masks, dtype = tf$float32)
  )) %>% 
  dataset_map(~.x %>% list_modify(
    img = tf$picture$resize(.x$img, dimension = form(128, 128)),
    masks = tf$picture$resize(.x$masks, dimension = form(128, 128))
  ))

Be aware how as soon as what these capabilities do, they free you of plenty of pondering (keep in mind how within the “previous” Keras strategy to picture preprocessing, you have been doing issues like dividing pixel values by 255 “by hand”?)

3. After transformation, a 3rd conceptual step pertains to merchandise association. You’ll typically need to shuffle, and also you definitely will need to batch the info:

 if (prepare) {
    dataset <- dataset %>% 
      dataset_shuffle(buffer_size = batch_size*128)
  }

dataset <- dataset %>%  dataset_batch(batch_size)

Summing up, utilizing tfdatasets you construct a pipeline, from loading over transformations to batching, that may then be fed on to a Keras mannequin. From preprocessing, let’s go a step additional and have a look at a brand new, extraordinarily handy approach to do function engineering.

Characteristic columns and have specs

Feature columns
as such are a Python-TensorFlow function, whereas feature specs are an R-only idiom modeled after the favored recipes package deal.

All of it begins off with making a function spec object, utilizing system syntax to point what’s predictor and what’s goal:

library(tfdatasets)
hearts_dataset <- tensor_slices_dataset(hearts)
spec <- feature_spec(hearts_dataset, goal ~ .)

That specification is then refined by successive details about how we need to make use of the uncooked predictors. That is the place function columns come into play. Completely different column varieties exist, of which you’ll see a couple of within the following code snippet:

spec <- feature_spec(hearts, goal ~ .) %>% 
  step_numeric_column(
    all_numeric(), -cp, -restecg, -exang, -intercourse, -fbs,
    normalizer_fn = scaler_standard()
  ) %>% 
  step_categorical_column_with_vocabulary_list(thal) %>% 
  step_bucketized_column(age, boundaries = c(18, 25, 30, 35, 40, 45, 50, 55, 60, 65)) %>% 
  step_indicator_column(thal) %>% 
  step_embedding_column(thal, dimension = 2) %>% 
  step_crossed_column(c(thal, bucketized_age), hash_bucket_size = 10) %>%
  step_indicator_column(crossed_thal_bucketized_age)

spec %>% match()

What occurred right here is that we instructed TensorFlow, please take all numeric columns (apart from a couple of ones listed exprès) and scale them; take column thal, deal with it as categorical and create an embedding for it; discretize age based on the given ranges; and eventually, create a crossed column to seize interplay between thal and that discretized age-range column.

That is good, however when creating the mannequin, we’ll nonetheless must outline all these layers, proper? (Which might be fairly cumbersome, having to determine all the fitting dimensions…)
Fortunately, we don’t must. In sync with tfdatasets, keras now supplies layer_dense_features to create a layer tailored to accommodate the specification.

And we don’t have to create separate enter layers both, resulting from layer_input_from_dataset. Right here we see each in motion:

enter <- layer_input_from_dataset(hearts %>% choose(-goal))

output <- enter %>% 
  layer_dense_features(feature_columns = dense_features(spec)) %>% 
  layer_dense(items = 1, activation = "sigmoid")

From then on, it’s simply regular keras compile and match. See the vignette for the whole instance. There is also a post on feature columns explaining extra of how this works, and illustrating the time-and-nerve-saving impact by evaluating with the pre-feature-spec manner of working with heterogeneous datasets.

As a final merchandise on the matters of preprocessing and have engineering, let’s have a look at a promising factor to come back in what we hope is the close to future.

Keras preprocessing layers

Studying what we wrote above about utilizing tfdatasets for constructing a enter pipeline, and seeing how we gave a picture loading instance, you’ll have been questioning: What about knowledge augmentation performance out there, traditionally, by keras? Like image_data_generator?

This performance doesn’t appear to suit. However a nice-looking resolution is in preparation. Within the Keras group, the current RFC on preprocessing layers for Keras addresses this matter. The RFC remains to be below dialogue, however as quickly because it will get carried out in Python we’ll comply with up on the R facet.

The thought is to offer (chainable) preprocessing layers for use for knowledge transformation and/or augmentation in areas akin to picture classification, picture segmentation, object detection, textual content processing, and extra. The envisioned, within the RFC, pipeline of preprocessing layers ought to return a dataset, for compatibility with tf.knowledge (our tfdatasets). We’re undoubtedly wanting ahead to having out there this kind of workflow!

Let’s transfer on to the subsequent matter, the frequent denominator being comfort. However now comfort means not having to construct billion-parameter fashions your self!

Tensorflow Hub and the tfhub package deal

Tensorflow Hub is a library for publishing and utilizing pretrained fashions. Current fashions may be browsed on tfhub.dev.

As of this writing, the unique Python library remains to be below improvement, so full stability shouldn’t be assured. That however, the tfhub R package deal already permits for some instructive experimentation.

The normal Keras concept of utilizing pretrained fashions sometimes concerned both (1) making use of a mannequin like MobileNet as a complete, together with its output layer, or (2) chaining a “customized head” to its penultimate layer . In distinction, the TF Hub concept is to make use of a pretrained mannequin as a module in a bigger setting.

There are two predominant methods to perform this, particularly, integrating a module as a keras layer and utilizing it as a function column. The tfhub README reveals the primary choice:

library(tfhub)
library(keras)

enter <- layer_input(form = c(32, 32, 3))

output <- enter %>%
  # we're utilizing a pre-trained MobileNet mannequin!
  layer_hub(deal with = "https://tfhub.dev/google/tf2-preview/mobilenet_v2/feature_vector/2") %>%
  layer_dense(items = 10, activation = "softmax")

mannequin <- keras_model(enter, output)

Whereas the tfhub feature columns vignette illustrates the second:

spec <- dataset_train %>%
  feature_spec(AdoptionSpeed ~ .) %>%
  step_text_embedding_column(
    Description,
    module_spec = "https://tfhub.dev/google/universal-sentence-encoder/2"
    ) %>%
  step_image_embedding_column(
    img,
    module_spec = "https://tfhub.dev/google/imagenet/resnet_v2_50/feature_vector/3"
  ) %>%
  step_numeric_column(Age, Price, Amount, normalizer_fn = scaler_standard()) %>%
  step_categorical_column_with_vocabulary_list(
    has_type("string"), -Description, -RescuerID, -img_path, -PetID, -Identify
  ) %>%
  step_embedding_column(Breed1:Well being, State)

Each utilization modes illustrate the excessive potential of working with Hub modules. Simply be cautioned that, as of right now, not each mannequin revealed will work with TF 2.

tf_function, TF autograph and the R package deal tfautograph

As defined above, the default execution mode in TF 2 is keen. For efficiency causes nevertheless, in lots of instances will probably be fascinating to compile components of your code right into a graph. Calls to Keras layers, for instance, are run in graph mode.

To compile a perform right into a graph, wrap it in a name to tf_function, as accomplished e.g. within the publish Modeling censored data with tfprobability:

run_mcmc <- perform(kernel) {
  kernel %>% mcmc_sample_chain(
    num_results = n_steps,
    num_burnin_steps = n_burnin,
    current_state = tf$ones_like(initial_betas),
    trace_fn = trace_fn
  )
}

# necessary for efficiency: run HMC in graph mode
run_mcmc <- tf_function(run_mcmc)

On the Python facet, the tf.autograph module mechanically interprets Python management stream statements into applicable graph operations.

Independently of tf.autograph, the R package deal tfautograph, developed by Tomasz Kalinowski, implements management stream conversion instantly from R to TensorFlow. This allows you to use R’s if, whereas, for, break, and subsequent when writing customized coaching flows. Try the package deal’s intensive documentation for instructive examples!

Conclusion

With that, we finish our introduction of TF 2 and the brand new developments that encompass it.

If in case you have been utilizing keras in conventional methods, how a lot adjustments for you is especially as much as you: Most every thing will nonetheless work, however new choices exist to put in writing extra performant, extra modular, extra elegant code. Specifically, take a look at tfdatasets pipelines for environment friendly knowledge loading.

When you’re a sophisticated person requiring non-standard setup, take a look into customized coaching and customized fashions, and seek the advice of the tfautograph documentation to see how the package deal may help.

In any case, keep tuned for upcoming posts exhibiting a number of the above-mentioned performance in motion. Thanks for studying!