Today, we’re going to look at the TensorFlow ecosystem in some depth. As an open-source ML framework released and developed by Google, it has built a formidable community in a relatively short period of time and has demonstrated some very interesting use cases.
What is TensorFlow?
TensorFlow is an open-source machine learning (ML) framework. It is essentially a software library for numerical computation using dataflow graphs. TensorFlow offers support for machine learning and deep learning and its numerical computation core is used across many different scientific domains.
TensorFlow is cross-platform (it runs on Linux, macOS and Windows) and its flexible architecture can run on multiple different CPUs, GPUs and TPUs, including mobile and embedded platforms and from desktops to clusters of servers, with optional modular CUDA and SYCL extensions for general-purpose computing on graphics processing units. The TensorFlow computations are expressed as stateful dataflow graphs.
Its distributed execution engine abstracts away the various supported devices and assures a high performance-core implemented in C++ for the TensorFlow platform. Python and C++ frontends sit atop the execution engine (the Layers API module offers a simpler interface for commonly used layers in deep learning models) and on top of that, higher-level APIs sit, including Keras, the Python deep learning library, and the Estimator API, which makes training and evaluating distributed models more straightforward.
TensorFlow now runs on version 1.0.0, which was released in February 2017.
Google’s Role in Developing and Open-Sourcing TensorFlow
TensorFlow was initially developed by the Google Brain team (as a replacement for its predecessor DistBelief), and it continues to be widely used at Google for research and production purposes. Like DistBelief, TensorFlow was originally intended for internal use only, but Google decided to release it under the Apache 2.0 open-source license in November 2015 in order to accelerate the evolution of AI via the support of the larger community. The machine learning system is general enough to be readily applicable to a wide variety of other domains, as we detail further below.
The Impact of ML at Google
At the TensorFlow Developer Summit this Spring, Jeff Dean, Head of AI at Google spoke to Laurence Moroney, Developer Advocate about the company’s work with TensorFlow and the broad impact of machine learning (ML) at Google. Dean spoke about his excitement at seeing how over the last 4-5 years, research in the core areas of machine learning have been turning into “things that are actually practical” and “can be used to solve problems across a really wide spread of domains”.
In his summit keynote, Dean highlighted the U.S. NAE Grand Engineering Challenges for the 21st Century and spoke in particular to two of the fourteen “grand engineering challenges”: “to advance health informatics” and “to engineer the tools of scientific discovery” and their relationship to AI. Dean sees TensorFlow as a key part of engineering the tools of scientific discovery as machine learning is “very clearly impacting many different scientific and engineering domains today”, such as how to use ML to tackle quantum chemistry problems. You can train an ML learning model to emulate an expensive quantum chemistry simulator, for instance, but you can get the same accuracy at 300,000 times faster.
Dean noted that four of the “grand challenges” were related to chemistry, such as making solar energy more affordable. Another area within the scientific discovery area is ML’s ability to automate the solving of new problems, which can help mitigate the relatively small number of engineers worldwide who know how to do this and thus democratize ML along with expanding the number of challenges it can address.
In terms of healthcare, Dean spoke about the significant realization and advances in computer vision in ML that can not only be applied to labeling Google photos, but also to medical imaging problems. The Google AI group has been working in many different areas of medical imaging from ophthalmology to dermatology, and beginning to see significant advances in all those areas.
The TensorFlow Community
As of late last year, TensorFlow had reached 21,000+ commits from outside contributors (in widely differing fields), and it continues to be in rapid development with over 890 external contributors adding to the code (again numbers accurate as of late last year).
Stack Overflow is an online help forum that is monitored by the TensorFlow team.
Google has also heavily invested in its online training systems, offering many online tutorials, such as:
Coding TensorFlow – an 8 part series on how to use JavaScript for machine learning in the browser. This is achieved through using TensorFlow.js, a JavaScript library for training and deploying ML models in the browser and on Node.js. The website includes samples, APIs and FAQs. The tutorial is split over eight parts.
The first three episodes provide a mini unit: (i) the first episode takes a basic look at what you need to get up-and-running with TensorFlow in the browser by constructing a simple model that fits its values to a line by learning that it is actually a line off of a small training set; demonstrating this by using TensorFlow to classify clothing images (ii) the second episode demonstrates how you prepare data for training (iii) the third looks at how to take the processed data seen in part 2 and build a neural network to perform classification.
Other videos in this series include coding “TensorFlow Lite” for Android and “TensorFlow Lite” for iOS.
There are numerous more TensorFlow-focused videos on the YouTube channel, including Tips of the Week and various real-world use case scenarios from “Using ML to predict insulin use for Type 1 Diabetes” to “TFX: an end-to-end machine learning platform for TensorFlow”.
TensorFlow on GitHub
In the TensorFlow GitHub org, TensorFlow itself is available along with an ecosystem of other repos, including models, serving, TensorBoard, Project Magenta, and others. TensorFlow APIs in multiple languages (Python, C++, Java, and Go) are also available; along with other bindings that the community has developed, including C#, Haskell, Julia, Ruby, Rust, and Scala.
Open-Source Models Available on TensorFlow
TensorFlow offers a variety of commonly used models as out-of-the-box solutions via https://github.com/tensorflow/models. The released code on many of these models includes the model graph and trained model weights, allowing developers to do their own fine-tuning using transfer learning.
Two examples of the available open-source models are:
tf-seq2seq: a sequence-to-sequence (seq2seq) model that is used in Google Translate. tf-seq2seq is an open-source seq2seq framework in TensorFlow that “makes it easy to experiment with seq2seq models and achieve state-of-the-art results”. Its codebase is clean and modular and in addition to machine translation, tf-seq2seq can be applied to any other type of sequence-to-sequence task, such as learning to produce an output sentence in response to an input sentence. This includes conversational modeling, speech recognition, image captioning and machine summarization.
Object Detection API: a machine learning model for computer vision that can localize and identify multiple objects in a single image, something which “remains a core challenge in the field”. The open-source codebase enables the construction, training and deployment of object detection models. Google describes its goals in designing the system as being to “support state-of-the-art models while allowing for rapid exploration and research”.
The TensorFlow Meets Series and Coca-Cola as a Real-World Use Case
TensorFlow Meets is a series of conversations between Laurence Moroney, the Google Developer Advocate and people in the business community working with TensorFlow.
TensorFlow is supporting Coca-Cola’s digital marketing campaigns, scanning unique codes from bottle caps that Coca-Cola customers can use to earn and win stuff on the company’s rewards website. Coca-Cola trained TensorFlow to classify the digits printed on the bottle tops so that customers can take a picture of them on their phones and easily scan them in online to find out if they have won a promotion.
Patrick Brandt, part of the Marketing Operations team at Coca-Cola, told Laurence Moroney during his TensorFlow Meet that Coca-Cola started working with TensorFlow because of its “approachable interface”, which “presents what would otherwise be very complex constructions in idioms that everyday programmers can understand” and then be used to build a deep learning graph. Coca-Cola’s marketing and developer team also found that normal OCR solutions couldn’t handle “the fidelity of the caps”.
Training TensorFlow on the Coca-Cola Bottle Classifier
Coca-Cola created a training set (both synthetic and real-world) to train TensorFlow involving three stages:
(i) Training – generating, gathering and labeling the data to create the neural network model
(ii) Production – running inference at scale to meet performance expectations
(iii) Active Learning – correcting inaccurate references to improve the model over time
The model was adjusted several times during development using the synthetic data set. The team eventually found a model that works efficiently in the real-world at scale. Coca-Cola’s performance requirement is a one second average processing time. TensorFlow has achieved that goal, which Brandt says makes for a “great user experience” on their site.
Why it was Successful for Coca-Cola
Moroney asked Brandt to imagine what would have happened if the company instead had applied a more heuristic model that instead used rules to interpret the codes. Brandt envisaged it probably working efficiently if the user was holding the cap and phone “perfectly”, but then running into problems… “the second they twist that thing in two degrees in any direction, that heuristic model would fall apart”.
All the variations of how the bottle cap and phone can be held (as the phone takes the picture of the cap) would need to be encoded into a heuristics-based system, which would be time-intensive, inefficient and prone to error. With a machine learning approach like that of TensorFlow, that translation appearance can be accommodated with a convolutional neural net.
Coca-Cola also found that the machine eye often works better than the human eye in terms of detecting the right digits. The model is trained to look through distortions on the bottle caps in a way that the human eye finds more challenging.
The soft drinks giant is now working on a number of other initiatives using TensorFlow, in particular building a computer vision platform. As part of that, this year, Coca-Cola launched a new digital icon they are putting on the exterior of their packaging that fits in with its other branded elements in terms of design, yet can also encode information for that product. Brandt described it as “a bespoke QR code” that is appealing from a visual perspective, featuring three concentric rings with a Coke bottle in the middle rather than the standard black-and-white barcode look.
All the Latest Developments
There are lots of new developments underway within TensorFlow both within Google and the wider community. Moroney outlined several use case scenarios at Cloud Next ‘18 in July, including:
- The discovery of extrasolar planets – Over the last year, through machine learning being able to spot patterns in the behavior of stars, it has led to finding new extrasolar planets and being able to count the number of planets in the sky via the Kepler 90i system, allowing the intricate mapping of solar systems;
- Early diagnosis of diabetic retinopathy – a diabetes complication that is caused by damage to the blood vessels of the light-sensitive tissue in the retina, which can lead to blindness if not discovered early. Through simple pictures of the cornea, ML has helped spot some of the markers of diabetic retinopathy so it can be diagnosed early and cheaply, which Google imagines being widely used on cell phones in developing countries in future use;
- Monitoring deforestation of the planet – an ML project in the Amazon used old Android phones to build TensorFlow ML models to spot the signs of illegal logging. By spreading these smartphones across the forest, illegal logging has been cut down;
- Google Arts app Selfie/Artwork – A trending social media use of ML earlier this year part in involved people taking a selfie on their phones and the ML app using patterns it sees in the selfie, matching that face with distinctive works of art…
Moroney went into some depth on everything new in TensorFlow from these interesting use case scenarios to the release of TensorFlow Lite for Mobile and Embedded Systems to what TensorFlow Hub offers. The full video is available here.