What is Generative AI?

June 15, 2023

Artificial intelligence has evolved from simply analyzing data to now being able to autonomously create new data and content. This generative capability is driving transformations across industries. In this post, we will dive into understanding what generative AI is, how it works, the state-of-the-art techniques behind it, and its diverse current and potential applications.

Generative AI refers broadly to AI systems that can produce novel, high-quality artifacts like images, video, text, code, designs, and more from scratch. Unlike traditional manually programmed software, generative AI relies on machine learning to train models on large datasets until they can generate new data similar to what they have seen before. The outputs are unique, realistic, and often indistinguishable from content created by humans.

From deepfakes to AI-written essays and music, generative AI is making tremendous strides. Under the hood, approaches like generative adversarial networks, variational autoencoders, diffusion models, reinforcement learning, and transformer neural networks enable modeling complex distributions in data to create new examples from them. The applications span content creation, process automation, augmenting human creativity, and more.

However, there are rising concerns regarding potential misuse and biases perpetuated by generative AI. As the capabilities continue to rapidly advance, it is crucial we work towards developing ethical, transparent, and controlled frameworks for generative models.

In this post, we will do a deep dive into how generative AI works, the key technologies powering it, where it is headed, and how we can steer progress responsibly. Let’s get started!

Generative AI refers to artificial intelligence systems that can generate new content and artifacts that are novel, high-quality, and realistic. Instead of being programmed with rules, generative AI relies on examining and learning from large datasets to produce original outputs.

Some key characteristics of generative AI:

Generates brand new content, rather than just classifying or labeling existing data. The outputs are not based on predefined templates or rules.
Uses machine learning techniques like neural networks to train on large datasets, allowing the models to learn the patterns and relationships in data.
Outputs are often probabilistic and reflect the statistical relationships learned from the training data. This allows variation and diversity in the generated content.
Generates artifacts that mimic styles, structures and patterns seen in the training data. This allows high-quality, realistic outputs.

Some major types and examples of generative AI:

Generative adversarial networks (GANs): GANs use two neural networks – one generates candidates while the other discriminates real from fake. This pushes the generator to create more realistic outputs. GANs have produced very convincing generated images and videos.
Variational autoencoders (VAEs): VAEs are neural networks that compress data into a latent space and can generate new data having similar characteristics to the training data. VAEs are good for creating diverse outputs.
Diffusion models: These generate data by starting with random noise and modifying it iteratively to introduce attributes of real data based on diffusion process modeling. DALL-E 2 uses a diffusion model to create novel images from text captions.
Reinforcement learning: RL agents are rewarded for actions that maximize a goal. This allows generative models to create content aimed at specific objectives, like maximizing user engagement.
Transformers: Transformer-based architectures like GPT excel at generating coherent, high-quality text by learning contextual relationships in language from large text corpora.

Generative AI has diverse applications across industries like generating code, chemical structures, art, 3D shapes, synthetic media and more. It promises to revolutionize content creation and automation. But it also raises concerns about misuse of realistic generated content and data privacy. Nonetheless, generative AI represents an exciting new frontier in AI research and application.

Generative adversarial networks (GANs)

Generative adversarial networks (GANs) are a powerful type of generative model that uses two neural networks competing against each other to generate new, synthetic data that closely resembles real-world data.

The two neural networks are:

Generator: This neural network generates new data instances (images, text, etc.) from random noise. It starts creating low-quality outputs but improves over time.
Discriminator: This neural network receives data instances from both the generator and real-world training data, and tries to determine which are fake (from the generator) and which are real.

The two networks play a minimax adversarial game – the generator tries to fool the discriminator by creating increasingly realistic data, while the discriminator tries to correctly classify the real and fake data. This process of competing forces the generator to improve continuously until the generated outputs become indistinguishable from real data.

GANs can produce incredibly realistic generated photos of human faces, animals, objects, and scenes that are very difficult to differentiate from real images. By sampling different points in the latent noise space, GANs can also generate highly diverse and unique outputs.

GANs are also used to generate ultra-realistic profile pictures on social media, create fake celebrity footage in forged videos, synthesize speech in the voice of a person with just a few samples, and other such applications. However, the potential for misuse of such realistic forged content has raised concerns.

Overall, the minimax game and adversarial training make GANs very effective at mimicking intricate patterns in data distributions and producing new data points that plausibly belong to those distributions. This makes them a popular architecture for multiple generative modeling tasks.

Variational autoencoders (VAEs)

Variational autoencoders (VAEs) are a type of generative model that use neural networks for compressing data into a lower-dimensional latent space and generating new data from sampling points in that space.

VAEs consist of:

An encoder neural network that compresses input data into a compact latent representation or code.
A decoder neural network that reconstructs the original input from the latent code.

During training, the VAE learns to optimize the encoder and decoder to allow converting data to and from the latent space, which captures the most salient features and variations in the data.

Once trained, the VAE allows:

Encoding any data into the latent space and decoding it back, which serves as a means of compression and reconstruction.
Generating entirely new data by sampling random points in the latent space and decoding them. The new samples retain similarities to the training data but represent novel variations.

Unlike GANs which require finding a fine balance during training, VAEs directly optimize for the ability to reconstruct data from the latent space, making them more stable to train. VAEs can generate diverse data by sampling in the latent space.

Applications of VAEs include:

Creating new human faces, animals, objects etc. with different variations.
Generating molecular structures and chemical compounds with desired properties.
Producing novel voices by conditioning the sampling on attributes like tone, accent etc.
Recommending product variations and designs based on initial concepts.

A major advantage of VAEs is the ability to intentionally guide the characteristics of generated data by manipulating the latent code vectors. Overall, VAEs offer a flexible framework for data-efficient generative modeling.

Diffusion models

Diffusion models are a class of generative models that create realistic data by gradually modifying random noise through a diffusion process. The process has two main steps:

Forward diffusion: The model starts with real training data and adds Gaussian noise to it iteratively resulting in detached noisy data. This destroys details while retaining high-level structure.
Reverse diffusion: Starts with pure noise and performs iterative refinement while removing some noise at each step. This gradually enhances details and introduces attributes of real data.

Mathematically, diffusion models employ a Markov chain to model how noise evolves over time (diffuses) through the data. By learning to reverse this diffusion, the model can start from noise and generate data with realistic attributes.

A key advantage is that diffusion models can condition the reverse diffusion on context like class labels or captions to directly generate targeted outputs.

DALL-E 2 uses a diffusion model conditioned on text captions to generate novel, high-quality images reflecting the caption meaning. It starts with random noise and slowly edits it over hundreds of steps while matching the evolving image to the caption at each point.

Other applications of diffusion models include creating synthetic voices, generating molecular structures with desired properties, producing realistic 3D shapes and rendering novel scenes or human poses based on descriptions.

Overall, diffusion models allow fine-grained control over the incremental generation process to craft high-quality, targeted results. Their ability to condition on context makes them suited for controllable generation tasks compared to methods that directly output complete results.

Reinforcement learning (RL)

Reinforcement learning (RL) is a technique that trains AI agents to take optimal actions in an environment to maximize cumulative rewards. This can be applied to train generative models as well.

In RL-based generative modeling:

The generative model produces candidate artifacts like images, text, music etc. based on current state.
The artifacts are evaluated against a reward function designed to assess quality, novelty, diversity and objectives like user engagement.
The generator model is reinforced to produce outputs that can obtain higher rewards in the future.

Over successive iterations, the generator learns to create artifacts that score highly on the specified reward criteria.

For example, an RL-trained generative model can:

Generate news headlines aimed at driving more user clicks by rewarding clickbait-sounding titles.
Synthesize product images or music that are more likely to elicit positive responses from target users based on their preferences.
Create levels in games that are engaging and challenging for players by using player metrics as rewards.

Unlike supervised learning which optimizes for resemblance to training data, RL directly optimizes for external goals specified via rewards. This makes RL suitable for generative modeling requiring purpose-driven, customized outputs.

However, designing the right rewards functions can be challenging. Bad rewards can lead to undesirable generated content. Overall though, RL provides a way to steer generative models towards practical objectives.

Transformers

Transformers are a class of neural network architecture that have driven major advances in generative modeling for text, becoming the dominant approach nowadays.

Transformers process input text by attending to context using a mechanism called self-attention. This allows modeling long-range dependencies in language better than recurrent neural networks.

Key advantages of transformers for generative text AI:

Self-attention learns contextual relationships between words and sentences across large corpora. This allows generating text that is coherent, logical and consistent.
Stacking multiple self-attention layers allows modeling hierarchical structure and long-term coherence in generated text.
Parallel processing of sequence positions increases speed and allows training on huge text corpora.
Conditioning the text generation on past context (previous tokens) allows more relevant, targeted continuation of text.

GPT-4 is a massive auto-regressive transformer network trained on over 1 trillion parameters. It achieves state-of-the-art performance in generative language tasks like continuing prompts with logical, human-like text.

Other transformer-based models like CTRL, GROVER, TransfoXL and GPT-Neo also demonstrate strong performance on text generation benchmarks, showing the power of self-attention.

Limitations include large computational requirements for training and generation, and potential for generating toxic or biased text if the training data contains such attributes.

Overall, transformers allow modeling the real-world complexity and nuance of language more effectively compared to earlier methods. This makes them indispensable for AI assistants, chatbots, summarization, story generation and other text-based generative applications.

Conclusion

In summary, generative AI refers to a cutting-edge class of artificial intelligence technologies that create novel, realistic artifacts like images, videos, text, 3D models, and more from scratch. Instead of following predefined rules, generative AI models develop creative capabilities by learning to represent and generate data that conforms to the patterns in their training datasets through techniques like neural networks and adversarial training.

Key generative AI methods highlighted in this post include generative adversarial networks, variational autoencoders, diffusion models, reinforcement learning agents, and transformer-based models. Each approach has its strengths and applications, from generating highly realistic synthetic media using GANs to creating coherent text using transformers like GPT.

The applications of generative AI span multiple industries and use cases, ranging from enhancing creativity for artists to automating repetitive tasks and augmenting human capabilities. However, concerns remain about potential misuse of forged media, biases perpetuated through data, and legal implications surrounding copyright and data rights. Tremendous research is focused on improving the capabilities, controllability, and social responsibility of generative AI systems.

Overall, generative AI represents an enormously promising field that is rapidly evolving and contributing towards more capable, autonomous AI systems. The next decade will likely see generative models become indispensable tools across many sectors and potentially even taking over creative jobs. Going forward, developing frameworks to ensure ethics, fairness and transparency should be prioritized to fully realize the benefits of this transformative technology.

John Arnott