Joel Lehman

The University of Texas at Austin, Computer Science, Post-Doc

Followers

Following

Public Views

I'm Joel, and I'm currently a post-doc at the University of Texas at Austin. My research goal is to write algorithms inspired by natural evolution that continually produce a wild diversity of interesting and complex things.
Supervisors: Risto Miikkulainen

less

Interests

Uploads

Papers by Joel Lehman

Improving Evolvability through Novelty Search and Self-Adaptation In: Proceedings of the 2011 IEEE Congress on Evolutionary Computation (CEC 2011). Piscataway, NJ: IEEE

A challenge for current evolutionary algorithms is to yield highly evolvable representations like... more A challenge for current evolutionary algorithms is to yield highly evolvable representations like those in nature. Such evolvability in natural evolution is encouraged through selection: Lineages better at molding to new niches are less susceptible to extinction. Similar selection pressure is not generally present in evolutionary algorithms; however, the first hypothesis in this paper is that novelty search, a recent evolutionary technique, also selects for evolvability because it rewards lineages able to continually radiate new behaviors. Results in experiments in a maze-navigation domain in this paper support that novelty search finds more evolvable representations than regular fitnessbased search. However, though novelty search outperforms fitnessbased search in a second biped locomotion experiment, it proves no more evolvable than fitness-based search because delicately balanced behaviors are more fragile in that domain. The second hypothesis is that such fragility can be mitigated through selfadaption, whereby genomes influence their own reproduction. Further experiments in fragile domains with novelty search and self-adaption indeed demonstrate increased evolvability, while, interestingly, adding self-adaptation to fitness-based search decreases evolvability. Thus, selecting for novelty may often facilitate evolvability when representations are not overly fragile; furthermore, achieving the potential of self-adaptation may often critically depend upon the reward scheme driving evolution.

downloadDownload free PDF

On the Critical Role of Divergent Selection in Evolvability

Frontiers in Robotics and AI, Aug 2, 2016

An ambitious goal in evolutionary robotics is to evolve increasingly complex robotic behaviors wi... more An ambitious goal in evolutionary robotics is to evolve increasingly complex robotic behaviors with minimal human design effort. Reaching this goal requires evolutionary algorithms that can unlock from genetic encodings their latent potential for evolvability. One issue clouding this goal is conceptual confusion about evolvability, which often obscures the aspects of evolvability that are important or desirable. The danger from such confusion is that it may establish unrealistic goals for evolvability that prove unproductive in practice. An important issue separate from conceptual confusion is the common misalignment between selection and evolvability in evolutionary robotics. While more expressive encodings can represent higher-level adaptations (e.g. sexual reproduction or developmental systems) that increase long-term evolutionary potential (i.e. evolvability), realizing such potential requires gradients of fitness and evolvability to align. In other words, selection is often a c...

downloadDownload free PDF

Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions

Creating open-ended algorithms, which generate their own never-ending stream of novel and appropr... more Creating open-ended algorithms, which generate their own never-ending stream of novel and appropriately challenging learning opportunities, could help to automate and accelerate progress in machine learning. A recent step in this direction is the Paired Open-Ended Trailblazer (POET), an algorithm that generates and solves its own challenges, and allows solutions to goal-switch between challenges to avoid local optima. However, the original POET was unable to demonstrate its full creative potential because of limitations of the algorithm itself and because of external issues including a limited problem space and lack of a universal progress measure. Importantly, both limitations pose impediments not only for POET, but for the pursuit of open-endedness in general. Here we introduce and empirically validate two new innovations to the original algorithm, as well as two external innovations designed to help elucidate its full potential. Together, these four advances enable the most open-ended algorithmic demonstration to date. The algorithmic innovations are (1) a domain-general measure of how meaningfully novel new challenges are, enabling the system to potentially create and solve interesting challenges endlessly, and (2) an efficient heuristic for determining when agents should goal-switch from one problem to another (helping open-ended search better scale). Outside the algorithm itself, to enable a more definitive demonstration of openendedness, we introduce (3) a novel, more flexible way to encode environmental challenges, and (4) a generic measure of the extent to which a system continues to exhibit open-ended innovation.

downloadDownload free PDF

Evolution through Large Models

arXiv (Cornell University), Jun 17, 2022

This paper pursues the insight that large language models (LLMs) trained to generate code can vas... more This paper pursues the insight that large language models (LLMs) trained to generate code can vastly improve the effectiveness of mutation operators applied to programs in genetic programming (GP). Because such LLMs benefit from training data that includes sequential changes and modifications, they can approximate likely changes that humans would make. To highlight the breadth of implications of such evolution through large models (ELM), in the main experiment ELM combined with MAP-Elites generates hundreds of thousands of functional examples of Python programs that output working ambulating robots in the Sodarace domain, which the original LLM had never seen in pre-training. These examples then help to bootstrap training a new conditional language model that can output the right walker for a particular terrain. The ability to bootstrap new models that can output appropriate artifacts for a given context in a domain where zero training data was previously available carries implications for open-endedness, deep learning, and reinforcement learning. These implications are explored here in depth in the hope of inspiring new directions of research now opened up by ELM.

downloadDownload free PDF

Rewarding Reactivity to Evolve Robust Controllers without Multiple Trials or Noise

Behaviors evolved in simulation are often not robust to variations of their original training env... more Behaviors evolved in simulation are often not robust to variations of their original training environment. Thus often researchers must train explicitly to encourage such robustness. Traditional methods of training for robustness typically apply multiple non-deterministic evaluations with carefully modeled noisy distributions for sensors and effectors. In practice, such training is often computationally expensive and requires crafting accurate models. Taking inspiration from nature, where animals react appropriately to encountered stimuli, this paper introduces a measure called reactivity, i.e. the tendency to seek and react to changes in environmental input, that is applicable in single deterministic trials and can encourage robustness without exposure to noise. The measure is tested in four different maze navigation tasks, where training with reactivity proves more robust than training without noise, and equally or more robust than training with noise when testing with moderate noise levels. In this way, the results demonstrate the counterintuitive fact that sometimes training with no exposure to noise at all can evolve individuals significantly more robust to noise than by explicitly training with noise. The conclusion is that training for reactivity may often be a computationally more efficient means to encouraging robustness in evolved behaviors.

downloadDownload free PDF

An Anarchy of Methods: Current Trends in How Intelligence Is Abstracted in AI

IEEE Intelligent Systems, Nov 1, 2014

downloadDownload free PDF

Improving evolvability through novelty search and self-adaptation

downloadDownload free PDF

Efficiently evolving programs through the search for novelty

A significant challenge in genetic programming is premature convergence to local optima, which of... more A significant challenge in genetic programming is premature convergence to local optima, which often prevents evolution from solving problems. This paper introduces to genetic programming a method that originated in neuroevolution (i.e. the evolution of artificial neural networks) that circumvents the problem of deceptive local optima. The main idea is to search only for behavioral novelty instead of for higher fitness values. Although such novelty search abandons following the gradient of the fitness function, if such gradients are deceptive they may actually occlude paths through the search space towards the objective. Because there are only so many ways to behave, the search for behavioral novelty is often computationally feasible and differs significantly from random search. Counterintuitively, in both a deceptive maze navigation task and the artificial ant benchmark task, genetic programming with novelty search, which ignores the objective, outperforms traditional genetic programming that directly searches for optimal behavior. Additionally, novelty search evolves smaller program trees in every variation of the test domains. Novelty search thus appears less susceptible to bloat, another significant problem in genetic programming. The conclusion is that novelty search is a viable new tool for efficiently solving some deceptive problems in genetic programming.

downloadDownload free PDF

Evolving a diversity of virtual creatures through novelty search and local competition

An ambitious challenge in artificial life is to craft an evolutionary process that discovers a wi... more An ambitious challenge in artificial life is to craft an evolutionary process that discovers a wide diversity of welladapted virtual creatures within a single run. Unlike in nature, evolving creatures in virtual worlds tend to converge to a single morphology because selection therein greedily rewards the morphology that is easiest to exploit. However, novelty search, a technique that explicitly rewards diverging, can potentially mitigate such convergence. Thus in this paper an existing creature evolution platform is extended with multi-objective search that balances drives for both novelty and performance. However, there are different ways to combine performance-driven search and novelty search. The suggested approach is to provide evolution with both a novelty objective that encourages diverse morphologies and a local competition objective that rewards individuals outperforming those most similar in morphology. The results in an experiment evolving locomoting virtual creatures show that novelty search with local competition discovers more functional morphological diversity within a single run than models with global competition, which are more predisposed to converge. The conclusions are that novelty search with local competition may complement recent advances in evolving virtual creatures and may in general be a principled approach to combining novelty search with pressure to achieve.

downloadDownload free PDF

Automatic Evolution of Multimodal Behavior with Multi-Brain HyperNEAT

An important challenge in neuroevolution is to evolve multimodal behavior. Indirect network encod... more An important challenge in neuroevolution is to evolve multimodal behavior. Indirect network encodings can potentially answer this challenge. Yet in practice, indirect encodings do not yield effective multimodal controllers. This paper introduces novel multimodal extensions to HyperNEAT, a popular indirect encoding. A previous multimodal approach called situational policy geometry assumes that multiple brains benefit from being embedded within an explicit geometric space. However, this paper introduces HyperNEAT extensions for evolving many brains without assuming geometric relationships between them. The resulting Multi-Brain HyperNEAT can exploit human-specified task divisions, or can automatically discover when brains should be used, and how many to use. Experiments show that multi-brain approaches are more effective than HyperNEAT without multimodal extensions, and that brains without a geometric relation to each other are superior.

downloadDownload free PDF

Revising the evolutionary computation abstraction

Though based on abstractions of nature, current evolutionary algorithms and artificial life model... more Though based on abstractions of nature, current evolutionary algorithms and artificial life models lack the drive to complexity characteristic of natural evolution. Thus this paper argues that the prevalent fitness-pressure-based abstraction does not capture how natural evolution discovers complexity. Alternatively, this paper proposes that natural evolution can be abstracted as a process that discovers many ways to express the same functionality. That is, all successful organisms must meet the same minimal criteria of survival and reproduction. This abstraction leads to the key idea in this paper: Searching for novel ways of meeting the same minimal criteria, which is an accelerated model of this new abstraction, may be an effective search algorithm. Thus the existing novelty search method, which rewards any new behavior, is extended to enforce minimal criteria. Such minimal criteria novelty search prunes the space of viable behaviors and may often be more efficient than the search for novelty alone. In fact, when compared to the raw search for novelty and traditional fitness-based search in the two maze navigation experiments in this paper, minimal criteria novelty search evolves solutions more consistently. It is possible that refining the evolutionary computation abstraction in this way may lead to solving more ambitious problems and evolving more complex artificial organisms.

downloadDownload free PDF

Novelty Search and the Problem with Objectives

Genetic and evolutionary computation, 2011

By synthesizing a growing body of work in search processes that are not driven by explicit object... more By synthesizing a growing body of work in search processes that are not driven by explicit objectives, this paper advances the hypothesis that there is a fundamental problem with the dominant paradigm of objective-based search in evolutionary computation and genetic programming: Most ambitious objectives do not illuminate a path to themselves. That is, the gradient of improvement induced by ambitious objectives tends to lead not to the objective itself but instead to dead-end local optima. Indirectly supporting this hypothesis, great discoveries often are not the result of objective-driven search. For example, the major inspiration for both evolutionary computation and genetic programming, natural evolution, innovates through an open-ended process that lacks a final objective. Similarly, large-scale cultural evolutionary processes, such as the evolution of technology, mathematics, and art, lack a unified fixed goal. In addition, direct evidence for this hypothesis is presented from a recently-introduced search algorithm called novelty search. Though ignorant of the ultimate objective of search, in many instances novelty search has counter-intuitively outperformed searching directly for the objective, including a wide variety of randomly-generated problems introduced in an experiment in this chapter. Thus a new understanding is beginning to emerge that suggests that searching for a fixed objective, which is the reigning paradigm in evolutionary computation and even machine learning as a whole, may ultimately limit what can be achieved. Yet the liberating implication of this hypothesis argued in this paper is that by embracing search processes that are not driven by explicit objectives, the breadth and depth of what is reachable through evolutionary methods such as genetic programming may be greatly expanded.

downloadDownload free PDF

On the Potential Benefits of Knowing Everything

Evolutionary algorithms are designed to find impressive solutions in complex search spaces. Meeti... more

downloadDownload free PDF

Exploiting Open-Endedness to Solve Problems Through the Search for Novelty

Artificial Life, 2008

This paper establishes a link between the challenge of solving highly ambitious problems in machi... more This paper establishes a link between the challenge of solving highly ambitious problems in machine learning and the goal of reproducing the dynamics of open-ended evolution in artificial life. A major problem with the objective function in machine learning is that through deception it may actually prevent the objective from being reached. In a similar way, selection in evolution may sometimes act to discourage increasing complexity. This paper proposes a single idea that both overcomes the obstacle of deception and suggests a simple new approach to open-ended evolution: Instead of either explicitly seeking an objective or modeling a domain to capture the open-endedness of natural evolution, the idea is to simply search for novelty. Even in an objective-based problem, such novelty search ignores the objective and searches for behavioral novelty. Yet because many points in the search space collapse to the same point in behavior space, it turns out that the search for novelty is computationally feasible. Furthermore, because there are only so many simple behaviors, the search for novelty leads to increasing complexity. In fact, on the way up the ladder of complexity, the search is likely to encounter at least one solution. In this way, by decoupling the idea of open-ended search from only artificial life worlds, the raw search for novelty can be applied to real world problems. Counterintuitively, in the deceptive maze navigation task in this paper, novelty search significantly outperforms objective-based search, suggesting a surprising new approach to machine learning.

downloadDownload free PDF

Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning

arXiv (Cornell University), Dec 18, 2017

Deep artificial neural networks (DNNs) are typically trained via gradient-based learning algorith... more Deep artificial neural networks (DNNs) are typically trained via gradient-based learning algorithms, namely backpropagation. Evolution strategies (ES) can rival backprop-based algorithms such as Q-learning and policy gradients on challenging deep reinforcement learning (RL) problems. However, ES can be considered a gradient-based algorithm because it performs stochastic gradient descent via an operation similar to a finite-difference approximation of the gradient. That raises the question of whether non-gradient-based evolutionary algorithms can work at DNN scales. Here we demonstrate they can: we evolve the weights of a DNN with a simple, gradient-free, populationbased genetic algorithm (GA) and it performs well on hard deep RL problems, including Atari and humanoid locomotion. The Deep GA successfully evolves networks with over four million free parameters, the largest neural networks ever evolved with a traditional evolutionary algorithm. These results (1) expand our sense of the scale at which GAs can operate, (2) suggest intriguingly that in some cases following the gradient is not the best choice for optimizing performance, and (3) make immediately available the multitude of neuroevolution techniques that improve performance. We demonstrate the latter by showing that combining DNNs with novelty search, which encourages exploration on tasks with deceptive or sparse reward functions, can solve a high-dimensional problem on which reward-maximizing algorithms (e.g. DQN, A3C, ES, and the GA) fail. Additionally, the Deep GA is faster than ES, A3C, and DQN (it can train Atari in ∼4 hours on one desktop or ∼1 hour distributed on 720 cores), and enables a stateof-the-art, up to 10,000-fold compact encoding technique.

downloadDownload free PDF

Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents

Evolution strategies (ES) are a family of black-box optimization algorithms able to train deep ne... more Evolution strategies (ES) are a family of black-box optimization algorithms able to train deep neural networks roughly as well as Q-learning and policy gradient methods on challenging deep reinforcement learning (RL) problems, but are much faster (e.g. hours vs. days) because they parallelize better. However, many RL problems require directed exploration because they have reward functions that are sparse or deceptive (i.e. contain local optima), and it is unknown how to encourage such exploration with ES. Here we show that algorithms that have been invented to promote directed exploration in small-scale evolved neural networks via populations of exploring agents, specifically novelty search (NS) and quality diversity (QD) algorithms, can be hybridized with ES to improve its performance on sparse or deceptive deep RL tasks, while retaining scalability. Our experiments confirm that the resultant new algorithms, NS-ES and two QD algorithms, NSR-ES and NSRA-ES, avoid local optima encoun...

downloadDownload free PDF

Designing neural networks through neuroevolution

Nature Machine Intelligence, 2019

downloadDownload free PDF

Case Study 2: Objectives and the Quest for AI

Why Greatness Cannot Be Planned, 2015

This chapter provides a case study in how objectives can hold back progress in a particular field... more This chapter provides a case study in how objectives can hold back progress in a particular field of scientific research. The field of artificial intelligence (AI) focuses on the ambitious aim of creating a human-level computer intelligence, an objective which is fascinating for the profound implications it entails if is ever achieved. Interestingly, viewing the dominant approaches to AI through the myth of the objective illustrates the insidious potential for objectives to deceive even scientific experts. In particular, AI research is guided by perceived progress towards the objective of human-level AI, and the criteria for publishing AI papers are themselves driven by performance and theoretical objectives. While on their surface these incentives seem to make sense, there’s no reason to suspect that the quest for AI is fundamentally different from other systems of innovation or achievement. Thus the chapter suggests non-objective criteria that might productively unseat traditional objective-based incentives in AI research.

Case Study 1: Reinterpreting Natural Evolution

Why Greatness Cannot Be Planned, 2015

This chapter, the first case study, reconsiders the drive behind natural evolution in light of th... more This chapter, the first case study, reconsiders the drive behind natural evolution in light of the myth of the objective. Much discussion of evolution in popular culture emphasizes survival and reproduction as a central objective of the process. This chapter turns this perspective on its head, painting natural evolution as an open-ended exploration that succeeds in spite of competition. In particular, we can view evolution as a diversifying search that accumulates an increasing collection of organisms that can survive and reproduce. In other words, surviving and reproducing is something satisfied from the origin of life. Thus the sheer majesty of evolution is in the great variety and complexity of ways to make a living, not narrowly in how well a particular organism can compete with others. The conclusion is evolution may be better viewed through an objective-free lens as water spilling through the space of possible ways to live, instead of as a red-toothed competition between organisms.

The False Compass

Why Greatness Cannot Be Planned, 2015

The question answered in this chapter is, why does the world work in such an unexpected way? Why ... more The question answered in this chapter is, why does the world work in such an unexpected way? Why is it often easier to succeed without objectives? To expose the way that objectives hamper discovery, this chapter surveys scientific experiments, natural evolution, and technological innovation. The problem is that the precursors to a great achievement often do not resemble that achievement. However, pursuing objectives relies upon exactly there being a recognizable pattern of steps to the desired objective that pursuing objectives relies one. This disconnect between the structure of great accomplishments and how accomplishments are pursued leads to deception, when focusing too much on reaching an ambitious objective actually prevents its accomplishment. While deception implies that we cannot simply accomplish anything we desire through brute force, by examining how innovation actually happens we can adopt a more realistic and (ultimately more interesting) perspective on accomplishment. Successful innovation proceeds more smoothly not through attempts to muscle one’s way towards a particular discovery, but by collecting stepping stones with an open mind and investigating to where they lead without preconceived judgement.

Joel Lehman

Uploads

Papers by Joel Lehman

Log In