GitHub - openai/universe: Universe: a software platform for measuring and training an AIs general in

Time 2021-06-04 14:10:52

Web Name: GitHub - openai/universe: Universe: a software platform for measuring and training an AIs general in

WebSite: http://universe.openai.com

ID:181074

Keywords:

measuring,for,and,

Description:

Universe: a software platform for measuring and training an AI's general intelligence across the world's supply of games, websites and other applications. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Go back Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Go back Launching Xcode If nothing happens, download Xcode and try again. Go back Your codespace will open once ready. There was a problem preparing your codespace, please try again. This repository has been deprecated in favor of the Retro (https://github.com/openai/retro) library. See our Retro Contest (https://blog.openai.com/retro-contest) blog post for detalis.universeUniverse is a softwareplatform for measuring and training an AI's general intelligenceacross the world's supply of games, websites and otherapplications. This is the universe open-source library, whichprovides a simple Gyminterface to each Universe environment.Universe allows anyone to train and evaluate AI agents on an extremelywide range of real-time, complex environments.Universe makes it possible for any existing program to become anOpenAI Gym environment, without needing special access to theprogram's internals, source code, or APIs. It does this by packagingthe program into a Docker container, and presenting the AI with thesame interface a human uses: sending keyboard and mouse events, andreceiving screen pixels. Our initial release contains over 1,000environments in which an AI agent can take actions and gatherobservations.Additionally, some environments include a reward signal sent to theagent, to guide reinforcement learning. We've included a few hundredenvironments with reward signals. These environments also includeautomated start menu clickthroughs, allowing your agent to skip to theinteresting part of the environment.We'd like the community's helpto grow the number of available environments, including integratingincreasingly large and complex games.The following classes of tasks are packaged inside ofpublicly-available Docker containers, and can be run today with nowork on your part:Atari and CartPole environments over VNC: gym-core.Pong-v3, gym-core.CartPole-v0, etc.Flashgames over VNC: flashgames.DuskDrive-v0, etc.Browser tasks ("World of Bits") over VNC: wob.mini.TicTacToe-v0, etc.We've scoped out integrations for many other games, includingcompleting a high-quality GTA V integration (thanks to Craig Quiter and NVIDIA), but these aren't included in today's release.Contents of this documentuniverseGetting startedTestingAdditional documentationGetting helpWhat's next?Getting startedInstallationSupported systemsWe currently support Linux and OSX running Python 2.7 or 3.5.We recommend setting up a conda environmentbefore getting started, to keep all your Universe-related packages in the same place.Install UniverseTo get started, first install universe:git clone https://github.com/openai/universe.gitcd universepip install -e .If this errors out, you may be missing some required packages. Here'sthe list of required packages we know about so far (please let us knowif you had to install any others).On Ubuntu 16.04:pip install numpysudo apt-get install golang libjpeg-turbo8-dev makeOn Ubuntu 14.04:sudo add-apt-repository ppa:ubuntu-lxc/lxd-stable # for newer golangsudo apt-get updatesudo apt-get install golang libjpeg-turbo8-dev makeOn OSX:You might need to install Command Line Tools by running:xcode-select --installOr numpy, libjpeg-turbo and incremental packages:pip install numpy incrementalbrew install golang libjpeg-turboInstall DockerThe majority of the environments in Universe run inside Dockercontainers, so you will need to install Docker (on OSX, werecommend Docker for Mac). You should be able torun docker ps and get something like this:$ docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMESAlternate configuration - running the agent in dockerThe above instructions result in an agent that runs as a regular python process in your OS, and launches docker containers as needed for the remotes.Alternatively, you can build a docker image for the agent and run it as a container as well.You can do this in any operating system that has a recent version of docker installed, and the git client.To get started, clone the universe repo:git clone https://github.com/openai/universe.gitcd universeBuild a docker image, tag it as 'universe':docker build -t universe .This may take a while the first time, as the docker image layers are pulled from docker hub.Once the image is built, you can do a quick run of the test cases to make sure everything is working:docker run --privileged --rm -e DOCKER_NET_HOST=172.17.0.1 -v /var/run/docker.sock:/var/run/docker.sock universe pytestHere's a breakdown of that command:docker run - launch a docker container--rm - delete the container once the launched process finishes-e DOCKER_NET_HOST=172.17.0.1 - tells the universe remote (when launched) to make its VNC connection back to this docker-allocated IP-v /var/run/docker.sock:/var/run/docker.sock - makes the docker unix socket from the host available to the container. This is a common technique used to allow containers to launch other containers alongside itself.universe - use the imaged named 'universe' built abovepytest - run 'pytest' in the container, which runs all the testsAt this point, you'll see a bunch of tests run and hopefully all pass.To do some actual development work, you probably want to do another volume map from the universe repo on your host into the container, then shell in interactively:docker run --privileged --rm -it -e DOCKER_NET_HOST=172.17.0.1 -v /var/run/docker.sock:/var/run/docker.sock -v (full path to cloned repo above):/usr/local/universe universe pythonAs you edit the files in your cloned git repo, they will be changed in your docker container and you'll be able to run them in python.Note if you are using docker for Windows, you'll need to enable the relevant shared drive for this to work.Notes on installationWhen installing universe, you may see warning messages. These lines occur when installing numpy and are normal.You'll need a go version of at least 1.5. Ubuntu 14.04 has an older Go, so you'll need to upgrade your Go installation.We run Python 3.5 internally, so the Python 3.5 variants will be much more thoroughly performance tested. Please let us know if you see any issues on 2.7.While we don't officially support Windows, we expect our code to be very close to working there. We'd be happy to take pull requests that take our Windows compatibility to 100%. In the meantime, the easiest way for Windows users to run universe is to use the alternate configuration described above.System overviewA Universe environment is similar to any other Gym environment:the agent submits actions and receives observations using the step()method.Internally, a Universe environment consists of two pieces: a client and a remote:The client is a VNCEnvinstance which lives in the same process as the agent. It performsfunctions like receiving the agent's actions, proxying them to theremote, queuing up rewards for the agent, and maintaining alocal view of the current episode state.The remote is the running environment dynamics, usually aprogram running inside of a Docker container. It can run anywhere --locally, on a remote server, or in the cloud. (We have a separatepage describing how to manage remotes.)The client and the remote communicate with one another using theVNCremote desktop system, as well as over an auxiliary WebSocketchannel for reward, diagnostic, and control messages. (For moreinformation on client-remote communication, see the separate page onthe Universe internal communication protocols.)The code in this repository corresponds to the client side of theUniverse environments. Additionally, you can freely access the Dockerimages for the remotes. We'll release the source repositories forthe remotes in the future, along with tools to enable users tointegrate new environments. Please sign up for our betaif you'd like early access.Run your first agentNow that you've installed the universe library, you should makesure it actually works. You can paste the example below into yourpython REPL. (You may need to press enter an extra time to makesure the while loop is executing.)import gymimport universe # register the universe environmentsenv = gym.make('flashgames.DuskDrive-v0')env.configure(remotes=1) # automatically creates a local docker containerobservation_n = env.reset()while True: action_n = [[('KeyEvent', 'ArrowUp', True)] for ob in observation_n] # your agent here observation_n, reward_n, done_n, info = env.step(action_n) env.render()The example will instantiate a client in your Python process,automatically pull the quay.io/openai/universe.flashgames image,and will start that image as the remote. (In our remotes documentation page, we explain other ways you can runremotes.)It will take a few minutes for the image to pull the first time. After that,if all goes well, a window like the one below will soon pop up. Youragent, which is just pressing the up arrow repeatedly, is nowplaying a Flash racing game called Dusk Drive. Your agentis programmatically controlling a VNC client, connected to a VNCserver running inside of a Docker container in the cloud, rendering aheadless Chrome with Flash enabled:You can even connect your own VNC client to the environment, eitherjust to observe or to interfere with your agent. Our flashgamesand gym-core images conveniently bundle a browser-based VNCclient, which can be accessed athttp://localhost:15900/viewer/?password=openai. If you're on Mac,connecting to a VNC server is as easy as running: openvnc://localhost:5900.(If using docker-machine, you'll need to replace "localhost" with theIP address of your Docker daemon, and use openai as the password.)Breaking down the exampleSo we managed to run an agent, what did all the code actuallymean? We'll go line-by-line through the example.First, we import the gym library,which is the base on which Universe is built. We also importuniverse, which registersall the Universe environments.import gymimport universe # register the universe environmentsNext, we create the environment instance. Behind the scenes, gymlooks up the registrationfor flashgames.DuskDrive-v0, and instantiates a VNCEnvobject which has been wrappedto add a few useful diagnostics and utilities. The VNCEnv objectis the client part of the environment, and it is not yet connectedto a remote.env = gym.make('flashgames.DuskDrive-v0')The call to configure() connects the client to a remoteenvironment server. When called with configure(remotes=1),Universe will automatically create a Docker image running locally onyour computer. The local client connects to the remote using VNC.(More information on client-remote communication can be found in thepage on universe internal communication protocols. More on configuring remotes is at remotes.)env.configure(remotes=1)When starting a new environment, you call env.reset(). Universeenvironments run in real-time, rather than stepping synchronouslywith the agent's actions, so reset is asynchronous and returnsimmediately. Since the environment will not have waited to finishconnecting to the VNC server before returning, the initial observationsfrom reset will be None to indicate that there isnot yet a valid observation.Similarly, the environment keeps running in the background evenif the agent does not call env.step(). This means that an agentthat successfully learns from a Universe environment cannot take"thinking breaks": it must keep sending actions to the environment at all times.Additionally, Universe introduces the vectorized GymAPI. Rather than controlling a single environment at a time, the agentcan control a fixed-size vector of n environments, each with itsown remote. The return value from reset is therefore a vectorof observations. For more information, see the separate page onenvironment semantics)At each step() call, the agent submits a vector of actions; one foreach environment instance it is controlling. Each VNC action is alist of events; above, each action is the single event "press theArrowUp key". The agent could press and release the key in oneaction by instead submitting [('KeyEvent', 'ArrowUp', True),('KeyEvent', 'ArrowUp', False)] for each observation.In fact, the agent could largely have the same effect by justsubmitting ('KeyEvent', 'ArrowUp', True) once and then callingenv.step([[] for ob in observation_n]) thereafter, without everreleasing the key using ('KeyEvent', 'ArrowUp', False). Thebrowser running inside the remote would continue to statefullyrepresent the arrow key as being pressed. Sending other unrelatedkeypresses would not disrupt the up arrow keypress; only explicitlyreleasing the key would cancel it. There's one slight subtlety:when the episode resets, the browser will reset, and will forgetabout the keypress; you'd need to submit a new ArrowUp at thestart of each episode.After we submit the action to the environment and render one frame,step() returns a list of observations, a list of rewards, alist of "done" booleans indicating whether the episode has ended,and then finally an info dictionary of the form {'n': [{},...]}, in which you can access the info for environment i asinfo['n'][i].Each environment's info message contains useful diagnosticinformation, including latency data, client and remote timings,VNC update counts, and reward message counts.We call step in what looks like a busy loop. In reality, thereis a Throttlewrapper on the client which defaults to a target frame rate of 60fps, or oneframe every 16.7ms. If you call it more frequently than that,step will sleepwith any leftover time.TestingWe are using pytest for tests. You can run them via:pytestRun pytest --help for useful options, such as pytest -s (disables output capture) or pytest -k expression (runs only specific tests).Additional documentationMore documentation not covered in this README can be found in thedoc folder of this repository.Getting helpIf you encounter a problem that is not addressed in this README pageor in the extra docs, then try our wiki page of solutionsto common problems -and add to it if your solution isn't there!You can also search through the issueson this repository and our discussion board to see if another user has postedabout the same problem or to ask for help from the community.If you still can't solve your problem after trying all of the abovesteps, please post an issue on this repository.What's next?Get started training RL algorithms! You can try out the Universe Starter Agent, an implementation of the A3C algorithm that can solve several VNC environments.For more information on how to manage remotes, see the separate documentation page on remotes.Sign up for a beta to get early access to upcoming Universe releases, such as tools to integrate new Universe environments or a dataset of recorded human demonstrations.Changelog2017-02-08: The old location for wrappers.SafeActionSpace has been moved to wrappers.experimental.SafeActionSpace. SoftmaxClickMouse has also been moved to wrappers.experimental.SoftmaxClickMouse2017-01-08: The wrappers.SafeActionSpace has been moved to wrappers.experimental.SafeActionSpace. The old location will remain with a deprecation warning until 2017-02-08.2016-12-27: BACKWARDS INCOMPATIBILITY: The gym monitor is now awrapper. Rather than starting monitoring asenv.monitor.start(directory), envs are now wrapped as follows:env = wrappers.Monitor(env, directory). This change is on masterand will be released with 0.21.0. Universe: a software platform for measuring and training an AI's general intelligence across the world's supply of games, websites and other applications. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.

TAGS:measuring for and 

<<< Thank you for your visit >>>

Universe: a software platform for measuring and training an AI's general intelligence across the world's supply of games, websites and other applications. - openai/universe

Hot Websites