An Overview of Yahoo Cloud Computing

2012 days ago, 911 views
PowerPoint PPT Presentation

Presentation Transcript

Slide 1

An Overview of Cloud Computing @ Yahoo! Raghu Ramakrishnan Chief Scientist, Audience and Cloud Computing Research Fellow, Yahoo! Investigate Reflects numerous exchanges with: Eric Baldeschwieler, Jay Kistler, Chuck Neerdaels, Shelton Shugar, and Raymie Stata and joint work with the Sherpa group, specifically: Brian Cooper, Utkarsh Srivastava, Adam Silberstein, Rodrigo Fonseca and Nick Puz in Y! Investigate Chuck Neerdaels, P.P. Suryanarayanan and numerous others in CCDI

Slide 2

Questions What is distributed computing? Level and utilitarian administrations What's it going to change? Programming plans of action, science, life what number mists will there be? 1, 2, 3, interminability What's new in distributed computing? HPC frameworks, ASPs, facilitated administrations, Multics (!) Emerging "cloud stack" to bolster a wide class of projects, including information escalated applications

Slide 3

SCENARIOS Pie-in-the-sky

Slide 4

Living in the Clouds We need to begin another site, Our site will give postings of things to deal, occupations, and so forth. Over the long haul, we'll include more elements And outline how more cloud capacities (and comparing framework parts) are utilized as required List of abilities/segments is illustrative, not thorough Our cloud gives a "dataset" deliberation FredsList doesn't stress over the basic segments

Slide 5

Step 1: Listings Scenario FredsList needs to store postings as (key, classification, portrayal) application DECLARE DATASET Listings AS ( ID String PRIMARY KEY, Category String, Description Text ) 5523442, childcare, Nanny accessible in San Jose 1234323, transportation, For deal: one bike, scarcely utilized 215534, needed, Looking for issue 1 of Superman comic book Simple Web Service API's Database PNUTS

Slide 6

Step 2: System Evolution Fred belatedly acknowledges costs are helpful data! application ALTER DATASET Listings ADD (Price Float) 5523442, childcare, Nanny accessible in San Jose 215534, needed, Looking for issue 1 of Superman comic book 32138, camera, Nikon D40, USD 300 1234323, transportation, For deal: one bike, scarcely utilized Simple Web Service API's Schemas are adaptable, and advance versus Database PNUTS Not each record in a dataset has values characterized for all fields proclaimed for the dataset

Slide 7

Federation of frameworks offering diverse capacities Step 3: Search FredsList's clients rapidly request catchphrase look application ALTER Listings SET Description SEARCHABLE "dvd's" "bike" "caretaker" Simple Web Service API's Database Search PNUTS Vespa Messaging Tribble

Slide 8

Federation of frameworks offering distinctive execution focuses Step 4: Photos FredsList chooses to include photographs/recordings to postings application ALTER Listings ADD Photo BLOB Simple Web Service API's Storage Database Search Foreign key photograph → posting MObStor PNUTS Vespa Messaging Tribble

Slide 9

Step 5: Data Analysis FredsList needs to break down its postings to get insights about class, do geocoding, and so forth. application ALTER Listings MAKE ANALYZABLE Hadoop program to produce favor pages for postings Hadoop program to geocode information Pig question to break down classes Simple Web Service API's Storage Compute Database Search Foreign key photograph → posting MObStor Grid PNUTS Vespa Messaging Tribble Batch send out

Slide 10

And at this point, Fred is worldwide, and needs geo-replication! Step 6: Performance FredsList needs to lessen its information get to inertness application ALTER Listings MAKE CACHEABLE Simple Web Service API's Storage Compute Database Caching Search Foreign key photograph → posting MObStor Grid PNUTS memcached Vespa Messaging Tribble Batch send out

Slide 11

Data Serving versus Examination Very unique workloads, prerequisites Data from serving framework would one say one is of numerous sorts of information (snap streams are another normal kind, as are syndicated encourages) to be dissected and coordinated The consequence of investigation frequently goes right again into serving framework

Slide 12

EYES TO THE SKIES Motherhood-and-Apple-Pie

Slide 13

Why Clouds? On-request foundation to make a crucial move in the OE bend: Do things we can't do Build all the more powerfully, more proficiently, more internationally, more totally, more rapidly, for a given spending Cloud administrations ought to do hard work of truly difficult work of scaling & high-accessibility Today, this is done at the application level, which is not beneficial

Slide 14

Requirements for Cloud Services Multitenant. A cloud benefit must bolster numerous, authoritatively inaccessible clients. Versatility. Occupants ought to have the capacity to arrange and get assets/QoS on-request . Asset Sharing. In a perfect world, save cloud assets ought to be straightforwardly connected when an inhabitant's arranged QoS is deficient, e.g., because of spikes. Flat scaling. It ought to be conceivable to include cloud limit in little additions; this ought to be straightforward to the inhabitants of the administration. Metering. A cloud benefit must bolster bookkeeping that sensibly attributes operational and capital uses to each of the inhabitants of the administration. Security. A cloud administration ought to be secure in that inhabitants are not made powerless in light of provisos in the cloud. Accessibility. A cloud administration ought to be exceptionally accessible. Operability. A cloud administration ought to be anything but difficult to work, with couple of administrators. Working expenses ought to scale directly or better with the limit of the administration.

Slide 15

Types of Cloud Services Two sorts of cloud administrations: Horizontal ("Platform") Cloud Services Functionality empowering inhabitants to fabricate applications or new administrations on top of the cloud Functional Cloud Services Functionality that is valuable all by itself to occupants. E.g., different SaaS occurrences, for example,; Google Analytics and Yahoo's! IndexTools; Yahoo! properties went for end-clients and private companies, e.g., flickr, Groups, Mail, News, Shopping Could be based on top of level cloud administrations or without any preparation Yahoo! has been putting forth these for quite a while (e.g., Mail for SMB, Groups, Flickr, BOSS, Ad trades)

Slide 16

Opening Up Yahoo! Look Phase 1 Phase 2 BOSS takes Yahoo's! open procedure to the following level by giving Yahoo! Look foundation and innovation to designers and organizations to help them manufacture their own particular hunt encounters. Giving site proprietors and designers control over the presence of Yahoo! List items.

Slide 17

BOSS Offerings BOSS offers two alternatives for organizations and engineers and has banded together with top innovation colleges to drive seek experimentation, advancement and research into cutting edge look. Scholastic Working with the accompanying colleges to take into consideration wide-scale inquire about in the hunt field: API A self-benefit, web administrations show for designers and new companies to rapidly manufacture and send new inquiry encounters. CUSTOM Working with outsiders to fabricate a more applicable, brand/website particular web seek encounter. This choice is together worked by Yahoo! also, select accomplices. College of Illinois Urbana Champaign Carnegie Mellon University Stanford University Purdue University • MIT Indian Institute of Technology Bombay University of Massachusetts (Slide politeness Prabhakar Raghavan)

Slide 18

Partner Examples

Slide 19

Horizontal Cloud Services Horizontal cloud administrations are establishments on which inhabitants fabricate applications or new administrations. They ought to be: without semantics. Must be "generic foundation," and not fixing to particular application rationale. May give the capacity to infuse application rationale through very much characterized APIs Broadly relevant. Must be extensively appropriate (i.e., it can't be expected for only maybe a couple properties). Blame tolerant over ware equipment. Must be assembled utilizing cheap item equipment, and ought to cover part disappointments. While every cloud benefit gives esteem, the force of the cloud worldview will rely on upon an accumulation of well-picked, inexactly coupled administrations that on the whole make it simple to rapidly create and work imaginative web applications.

Slide 20

Yahoo! Cloud Stack EDGE Horizontal Cloud Services YCS YCPI Brooklyn … WEB Horizontal Cloud Services VM/OS yApache PHP App Engine APP Provisioning (Self-serve) Monitoring/Metering/Security Horizontal Cloud Services VM/OS Serving Grid … Data Highway STORAGE Horizontal Cloud Services Sherpa MOBStor … BATCH Horizontal Cloud Services Hadoop …

Slide 21

Yahoo! CCDI Thrust Areas Fast Provisioning and Machine Virtualization: On request, convey an arrangement of hosts imaged with coveted programming and designed against standard administrations Multiple hosts might be multiplexed onto the same physical machine. Clump Storage and Processing: Scalable information stockpiling upgraded for group handling, together with computational abilities Operational Storage: Persistent capacity that backings low-inertness overhauls and adaptable recovery Edge Content Services: Support for managing system topology, correspondence conventions, reserving, and BCP Rest of today's discussion

Slide 22

Web Data Management CRUD Point queries and short outputs Index composed table and irregular I/Os $ per idleness Scan arranged workloads Focus on successive circle I/O $ per cpu cycle Structured record stockpiling (PNUTS/Sherpa) Large information examination (Hadoop) Object recovery and spilling Scalable document stockpiling $ per GB Blob stockpiling (SAN/NAS)

Slide 23

Hadoop: Batch Storage/Analysis Why is cluster preparing essential? Whether it's reaction forecast for promoting machine-learned importance for Search, or substance advancement for group of onlookers, information serious registering is progressively key to everything Yahoo! does Hadoop is vital to tending to this need Hadoop is a contextual investigation in our cloud vision Processes huge measures of information Provides flat scaling and adaptation to internal failure for our clients Allows those