Verticalmove is responsible for building one of the most cutting-edge technology groups throughout Microsoft. The Experimentation and Analysis (ExP) group empowers every major product within Microsoft to objectively influence iterative changes to these products and their features through intelligent, data-driven, controlled experiments.
The internet connectivity of client software (e.g., apps running on phones and PCs), web sites, and online services provide an unprecedented opportunity to evaluate ideas quickly using controlled experiments, also called A/B tests, split tests, randomized experiments, control/treatment tests, and online field experiments. Unlike most data mining techniques for finding correlational patterns, controlled experiments allow establishing a causal relationship with high probability. Experimenters can utilize the Scientific Method to form a hypothesis of the form "If a specific change is introduced, will it improve key metrics?" and evaluate it with real users.
The unofficial shorter story...
It’s highly likely that if you use *ANY* major online service, Microsoft or not, you’re involved in an anonymous controlled experiment right now. Bing as an example conducted more than 10,000 A/B controlled experiments last year, obsessed in making Bing one of the best online experiences for it’s users in the world.
You can read further about the ExP group via these links below:
What if your job description were simply “Make tomorrow better?” That’s the essence of roles within our team.
The Analysis and Experimentation (A&E) team is part of the Artificial Intelligence and Research Group. We help just about every Microsoft product you can think of – Xbox, Cortana, Bing, Bing Ads, Windows, Office, Skype, and more – We measure whether changes to features, rankers, models, or content actually help people. It’s common for these experiments we help run to uncover new opportunities worth tens of millions of dollars or help hundreds of millions of people.
Our 30 or so data scientists help sift through petabytes of data to find sometimes tiny but important changes that help customers. Some teams at Microsoft that we work with are new to A/B testing and need to learn about what questions they can answer with experiments and how to answer them. Others are more sophisticated, running almost everything on their own, only coming to us for advice on experiments and/or asking for help as data detectives to understand mysterious movements and their possible causes.
Our data scientists do a blend of coding, statistics, and research. The old joke is that 80% of data science is data cleaning, and new logs and data pipelines often need quite a bit of coding for cleaning and debugging. Coming up with new metrics, especially metrics that turn raw data into measures of people’s happiness, requires an understanding of statistics, as does debugging mysterious metrics movements as real impacts or issues in the data. And, running one of the largest experimentation platforms in the world means we’re often researching new ways to detect small signals in the noise or improve our ability to rapidly provide insights, some of which we publish at academic conferences.
If touching just about every Microsoft product you can think of and helping hundreds of millions of people with coding and statistics excites you to, let’s talk more! We’d love to have you join us.
• Work directly with application teams/partners (internal clients such as Xbox, Skype, Office) to understand their offerings/domain and help them become successful with data so they can run controlled experiments (a/b testing).
• Leverage your statistical and computational knowledge to build algorithms for calculating variances.
• Understand the data generated by experiments, and producing actionable, trustworthy conclusions from them.
• Handle large amounts of data using various tools, including your own. We prefer C#, Python and SQL but are open to all OO programming languages.
• Build data manipulation, data processing, and data visualization tools and share these tools across Microsoft.
• Apply data analysis, data mining and data processing to present data clearly and develop experiments (ab testing)
• Ensure high-quality data and understand how data is generated out experimental design and how these experiments can produce actionable, trustworthy conclusions.
• Work with development team to build tools for data logging and repeatable data tasks to accelerate and automate data scientist duties.
• Assist senior management in making key business decisions.
• 3+ years of experience working with large data sets or doing large scale quantitative analysis
• 5+ years of professional experience.
• Expert SQL scripting required.
• Strong coding abilities. Preference towards knowledge of one of the following open source languages: Scala, Java, C++ or C#. We use Python and C#.
• Experience working with Hadoop, Pig/Hive, Spark, MapReduce
• Experience manipulating large data sets through statistical software (ex. R, SAS) or other methods.
• Ability to drive projects.
• Fundamental understanding of statistics – hypothesis testing (t-test, p-values), confidence intervals, regression, classification, and optimization.
• Strong algorithmic problem-solving skills.
• Superior verbal, visual and written communication skills to educate and work with cross functional teams on controlled experiments.
• A willingness to learn, share, and improve.
• Experimentation design or A/B testing experience is preferred.
• Bachelor’s or Master’s degree in Computer Science, Math, Physics, Engineering, Statistics or other technical field. PhD preferred.