University of Wisconsin - Madison, High Energy Physics

Position ID:

Univ-High Energy Physics-INT [#29772]

Position Title:

Computing - DevOps Intern

Position Location:

Illinois, United States of America

Subject Areas:

Computer Science / Databases, Computer Systems, Computer Architecture, Distributed Systems and Networking, Operating Systems, Scientific Computing
Computer Science and Electrical Engineering / Admin
Particle Physics / experiment

Appl Deadline:

2025/03/17 11:59PM help popup

(posted 2025/02/21, listed until 2025/06/19)

Position Description:

CMS Site Support and Tier-0 Operator

Overview

The Compact Muon Solenoid (CMS) site support and Tier 0 teams are looking for a highly motivated and skilled DevOps Engineer to join our effort. This person would be both running the Tier-0 and working with site admins of 70 computing sites around the world to ensure they are performing at capacity.

Program Description

CMS is an experiment that investigates a wide range of physics using the Large Hadron Collider (LHC) at CERN in Geneva Switzerland. CMS Computing Operations manages these central workflows starting with processing freshly recorded data at CERN, also known as Tier-0 Processing, to re-processing data and producing Monte Carlo simulations at worldwide distributed computing centers. The CMS Collaboration operates a vast, distributed computing infrastructure to record, process, and analyze petabyte-scale data across approximately 70 global computing sites. This infrastructure utilizes sophisticated grid technologies and state-of-the-art software systems to achieve high throughput for data processing and transfer. CMS requires expert Site Support Operators to monitor site performance and ensure the uninterrupted operation of services that are critical for delivering accurate physics results.

The first stage of the processing infrastructure of CMS is handling data just recorded by the detector. The so-called Tier-0 converts the binary data recorded by the detector into a custom file format best suited for processing and analysis, and performs the very first reconstruction pass to ready the recorded data for physics analysis. It also repacks the data into larger files suited for transfer to other GRID sites, tape storage and large-scale processing. The Tier-0 is developed with a very high degree of automation. Changing data taking conditions of the LHC and the detector make it necessary though to constantly monitor the infrastructure by experts and handle all arising issues quickly. A good overview is shown in the following animation: http://cds.cern.ch/record/1541893?ln=en ;

Job Description

CMS is looking for a DevOps engineer based at Fermilab to operate the Tier-0 and monitor site performance and ensure the uninterrupted operation of services that are critical for delivering accurate physics results. The candidate will be trained in all technologies used for both Site Support and Tier-0 operations. The successful candidate will work with admins from 70 data centers around the world to help optimize resource usage, and debug site issues, as well as contribute to the operations of the Tier-0. A site support/Tier-0 operator is monitoring the Tier-0 system, maintains its health and debugs problems together with site admins and system developers. Sometimes flexibility in the time schedule is required. They will need to be able to handle time critical tasks and services and take over mission critical monitoring responsibilities. The successful candidate will be able to work on the further optimization and automatization of the system and is expected to be able to work both within an international team, and independently to solve problems and implement solutions quickly and effectively. The monitoring infrastructure for sites and services is provided by CERN and shared by all LHC experiments. It is based on Nagios, HDFS, ElasticSearch, InfluxDB, Kibana, and Grafana. Probes are experiment specific and the successful candidate is expected to contribute to maintenance, enhancing, and development of additional probes as time permits.

Technical competencies:

Bachelor's degree in computer science or engineering or similar
Experience with software development and code design

Python
Shell scripting
Experience with SQL databases (Oracle knowledge is a plus)
Any experience with NoSQL databases is a plus
Experience with common web technologies like HTML, CGIi, and JavaScript
Experience with Nagios and/or Grafana

Experience with Linux and Grid/Cloud infrastructures and technologies

Proficiency with Linux administration, services, and troubleshooting
Experience with batch systems and high throughput computing on Grid/Cloud
Experience with ticketing systems and computing service support

Behavioral Requirements

Good communications skills

Must be fluent in English, both written and oral

Strong organizational skills
Demonstrating flexibility: adapting quickly and resourcefully to shifting priorities and requirements.
Ability to work independently
Working in teams:

ability to work with an international team
handing duties off between time zones

Learning and sharing knowledge: keeping up-to-date with developments in your own field of expertise and readily absorbing new information.

Details

This is an opening for a term Research Intern hired through University of Wisconsin Madison (UW-Madison) and stationed at the Fermi National Accelerator Laboratory in Batavia, Illinois, USA. The successful candidate is required to have a Bachelor's degree in engineering, computing, mathematics or natural sciences. The stay is limited to 1 year with the possibility for an extension for a second year.

The successful candidate is provided with:

Salary as determined by experience/qualifications
Benefits (e.g. health, dental and vision) are included.
Travel to Fermilab.
Paid legal holidays as determined by UW-Madison.
The UW-Madison visa office provides assistance to acquire a visa, if needed.
Reasonable accommodations for qualified individuals with disabilities.

The University of Wisconsin is an Equal Opportunity and Affirmative Action Employer. Qualified applicants will receive consideration for employment without regard to, including but not limited to, race, color, religion, sex, sexual orientation, gender identity, national origin, age, pregnancy, disability, or status as a protected veteran and other bases as defined by federal regulations and UW System policies. We promote excellence through diversity and encourage all qualified individuals to apply. For more information regarding applicant and employee rights and to view federal and state required postings, click here.

Application Materials Required:

Submit the following items online at this website to complete your application:

Cover letter
Curriculum Vitae

Further Info:

Department of Physics
1150 University Ave
Madison, WI 53706