Senior Software Engineer

Wells Fargo is seeking a Senior Software Engineer.

We are looking for an experienced and fully hands-on Application Support Engineer with deep understanding of Reliability Engineering (SRE) principles, and considerable practical experience implementing them in complex enterprise environments, to join our WIMT Platform team. This Application Support Engineer / SRE will assume Operational Ownership (in partnership with Software Development and Business owners) of a subset of WIMT systems and applications, with the primary focus on improving Reliability and Supportability -- by acquiring deep understanding of the in-scope systems, and using SRE principles to operate them.

Ideal candidate enjoys and takes pride in solving complex technical problems, using creative approaches to troubleshooting and weakness analysis, automating routine or risky manual operations, sharing knowledge and experience with colleagues, and leading others by personal example. Candidate will join a passionate engineering Team that combines various skillsets and experiences as Systems Engineers, Software Engineers and Infrastructure Engineers, to achieve strategic reliability and everyday operational health goals. Additional key responsibilities are listed below.

In this role, you will:

• Lead moderately complex initiatives and deliverables within technical domain environments

• Contribute to large scale planning of strategies

• Design, code, test, debug, and document for projects and programs associated with technology domain, including upgrades and deployments

• Review moderately complex technical challenges that require an in-depth evaluation of technologies and procedures

• Resolve moderately complex issues and lead a team to meet existing client needs or potential new clients needs while leveraging solid understanding of the function, policies, procedures, or compliance requirements

• Collaborate and consult with peers, colleagues, and mid-level managers to resolve technical challenges and achieve goals

• Lead projects and act as an escalation point, provide guidance and direction to less experienced staff.

• Operational Ownership / Application Support:

• Maintain system operational knowledge (functional and technical)

• Understand and monitor system operation, ensure optimal availability, functional health, and performance (driven by SLO/SLA)

• Triage alerts, respond to incidents, perform root cause analysis (troubleshooting)

• Handle users' questions and requests related to business systems (not a Desktop Support)

• Change requests implementation (manual deployment steps, overall deployment coordination)

• BCP planning and implementation

• Ensure continuous improvements of operational processes and methods

• Reliability Engineering:

• Analyze system's monitoring and observability needs (technical, functional, business), and create or adjust logging, monitoring, alerting and analytics solutions to cover those needs

• Use understanding of software engineering (system code) and infrastructure to improve the depth and quality of root cause analysis (troubleshooting)

• Partner with Architecture, Infrastructure and Development teams to influence decisions that impact reliability and supportability

• Identify routine or risky manual operations, and create automation solutions (scripting, tooling) or influence fixing the sources of manual work (as appropriate)

• Drive deeper post-incident reviews for major incidents, to learn and improve

• Engage in weakness research and analysis, and architectural reviews, to use deep knowledge of production operation to suggest improvements

• Use deep knowledge of production operation to create detailed high-quality stories and tasks on DEV owners' backlog, with the focus on reliability and supportability

• Ensure continuous improvements of systems' reliability and supportability

Required Qualifications:

• 4+ years of Software Engineering experience, or equivalent demonstrated through one or a combination of the following: work experience, training, military experience, education

Desired Qualifications:

• 12+ years of software engineering experience

• 5+ years of application production support experience

• Education BS/BA degree or higher

• An industry-standard technology certification

• Strong verbal, written, and interpersonal communication skills

• 3+ years of experience with Cloud technologies    

• Knowledge and understanding of Site Reliability Engineering (SRE) concepts         

• 3+ years of Agile experience            

• Advanced scripting skills specifically around automation, log rotation, data collection, error collection and alerting

• Scripting and automation experience

• Experience with complex business logic and dependencies

• 3+ years of CI/CD automation and configuration experience (DevOps / pipeline automation)

• 3+ years of experience with ITSM processes (e.g. Incident Management, Change Management, Asset Management and Configuration Management)  

• Hands-on experience with writing / maintaining technical documentation such as fixlogs, runbooks, knowledge base, architectural diagrams

• Hands-on experience with system administration across multiple platforms

• Hands-on experience with one or more software development languages: Java, JavaScript, Ruby, Python, JSON, Angular, NodeJS, .Net/C#

• Hands-on experience with one or more CI/CD automation tools: Jenkins, Gradle, Maven, Git, SonarQube, Artifactory, Ansible, Puppet, Apigee

• Hands-on experience with one or more process management and scheduling tools: Autosys, ControlM

• Hands-on experience with one or more Monitoring/Observability/APM/Analytics tools: Splunk, Elastic, Kibana, Grafana, Prometheus, AppDynamics, Dynatrace, New Relic, DataDog, Kafka, CloudWatch, Jaeger, Zipkin, Big Panda, TrueSight

• Hands-on experience with one or more Server OS: Windows, Linux, Unix, Mainframe

• Hands-on experience with one or more Cloud and virtualization technologies: Azure, GCP, AWS, PCF, PKS, Kubernetes, OpenShift, VMware

• Hands-on experience with one or more Data storage, management and messaging technologies: Kafka, IBM MQ, Apache Airflow, Logstash, Spark, Oracle, SQL, MongoDB, Cassandra, Hadoop, Cloudera, AWS EMR, S3

• Hands-on experience with one or more Testing Frameworks: Selenium, JMeter, Blazemeter, Performance Center, Perfecto, Cucumber, Gherkin, ALM, Gremlin, Chaos Monkey, Chaos Toolkit, Simian Army, Toxi Proxy

• Working knowledge of TCP/IP networking, experience analyzing packet captures to assist in troubleshooting

• Working knowledge of Internet technologies: routing, NAT, firewalls, load-balancing, proxies, web servers

Job Expectations:

• Ability to work additional hours as needed

• Ability to work on call as assigned

• Flexibility to work in a 24/7 environment, including weekends and holidays