Introduction
I’m a scientific programmer who’s been working in the Oil and Gas Software industry for over a decade. I can think programmatically. I’ve written thousands and thousands of lines of code. I know 5+ programming languages. But I know my limits! I am not a commercial software engineer.
Many of you may also be like me – you might have a science (or better, data science) background; you might have excellent Python or Matlab fluency; you may have done some HPC work using clusters with thousands of nodes. Your friends and family might describe you as a programmer!
As you transition into commercial applications of software development, it’s important to keep the following five points in mind. Commercial Software Engineers:
Focus on stability
One of the first and most important differences between scientific programming and commercial engineering is a focus on stability. In scientific programming, if my code crashed or numerical instabilities caused poor results, I just fired it back up and tried again. For commercial applications, software defects can literally cost lives (eg, 737 MAX).
Commercial engineers must anticipate that software inputs (data, measurements, or manual user inputs) may frequently contain errors or be otherwise problematic. They carefully engineer validation methods to screen for these problems, identify critical issues, and gracefully programmatically handle any subsequent errors that would cause the system to fail. A lot of development time is spent thinking through infrequent, low probability edge cases that can cause stability problems that could have major impacts.
For example, those of us who remember the early days of the Windows operating system are familiar with the famous “Blue Screen of Death”. While still achievable (I should get a badge for that!), that it is a much less frequent occurrence is a testament to how Microsoft’s software engineers have developed methods to handle critical errors while keeping the system up and running.
Focus on scalability
When I think of scalability, I’m usually thinking about how many more nodes can I get to run a big calculation. Commercial developers, particularly for cloud applications, think a little differently. In this context, scalability is how the system’s performance is impacted in response to changes in the application’s processing demands.
For example, consider applications that require access to the same data storage. As the number of applications increases, can the storage respond to maintain consistent (and acceptable) application performance? Or, from a system consistency point of view, if two users are trying to access or change the same underlying data simultaneously, how does the system respond? What if there are thousands of users?
Or, think about quantity of data. The software may work great for a few hundred thousand of data points. Will it perform as well when there are tens of millions of data? What about billions? How will the software performance change over the next 10 years as IOT data sources proliferate?
Commercial software engineers must think beyond early applications of their software products and anticipate future states where stresses are considerably larger.
Focus on efficiency
If you’re used to writing code to run locally on your laptop like me, you may have thought about things like memory efficiency and tradeoffs between RAM and hard disk storage to optimize performance. But you probably haven’t thought much about cost because your laptop is a commodity product and you paid for it a year or two ago.
While atomized costs per byte or computation may be only fractions of pennies, as applications scale to thousands of concurrent users these costs add up quickly and commercial developers need to pay attention to if their software cost model fits their customer revenue model!
Focus on maintainability
argB0temp – yep, I’ll admit it, that’s a variable name I have floating around in one of my (very old) fortran90 codes. As I write this, I still can’t decide if it means “temporary” or “temperature.” It is not meaningful to me today, but surely it was important 20 years ago.
Commercial software developers understand that the code they write lives on far beyond their involvement with the project, and therefore focus on the maintainability of the code base. It’s a broad topic, and for simplicity I’ll just define it as the ability for a future person (including future you!) to both understand, troubleshoot, and improve an existing piece of software.
This goes beyond densely commenting your code explaining every line. It includes creating and updating code style guides (variable naming conventions, file names, etc.), philosophical discussions of library architecture (what’s going to be a function, what’s going to be a class or component), estimates of complexity (do I need to break this code up or consolidate), and anticipating future changes and documenting technical debt (choices have been made… we’ll pay for it later!).
Ultimately, code maintenance is a tradeoff between overhead today and overhead tomorrow, and it’s easy to procrastinate! The best commercial developers make sure to budget time today to improve maintainability when they are intimately involved with the project so that future developers can more quickly make value adding improvements.
Focus on user experience
Finally, scientific programmers usually have just one customer – themselves! We therefore take a lot of shortcuts like creating shell scripts to configure and run software that would be completely untenable in a commercial product.
Commercial software engineers, on the other hand, are laser-focused on the end-user experience. Can we make the application easy and pleasing to use? Can we make it intuitive – will users automatically know how to interact with the application? Can we reduce the number of interactive steps needed to see key results? Can we anticipate how the user will interact with the software and places they may struggle? Are there small changes we can make to the software that make the product more valuable to the customer?
Enhanced user experience ends up being the number one driver for product adoption, and commercial developers are happiest when other people are successfully using the software tools they’ve built.
What do you think? Have you experienced any of these issues? Let me know below!