Blog
-
Livable Code
April 19, 2018
This was originally an entry in my Rails Conf note series but this talk deserved it’s own post. Sarah Mei give a really insightful talk with a unique way to view software. I will attempt to summarize it here but I would recommend listening to the talk when it comes out.
Summary
Think of software as a creative and iterative process over a purely technical endeavour. The talk began talking about Conway’s Law: The structure of the system a team or organization builds will match their communication structure. This means messiness in communication will translate into messiness in the code base. Often software teams work as independent technical teams when they creative teams. Programming can be creative and the systems we build rely on communicating and novel solutions.
Software is regularly compared to architecture, but Sarah argues that this is wrong. Architecture has three well defined stages:
- Design
- Construction
- Maintenance
These three stages are handled by separate teams and require separate skills. Software used to be built like that, but not any more. Now we have libraries and frameworks that provide the structure. Instead we are living inside of these buildings that are the technologies we choose. These technologies give us the outline and walls with limitations and design choices we must live with or work around. We now focus on making our code livable.
Inside our “home” of code we need to create value and solve problems. Also the happiness that we feel depends on who we live in the code with. Everyone has their own quirks and habits, but together you work and live together. This living together in a house analogy was extended to hoarding and messiness. When you let laziness seep in, little decisions and messes compound until it’s an episode of Hoarders.
The answer to this isn’t to through out the house and start again, the same habits and behavior that made the mess will slowly creep in to the new system. To prevent this mess approach coding with the livable code mindset rules:
- Don’t make it worse.
- Improvement over consistency: but be sure to communicate the direction and goal.
- Inline everything: do the easy things instead of saying you’ll come back to it.
- Talk more about the code.
- Don’t always ask for permission, but be upfront with what you are doing
- Don’t ask for forgiveness
- Be open to feedback but don’t always listen
Code gets messy if care is not regulary cleaned and organized. Blog post and book sample code is comparable to staged homes. Those aren’t livable, just like the showrooms and photos of houses for sale hide the clutter and the things you actually need in a home. This analogy really resonated when she pulled up a photo of a bedroom on a house listing and picked apart how it wasn’t livable. No chargers on the end tables, no dresser, bed was on the floor to make the room feel spacious, etc. This is staged, just like overly abstracted code.
Modern software development is building and using generic components that together build a home, and we live there so keep it livable.
Takeaways
- Think of code as a home, it needs to be livable, you should organize and clean but a little mess is okay.
- Value continous improvement over consistency and correctness.
- Rewriting code will not prevent the next codebase from getting messy, instead fix the process.
Sources
-
Rails Conf 2018 Day 2
April 18, 2018
I visited Rails Conf and Pittsburgh this week and I wanted to post my notes from the talks I went to. Here are the notes and takeaways from day two.
Keynote: The Future of Rails 6: Scalable by Default
Speaker: Eileen M. Uchitelle
Notes
Rails is scalable out of the box up to a point. Once Rails gets to Github size though there are some workarounds and optimizations most teams do. These optimizations are going to be built in to Rails 6.
Some Rails 6 scalable by default improvements:
- Multiple database support with a 3 teir database.yml
- Faster test suite
- Parallel testing out of the box.
When companies try to solve problems they often only look inward. Eileen encouraged developers to think about building general solutions and open sourcing them to help the community. Scalablility problems face a lot of people. This encouraged me to reinitiate open sourcing a project that addresses one issue of scaling.
Takeaways
- Some cool new features are coming in Rails 6 which should help tests run faster and databases to be easier to work with.
- When extending Rails to support your scale, take a look at what you are building and try to make it generic and open sourced.
So You’ve Got Yourself a Kafka: Event-Powered Rails Services
Speaker: Stella Cotton
Notes
Kafka is used to stream data for data pipelines and event driven applications and pipelines. Kafka guarantees at least once delivery within partitions. Think of a Kafka partition as a long log file with indexes and guarantees of ordering, a Kafka cluster is made up of many partitions. Applications will have multiple partitions but you should put related events on the same partition for ordering. Kafka improves speed and indepence over using RPC for distributed events processing but it does remove the explicit dependencies.
Martin Fowler defines four types of event systems in his blog post: What do you mean by “Event-Driven”?
- Event created: Only send an event an id. Downstream services will call sender if they need more information.
- Event + Information: Event with id, event and state changed information. No reliance on calling the sender, a downstream server has all it needs.
- Event Sourcing: All state changes are admitted as events and you can rebuild the state of the world from replaying all events.
- Command Query Responsibility Segregation(CQRS): Split events into read and write. Good reference here: CQRS
Suggested using Avro for Kafka schemas. Scaling can be as easy as adding more consumers but there needs to be metrics on latency. Consumers that are slower than the requests coming in or paused for a while may have a hard time catching up.
Takeaways
- Kafka can help scale applications but be careful about hidden dependencies or coupling.
- Metricisize and make sure consumers keep up with producers and failures can be recovered from.
Postgres 10, Performance, and You
Speaker: Gabe Enslein
Gabe is on the Heroku Postgres team and summarized some of the cool things coming to Postgres 10.
Notes
- Native partitioning in default installation.
- Tables can be created from partitions
- Hash indexes are now first class citizens and don’t have corruption issues. This is big! When asked who uses hash indexes no one raised their hands, we don’t because of the WAL log warning and limited support.
- Much better parallel querying support. Gather is now ordered, this means parallel scans and merge joins can have natural ordering.
Takeaways
- Partition support and better parallelization of queries is coming.
- Rails 4 doesn’t support any of the new features and no native Ruby gems support the partitoning or new Postgres specific indexes yet.
Five Sharding Data Models and Which is Right
Speaker: Craig Kerstiens
Notes
What is sharding? Sharding is seperatinga large database into smaller faster databases. Tables on different nodes allow performance gains. Tips for sharding:
- Shard on hash or range but still hash ranges so that they are distributed across nodes. This helps prevent clustering.
- Define the number of shards up front and go larger than you expect to ever go. This way you can add new nodes without needing to reshard.
- Can also bucket time series into tables to make easy bucketing and drop old data.
Their are five general ways to colate data:
- Geography: For when there is clear geographic boundaries. Example Uber, Instacart
- Multitenant: Each customer has their own shard. Will not work well if one customer takes up disproportianate amount of database, >10% may mean sharding doesn’t help much.
- Entity ID: sharding on an id if there aren’t joins that are needed. Best for aggregations.
- Graph Database: shard on a few relation types and replicate duplicate data. Check out paper TAO distributed graph datastore.
- Time Series: event data and metrics can be sharded by a time period. Works best when dropping older data.
Takeaways
- Not sharding is easier, try to wait but when you commit plan for the big scale.
- When looking to pick a sharding strategy. Use process of elimination instead of guessing which way would work best.
Ales on Rails: Making a Smarter Brewery with Ruby
Speaker: Ben Shippee
This was more of a fun talk without many takeaways but fun to see someone building a cool home built system for managing a brewery.
Notes
Rails application was custom built for Brew Gentleman. The goals were to make managing the brewery easier. Features that they built to automate the brewery:
- What’s on tap, kicked with easy management for workers.
- PDF menu and label generators generators.
- QR Code inventory management system.
- Special releases reservations.
- Lot’s more!
Takeaways
- Sounds like a fun project and I definitely would have fun doing something similar.
Containerizing Rails: Techniques, Pitfalls, & Best Practices
Speaker: Daniel Azuma
Blog post from speaker of the talk here: Containerizing Rails: Techniques, Pitfalls, and Best Practices (RailsConf 2018)
Notes
Tips for containerizing your application:
- Read and understand the base image.
- Combine update, install and clean commands in one run line to prevent bloat of image.
- Use multi stage Docker files to have an image for building dependencies and then copy over the built app without development dependencies.
- Set locale in the Dockerfile to potentially avoid some weird Ruby string errors.
- Run your app under an unprivileged user still!
- Prefer the exec form:
CMD ["bundle", "exec", "rails", "s"]. This ensures that the stop signals are sent to the program and not the shell. - Can get around 6 with by prefacing cmd with
exec. - Avoid using onbuild because it makes assumptions about how image is used.
- Always specify resource constraints to help Kubernete’s plan workload.
- Avoid preforking in a container, instead have one process per container.
- Scale by adding containers.
- Send logs outside the container, either an agent or standard out.
Takeaways
- Keep your containers small, defined workloads and constrained within limits.
- Reduce image size by using build images and running multiple commands in line.
-
Rails Conf 2018 Day 1
April 17, 2018
I visited Rails Conf and Pittsburgh this week and I wanted to post my notes from the talks I went to. Here are the notes and takeaways from day one.
Keynote: Fixme
Speaker: DHH
We wouldn’t have Rails without the focus and work of DHH and this talk was a great start to the conference. The talk focused on learning and making Rails easy and accessible to everyone through conceptual compression.
Notes
Conceptual Compression
Technology is constantly changing but at the same time it is staying the same. Some technologies change but others change the same. Rails attempts to reduce the mindspace needed develop for the web.. DHH gave the example of ActiveRecord allowing people to build websites without needing to understand SQL. This lowers the barrier to entry. We no longer need a DBA’s specialized knowledge and can just use the developers. We can now get a lot more done without worrying about details that our predecessors had to. All of this is enabled by conceptual compression, taking complicated problems, solving them and then abstracted them so day to day development doe not rely on the specialized knowledge.
Embrace Leaky Abstraction
Rails has tons of leaky abstractions, and DHH is proud of this. It’s an important tool for getting basic functionality out. Day one doesn’t have a perfect solution but you can help manage away some complexity. Where complexity leaks through users can dive deeper when they need. Defined this as JIT(Just in Time) learning.
Rails is Inclusive
The internet has been colonized by large companies. Rails attempts to keep the bar of entry low to encourage growth and new opportunities. Low barriers to entry and more businesses is benificial for everyone.
Takeaways
- Conceptual Compression: software is improving and becoming easier; advanced topics can be reduced to simpler extractions that can be learned later.s
- JIT Learning allows developers to go far without knowing everything, learn more when you need it.
- Embrace leaky abstractions to help enable just in time learning.
Continuous Deployments and Data Sovereignty: A Case Study
Speaker: Mike Calhoun
Notes
Different countries have different data laws. Working in a place like Australia means users Private Identifying Information can not leave the country. This complicates the deployment and development structure. Originally they tried doing separate branches per region it wasn’t practical. Ended up just doing completely separate deployments per region, with little interoperability.
If cross region support is required One solution is to have separate regional deployments and then a service that has a (region_id, user_id) pair but none of the identifying information.
Takeaways
- Think hard about the audience before building and structuring your app if you will be multi-region.
- Consult lawyers.
Scaling the Software Artisan
Speaker: Coraline Ada Ehmke
Notes
Coraline argued developers are software artisans not software engineers. Creating custom solutions is being commoditized away. More and more software is generic and composed of smaller open source or closed source subsystems, this is like manufacturing standardizing on parts. “The job of the future is not your job today” As a software artisans career progresses it’s less about the code, instead you are asking four questions:
- Innovate:
- Should we solve this with code?
- Can we fix this with better communication?
- Is this something we should build ourselves?
- Should we extract this into a library?
- Standardize:
- How do we want to do things?
- How do we build consensus? Be inclusive and allow dissent
- How do we spread best practices?
- Mutlitply
- Do we have a good mix of people?
- Are we pairing people effectively?
- What can I do to level the team up?
- Be Ethical:
- What are the consequences and reprocussions of this code?
- Remember that bias, selling data and algorithms aren’t neutral.
Takeaways
- Software is changing and being commoditized and our jobs will change.
- Stay on top of the change by innovating, standardizing, multiplying and being ethical
- We are accountable for the software we create
Down The Rabbit Hole: An Adventure in Legacy Code
Speaker: Loren Crawford
Notes
Legacy code: system of working software written by many.
When deciding whether to change or not change, if no one is reading it then don’t change it.
“The hurrior I go, the more behinder I get” ~ Lewis Carrol
Most of the material covered in this talk I already summarized here: Working Effectively with Legacy Code
Takeaways
- Watch: A Common Taxonomy of Bugs and How to Catch Them
- Read books: “Refactoring”, “Working Effectively with Legacy Code”, “Ruby Object Oriented Design”
The Life and Death of a Rails App
Speaker: Olivier Lacan
Olivier Lacan worked at Code School and he walked through the stages of birth through death of their startup and team. I really enjoyed this talk.
Notes
Sucess for them came from
(timing + content) * audience.Stages of a company/software project:
- Growth: As the company grew they added dependencies: both human(customers) and software.
- Butterfly: When they started growing they got to a stage where they started to think they were special and justifying why they needed to do something different. Spoilers you aren’t different.
- Adulthood: They recognized that the decisions they made were not the only solution.
Faced team instability and lack of ownership. Making diverse hire’s helped with to counteract this and other problems. Introduced the concept of immunity for new team members, with permission to question and criticize decisions. I think this immunity should be indefinite.
You need to distrust the process. What is right for the company changes and process makes things comfortable but not necessarily efficient. Continue to question choices.
Some warnings:
- Beware of not invented here syndrome, there are a lot of great tools out there. Remember you aren’t special.
- Devalue code and overvalue your people.
- Security is important and bug bounty programs are a good start.
- Not listening to security issues isn’t a security strategy.
“Sunset” and “Acquihire” are euphemisms and hide the truth that an application is dying or being killed.
Takeaways
Post-Mortem:
- The legacy is the people the team and customers that you built and reached.
- Ditch toxic people ASAP.
- Nurture good people.
- Share ownership and allow mistakes.
- Involve yourself in the ecosystem you live in, i.e. open source, community etc.
- Accept mortality (never enough money, time, talent, etc).
- Share healthy habits and ideas.
- Focus on joy.
- Learn by doing.
- Just In Time learning is okay, learn and discover as you grow.
Knobs, Levers and Buttons Tools for failing gracefully
Speaker: Amy Unger
Amy is a software engineer at Heroku and talked about managing failure in the software like pilots in a cockpit.
Notes
Lots of types of knobs, levers and buttons to help manage failure.
- Maintenance Mode: big on/off lever to put your site offline if there are issues
- Read Only Mode: requires splitting writes and reads but can be useful for some applications. For example you can disable posting/comments but enable browsing.
- Feature Flags: Disable failing features behind a switch.
- User: per user feature flags, less useful when tons of users.
- Global: disable a feature for everyone
- Group: disable a feature for groups, i.e. admins, partners, enterprise customers
- Rate Limiting: Load shedding. Best implemented with a default value and have ratio modifiers on the individual level to allow graceful scaling of load.
- Stop Work: Being able to turn off non-critical work is important in error situations.
- Known Unknowns: When you want to roll something out but are unsure of the consequences, github/scientist or similar can help.
- Circuit Breakers: Responsive shut offs and hard shutoffs can help manage failure.
Takeaways
There are a lot of different ways to manage failure to use depending on requirements. Using ratios for load shedding was something I had never considered and sounds valuable and easy.
Turbo Boosting Real-world Applications
Speaker: Akira Matsuda
This talk was a dive into some slow parts of Rails and hacks to make things faster. Most of this was around adding parallelization to database queries and view rendering.
Takeaways
There are some ways to parallize sending database queries to speed up responses but you have to be careful of database connections. The futures pattern is easy to use in Ruby and can enable async work
Thread.new { do_something }.value
Keynote: Rails Doesn’t Scale
Speaker: Mark Imbriaco
Clickbait talk title that was less focused on scaling code and more on scaling teams.
Notes
The social side of scaling an application isn’t as interesting as scaling the team. If you can scale a team effectively than they can scale your application. Hero culture doesn’t scale; teaching scales as you add people. So scale up your team with good teaching practices.
Takeaways
- Just because you learn a lesson once doesn’t mean you won’t repeat it.
- People are what scale technology, so scale your people.
-
She Connects Women in Tech
March 22, 2018
Venture Cafe had an event She Connects, a night celebrating women entrepreneurs. I attended a town hall on women in tech and here are my notes. The forum was broken into four categories, pipeline, mentoring, leadership and strategic partnerships.
Pipeline
Encouraging women in tech, bringing and keeping them in the workforce.
- Learning and motivation does not come from expensive kits, it can just come from encouragement, role models, and learning fundamentals.
- A suggestion was raised to socialize young boys and girls to see females as engineers. Having boys see women engineers can go a long way for them accepting their female peers as future engineers.
- Job descriptions need to be inclusive and not signalling male culture. Example given was stereotypical male nerd culture like Star Wars should not be in a job description.
Mentoring
Mentoring and growing women in tech.
- Mentoring is a two way relationship, there needs to be a reciprocal benefit. How can you add value for your mentor?
- Male mentors are just as important as female mentors. Males have the ability to make a difference. Males may also have privilege they can leverage to help their mentee.
- If you benefited from a mentor, consider giving back and “sending the elevator back down”
Leadership
How women in tech can be a leader.
- Part of being a leader is confidence.Confidence is knowing the value you represent. Know what experience, education, and skills you have.
- Leaders are forward thinking. Work from where you want to be in a year backwards and then execute. Apply this same strategy to the business.
- Passion is important in a leader and your team members can tell if you are passionate.
- Female leadership often looks different than the male assertive culture. Work to redefine leadership to be more nurturing and reflective.
Strategic Partnerships
Connecting women to opportunities that may be typically for men
- Women are given less opportunities than men because of culture and assumptions.
- Forums focused on women entrepreneurs are a great way to be open, give back and keep the conversation flowing.
- Some aspects of start-up culture like pitch competitions are more suited for male bravado and can be exclusive.
-
Working Effectively with Legacy Code
March 04, 2018
I have started working my way through the book Working Effectively with Legacy Code. I have been recommended this book multiple times and purchased it a while ago but hadn’t gotten around to reading at. A few take-aways I got from the first few chapters:
What is a unit test?
According to the author a unit test isn’t a ‘unit’ if:
- It talks to a database
- Makes network calls
- Touches the file system
- Relies on a special environment configuration/setup
Rules for changing legacy code
The author identifies the following steps to making changes.
- Identify change parts
- Find the test points
- Break the dependencies.
- Write tests for this code
- Make changes and refactor
-
2018 Goals Progress
February 04, 2018
I didn’t create a post outlining 2018 goals because it felt a little cliché. Now that I’m a month into my goals and the hub-bub and planning has died down I wanted to document my public goals.
Fitness Goals
- Run a half marathon
- Climb at V5 bouldering/5.12 top rope
- Go on a backpacking trip
Professional Goals
- Blog regularly(minimum 6 posts)
- Attend a tech conference
- Launch two simple projects
- Improve writing through blogging and reading
Learning Goals
- Take two online classes in new subjects
- Read 40 books
Hobby Goals
- Make 24 new dishes
- Learn some Korean
I hope to follow through with these goals and will be blogging about my progress.