Scalable web architectures

Ive been hearing this word “sharding” more and more often, and its spreading like fire. Theo Schlossnagle, the author of “Scalable internet architecutres” argues that federation is form of partitioning, and that sharding is nothing but a form of partitioning and federation. Infact, according to him, Sharding has already been in use use for a long time.

I’m not a dba, and I don’t pretend to be one in my free time either, so to understand the differences I did some research and found some interesting posts.

The first time I heard about “Sharding” was on Been Admininig’s blog about Unorthodox approach to database design (Part I and Part II). Here is the exact reference…

Splitting up the user data so that User A exists on one serverwhile User B exists on another server, each server now holds a shard ofthe data in this federated model.

A couple of months ago Highscalability.com picked it up and made it sound (probably unintentionally) that sharding is actually different from Federation and Partitioning. Todd’s post also points at Flickr using sharding.The search for Flickr architecture lead me to Colin Charles’ post about Federation at Flickr: A tour of the Flickr architecture where he does mention shards as a component of Federation key. Again no mention of Sharding being anything new.

Federation Key Components:

• Shards: My data gets stored on my shard, but the record ofperforming action on your comment, is on your shard. When making acomment on someone elses’ blog
• Global Ring: Its like DNS, you need to know where to go and whocontrols where you go. Every page view, calculate where your data is,at that moment of time.
• PHP logic to connect to the shards and keep the data consistent (10 lines of code with comments!)

Based on the discussions on these and other blogs, “Shards” sounds more like a terminology used to describe fragments of data which is federated across multiple databases instead of an architecture by itself. I think Theo Schlossnagle has a valid argument. If any of you disagree I’m interested to hear what you have to say. A clearer definition between sharding and federation would be very helpful as well.

Here are more references to Shard/Sharding.

• Hibernate Shards: Hibernate Shards allows you to continue using the Hibernate APIs you know and love: SessionFactory, Session, H Criteria, Query. If you already know how to use Hibernate, you already know how to use Hibernate Shards.
• Scaling Digg… Shards and DB: Another discussion on Digg here.
• Notes on sharding, unique keys, foreign keys
• Sharding databases with MySQL
• Database sharding helps high-traffic sites
• Sharding?

Popularity: 44%

Patrick Lenz, founder and lead developer of freshmeat.net was also responsible for the relaunch of another website eins.de which recently moved from php to ruby. Eins.de site serves about 1.2 million dynamic pages a day. He wrote a series of articles describing how they redesigned the site to scale for growth. I found these articles very informative with a extreemly mature discussion of the colorful world of scalability.

• The adventures of scaling, Stage 1
• Questions and answers for Stage 1
• The adventures of scaling, Stage 2
• The adventures of scaling, Stage 3
• The adventures of scaling, Stage 4

Here is the final summary of what they ended up after 4 months of optimizations

Systems optimization

• Use Linux 2.6 instead of 2.4
• Use self-compiled Ruby 1.8.4 instead of all else
• Use MySQL-supplied binaries
• Use lighttpd 1.4.11 instead of all else
• Use memcache-client instead of Ruby-MemCache
• Use a smaller amount of dispatchers
• Watch your dispatchers

Code optimization

• Avoid components
• Use memcached to store expensive computations
• Use memcached for sessions
• Don’t use live-previews if your site is popular
• Use the exception notification to be aware of raised exceptions

Popularity: 21%

In my other life I work with a medium scale web application which has had many different kinds of growing problems over time. One of the most painful one is the issue about “statelessness”. If I could only give one recommendation to anyone building a brand new web application, I’d say “go stateless“. But going stateless is not the same as going session-less. One could implement a perfectly stateless web architecture which still uses sessions to authenticate, authorize and track user activity. And to complicate matters further, when I say stateless, I really mean that the server should be stateless, not the client.

Basic authentication
Most interactive web applications today which allow you to manipulate data in some form require authentication and authorization mechanisms to control access. In the good old days “Basic Authentication” was the most commonly used authentication mechanism. Once authenticated, the browser would send the credentials to the server with every subsequent HTTP request. Over time, cookie and URL based session tracking mechanisms took over, which is what we are using in most of our web based applications today. In a one-server farm, you probably don’t need to pay attention to how these authentication, authorization or tracking mechanisms works. But once you have multiple servers in the farm, you do need to verify that a clients authenticated to one of the server is considered authenticated by all of the servers. Since, in “Basic Authentication”, the browser sends the authentication credentials with every request it doesn’t matter which server your session requests goes to. Each request is treated as a new session and credentials are verified every single time.

Transition to sessions
As traffic and security concerns grew, some of the smarter web applications switched to using “sessionids” in cookies and URLs. In this design, servers issue a sessionid (like a train/plane ticket) for every successful authentication and requests the client to send the sessionid instead of the authentication credentials for every subsequent request. This not only reduces the chances of someone sniffing your credentials, it also reduces the work done on the server because all it has to do now is check if the sessionid was issued by it in the last few minutes/hours (and is active), instead of validating the user against the all the users in the database for every single request.

Session replication
To understand which HTTP requests are part of which session, the server has to issue a unique HTTP session identifier to each of the user sessions. As it happens most web application servers also maintain an internal “sessionid” for each session, so inserting this sessionid in the cookie/url seemed to be the most natural thing to do. Unfortunately, if you have done any web programming, you will notice that unless you do something special, these default sessionids are local to each application server. Without doing special magic, like setting up application server in clustering mode, or persisting session information on shared resource (database/NFS) it would be difficult for one server to know which sessions are active on other application servers. This is where the scalability problem comes in. Replicating a lot of session information between 2 nodes might be simple, and 10 nodes might be possible.. but this architecture is not very scalable without a little extra work.

Anatomy of a session
“Sessions”, have three primary purposes.

1. It helps group together HTTP requests coming from same browser.
2. It can cache authentication credentials and key user info after user authenticates.
3. It allows caching of other information in session which might be temporary in nature.

If you have ever tried to create a new user account on a website, chances are that you would have gone through a couple of pages before the registration was completed. Most web applications keep the information from intermediate pages in temporary session variables within the application server. Developers love to use temporary session variables for such information and this approach of storing temporary information can grow into a beast which cannot be replicated or shared using shared resource. To design a scalable website now, not only would one have to persist the sessionid and authentication credentials onto some kind of shared resource, one would also have to figure out what to do with these temporary session variables.

Since sessions are created only once per session, and since authentication happens only once too, it should be possible to keep that kind of information in a central database or shared network cache like memcached. The temporary session variables on the other hand can be very noisy (changes frequently) and may need not be critical enough to be available to the other servers. One needs to find a balance between good performance and reliability. I know organizations which have decided to accept some level of data loss with temporary session variables to improve over all performance.

But the key thing to remember here is that every time a session is created or authenticated it should be copied over to some shared scalable resource. It could be something like a cluster of memcached servers, or a cluster of replicated database servers. This will guarantee that if the user does switch application servers the requests can be authenticated by checking this shared resource before continuing with what they were doing.

Session vs State
The information about where the user is or was within an application at any given moment is called the “state”. If that information is kept only on the server and is not replicated you will probably see errors of some kind when servers fail or switches user to another server. If the application server does persist the sessionid and user credentials in a central database, then, as far as the session is considered it wouldn’t matter which server the user goes to. Most of the new web applications today try to maintain the state information within the browser itself which frees up the server from the responsibility of storing, maintaining and replicating state information across all the servers.

Final thoughts

Loadbalancers: If you are doing what I did, you’d be calculating the probability of a user switching servers mid-session. In our environment we noticed about one in 50 servers fail every month. Such disasters will force all sessions created on that server to another server. Occasionally we also see issues with loadbalancer, which can reset all sticky sessions across all servers. This isn’t that bad. But, if you are running a CPU intensive web application, session stickiness can at times make some servers more loaded than others and if your traffic grows enough to need multiple loadbalancers, or have multiple global sites running in active-active mode, you will eventually need the ability handle server switching/failovers.

REST (Representational State Transfer): The REST architecture does recommend some of the ideas I addressed above, but in its strictest interpretation one is not allowed to create Cookies. In my personal opinion a Cookieless world, though ideal, is not pragmatic.

Popularity: 32%

Here is an interesting contribution on the topic from Preston Elder 

I’ve worked in multiple extremely super-scaled applications (including ones sustaining 70,000 connections at any one time, 10,000 new connections each minute, and 15,000 concurrent throttled file transfers at any one time - all in one application instance on one machine).

The biggest problem I have seen is people don’t know how to properly define their thread’s purpose and requirements, and don’t know how to decouple tasks that have in-built latency or avoid thread blocking (and locking).

For example, often in a high-performance network app, you will have some kind of multiplexor (or more than one) for your connections, so you don’t have a thread per connection. But people often make the mistake of doing too much in the multiplexor’s thread. The multiplexor should ideally only exist to be able to pull data off the socket, chop it up into packets that make sense, and hand it off to some kind of thread pool to do actual processing. Anything more and your multiplexor can’t get back to retrieving the next bit of data fast enough.

Similarly, when moving data from a multiplexor to a thread pool, you should be a) moving in bulk (lock the queue once, not once per message), AND you should be using the Least Loaded pattern - where each thread in the pool has its OWN queue, and you move the entire batch of messages to the thread that is least loaded, and next time the multiplexor has another batch, it will move it to a different thread because IT is least loaded. Assuming your processing takes longer than the data takes to be split into packets (IT SHOULD!), then all your threads will still be busy, but there will be no lock contention between them, and occasional lock contention ONCE when they get a new batch of messages to process.

Finally, decouple your I/O-bound processes. Make your I/O bound things (eg. reporting via. socket back to some kind of stats/reporting system) happen in their own thread if they are allowed to block. And make sure your worker threads aren’t waiting to give the I/O bound thread data - in this case, a similar pattern to the above in reverse works well - where each thread PUSHING to the I/O bound thread has its own queue, and your I/O bound thread has its own queue, and when it is empty, it just collects the swaps from all the worker queues (or just the next one in a round-robin fashion), so the workers can put data onto those queues at its leisure again, without lock contention with each other.

Never underestimate the value of your memory - if you are doing something like reporting to a stats/reporting server via. socket, you should implement some kind of Store and Forward system. This is both for integrity (if your app crashes, you still have the data to send), and so you don’t blow your memory. This is also true if you are doing SQL inserts to an off-system database server - spool it out to local disk (local solid-state is even better!) and then just have a thread continually reading from disk and doing the inserts - in a thread not touched by anything else. And make sure your SAF uses *CYCLING FILES* that cycle on max size AND time - you don’t want to keep appending to a file that can never be erased - and preferably, make that file a memory mapped file. Similarly, when sending data to your end-users, make sure you can overflow the data to disk so you don’t have 3mb data sitting in memory for a single client, who happens to be too slow to take it fast enough.

And last thing, make sure you have architected things in a way that you can simply start up a new instance on another machine, and both machines can work IN TANDEM, allowing you to just throw hardware at the problem once you reach your hardware’s limit. I’ve personally scaled up an app from about 20 machines to over 650 by ensuring the collector could handle multiple collections - and even making sure I could run multiple collectors side-by-side for when the data is too much for one collector to crunch.

I don’t know of any papers on this, but this is my experience writing extremely high performance network apps

[ This post was originally written by Preston on slashdot. Reproduced here with his permission. If you have an interesting comment or a writeup you want to share on this blog, please forward it to me or submit a comment to this post ]

Tags: scalability

Popularity: 28%

A single server, today, can handle an amazing amount of traffic. But sooner or later most organizations figure out that they need more and talk about choosing between horizontal and vertical scaling. If you work for such an organization and also happen to manage networking devices, you might find a couple of loadbalancers on your desk one day along with a yellow sticky note with a deadline on it.

Loadbalancers, by definition, are supposed to solve performance bottlenecks by distributing or balancing load between different components its managing. Though you would normally find loadbalancers in front of a webserver, a lot of different individuals have found other interesting ways of using it. For example I know some organizations have so much network traffic that they can’t use just one sniffer or firewall to do their job. They ended up using some high end loadbalancers to intelligently loadbalance network traffic through multiple sniffers/firewalls attached to them. These are rare, so don’t worry about it.

But in general, if you are having a web scalability issue, you would probably start looking at hardware solutions first. And if you do, here are my recommendation on questions you need to ask before you investigate how to set it up.

• Is the loadbalancer really needed ?
  • Identify the bottlenecks in the system. Adding webservers/appservers will not solve the problem if database is the bottleneck.
    • BTW, If you can replicate read only database to multiple servers, you might be able to use a loadbalancer to balance that traffic…
  • If most of the traffic is due to static content, investigate apache’s ability to set better cachable headers. Or use a CDN(Content distribution network) like akamai/limelight to offload cachable objects.
  • DNS and SMTP are examples of protocols which are designed to be automatically loadbalanced. If an SMTP server fails to respond, the DNS protocol will help an SMTP client to find alternative SMTP servers for a destination. If your organization has controls over the protocol they are using, they should investigate the possibility of using DNS as the loadbalancing mechanism.
• Is it a software or a hardware loadbalancer?
  • Most people can’t think of loadbalancer as anything else but a dedicated hardware. Interestingly, chances are you are already using software loadbalancers in your network without knowing it (like the DNS loadbalancer I mentioned above). I’ve seen a decline in commercial web software loadbalancers in the recent years, which has been replaced by open source components like mod_jk loadbalancer, perlbal,etc.
  • In this particular writeup, I’m going to focus on hardware loadbalancer to keep it short.
• Do you need failover ?
  • If you need loadbalancer failover, you should be buying in pairs
  • Some loadbalancer work in Active-Active mode, where both loadbalancers can be functional, while others allow only Active-Passive mode.
  • The term “failover” can mean many things. Make you you ask the right questions to understand what a loadbalancer really offers.
    • It could mean TCP session failover where every single TCP connection will be failed over.
    • It could also mean HTTP session failover (where one session is defined by one unique Cookie). If a loadbalancer supports only this mode, every single user connected to the loadbalancer will notice a blip when the primary loadbalancer dies. More often than not, this is what a loadbalancer vendor usually provides. And unfortunately not everyone in pre-sales is aware of this “minor” detail.
• How do you want to do configuration management ?
  • Not all brands are made equal. Some are easy to manage and others aren’t. I personally prefer CLI over GUI on any day.
  • But more than the CLI/GUI, I want the ability to revision control and compare different versions of configuration files. The current brand we use at work, unfortunately, doesn’t provide an easy way to do this. If you are supporting a big operation with multiple loadbalancers and have to replicate same/similar setup in multiple places, then this is something you shouldn’t compromise on.
• How many servers are we talking about ?
  • If your loadbalancer device has ports on it to attach the servers physically to it, you should make sure you don’t have more servers than the number of ports
  • In most cases, however, you can add a 100BaseT switch and add more servers to it. But regardless, having an approximate number of servers will help you decide some of the other questions later.
• Will all of these server be part of the same farm/cluster ?
  • Some organizations setup different websites on the same loadbalancer and set it up in such a way that the loadbalancer distributes load to different farm/cluster of servers depending on which domain name the client requested for.
  • Some loadbalancers can also inspect the URL and send requests with different paths (in the same domain) to different server farms. For example “/images” could go to one server and “/signup” could go to another one.
  • There are still others who might keep a set of servers on standby mode (not active) waiting to be brought up automatically in an event of problems with the primary clusters.
• Will all of the servers have the same weights when loadbalancing ?
  • For example are there some servers which should get more traffic than others because they are faster ?
• Is session stickyness important ?
  • If a user ends up on a particular webserver, is there any reason why that user should continue to stay on that webserver ? There are different kinds of session stickiness which I can think off.
  • • The most popular kind is probably IP based stickiness where traffic from same IP always goes to the same webserver. This was a great way of loadbalancing until companies like AOL decided that they will loadbalance outgoing traffic using different proxy servers, effectively changing source IP address of traffic coming from the same client.
    • My favorite session stickiness mechanism is Cookies. Cookies can be used as session tracking IDs to associate a user session with a particular webserver. There are many different ways of implementing this of which these are the few interesting ways I’ve used
      • Allow the loadbalancer to set a cookie for you in each session without the tracking cookie.
      • Most web application servers like PHP and java use cookie names like “PHPSESSIONID” or “JSESSIONID” which is an excellent session identifier which a loadbalancer can track.
      • There are a few other interesting cookie options… but I’d rather not discuss it here at this moment.
  • If you really need session stickyness, you should investigate further if your application is really horizontally scalable. Most often, sticky session feature is used as a bandaid to temporarily give an impression of scalable web app design, which could, in future, prove disastrous.
  • Session stickiness comes with some other configuration baggage as well.
    • You need to decide how long an idle session should be considered active before its shutdown. If you have a lot of traffic and if you set this session timeout to be too high, it can quickly fill up your memory.
    • If possible set the timeout to be as close to the application timeout on your app server.
• Does the traffic have to be encrypted using SSL ?
  • Some loadbalancers have built in SSL engines.
  • Others have capabilities to offload them to SSL accelerators.
  • One could also set it up in such a way that traffic is decrypted on the apache server after the loadbalancing is done. Please be cautious here. If you decide to do SSL decryption on the webserver, you are effectively disallow the loadbalancer to inspect HTTP packets which can be otherwise used to make intelligent routing decisions
• Do you need compression ?
  • Some Load balancers which come with SSL engines support on-the-fly compression which can significantly speed up user experience if you have a lot of compressible objects.
• Debugging
  • Whether you like it or not, one day you will have some problem with your loadbalancer and you will be asked by their support team to get some sniffs for them. This usually is a painful process, especially if the equipment is in production network. One feature which can simplify this is allowing the device to sniff on itself. This is not a must-have, but probably a like-to-have feature.
• Other services
  • Checklist for other services… if you want
    • NTP - Have its time always be in sync with rest of your network
    • Syslog - Ability to send syslog messages.
    • Mail - Ability to send mails when problems happen
    • SNMP - Monitoring and traps
• External factors
  • A standard sized company investing in loadbalancers usually don’t invest on a single loadbalancer. They usually buy a pair for production network, and another pair for QA or staging networks. By the time you add the support costs, it gets very expensive. If you make an investment like that, make sure the company selling that to you is still alive a couple of years down the line to support you.
  • Loadbalancer’s are not as reliable as phones. Finding bugs in a loadbalancer is much easier than you think. If you don’t have a reliable support team who is willing to help you patch the code fast enough, you might see some downtime or performance issues
  • At the same time, if the company does release a lot of patches on a weekly or monthly basis, you should find out how stable its code is. I usually ask how long it has been since the last stable release.

Popularity: 26%

TypePad was and probably is one of the first and largest paid blogging service in the world. In a presentation at OSCON 2007 , Lisa Phillips and Garth Webb spoke about TypePad’s problems in 2005. Since this is a common problem with any successful company I found it interesting enough to research a little more.

TypePad was, like any other service, initially designed in the traditional way with Linux, Postgres, Apache, mod_perl, perl as the front end and NFS storage for images on a filer. At that time they were pushing close to 250mbps (4TB per day) through multiple pipes and with growing user base, activity and data they were growing at the rate of 10 to 20% per month.

Just before the planned move to newer better data center, sometime in Oct 2005, TypePad started experiencing all kinds of problems due to its unexpected growth. The unprecedented stress on the system caused multiple failures over the next two months which ranged from hardware, software, storage to networking issues. While at times it made reading or publishing services to be completely unavailable, it also caused sporadic performance issues with statistic calculations.

One of the most visible failures was in December of 2005 when during a routine maintenance, in the middle of the process of adding redundant storage, something caused the complete storage cluster to go offline which caused the entire bank of webservers serving the webpages went down . Because they had separate storage cluster for backend database, it wasn’t affected by the outage directly.

Its at times like these that most companies fail to communicate with their users. Sixpart, fortunately, understood this early and did its job well.

Today Typepad’s architecture is similar to the one of Livejournal with users distributed over multiple master-master mysql replication. They have partitioned the database by UserIDs and have a global database to map UserIDs to partitions. They use Mysql 5.0 with InnoDB and Linux Heartbeat for HA.

The images though they decided to switch from a NFS storage to Perlbal ( Perl-based reverse proxy load balancer and web server) +MogileFS (open source distributed file system) which can scale much better with lower overhead over commodity hardware. Look at the image on the right which how Typepad served images in the transition phase from NFS to MogileFS. Follow the arrows with numbers to see how the requests go through within the network. For an image stored on MogileFS (Mogstored), the app server talks to MogileDB through mod_perl2 first (Step 3,4). MogileDB/mod_perl2 sends a Perlbal internal redirect(Step 5,6,7) to the actual image resource which is located on Mogstored(step 8,9).

Since most of the activity on the blogs are read only operations, it made sense to add memcached early into the process to ease load on a lot of components.

memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.

In another interesting approach to scalable architecture they recognized the fact that one of the most write intensive operations was commenting system which made them experiment with “The Schwartz“. This technology helped them use a queuing mechanism which could reliably delay write intensive operations to the database effectively allowing it to scale more.

The Schwartz is taglined “a reliable job queue system” and was originally developed as a generic job processing system for Six Apart’s hosted services. It is used in production today on TypePad, Livejournal and Vox for managing tasks that can be performed by the system without user interaction.

References

http://www.sixapart.com/typepad/news/2005/10/to_our_customers.html

http://www.niallkennedy.com/blog/archives/2005/12/typepad-outage-details.html

http://www.movabletype.org/documentation/administrator/publishing/publish-queue.html

Popularity: 27%

There was a major skype outage last week and though there is an “official explaination” and other discussions about it floating around, I found this comment from one of the GigaOm readers more interesting to think about. Now this particular description may not accurately describe the problem (which might be speculation as well) but it does describe , in a few words, how skype’s p2p network scales out. You should also take a look at the detailed discussion of the skype protocol here.

Number of Skype Authentication servers:
Count == 50; // Clustered
Number of potential Skype clients:
Count = 220,000,000 // Mostly decentralized
Number of SuperNode clients to maintain network connectivity:
Count = N / 300 at any one time.

• If there are 3.0 million users online then the ratio is 3,000,000 / 300 = 10,000 == Supernodes available
• Supernodes are bootstraps into the network for normal first run clients (”and handle routing of children calls”).
• Supernodes maintain the network overlay via a DHT(”Distributed Has Table”) “type” method. // This is normally very slow and done over UDP
• If a client cannot find a Supernode, regardless of authentication via central server then is NOT allowed on the Skype network.

Lack of Supernodes mean lack of network connectivity regardless of successful login via “central server”.
You CAN be a Supernode but not have full network connectivity because you have only a portion of the “Distributed Index Data aka DHT”.
MOST people that become Supernodes will bail out if they cannot keep a clear route (”aka calls bail out, client restarts and aborts Supernode status, thus booting it’s 300 - 500 Children and putting them into a “Connecting mode”.

Children that are trying to “Connect” are unable to do anything unless they have a “Supernode” as a parent. // No calls, No IM….

The overview of this is as follows:

Skype introduced a flaw into the network that dealt with “routing” and “fucked” the “decentralized data store aka DHT” this in turn ran clients on a RANDOM search of Supernodes which at this point were well booted off of the network.

In the End:
It is a huge cycle, no matter how many bugs they “fix” in the “central servers” it will take many days for N nodes to become Supernodes so they can route X data from peer A to peer B. This is NOT minor, a fix to the centralized server code base to relay data to N Supernodes there is lack there of, resulting of a very segregate network. Right now there are approximatly 10,000 sub Skype networks instead of 1 Single “in sync” network. When this “data store(see DHT) is in sync globally then the Skype network will be again STABLE.

I know this is very broad but, unless magically all of said nodes can recreate the “single overlay (DHT)” then nothing will be in sync. You will see delayed messaged, delayed or incorrect profiles and presence.

My take, in the end is give it 48 more hours and it may be semi-stable, but hey this is what you get with using end users as your own redundancy…

Yours…

Popularity: 28%

There is a very interesting interview with Markus Frind, the one man army behind the website PlentyOfFish.com. The site boasts of traffic higher than match.com, about 30 million page views a day, and runs on a single webserver with a couple of database servers. Markus has found interesting ways of surviving different kinds of problems he had. Here is the direct link to interview in wmv format.

Popularity: 11%

A very interesting Google talk about designing a good API. This may not seem like a scalability issue, but if you really want to host a horizontally scalable system you need to have a good scalable API design to go with it.

Every day around the world, software developers spend much of their time working with a … all variety of Application Programming Interfaces (APIs). Some are integral to the core platform, some provide access to widely distributed frameworks, and some are written in-house for use by a few developers. Nearly all programmers occasionally function as API designers, whether they know it or not. A well-designed API can be a great asset to the organization that wrote it and to all who use it. Good APIs increase the pleasure and productivity of the developers who use them, the quality of the software they produce, and ultimately, the corporate bottom line. Conversely, poorly written APIs are a constant thorn in the developer’s side, and have been known to harm the bottom line to the point of bankruptcy. Given the importance of good API design, surprisingly little has been written on the subject. In this talk, I’ll attempt to help you recognize good and bad APIs and I’ll offer specific suggestions for writing good ones.

Popularity: 10%

Popularity: 11%