Russell Beattie

I'm starting to come around to the fact that even though MySQL is a kick ass db and Orion Server is a great Web and App server, both which will allow you to pound on them pretty relentlessly, the fact is that if you want to really scale your website's content, it needs to be static.

Independant developers like myself don't have the resources to be sloppy... so we have to design our apps to be as efficient as possible, cacheing pages to disk and using a web server like Apache or Zeus which can handle eBay-level hits. Yes, "hardware is cheap" but that's "cheap" if you're a corporation, if you're an individual like myself, you have to work within the boundaries of your hosting company or hosted server.

Right now at my hosting server John Companies I have ample bandwidth to play with - 40 gigs a month (!!!) - but because my server is virtual, my memory and processing power is limited. Before I was pooling my db connections on this site, I just let each page open its own connection and didn't worry about it (doh!). If 500 people/aggregators hit my site at the same time, you can imagine the memory requirements that MySQL would suddenly require and hold on to for a while. This wasn't an issue until Mobitopia arrived, but then John rightfully sent me a nasty gram (or two) about my memory hog apps and I got off my ass and fixed my code.

This is a good lesson - it's taught me to start thinking about scalability from the step one. I need to start thinking about supporting 10s or even 100s of thousands of users from my server alone (is this possible?), the more scalability I tweak out of this one system the better it will be later if I need to duplicate it. Plan for success and modularize like Yahoo/Google.

Dynamic content is the "easiest" to generate, but it just requires a lot of horsepower (dare I say it? Especially for Java. Doh! :-) ). I say easiest in quotes because I'm discovering a bunch of bugaboos on both this site and Mobitopia while using db pooling, page-caching, 304 messages, etc. But still, throwing a db up is the most straight forward way store data, and then you can slice it and dice it to your heart's content. This is how any server side app I can imagine would function regardless, it just would have an automated cacheing layer in front that would take any dynamic content and generate static pages to be served.

Secondly, after getting some inside scoop on what technology one of the big telecoms is using for its portal, I'm realizing how important it is if you're aiming towards the mobile market to integrate XML and XSL transforms right at the beginning. It's pretty obvious if you think about it (which I hadn't): WML, XHTML, and MMS (SMIL) messages are all XML - if you're generating content for these devices "by hand" you're just asking for trouble. Mobile content is XML content.

So that's what I'm starting to think about on the server side. Creating an application server that will allow dynamic generation of content, but to be cached to disk via some sort of XML transformation process. It looks like "miniblog" is going to have to get an overhaul pretty soon.

-Russ