Tag Archives: TechMission

We Are Made of Love: Getting Back to the Simple Things

After giving it some thought I have decided to begin steering my blog back to my personal life – and not just blog about technology stuff. I honestly haven’t come up with a great solution for blogging about my personal life, my thoughts, and generally what I’m doing and thinking about doing in the technology industry (like the post I wrote back in March on Technology Solutions for Nonprofit Organizations) versus raw technical topics that 3/4 of my friends & family would have no interest in (and hardly any understanding about).

So if you are one of those 3/4, I apologize.

As I am continuously working on ideas and plans for launching Develop CENTS into a full-time venture that will provide web hosting and technical support to nonprofit organizations around the world, I do plan on blogging about pure, raw technical topics. But those will be on Develop CENTS’ (edit: was Smooth Stone Services) website from now on (that website has recently been redesigned and upgraded with more features including an interactive forum & wiki as well as a blog for site administrators – like me).

I’ve had an interesting past few days, as well as an interesting few months. As many of my friends know, I am now officially in my last week of work at TechMission, the nonprofit organization I worked with as an AmeriCorps intern, for the past year. Next Wednesday, June 2nd, will be my last day. On Monday, June 14, I will begin a new internship with a freaking-awesome company called Acquia that provides Drupal services. I’m pumped.

In addition to my time at TechMission, I have worked hard on my own business plan and in building a stronger foundation for Develop CENTS to eventually turn into my full time business / ministry. All of this time working has kept me busy to the point of not doing much else. Building your own business on top of working a full time job is hard! But this is what I plan on doing for at least the next year or two – and maybe longer.

But right now, I feel like I’m on a bunch of different rabbit trails. Here’s the main point of my blog tonight: I’m getting really excited for my future (whatever that is), but also have a lot of things on my mind. Just a few minutes ago, I set the following to be my Facebook status message:

I forgot how much of what encourages me: a) talking to old college friends on the phone; b) listening to Sleeping at Last (and other music – I don’t turn my stereo on much); c) Taking an evening to “chill” – do a bit of yard work, eat a good dinner, don’t have anywhere to go; or d) All of the Above. If you answered “D” then you win… well nothing. But it’s been a good night.

I have a friend getting married in under two weeks. For the first time in a long time, I cleaned my room yesterday. After being literally sick for about a month, I think I’m finally starting to feel better. And tonight – I took time to chill. Life is good.

Listening to Sleeping at Last a few minutes ago, their song Needle & Thread came on. I’ve never been one to pay attention to lyrics much, but a certain part of the song always sticks out to me whenever I listen to it: We are made of love.

I looked up the lyrics tonight, and they are beautiful. Here’s an excerpt:

That we are made of love,
And all the beauty stemming from it.
We are made of love,
And every fracture caused by the lack of it.

“You were a million years of work,”
Said God and His angels, with needle and thread.
They kissed your head and said,
“You’re a good kid and you make us proud.
So just give your best and the rest will come,
And we’ll see you soon.”

As I move on from TechMission to my next adventure at Acquia, I want to give my best in all I do. But more than that, I want to give my best in everything I do – building my own business, eventually doing “community development”, and even simply living life.

It’s the simple things in life – taking time to relax, read the Bible, do something physical, listen to good music, talk to old college roommates on the phone – that I don’t do nearly as often as I think I should. But, as Sleeping at Last mentions in another one of their songs, “You are meant for amazing things.”

Life is Short. Why waste it by staying too busy all the time?

This was edited on Monday, Dec 17, 2012 to change the business name from Smooth Stone Services (old) to Develop CENTS (new).

Share

MSN Bot Behaving Poorly

MSNBot Behaves Poorly

As I was going through my old emails at work today (I’m still at TechMission, and will be there for at least another month), I came across a write-up that I composed and sent to the other three members of our tech team (we manage the technical aspects of TechMission’s websites, and we maintain the web server). I wrote this last fall and had meant to post it onto my blog, but forgot about it.

This is some research that I conducted, and my recommendations into addressing a high server load problem that we were having at the time. Note that my entire time at TechMission has been in the role of an Americorps intern, and everything that I have done in this role, including my work indicated in this blog post has been completely self taught in the recent past.

There was a problem…
Last fall (2009), TechMission’s servers were fairly unstable in terms of performance. Our websites were slow, server load would routinely be above 5.0 on a 5 and 15 minute average, and we constantly had to restart Apache.

After I did some research into why we were having so many problems, I found that our website was being hammered by Robot crawlers that were not respecting all of our robots.txt directives. One of these robots, surprisingly, was the crawler used by MSN.

MSNbot caused TechMission’s server load to rise to very high levels. In the 3rd week of September, we had two days where top reported our average server load during business hours at hovering between 10 and 20. For a normal server, the ideal server load would be under 1.0. In effect, we were experiencing DDoS symptoms.

When our load first increased to high levels, we did not know what the cause was. And so in our search for this information, one of my coworkers checked our WHM Apache logs, and suggested that I do the same. As I scanned the document, I noticed that several IP addresses in the same range were showing up multiple times throughout the status log. I immediately became suspicious, because this log is a snapshot of the current activity on the server – processes that are literally happening at the time the log is loaded.

I went to www.projecthoneypot.org and searched for several of these specific IP addresses. All of these addresses were associated with the same “user”: msnbot. I then went into one of my open putty sessions and issued netstat | grep msn and found several current connections to the server.

We found a solution…
I decided to try my theory out that MSNbot was the cause of our high server load. After getting approval from my coworkers (I was only an intern at TechMission) I went into WHM and added these IP addresses to our blacklist. Server load dropped like a rock, from 20 down to under 10 within a 2-3 minute time period.

Other people have experienced similar issues
According to several sources, msnbot is widely known to behave poorly. On April 16th, 2009, a blog posting was published which gave proof that msnbot used the wrong robots.txt file when indexing a website. Instead of using the right robots.txt, it has been known to follow the instructions of a completely different (unknown) website.[1] In February, other people complained of this same problem.[2]

The phenomenon of the msnbot slowing servers down is not new. In 2006, an article was published with a detailed report on how several webmasters and server administrators have experienced denial of service (DDoS) symptoms as a result of the bot.[3]

Traffic Sources
Approximately 76% of our traffic for www.urbanministry.org comes from search engines. Out of this, 68% comes from Google, and 4% comes from Yahoo.[4] From July 1st to August 31st of this year, Bing provided 2,695 visitors to our site, and ranked as the 3rd contributing search engine (behind Google and Yahoo). From October 6, 2008 through today, October 6, 2009, Bing ranks 5th among search engines, and provided 5,435 visitors to our site. Out of these visitors, we had a 59% bounce rate.

Recommendations
Based on the research cited above, I have a couple of ideas. First of all, we need to do more research to find out if by blocking msnbot, our traffic from Yahoo will eventually be affected, since Microsoft and Yahoo have begun partnering together. On July 29th, 2009, this announcement was made public.[5]

Since we have more aggressive robots.txt instructions, perhaps we could begin to unblock a few of the MSN IP addresses (not all of them) and see what happens. I think it would be interesting to create a log of all MSN connections on our server, and find out what it does. We do know for a fact that currently, not all IP addresses are blocked, as I have occasionally seen the bot show up under different IP addresses than the ones we have blocked.

Based on the data that we obtain by unblocking a few more MSN IP addresses and log all of the MSN connections, I think that we could come back in approximately another month or two and determine whether our robots.txt instructions are being followed.

Sources
[1] http://www.chewie.co.uk/seosem/msnbot-20b-is-ignoring-robotstxt-and-no-index-meta-tags/.

[2] http://www.webmasterworld.com/search_engine_spiders/3839742.htm

[3] http://www.masternewmedia.org/news/2006/07/05/server_slowdown_problems_possible_causes.htm

[4] TechMission’s Google Analytics

[5] http://www.searchmarketing.com/searchmarketing/2009/07/microsoft-yahoo-partnership-part-1-of-3.html

Share