Never been one to let the carrier drop

So, I’ve deployed many a application to EC2 (amazon’s cloud service)

I think that the good things about it are well covered on Amazon’s pages talking about it. There are, however, some bad sides they don’t mention:

1) Amazon is not subject to Moore’s law. In the time one of my customers has been on there, CPU power has quadrupled for the price and so has memory, and disks have improved by two orders of magnitude thanks to cheap and reliable SSDs. However, Amazon continues to charge the same price they did three years ago, and does not offer a SSD at all. They offer the option to purchase a specified number of IOPS, but the price is awful – you could buy a SSD every month for what they’re charging for a mere 1000 IOPS.

2) Amazon is not as reliable as one might like. Several times we’ve had instances get wedged in ways that required filling out a ticket and waiting several hours to repair. And today, every single EBS volume went down at 11 AM, and half of them are not back and it’s already 3:30

3) There’s no one to call. With managed hosting or colocation there’s usually someone you can get on the phone when everything is breaking. At amazon, all you can do is fill out a feedback form, and you don’t even get a ticket # or a response.

4) If a disaster does happen, everything gets very sluggish because everyone is trying to get into the web interface to fix things at once, and everyone is rebuilding their servers from AMIs at once.

5) Disk space at amazon is *expensive*. You basically buy the hard disk every few months.

For those of you who don’t know, wshr is a internet radio broadcast I do a few times a year (sometimes more often). It contains a mix of covers and originals, and sometimes involves other musicians and sometimes doesn’t. This particular broadcast was just me (with the occasional help added by Allie the cat)

http://sheer.us/stuff/wshr/wshr-100712.mp3

Includes:

House of the rising sun
Sound of silence
Scarborough fair
Love You
Starshine
Fading like a flower
My poor allie-cat
two unnamed original tracks
Ode To Joy
City of new Orleans
1%
Rainbow Connection
Galileo
Winter
Walking in Memphis
Sympathy for the devil
Southern Cross
Dry County
Who’s going to ride your wild horses?
Angel from Montgomery
Joy To The World
Over the rainbow
Brahm’s Lullabye

I will be performing with Mike, Bruce, and Art at Cafe Racer (http://caferacerseattle.com/) at a benefit concert for the friends and family of the victims. The show is on Saturday, Sept 15th. We go on at 9p. Door is $5, but I have a few guest list entries if anyone is feeling broke but still wants to come out.

Occasionally I feel like I’m pretty much one of a kind.

I would guess that there are a lot of zeros after the decimal point for the percentage of the U.S. that is as far left as I am. I’m so far left that sometimes I meet myself coming back from the right. I really would like us to figure out ways to give everyone everything – and I’m serious about it. I think it can be done. I think it can be done just by creating the right software to run on our minds.

Of course, since we don’t live in matrix-world, loading software into our minds is a challenging thing. I don’t have any real suggestions yet about how to achieve this. But I don’t see any reason why the reality we experience has to be that connected with what our bodies are doing, and we have processor power to burn. Simultaneously giving us everything we’ve ever wanted and having our bodies arrange for care and feeding of ourselves looks immanently achievable from where I sit.

So, apparently when it rains it pours. I was originally going to just do this track as a test to see if I could record multitrack while I had ivory up, to see if I could use ivory instead of my keyboard’s internal TG in order to get my keyboard’s internal drum machine on a different stereo pair so I could record me and mike’s musical stylings in *true* multitrack

I decided to do “Dry County”, since Gayle had mentioned she liked my version better than Bon Jovi’s

But I couldn’t do it without blowing take after take. So I decided to cheat a little and record the piano part and drums first, and then the vocals.

And, once you get started doing studio tricks, it’s hard to stop. Before I knew it I had a cello, a voilin, two different synthesizers, some filler drums, four vocal tracks, and it was starting to sound awfully good to me.

So I’m kicking it over to bunne for remixing – since he is much more in tune with mixdown than I am – but I thought I would share the rough track with my fans out there.

Here it is: http://sheer.us/stuff/2012/Sheer/DryCounty-Rough.mp3

The *very* astute will notice the lyrics are not quite the same as the original. Bonus points to whoever comments on the difference first. (Except for Gayle, since she already knows because I told her about the change when I was dithering over whether to make it or not)

So, it’s been a long time since I posted anything music-related. I got partial multitrack (the keyboard and the drums were on the same stereo pair since I was using the integrated drum machine in my new keyboard) recordings of me and Mike Mesford playing three songs. I’ve made a rough mix of them, which is available at http://www.sheer.us/stuff/2012/SheerAndMike/.

I don’t know how I feel about them. Usually when I’m working on a mixdown, I either love the material or hate it. With these tracks, I sort of went back and forth. I do think that ‘The way it is’ is worth a listen, especially if you’re a Dead fan since it has that sort of trippy laid back jam feeling to it.

I’m giving the stems to a friend of mine who’s more of a audio guru than me, and it will be interesting to see what he does with it.

While sites are in development, they often only have a few users and can go many, many minutes without being accessed. When being used this way, you will notice that some sites take a inordinately long time to come up when using IIS 7

The reason for this is that the app server for those sites is timing out. Going into advanced settings for the application pool and setting IdleTimeout to 0 will solve this problem nicely.

The downside? You’re using some RAM to run the app server. My theory is that it’s actually caching the CLR translated to native bytecode. But for most sites, this is not a meaningful amount of RAM compared to what a modern server has available – maybe 100 megabytes. And the performance improvement is well worth it.

Below is a email I would have sent to HP – had they not given me a fake email address when I asked for a address to send them the solution to.

I’ve worked tech support. I know how hard it can be. And I know that no tech support person would *ever* want to walk someone through the steps I outline below. But you shouldn’t lie to your customers. I’m sure the person I talked to knew it was possible. He could have told me “I’m not allowed to answer that question because HP would rather not spend money on my time to walk you through this.” Which would have been the truth, and I would have been all right with.

The company I worked for doing tech support had a policy of lying to customers. They would commonly tell them they needed a special type of printer cable for “bidirectional” support, when in fact the majority of the problems the customers were having had nothing to do with that.

They also lied to vendors.. but that’s another story. Anyway, I don’t really blame the individual I talked to, and while I started to vent my frustration at him for either A: lying to me or B: not knowing what he was talking about, I quickly reined myself in because I realized I had zero interest in hurting him or making his day worse, and he was the victim of a much larger system as much as I was.

That said.. if anyone else out there is trying to do this migration, here’s how I made it work. Took me about a hour and a half to figure this out – if I can save someone else the trouble, great!

——-

I called tonight (from XXX-XXX-5701) to ask how I could install Windows 7 from the system disk that the computer shipped with (1.5T) to a high performance SSD (Smaller. But much larger than the 40G in use). Your technician, while very polite, was pretty much clueless. He was able to identify that the reason I wasn’t able to use your recovery utility was that the disk geometry was wildly different, but he couldn’t tell me any solution other than purchasing windows install disks for $200.

I thanked him for his time, and offered to email him the solution once I found it. (I’m a career sysadmin and programmer. I knew it was possible. Just not how, yet.) Here it is.

Next I did what I really should have done first, and googled for my problem. The first hit was http://sonic-media.dk/?p=103, which describes how to do exactly what I’m doing. Of course, there are caveats.

To do this, you need a external USB disk with enough space to fit all the bloatware that HP installs on the system. 64G is probably the minimum I would do – but I had a 1T lying around, which let me keep several instances of the backup to try out.

In order to make the NTFS partition *fit* on the SSD, it needs shrunk. This can be done from inside disk manager (see http://www.sevenforums.com/tutorials/2672-partition-volume-shrink.html), but first you have to disable system protection, virtual memory, and hibernation. (links to how to do all that included)

(optional) You also have the option to delete the system restore partition. I know, tech support is groaning at me about this, but remember your average person bright enough to know they want a SSD is also bright enough to write that data off somewhere else first. Remember that SSDs are *expensive* – you want every gigabyte to go to something you’re going to use, if not every day, at least occasionally. So yes, obviously, make a backup first. (In fact, that’s kind of the whole point of the first article. You’re using the Windows Image Backup tool to make a backup and restore it onto your new media). Then delete it. You can always restore to your original system disk with that partition once you’re done with this exercise.

For a boot disk (the windows backup tool asks you if you want to make one) I suggest using nonvolatile media like a DVD, as I discovered to my chagrin that if you pull the USB drive while shutdown is occuring using USB restore media, you don’t have USB restore media any more. I like my restore media to stay restore media.

Obviously, this is not something you want to walk Grandma through over the phone. But for advanced users, you could forward this email to them rather than telling them that it’s not possible to do what they want to do. SSDs are a great improvement over spinning disks, speed wise, because they have zero seek time – and as the cost drops, you’re going to see more and more people wanting to do what I just did. (Of course, eventually the capacity will be large enough that this issue won’t come up)

Finally, the Vertex SSD I had purchased came already formatted NTFS. For some reason, the system restore utility – even though it said it would partition and format the disk.. got very upset about this. So, I followed the directions in http://answers.microsoft.com/en-us/windows/forum/windows_7-system/no-disk-that-can-be-used-for-recovering-the-system/e855ee43-186d-4200-a032-23d214d3d524 to use the recovery console to wipe the disk.

Also, of course, most of us can figure this stuff out. 😉 I just hoped that calling tech support would save me having to run down all the information myself.

P.S. the computer now takes longer to go through the BIOS stuff than it does to boot the OS. 😉 Windows 7, from 0 to fully online in 6 seconds. Now perhaps you understand why I bothered?

Recently, one of the sites I admin for came under what I would refer to as a DDOS attack by http://www.80legs.com/webcrawler.html.

This claims to be a ordinary web spider, but it does some things that other web spiders don’t:

1) It makes between 20 and 100 connections to the server, from different IP addresses
2) It makes requests as fast as the server will answer

Now, for a web server with flat files, this is fine. But this particular web server had very complex database content that involved a lot of joins and multiple queries to build each page. It runs on a fairly powerful box – four of them, actually – but it still wasn’t up for 100 connections querying as fast as it would respond. I think probably most database-driven sites would have some problems with this.

As 80legs points out on their web site, blocking them by IP will not work because they are a distributed engine spanning thousands of IPs. Kind of like a botnet. And their indexing is user-driven.. that is, you can pay them to index a particular site for you. Good way to mess with your competitors. 😉

Anyway, my solution was simple and elegent. We already use haproxy to distribute load among the web servers, so I just pulled out the ‘tarpit’ and wrote a quick regex. For those of you not familiar with haproxy, it’s a single threaded non blocking daemon (Oh, i love those! Just like ew-too!) that proxies web requests to servers, automatically adjusts when servers go down, and has a bunch of neat features. It’s free software, and it has worked extremely well for us.

Anyway, I stuck the following in haproxy.cfg:

reqitarpit ^User-Agent: .*www.80legs.com.*

Goodbye, 80legs. Have fun hanging out in 30-second-delay-for-any-request land 😉

For those of you who haven’t set up haproxy before, it’s pretty trivial. It can run on the same box as your web server and just attach to a different interface (i.e. bind the webserver to localhost and it to the outside interface) or a different machine, or whatever. It’s a very lightweight load, as STNB things tend to be.

Random factoid for those of you not familiar with ew-too – the reason ew-too was written STNB is that it was originally designed to run on university computers, and be such a light load that the administrators never noticed it – on machines that were the equivalent of a 486. With a hundred people or more connected. STNB is a very clever approach for situations that it works for.

A confusing problem I’ve run across several times. I wanted to share it with you, because it has me occasionally in mental agony and every once in a while you send me responess that are pure gold in clearing my confusion or building me up.

Let’s say you have person A and person B. Person A is a friend. Person B might be their parents, their spouse, another friend, or any number of possibilities.

Person B tells you not to talk to person A. They might tell you to delete their phone number from your phone, or to never email them again.

In the past, I have always complied with these requests from Person B. Today, I decided to respond to a email from a Person A who has been friends with me for years, despite the Person B request six months ago that I not talk to Person A because Person A was having delusions about who I was and what I represented.

Now, I have a specific case in mind here – although I’ve seen this pattern many, many times in my life – and I want to talk more about this case. I don’t agree with person B. If Person A was confused and thinking I was going to marry them and solve all their problems, I think I *should* talk to them.. to kindly and gently explain that that’s not my path right now.. I love them, I hope they get what they need, but I can’t be that person for them. I wouldn’t want that person to just cut me off with no explanation. But, I complied with Person B’s request because I was afraid of what person B might do.

Now I agree if Person A says please don’t contact me, I shouldn’t contact them. I have a hard time with these some times for a long list of reasons that I’d love to go into with you at a later date, but, I at least agree that I shouldn’t be contacting them.

My struggle is this: The aforementioned incident left a Person-A shaped hole in my life. Person A is my friend, and we share many common interests and I didn’t want them gone.

The angry part of me is saying, What business is it of Person B (their parents, in this case, but Person A is of age) whether me and Person A are friends. They may feel that my friendship is hurting person A, but if so they should explain that in enough detail for me to understand how, not just say “Don’t talk to person A”

I think I hear a threat where there isn’t any. My irrational fear side sees.. weell, are they going to take out a restraining order against me, have me arrested, come gun me down.. all sorts of things that in the real world do not seem to happen to me.

It’s just upsetting. I don’t know the “right” thing to do, and I am conflicted between that part of me that says complience with any request is the “right” thing to do and the part of me that says treating Person A in the way I would choose to be treated is the “right” thing to do.