Archive for the 'Software' Category

Brogrammers Making the World Better Through…

In just the sort of Web 2.0, Hacktivist, Hipster, Social Media, viral-video launch you could expect from a hungry start-up, Mike Judge’s HBO Series Silicon Valley Episode 1 was posted to YouTube this week: http://youtu.be/VvkmsI54ss4 and is wonderful.

Much like Judge’s other well-known (if under-appreciated) efforts and cult-culture hits, Office Space and Idiocracy, Silicon Valley is acidic, hyper-real, and both simultaneously painful and hilarious to watch.

2 minutes into the first episode I was laughing so hard I was grabbing a pencil to record the many memorable lines/scenes/scenarios – here are just a few:

  • Tech companies making the world better through…
    • “Constructing hierarchies for maximum code reuse and extensibility”
    • “Through minimum message-oriented transport layers”
    • “Claiming to make the world better all the time”
  • Spot-on depictions of…
    • Brogrammers, stereotypical party-of-5 developers, nap-pods, “hacker hostels/incubators,” launch parties, tech T-shirts (H.T.M.L), the Google-bus, TED-talks : e.g. “College is Snake Oil,” vapid VPs, supposed tech gurus/visionaries throwing money at any silly idea, and many more.
  • Frighteningly realistic and hilarious developer quotes such as
    • “I put the prototype up on Github”
    • “I memorized my hexadecimal time tables when I was like 14” – “go ahead ask me what 9xF is?” “fleventy-flive”
  • Miscellaneous biting indictments of Silicon Valley and the culture at large
    • “Kid Rock is the poorest person here”
    • “Inferior products win out all the time”

Keep up the good work & maybe there will be at least a few people who can appreciate and understand such good humor and perhaps more importantly who are actually still willing to reward such efforts & quality by actually paying for something (full disclosure – I am too cheap to pay for HBO – but I *will* watch it when it comes to Netflix).

 

Advertisements

The Pragmatic and Practical Design Template (Discussion)

Discussion

(Just Jump to Design Template)

We are all designers at heart. When we are little we construct models with play-doh and virtual worlds with crayons. But somewhere along the way, particularly in engineering domains, we lose this view of design as a simple model of reality and instead insist that a model must be as close to reality a possible. A design is a plan -or probably more correctly “a guess”- for something you want to create. Its only purpose is to get you from the “thinking about” to the “building” stage as quickly-and reasonably-as possible. For some domains, space travel and nuclear power for instance, you need this stage to be much more precise and deliberate, but fortunately few domains require this level of fidelity/reliability. Particularly in quick-to-release and prototype software, you need the design stage to be quick yet useful and add value. You don’t want to invest a bunch of effort in a model that isn’t the final thing you are going to use, but you do want to show that you have at least thought about that thing in advance.

I’ve been thinking about Software Design lately because I need people to do (and want to do) this activity, i.e. show that they have thought about and planned things beforehand, but even after many years I’m still not sure the best means to achieve this end. I’ve read lots and lots of design documents and rarely are they instructive or useful. These things can cost a lot of money and time to produce, but rarely do they return tangible benefits – mainly because most of the software designs/plans/guesses are out of date as soon as they are written. In fact, the only designs I even remember actually reading and using are the ones that defined some type of data interface contract, and that was because this contract didn’t change (or change much).

So maybe it is time for a different approach to the standard “Design Document.” Perhaps a better approach is just a couple back-of-the-napkin, top-level diagrams + a simple FAQ that answers some of the most important questions. The top-level diagrams show that you have thought about this at some level of abstraction and that you can show the major parts. The FAQ is used to make a convincing argument for building this thing and the approach used. The right set of questions might even be applicable to a wide range of projects, from remodeling your bathroom, or having a medical procedure, to multi-million dollar software projects.  This FAQ answers the basic and paramount questions of “Would you do this yourself? And spend your own money on this?”

For any design or other explanatory write-up, the FAQ questions you should be able to answer (and answer clearly and concisely) if what you are planning to work on is useful and worthwhile are:

(Jump to Design Template)

 

Design Tools

Design Tools (CC2 – Paul Stein – Original Image: https://flic.kr/p/qhsUF)

The Pragmatic and Practical Design Template

The Pragmatic and Practical Design Template

For discussion/background see this post.

Note: any answer here should be less than 3 sentences, the shorter/clearer/more concise the better – 1 sentence even being optimal. If it requires more, then you probably aren’t using understandable & clear user-focused language. Avoid references to specific technology as much as possible –e.g. instead saying “XML” just say “a file” or “a format” – terms everyone is going to understand. Not all of these questions below are applicable to every case, so don’t try to answer if they don’t fit/apply.

1.           What problem are you solving?

a.            Was something not possible before that should be possible now? Or that you couldn’t do another way? What do you hope to gain?

b.            What is it for? Who benefits? What is the motivation behind it?

2.            What is your high-level plan for solving the problem?

a.            What is your plan?

b.            Commitment: When (what specific date) are you committing to do this by?

c.             Show your high level diagram (typical Level of Detail expected: whiteboard/back-of-envelope/napkin)

3.            Why are you doing things this way?

a.            What are the primary drivers?

4.            Is what you are creating adding value?

a.            Can someone get more out of this approach/thing than they could before?

5.            Will this change behavior?

a.            Is what you are working on really going to change anything? Will someone *want* to use this because it makes their life simpler/easier/more productive?

6.            Is there any alternative or easier way?

a.            What are the alternatives? Are they easier? Cheaper? Quicker?

b.            Does something similar already exist?

c.             How are you able to get more value out of this design than you could with something simpler/easier/quicker?

d.            If this is something hard-to-understand or explain above: What is driving the complexity of this design and is this really the simplest way? Are you using customer-focused language? Have some steps been over-complicated?

7.            Is this actually useful?

a.            Are you making something useful or just making something?

8.            Is what you are doing really worth it?
Remember: don’t go throwing good time after bad work and you can’t get back the money/effort you already expended, so don’t necessarily consider the resources already invested (“sunk costs”).

a.            Is the time, energy, effort, cost, etc. worth what you hope to gain?

b.            What do you hope to gain?

9.            Are there any other dependencies or risks to consider?

a.            Does what you are doing depend on anyone/anything else at all? What are these dependencies / risks? Can they be minimized (by isolating some part of the system)?

10.          Is what you are doing usable in different environments? Is so, what are these things that won’t change in this new environment?

a.            Note: This is the lowest priority for getting work done quickly, but remember to focus on what won’t change as this project matures/ages: good workflow, data model design, portable data – these don’t change & can be useful and used anywhere.

b.            For example, if you do something in .NET, is the design still solid/applicable and is the back-end data still useful if I have to implement this in Python or JavaScript? Code itself is actually fairly useless, but good design, data, and concepts apply anywhere.

 

These questions together add up to what is frequently referred to as a “Concept” or “Design” Document – but really it is just a set of questions that people have about any endeavor. Put even more simply what they usually want to know is

Global:

  • “Would you spend your own money on this?”
  • “Would you do this yourself?”

Specific:

  • “What are you doing?”
  • “Why are you doing it?”
  • “How much is it going to cost?”
  • “How will I benefit?”
  • “How are you going to do it?”
  • “When are you going to be done?”
  • “What is going to stop you from getting it done on time?”
  • “If you have problems when are we going to know whether we should stop trying to do it”
  • and so forth.

Keeping the language clear, concise, simple, and universally understandable is the hardest part. Avoid technical terms and instead remember to write from the perspective of someone who knows very little about that thing and wants to quickly and easily learn. That is your audience. Especially don’t write from your own perspective, or from someone who is like you and already knows the subject, that person doesn’t even need such a doc.

For discussion/background see this post.

References: heavily adapted from, but inspired by, 37 Signal’s Book Rework “When to Quit” Chapter. Great set of graphics from this book here.

 

Design Tools

Design Tools (CC2 – Paul Stein – Original Image: https://flic.kr/p/qhsUF)

Impostor Syndrome and the Trap We Set for Ourselves

“No one can make you feel inferior without your consent”- Eleanor Roosevelt

“Evaluate yourself by your own standards, not someone else’s” – Life’s Little Instruction Book (but just a paraphrase of Galatians 6:4)

“Stop trying to keep up with the Joneses” – Everyone

Something was going on in the Tech World last week, in an industry that is perpetually ignoring the past (and even the present) and is continually looking for the next best thing; I seemed to notice an unprecedented level of self-reflection(or perhaps I was just tuned to that frequency myself).  Jeff Atwood published the thoughtful “Beware the trap we set for ourselves” in which he ponders “the opinions of other people matter, but they are the traps we set for ourselves” and Julie Bort published a very good piece on Impostor’s Syndrome – which also had a link to a very good blog post with a female developer’s point of view (excellent illustration from this post reused below). These were all great posts – they are things for which I never had a name for or identified the root cause of, but boy did I notice the symptoms everywhere.

Insecurity can be a good thing – it can inspire you to work and try harder, to question and improve things that truly need questioning and improving.  And the advice is everywhere, even if we exclude the voices in our own heads: “Work Harder” “Don’t let your team/company down” “Only the paranoid survive” “Anyone can do it” But like any good trait, it can be pushed too far until it gets twisted into a bad one. Being hard working is a good trait, being a workaholic is not.  Being thrifty is a good trait, being cheap is not. And so on – and so it goes with insecurity.

Much like there is a special-purpose part of the brain that allows you to remember, in explicit detail, every embarrassing moment experienced in your life, there is also a special part of the brain that seems to encourage and reinforce feelings of insecurity. Most people who care about improving themselves seem to also be pre-programmed to think that they are never good enough.  This is just human nature, you need to acknowledge that these feelings exists, plan for them, know when they are helpful and more importantly know when to ignore them, and move on.

Technology changes, jobs change, but the lessons we’ve already learned don’t. Work hard and honestly 8-5, do the best job you are able, be able to handle constructive criticism, but in the end don’t worry excessively what you or others think, and don’t allow self-doubt to cloud your opinion of yourself. You need to be a person that you respect and admire. Don’t get trapped into judging yourself by unrealistic or someone else’s standards.

 

What I Think I Know - What I Actually Know

Overcoming Impostor Syndrome – https://medium.com/tech-talk/bdae04e46ec5

The Top 3 Most Influential Software Blogs

It truly amazes me that many of the sites that I use every day (I am thinking of sites like Stack Overflow and Github here) were inspired/influenced/created by a very surprisingly small number of people (and I suspect that this is also true of most other industries). 37 Signals taught the core concepts of the software business & project management (that entrepreneurs are really “starters,” that businesses need to be paid for/sustainable, and many other simple, common sense, yet truly inspiring concepts that they captured in their books ReWork and Remote), Jeff Atwood and Joel Spolsky taught the importance of software writing and of creating a community. There are probably many other inspirers to choose from, but these are probably the best inspirational/informational blogs to read if you were forced to pick a true top 3. Even though they may not always be actively maintained today (mainly because of the success of their founders) – if you have never read, then reading their “greatest hits” will still be truly illuminating, inspiring, & educational:

ArcGIS Python Script Debugging as it Outta Be

Setting up debugging of ArcGIS Python Geoprocessing Scripts used to be quite an exercise in frustration. 

However, with the Komodo IDE (for scripting languages like Python and Perl), I was pleasantly surprised this is now as simple as

  1. Download and Install the Komodo IDE from Active State ($300 to use after the 21 day free trial)
  2. In the IDE settings, set the User Environment Variable:
    PYTHONPATH=C:\Program Files\ArcGIS\bin
  3. Copy/paste any arguments/parameters you want to test with from the ArcCatalog/ArcToolbox command line output into the Debug Command Line Arguments

And it just works.  How come this doesn’t happen more often with software? 

Your Cheatin’ File Formats

A privacy problem that seems to go largely unnoticed is the issue of personal data that is hidden away in computer documents without their creators’ knowledge.  In fact, nearly all of the most common and popular document formats use such metadata to tuck away all sorts of nifty descriptive information about the document.  Here are just a few examples:

  • When it was created/changed
  • Who made the changes based on User Name or other Operating System-captured name
  • Applications used – including watermarking or similar identifying information tying a document directly back to the exact copy of software or hardware that created it
  • And on and on

Unless you use only text (.txt) files to store data, then odds are pretty good that your documents (MS Word, PDF, JPEG, etc.) have gobs of this type of extra information attached.  And in most cases, while perhaps overdone by complex document formats, this additional document information is intended to be a useful thing and not stored for any nefarious, privacy-intruding purposes.  

However, privacy issues can quickly arise when these documents are then published to the web.  In this scenario, they can reveal personal information through their metadata that their users never desire or intend to be published. 

A perfect example of this situation that has entered the annals of Web Lore is the Cat Schwartz (of  circa-2000 TechTV fame) cropping wardrobe malfunction.  An original topless image was cropped to just an innocuous head shot and posted to her blog, but oops, the metadata thumbnail still contained the original uncropped topless photo.  Just a small, yet-shocking example of hidden metadata stored in only one such complex and ubiquitous Internet data exchange format – in this case a JPEG with EXIF metadata. 

So what are users to do that want to “scrub” all personal information and metadata from their documents before posting to the web?  Unfortunately, there appear to be no easy, one-size-fits-all solutions to this problem.  Application vendors have little to gain and much to lose by stripping out such metadata.  These applications need to have access to this metadata to provide increased functionality and the market appears to make it clear that users value this functionality over privacy.  Even when vendors do provide mechanisms to eliminate such data, they make it cumbersome and onerous.  Third party solutions often only work on one specific complex data format. 

Windows Vista surprisingly does provide a mechanism for doing this (Properties | Advanced | “Remove Properties and Personal Information“), but this only removes some of the obvious metadata that Windows can identify and does nothing with vendor specific data.  Also, you have to actually manually select the file(s) – it can’t recursively cleanse subfolders.

Take the simplest of examples: How do I remove personal data from my JPEGs before I post to public photos sites?

The Windows Vista “Remove Properties” tool doesn’t help because it only handles a few of the obvious EXIF data items (like Title, Author, Tags, etc.), but there are literally hundreds of others unhandled (even the very obvious ones like “Taken On” date and editing application).  Thus for even this simplest example, the user is forced to turn to a third party tool like ExifTool – an impressive, but somewhat geeky and command-line driven EXIF metadata utility that includes a cleaner.  One could also save the JPEG to a different format that doesn’t support EXIF metadata like BMP or PNG, but get ready for some serious size bloat as the compression is lost.

To “quickly” achieve this, I just gave up and wrote my own (C# source code below-now how’s that for geeky?) – but it is only a marginal success because it only handles the Text metadata.  When I tried to just remove all metadata, I got some troublesome results (the compression was removed, or the changes were just ignored because they caused inconsistencies).  This is a worrisome example of how even someone who is actively committed to removing all of this information can be thwarted.  But I figured the text attributes included most of information that someone might want to scrub anyway (like dates, programs, etc.). 

So there is one complex data format partially down, thousands more to go.  Privacy really shouldn’t be this hard folks…

  
// Disclaimer: Use of this code is done so entirely at your own risk.
// This software is provided "as is" without warranty of any kind
// C# Snippets/Class to remove image text metadata from a jpeg file
// Note: removing non-text metadata can have undesired effects of
// altering the compression or other image characteristics
class ExifTextCleanser
{
   public static void RemoveImageTextPropertyItems(Image image)
   {
        foreach (PropertyItem pi in image.PropertyItems)
        {
            // if it's text, remove it
            if (pi.Type == 2) // 2 = Text
            {
                image.RemovePropertyItem(pi.Id);
            }
        }
   }

   public static void PrintImageTextProperties(Image image)
   {
    Console.WriteLine("properties id count=" + image.PropertyIdList.Length);
    Encoding encoder = new ASCIIEncoding();

    // Print all Image PropertyItems
    foreach (PropertyItem pi in image.PropertyItems)
    {
       if (pi.Type == 2) // 2 = Text
       {
        string textProperty = encoder.GetString(pi.Value);
        Console.WriteLine("Property, ID=" + pi.Id + ", value=" + textProperty);
       }
    }
   }

   public static void CleanseJpeg(string originalFileName, string newFileName)
   {
    if (!(originalFileName.ToLower().EndsWith(".jpg") ||
          originalFileName.ToLower().EndsWith(".jpeg")))
    {
     Console.WriteLine(originalFileName + " not a JPEG.");
     return;
    }

    Bitmap bitmap = new Bitmap(originalFileName);

    PrintImageTextProperties(bitmap); // take a peek at this metadata info

    RemoveImageTextPropertyItems(bitmap); // then nuke it

    // save the cleansed version of the file
    bitmap.Save(newFileName);
   }
}