See no evil, hear no evil, speak no evil, code no evil

Data can be abused. The rights of individuals can be violated. Bits of disparate information can be tracked without a customer’s knowledge, and used to piece together identities or other profile information that a customer did not intend to divulge. Thanks to Big Data and other analytics, patterns can be tracked that would dismay customers or partners.

What is the responsibility of the software development team to make sure that a company does the right thing – both morally and legally? The straight-up answer from most developers, and most IT managers outside the executive suite, is probably, “That’s not our problem.” That is not a very good answer.

Corporations and other organizations have senior managers, such as owners, presidents, CEOs and board of directors. There is no doubt that those individuals have the power to say yes – and the power to say no.

Top bosses might consult with legal authorities, such as in-house counsel or outside experts. The ultimate responsibility for making the right decision rests with the ultimate decision-makers. I am not a lawyer, but I expect that in a lawsuit, any potential liability belongs with managers who misuse data. Programmers who coded an analytics solution would not be named or harmed.

This topic has been on my mind for some time, as I ponder both the ethics and the legalities implicit in large-scale data mining. Certainly this has been a major subject of discussion by pundits and elected officials, at least in the United States, when it comes to customer info and social-media posts being captured and utilized by marketers.

Some recent articles on this subject:

Era of Online Sharing Offers Benefits of 'Big Data,' Privacy Trade-Offs

The Challenge of Big Data for Data Protection

Big Data Is Opening Doors, but Maybe Too Many

What are we going to do in the face of questionable software development requirements? Whether we are data scientists, computer scientists or other IT professionals, it is quite unclear. A few developers might prefer to resign rather than write software they believe crosses a moral line. Frankly, I doubt that many would do so.

Some developers might say, “I didn’t understand the implications.” Or they might say, “If I don’t code this application, management will fire me and get someone else to do it.” Or they might even say, “I was just following orders.”

Hurray for COBOL and the mainframe

Perhaps I’m an old fogey, but I can’t help but smile when I see press releases like this: “IBM Unveils New Software to Enable Mainframe Applications on Cloud, Mobile Devices.” 

Everything old will become new again, as the late Australian musician Peter Allen famously sang in his song of that name.

Mainframes were all the rage in the 1960s and 1970s. Though large organizations still used mainframes as their basis of their business-critical transaction systems in the 1990s and 2000s, the excitement was around client/server and n-tier architectures built up from racks of low-cost commodity hardware.

Over the past 15 years or so, it’s become clear that distributed processing for Web applications fit itself into that clustered model. Assemble a few racks of servers and add a load-balancing appliance, and you’ve got all the scalability and reliability anyone needs.

But you know, from the client perspective, the cloud looks like, well, a thundering huge mainframe.

Yes, I am an old fogey, who cut his teeth on FORTRAN, COBOL, PL/1 and CICS on Big Blue’s big iron (that is to say, IBM System/370). Yes, I can’t help but think, “Hmm, that’s just like a mainframe” far too often. And yes, the mainframe is very much alive. {http://www-03.ibm.com/software/products/us/en/entecoboforzos}

IBM’s press release says that,

Today, nearly 15 percent of all new enterprise application functionality is written in COBOL. The programming language also powers many everyday services such as ATM transactions, check processing, travel booking and insurance claims. With more than 200 billion lines of COBOL code being used across industries such as banking, insurance, retail and human resources, it is crucial for businesses to have the appropriate framework to improve performance, modernize key applications and increase productivity.

I believe that. Sure, there are lots of applications written in Java, C++, C# and JavaScript. Those are on the front end, where if  a database read or write fails, or a non-responsive screen is an annoyance, nothing more. On the back end, if you want the fastest possible response time, without playing games with load balancers, and without failures, you’re still looking at a small number of big boxes, not a large number of small boxes.

This fogey is happy that the mainframe is alive and well.

Big Expectations and Big Challenges for Big Data

According to IDG Research, 80% of business leaders say that Big Data should enable more informed business decisions – and 37% say that the insights provided by Big Data should prove critical to those decisions.

A February 2013 survey on Big Data was designed and executed jointly by IDG Research and Kapow Software, which sells an integration platform for Big Data. As with all vendor surveys, bear in mind that Kapow wants to make Big Data sound exciting, and to align the questions with its own products and services.

That said, the results of the survey of 200 business leaders, are interesting:

• 71% said that Big Data should help increase their competitive advantage by keeping them ahead of market trends

• 68% said that Big Data should improve customer satisfaction

• 62% believe Big Data should increase end-user productivity by providing real-time access to business information

• 60% said Big Data should improve information security and/or compliance

• 55% said Big Data should help create compelling new products and services

•33% said Big Data should help them monitor and respond to social media in real time

Those are big expectations for Big Data! The results to date… not so much. The study revealed that only one-third of organizations surveyed have implemented any sort of Big Data initiative – but another third expect to do so over the next year.

What are the barriers to Big Data success? The study’s answers:

• 53% say a lack of awareness of Big Data’s potential

• 49% say concerns about the time-to-value of the data

• 45% say having the right employee skills and training

• 43% say ability to extract data from the correct sources

Get ready for the #SDTimes100

The software development world keeps on changing. Just when we think we get a handle on something as simple as application lifecycle management, or cloud computing, or mobile apps, we get new models, new innovations, new technologies.

Forget plugging pair programming or continuous delivery or automated testing before checking code into the repository. The industry has moved on. Today, the innovation is around DevOps and Big Data and HTML5 and app stores and… well… it keeps changing.

Tracking and documenting those changes – that’s what we do at SD Times. Each year, the editors stop, catch our breath, and make a big list of the top innovators of the software development industry. We identify the leaders – the companies, the open-source projects, the organizations who ride the cutting edge.

To quote from Rex Kramer in the movie Airplane, the SD Times 100 are the boss, the head man, the top dog, the big cheese, the head honcho, number one...

Who are the SD Times 100? This week, all will be revealed. We will begin tweeting out the SD Times 100 on Thursday. Follow the action by watching @sdtimes on Twitter, or look for hashtag #SDTimes100.

After all the tweeting is complete, the complete list will be published to www.sdtimes.com. Be part of the conversation!

In-memory databases poised for takeoff

The classic database engines – like the big relational behemoths from IBM, Microsoft and Oracle – store the data on disk. So do many of the open-source databases, like MySQL and PostgreSQL, as well as the vast array of emerging NoSQL databases. While such database engines keep  all the rows and columns on the relatively slow, disks, they can boost performance by putting some element, including indices and sophisticated predicted caches, on faster solid-state storage or even faster main memory.

From a performance perspective, it would be great to store everything in main memory. It’s fast, fast, fast. It’s also expensive, expensive, expensive, and in traditional services, is not persistent. That’s why database designers and administrators leverage a hierarchy: A few key elements in the fastest, most costly main memory; more data in fast, costly solid-state storage; the bulk in super-cheap rotating disks. In some cases, of course, some of the data goes into a fourth layer in the hierarchy, off-line optical or tape storage.

In-memory databases challenge those assumptions for applications where database response time is the bottleneck to application performance. Sure, main memory is still fabulously expensive, but it’s not as costly as it used to be. New non-volatile RAM technologies can make main memory somewhat persistent without dramatically harming read/write times. (To the best of my knowledge, NVRAM remains slower than standard RAM – but not enough to matter.)

That’s not to say that your customer database, your server logs, or your music library, should be stored within an in-memory database. Nope. Not even close. But as you examine your application architecture, think about database contents that dramatically affect either raw performance, user experience or API response time. If you can isolate those elements, and store them within an in-memory database, you might realize several orders of magnitude improvement at minimal cost — and with potentially less complex code than you’d need to manage a hierarchical database system.

Not long ago, in-memory databases were a well-kept secret. The secret is out, according to research from Evans Data. Their new Global Development survey says that the developers using in-memory databases has increased 40% worldwide from 18% to 26% during the last six months. An additional 39% globally say they plan to incorporate in-memory databases into their development work within the next 12 months.

Power down… or airplane mode?

Like many of you, I travel with a vast array of personal electronic devices – so much that my briefcase bulges with screens, batteries, cables and charging bricks. Some devices are turned off when I’m on an airplane – and some aren’t, often because I forget.

Take this week, for example. I am working out of SD Times’ New York headquarters, instead of my usual office near San Francisco. What did I bring? A 13-inch mid-2011 MacBook Air notebook, an iPad Mini with Logitech Ultrathin Keyboard, a Google Nexus 7 tablet, a Galaxy Nexus phone, a Virgin Mobile MiFi access point, Bose QuietComfort 15 noise-cancelling headphones, RocketFish RF-MAB2 Bluetooth stereo headset, a Microsoft Notebook Optical Mouse 3000, a USB hub, and an HP-15C calculator. Oh, let’s not forget the Canon PowerShot S100 digital camera. And my Pebble watch.

All that for a five-day trip. A bit excessive? Maybe.

I can guarantee that not every device is powered down during a flight. Yes, the flight attendants ask passengers to turn devices all the way off, and I have good intentions. But there’s a good chance that the laptop is sleeping, that some tablets and the phone might in airplane mode instead of off, I might have forgotten to slide the switch on the Logitech keyboard, and so-on.

Think about all the electronic noise from those electronics. Think about all the potential interference from the WiFi, cellular and Bluetooth radios, the GPSes in the phone and Google tablet… yet it doesn’t seem to make a tangible difference.

I’m not alone in failing to turn off every personal electronic device. According to a new study by the Consumer Electronics Association,

Almost one-third (30 percent) of passengers report they have accidently left a PED turned on during a flight. The study found that when asked to turn off their electronic devices, 59 percent of passengers say they always turn their devices completely off, 21 percent of passengers say they switch their devices to “airplane mode,” and five percent say they sometimes turn their devices completely off. Of those passengers who accidently left their PED turned on in-flight, 61 percent said the device was a smartphone.

At least I have good intentions. Many travelers intentionally keep playing games with their phones, hiding them when the flight attendant walks by, taking them out as soon as the uniformed crewmember stops looking.

That doesn’t change the reality that devices are left turned on — and the flights appear to be perfectly safe. It’s time for the U.S. Federal Aviation Administration, and the U.S. Federal Communications Commission, to stop the ban on using electronic devices during takeoff, landing, and flying at altitudes under 10,000 feet.

Let's boost developer velocity by 30x

Not long ago, if the corporate brass wanted the change major functionality in a big piece of software, the IT delivery time might be six to 12 months, maybe longer. Once upon a time, that was acceptable. Not today.

Thanks to agile, many software changes can be delivered in, say, six to 12 weeks. That’s a huge improvement — but not huge enough. Business imperatives might require that IT deploy new application functionality in six to 12 days.

Sounds impossible, right? Maybe. Maybe not. I had dinner a few days ago with S. “Soma” Somasegar (pictured), the corporate vice president of Microsoft’s Developer Division. He laughed – and nodded – when I mentioned the need for a 30x shift in software delivery from months to days.

After all, as Soma pointed out, Microsoft is deploying new versions of its cloud-based Team Foundation Service every three weeks. The company has also realize that revving Visual Studio itself every two or three years isn’t serving the needs of developers. That’s why his team has begun rolling out regular updates that include not only bug fixes but also new features. The latest is Update 2 to Visual Studio 2012, released in late April, which added in new features for agile planning, quality assurance, line-of-business app developer, and improvements to the developer experience.

I like what I’m hearing from Soma and Microsoft about their developer tools, and about their direction. For example, the company appears sincere in its engagement of the open source community through Microsoft Open Technologies — but I’ll confess to still being a skeptic, based on Microsoft’s historical hostility toward open source.

Soma said that it’s vital not only for Microsoft to contribute to open source, but also to let open source communities engage with Microsoft. It’s about time!

Soma also cited the company’s new-found dedication to DevOps. He said that future versions of both on-premises and cloud-based tools will help tear down the walls between development and deployment. That’s where the 30x velocity improvement might come from.

Another positive shift is that Microsoft appears to truly accept that other platforms are important to developers and customers. He acknowledges that the answer to every problem cannot be to use Microsoft technologies exclusively.

Case in point: Soma said that fully 60% of Microsoft developers are building applications that touch at least three different platforms. He acknowledged that Microsoft still believes that it has the best platforms and tools, but said, “We now know that developers make other choices for valid reasons. We want to meet developers where they are” – that is, engaging with other platforms.

Soma’s words may seem like a modest and obvious statement, but it’s a huge step forward for Microsoft.

Mobile developer mojo

Tickets for the Apple Worldwide Developer Conference went on sale on Thursday, April 25. They sold out in two minutes.

Who says that the iPhone has lost its allure? Not developers. Sure, Apple’s stock price is down, but at least Apple Maps on iOS doesn’t show the bridge over Hoover Dam dropping into Black Canyon any more.

Two minutes.

To quote from a story on TechCrunch,

Tickets for the developer-focused event at San Francisco’s Moscone West, which features presentations and one-on-one time with Apple’s own in-house engineers, sold out in just two hours in 2012, in under 12 hours in 2011, and in eight days in 2010.

Who attends the Apple WWDC? Independent software developers, enterprise developers and partners. Thousands of them. Many are building for iOS, but there are also developers creating software or services for other aspects of Apple’s huge ecosystem, from e-books to Mac applications.

Two minutes.

Mobile developers love tech conferences. Take Google’s I/O developer conference, scheduled for May 15-17. Tickets sold out super-fast there as well.

The audience for Google I/O is potentially more diverse, mainly because Google offers a wider array of platforms. You’ve got Android, of course, but also Chrome, Maps, Play, AppEngine, Google+, Glass and others beside. My suspicion, though, is that enterprise and entrepreneurial interest in Android is filling the seats.

Mobile. That’s where the money is. I’m looking forward to seeing exactly what Apple will introduce at WWDC, and Google at Google I/O.

Meanwhile, if you are an Android developer and didn’t get into Google I/O before it sold out – or if you are looking for a technical conference 100% dedicated to Android development – let me invite you to register for AnDevCon Boston, May 28-31. We still have a few seats left. Hope to see you there.

Coping with the data

As I write this on Friday, Apr. 19, it’s been a rough week. A tragic week. Boston is on lockdown, as the hunt for the suspected Boston Marathon bombers continues. Explosion at a fertilizer plant in Texas. Killings in Syria. Suicide bombings in Iraq. And much more besides.

The Boston incident struck me hard. Not only as a native New Englander who loves that city, and not only because I have so many friends and family there, but also because I was near Copley Square only a week earlier. My heart goes out to all of the past week’s victims, in Boston and worldwide.

Changing the subject entirely: I’d like to share some data compiled by Black Duck Software and North Bridge Venture Partners. This is their seventh annual report about open source software (OSS) adoption. The notes are analysis from Black Duck and North Bridge.

How important will the following trends be for open source over the next 2-3 years?

#1 Innovation (88.6%)
#2 Knowledge and Culture in Academia (86.4%)
#3 Adoption of OSS into non-technical segments (86.3%)
#4 OSS Development methods adopted inside businesses (79.3%)
#5 Increased awareness of OSS by consumers (71.9%)
#6 Growth of industry specific communities (63.3%)

Note: Over 86% of respondents ranked Innovation and Knowledge and Culture of OSS in Academia as important/very important.

How important are the following factors to the adoption and use of open source? Ranked in response order:

#1 – Better Quality
#2 – Freedom from vendor lock-in
#3 – Flexibility, access to libraries of software, extensions, add-ons
#4 – Elasticity, ability to scale at little cost or penalty
#5 – Superior security
#6 – Pace of innovation
#7 – Lower costs
#8 – Access to source code

Note: Quality jumped to #1 this year, from third place in 2012.

How important are the following factors when choosing between using open source and proprietary alternatives? Ranked in response order:

#1 – Competitive features/technical capabilities
#2 – Security concerns
#3 – Cost of ownership
#4 – Internal technical skills
#5 – Familiarity with OSS Solutions
#6 – Deployment complexity
#7 – Legal concerns about licensing

Note: A surprising result was “Formal Commercial Vendor Support” was ranked as the least important factor – 12% of respondents ranked it as unimportant.  Support has traditionally been held as an important requirement by large IT organizations, with awareness of OSS rising, the requirement is rapidly diminishing.

When hiring new software developers, how important are the following aspects of open source experience? Ranked in response order:

2012
#1 – Variety of projects
#2 – Code contributions
#3 – Experience with major projects
#4 – Experience as a committer
#5 – Community management experience

2013
#1 – Experience with relevant/specific projects
#2 – Code contributions
#3 – Experience with a variety of projects
#4 – Experience as a committer
#5 – Community management experience

Note: The 2013 results signal a shift to “deep vs. broad experience” where respondents are most interested in specific OSS project experience vs. a variety of projects, which was #1 in 2012.

There is a lot more data in the Future of Open Source 2013 survey. Go check it out. 

Big Data and PC Sales Data

Last week, we held the debut Big Data TechCon in Cambridge, Mass. It was a huge success – more attendees than we expected, which is great. (With a debut event, you never really know.)

We had lots of sessions, many of which were like trying to drink from a fire hose. That’s a good thing.

A commonality is that there is no single thing called Big Data. There are oodles of problems that have to do with capturing, processing and storing large quantities of structured and unstructured data. Some of those problems are called Big Data today, but some have evolved out of diverse disciplines like data management, data warehousing, business intelligence and matrix-based statistics.

Problems that seemed simple to solve when you were talking about megabytes or terabytes are not simple when you’re talking about petabytes.

You may have heard about the “Four V’s of Big Data” – Volume, Velocity, Variety and Veracity. Some Big Data problems are impacted by some of these V’s. Other Big Data problems are impacted by other V’s.

Think about problem domains where you have very large multidimensional data sets to be analyzed, like insurance or protein folding. Those petabytes are static or updated somewhat slowly. However, you’d like to be able to run a broad range of queries. That’s an intersection of data warehousing and business intelligence. You’ve got volume and veracity. Not much variety. Velocity is important on reporting, not on data management.

Or you might have a huge mass of real-time data. Imagine a wide variety of people, like in a social network, constantly creating all different types of data, from text to links to audio to video to photos to chats to comments. You not only have to store this, but also quickly decide what to present to whom, through relationships, permissions and filters, but also implement a behind-the-scenes recommendation engine to prioritize the flow. Oh, and you have to do it all sub-second. There all four V’s coming into play.

Much in Big Data has to do with how you model the data or how you visualize it. In non-trivial cases, there are many ways of implementing a solution. Some run faster, some are slower; some scale more, others scale less; some can be done by coding into your existing data infrastructure, and others require drastic actions that bolt on new systems or invite rip-and-replace.

Big Data is fascinating. Please join us for the second Big Data TechCon, coming to the San Francisco Bay Area in October. See www.bigdatatechcon.com.

While in Cambridge wrapping up the conference, I received an press release from IDC: “PC Shipments Post the Steepest Decline Ever in a Single Quarter, According to IDC.”

To selectively quote:

Worldwide PC shipments totaled 76.3 million units in the first quarter of 2013 (1Q13), down -13.9% compared to the same quarter in 2012 and worse than the forecast decline of -7.7%.

Despite some mild improvement in the economic environment and some new PC models offering Windows 8, PC shipments were down significantly across all regions compared to a year ago. Fading Mini Notebook shipments have taken a big chunk out of the low-end market while tablets and smartphones continue to divert consumer spending. PC industry efforts to offer touch capabilities and ultraslim systems have been hampered by traditional barriers of price and component supply, as well as a weak reception for Windows 8. The PC industry is struggling to identify innovations that differentiate PCs from other products and inspire consumers to buy, and instead is meeting significant resistance to changes perceived as cumbersome or costly.

The industry is going through a critical crossroads, and strategic choices will have to be made as to how to compete with the proliferation of alternative devices and remain relevant to the consumer. 

It’s all about the tablets, folks. That's right: iPads and Android-based devices like the Samsung Galaxy, Kindle Fire, Barnes & Noble Nook and Google Nexus. Attempts to make standard PCs more tablet-like (such as the Microsoft Surface devices) just aren’t cutting it. Just as we moved from minicomputers to desktops, and from desktops to notebooks, we are moving from notebooks to tablets.

(I spent most of the time at the Big Data TechCon working on a 7-inch tablet with a Bluetooth keyboard. I barely used my notebook at all. The tablet/keyboard had a screen big enough to write stories with, a real keyboard with keys, and best of all, would fit into my pocket.)

Just as desktops/notebooks have different operating systems, applications, data storage models and user experiences than minicomputers (and minicomputer terminals), so too the successful tablet devices aren’t going to look like a notebook with a touchscreen. Apps, not applications; cloud-based storage; massively interconnected networks; inherently social. We are at an inflection point. There’s no going back.

How do you interpret IDC’s results? Write me at feedback@bzmedia.com.