I find the article troubling since someone will probably take it as plausible and a path to the future.  As a sociologist of the Internet, I appreciate the value of the vaste troves of data now available.  The scale of data and the computational power to analyze it should propel science in all fronts forward more rapidly than we have previously experienced, but it cannot do so without thoughtful theory.

The Wired article is unclear as to whether it reads the message of “correlation is enough” as a justification for massive data mining or if the author reads this progress to mean hypotheses prior to data analysis are a thing of science past.  Having been involved in a number of research projects over the years, I believe the scientific method of deriving theory deductively, then collecting appropriate data and testing only those hypotheses is pragmatically little more than rhetorical device.  Given the costs of data collection, finding data that’s reasonably relevant to the broad research interests usually is a first step, followed by exploratory analysis, and then finally framing the findings with the classic format of theory and hypothesis testing.

Inductive science works well in this fashion.  What the article seems to suggest, and may lead researchers to accept, is that theory itself can be neglected and only the statistically significant results from a rigorous data mining algorithm matter.  This sort of data mining would lead to bans on the public sale of ice cream (increased ice cream sales are highly correlated with incidents of rape) and immigration restrictions on the Mississippi Delta (the population of Louisiana and the area of the state are significantly negatively correlated).  Without theory and reasonable thought, the true causal mechanisms are left out of these models.

When it comes to the advertising revenue businesses like Google are really interested in, this difference is inconsequential.  Tertiary variables also correlated with the underlying causal mechanism can work well.  They’re the very foundation of stereotypes, racial profiling, biased polling, and direct mailing campaigns.  If we know someone supports the ACLU, they’re more likely than a random American to be likely to support the Sierra Club.  If we know an individual drives a Prius, they probably are more likely to use airplanes more often than Greyhound.

When the comparison is “random”, such ’statistically different from zero’ findings in exploratory data analysis are fine, and will make Google a lot of money.  When it comes to science, however, we still need theory to tell us that higher temperatures and increased gregarious behaviors lead to both ice cream consumption and frequency of rape cases, rather than the two being in any way related to one another, and that as time progresses more people live (at least pre-Katrina) in Louisiana while the delta is eroding into the Gulf of Mexico.  Without theory, science would suffer greatly from accepting coincidental correlations as causal and overlooking true causal mechanisms because the right data was not being analyzed.

Correlations may be enough for marketing, but they are not sufficient for Science.

These projects point towards a world in which computers and mobile devices do more than connect us to the network, they will sense the world around us and reason about both our location and who is with us.  Combined with back-end data mining, new mobile sensor studded devices are coming that will alter the nature of social interaction in its last, most analog hold out: face-to-face, co-present interaction.

I want to explore this change in the nature of what the sociologist Erving Goffman referred to as the “interaction order“.  We are living through the early stages of the “electrification of the interaction order”, a time in which the ways we interact with one another is changed dramatically by the availability of mobile social information networks.  Online social networking, content sharing and discussion systems have effects that are multiplied when channeled through a device carried by every person and active in every interaction, however fleeting.  

Imagine going to a business meeting or conference and having Facebook suggest that you link to the people you spent the longest time talking to.  Mobile social computing will add more content to the torrent already generated by “desktop” experiences.  Some projects are already digging into this area: good examples include companies and products like nTag.com, SpotMe.Com, and the many trail and path tracking applications now appearing in the iPhone AppStore.  Scott Counts and I wrote about a location based social networking application that demonstrated many of these features as well as search and matching features that have yet to appear in the first wave of production systems.

A first step in this direction is to focus more on the analytic back-ends that will be needed for the management of all forms of social media repositories.  Community analysis servers that provide a dashboard of community health and activity indicators will be a critical differentiating feature for community hosts, managers, and leading participants.  Successful communities will be those that can cultivate contribution the best while managing conflict at the lowest cost.  Once desktop bound social encounters are channeled through an analytics console more real-world events sensed by mobile devices can be added to the mix.  

I am looking forward to some time to push back and reflect more about these changes while looking around for new ways to explore them.  I will take some time to get my family settled into our new home in California.  I hope to catch up with many people!  I will also be visiting Yale, University of Maryland and Berkely for talks this fall.  I plan to attend the Microsoft Research Social Computing Symposium in Redmond (it will be good to be back!) and the Conference on Information and Knowledge Management (CIKM) in Sonoma.

My old masmith@microsoft.com email address is no longer active, so please contact me at marc.smith.email at gmail.com.  

I look forward to staying in touch with my many friends and colleagues at Microsoft while finding the time now to meet with a wide range of people interested in social media.  

(Excel) .NetMap (version .57) is NOW ON CODEPLEX: http://www.codeplex.com/netmap  

Over 2,000 downloads to date! (See: http://www.codeplex.com/netmap/stats) 

(Excel) .NetMap is an Add-in that Tony Capone is building that provides a new chart type, the network or directed graph, to the collection supported by Excel.  Using a template worksheet users can enter “edge lists” that describes sets of relationships between people or entities.  The team also includes Natasa Milic-Frayling, Eduarda Mendes Rodrigues, Adam Perer, Ben Shneiderman, and Eric Gleave.

Features in the current version:

• Bug fix: Out-of-memory exception when calculating graph metrics on large edge worksheet.   .NetMap was reading the entire worksheet in one step, which led to an exception when not enough memory was available to do this.  It now reads the worksheet in 5000-row chunks.  (This fix was also included in the private 1.0.1.53 build.)
• Bug fix: Writing graph metric results for large edge lists was very slow.  (This fix was also included in the private 1.0.1.53 build.)
• Bug fix: In the class libraries, NetMapControl didn’t allow vertices and edges to be selected from within graph initialization code.
• Bug fix: In the class libraries release, the Adapters DLL was missing.
• Feature: There is a new “Import Edges” button in the ribbon.  You specify an open workbook you want to import an edge list from, the columns to import, and which columns are Vertex 1 and Vertex 2.
• Feature:.NetMap is no longer in Excel’s Add-Ins ribbon tab.  It now has its own tab. 
• Feature:.NetMap now uses a slimmed-down .NET Framework version, called the “.NET Framework Client Profile.”  It speeds up setup on machines that don’t already have a version of the .NET Framework. 
• Bug fix: The Add-Ins tab sometimes failed to appear when switching among .NetMap and non-.NetMap workbooks.  In my tests, the new .NetMap tab is always visible when a .NetMap workbook is active; it doesn’t disappear. 
• Bug fix: Setup failed on some XP machines.  If an XP machine doesn’t have the .NET Framework at the time Office 2007 is installed, the Office setup doesn’t install the Office 2007 Primary Interop Assemblies (PIAs) required by .NetMap.  (This isn’t a problem on Vista, which comes with .NET Framework 3.0.)  The .NetMap setup now checks for the PIAs and installs them if they are missing.
• Feature: A new “Graph Type” drop-down in the Ribbon can be set to either Directed or Undirected.  This determines whether arrows are drawn on the graph.  Also, several of the graph metrics are calculated differently depending on the graph type.
• Feature: Some of the graph metrics are inaccurate when the graph has duplicate edges, and you’ll now be warned about this before the graph metrics are calculated.  If you choose to calculate the graph metrics anyway, the inaccurate ones will be highlighted in red. 
• Feature: You can turn off the duplicate edge warning. 
• Feature: A new Merge Duplicate Edges button in the graph does just that.  It adds a Tie Strength column to the Edges worksheet in the process. 
• Feature: The graph pane is now visible by default.
• Feature: Minor change in the .NetMap API: Layouts that are not implemented yet now thrown a NotImplementedExceptionRight-click controls over more attributes of nodes
• Feature: Two new buttons have been added to the Excel ribbon.  Click “Calculate Graph Metrics” to calculate one or more selected graph metrics, including the new betweenness centrality and clustering coefficient metrics. 
• Feature: To select which graph metrics to calculate, click the down-arrow to the right of the “Calculate Graph Metrics” button, then click the “Select Graph Metrics” button.
• Feature: The old “Calculate Vertex Degrees” button has been removed.  Vertex degrees are now calculated with the “Calculate Graph Metrics” button.
• Feature: New controls over which attributes of a node are displayed
• Feature: Calculation of clustering coefficients for each node 
• Feature: Calculation of “betweenness centrality” and “clustering coefficient” graph metrics.  Click the new “Calculate Graph Metrics” button in the Excel ribbon to calculate the new metrics.
• Feature: Two new buttons have been added to the Excel ribbon.  Click “Calculate Graph Metrics” to calculate one or more selected graph metrics, including the new betweenness centrality and clustering coefficient metrics.
• Feature: To select which graph metrics to calculate, click the down-arrow to the right of the “Calculate Graph Metrics” button, then click the “Select Graph Metrics” button.
• Feature: The old “Calculate Vertex Degrees” button has been removed.  Vertex degrees are now calculated with the “Calculate Graph Metrics” button.

Future work areas: 

• Clustering features stage 1.
• Review WPF for adding Zoom and Scale graph features.
• Automated time sliced data: ex: daily slices of an email list
• Filtering and clustering of nodes: ex: manual and automated groupings of nodes
• Smart selection: a “lasso” (freeform selection) loaded with a query/criteria allows users to select only those nodes within a bounding region that pass a set of test conditions.
• Tag or categorize nodes and node collections: label some nodes “teachers” or “students” any with any other label that could be used to cluster nodes.
• Cluster nodes by shared attributes and labels: allow some node to become merged or composited into clusters
• Additional layout plug-ins: collaboration opportunities with Lev Nachmanson and Tim Dwyer.
• Projecting layouts onto “semantic substrates”: ex: plot nodes into a scatterplot
• Double-Click create a node; click-drag, create an edge
• Improved label control over location, font, size, color, opacity, etc.
• Collaboration with the Pajek team!

Excerpt from the workshop description:

Participation in online, social networking activities has become highly popular in contemporary society. Commercial websites integrating with a range of Web 2.0 tools have created a new discourse, replacing the static, top-down nature of Web 1.0. Web 2.0 is also changing the way we do research. It has been envisioned that a well-designed social networking site can facilitate communications between scientists at different physical locations and in different disciplines, and can encourage them or at least make it easier for them to share their data and findings, and possibly recreate and reuse these resources. Research 2.0 is the term commonly used to describe the extension of Web 2.0 tools to support academic and other research. But do all lessons we have learnt from generic social networking sites apply to scientific social networking ones? Or are there any substantial differences between the two, given the specific needs of users working in scientific field?

Format of the workshop

This full-day workshop aims to map current territory of Research 2.0 (What Web 2.0 applications exist in research and how have they been adopted), and to identify the opportunities and challenges in the development and implementation of Research 2.0. It will consist of a number of short papers and a discussion session identifying promising research directions and initiating interdisciplinary collaboration.

It’s an interesting train of thought. I particularly like the bit at the end:

I was having dinner with a group of friends about a month ago, and one of them was talking about sitting with his four-year-old daughter watching a DVD. And in the middle of the movie, apropos nothing, she jumps up off the couch and runs around behind the screen. That seems like a cute moment. Maybe she’s going back there to see if Dora is really back there or whatever. But that wasn’t what she was doing. She started rooting around in the cables. And her dad said, “What you doing?” And she stuck her head out from behind the screen and said, “Looking for the mouse.”

Here’s something four-year-olds know: A screen that ships without a mouse ships broken. Here’s something four-year-olds know: Media that’s targeted at you but doesn’t include you may not be worth sitting still for. Those are things that make me believe that this is a one-way change. Because four year olds, the people who are soaking most deeply in the current environment, who won’t have to go through the trauma that I have to go through of trying to unlearn a childhood spent watching Gilligan’s Island, they just assume that media includes consuming, producing and sharing.

It’s also become my motto, when people ask me what we’re doing–and when I say “we” I mean the larger society trying to figure out how to deploy this cognitive surplus, but I also mean we, especially, the people in this room, the people who are working hammer and tongs at figuring out the next good idea. From now on, that’s what I’m going to tell them: We’re looking for the mouse. We’re going to look at every place that a reader or a listener or a viewer or a user has been locked out, has been served up passive or a fixed or a canned experience, and ask ourselves, “If we carve out a little bit of the cognitive surplus and deploy it here, could we make a good thing happen?” And I’m betting the answer is yes

Sociologists often study industrial revolutions. I wonder if any of them have thought of free time as a key predictor for revolutionary activity. . .

Colour-code messages from known senders

Somewhere in the guts of your email client is a simple tool for adding “rules” or “filters” for the mail you send and receive. Here’s a simple pair that have made my mail more manageable: first, add to your address book everyone who receives mail from you; second, change the colour of messages from known senders to a different tone from your regular mail (I use a soothing green).

This lets you tell, at a glance, whether a message is from someone you’ve seen fit to send a message to in times gone by. This is particularly useful for picking misidentified spam out of your spam folder: anything from a known sender that your mailer mistakenly stuck in there is probably worth a closer look.

It’s a simple concept, right? People you talk to are probably more important to you than people you don’t talk to. Yet email clients don’t really do a good job with this. They sort by date. Or by subject. Or author. If you want to know which messages are from senders who get mail from you, you have to set up a rule based on some grouping function in your address book.

A few years back, Marc’s group released a little app, called the Social Network and Relationship Finder (SNARF), which allowed you to view messages - in all their threaded glory - sorted by a few simple relationship measures. You could look at mail from folks you replied to. You could sort by how often you replied to them. You could sort by how often they sent you mail. And so on.

SNARF is only available if you use Outlook, the UI isn’t anything pretty to look at, and it had its functional limitations (the acronym got a few unflattering comments, too). Still, people who used it seemed to like it, and it was certainly handy for quickly locating the critical emails in a mass of unread messages.

I’d like someone to develop this functionality in a full-featured mail client, because having to run a separate app is a serious drawback. In general, I think messaging software - whether it’s a social media app or a system like email or IM - should allow the user to access, understand, and use the structure of relationships around them in order to improve their experience with the system. This is obviously not an easy task, but it’s something for us to keep in mind.

I’ve recently been reading some articles about e-research, e-science, and e-social science. Essentially, scholars in science and technology studies, sociology of science, computer supported cooperative work, and other fields have been discussing new communications technologies and their impact on scientific advancement and discovery for some time. The crux of the matter is the tension between benefit and management. The technologies allow for rapid collaboration, and that can be great, but if it’s not properly managed - and if users don’t have the right tools and policies to engage in productive collaboration - all kinds of problems can result. From simple inefficiency to disputes over credit for discovery, use of these collaborative tools in the scientific process is not a simple matter of getting people accounts and convincing them to use the wiki, or the blog, or whatever.

Finding and solving these problems is an interesting space for research and development. How might scientists benefit from the use of social media tools? What types of research are the best fit for online collaboration? How can we build and design tools and collaboration environments that allow scientists to pursue their research without having to spend a lot of energy in managing the technology? Some of the relevant research questions in this space have been addressed in last year’s special issue of the Journal of Computer-Mediated Communication on e-Science and at past CSCW meetings. There’s been plenty of work at the Oxford Internet Institute on e-science as well, ranging from general questions on collaboration in the social sciences to specific questions on infrastructure and policy.

Joroff makes an astute observation. Recent advances in communication media have blurred the boundaries between people from formerly divergent backgrounds. Due to near-instantaneous and inexpensive mechanisms for exchanging information, cultural and social exchanges happen more rapidly, and with a longer geographical reach, than they did before. Research areas which appeared unrelated in the past are now connected by related interests. In many cases, these have existed for some time but are only now becoming apparent as keyword-based searches of fully-indexed online archives turn up relevant results from a broad range of disciplines. In other cases, the advent of these new communications media have created areas where formerly disparate disciplines intertwine.

Sociology might be the clearest example of a discipline which can experience rapidly shifting boundaries, and an area where the most successful will necessarily bridge multiple areas of knowledge and expertise. Even in the early days of the discipline, the giants in the field (e.g. Max Weber) were multi-disciplinary. Now, online repositories of data include a ton of information about formal and informal social interaction. Computer scientists, physicists, mathematicians, and social scientists are all interested in exploring and understanding these patterns. Although their specific research and practical applications differ, the basic questions are largely similar, and insights in one field often apply across multiple disciplines.

Indeed, over the past few years some wonderful pieces of sociological research have come from scientists in different fields. Lada Adamic and Natalie Glance, physicists by training, published a paper on polarization and the link structure of political blogs. A group of computer scientists at Cornell University presented research at KDD on group formation in large social networks. Marc’s recent post on CIKM shows the extent to which computer scientists and scholars in other technical fields are starting to focus on sociological topics.

These examples highlight a trend that social scientists - and sociologists in particular - should note. Technically skilled researchers with the ability to process and analyze large-scale online data sets are doing some of the best sociology out there. They’re able to do this because they are capable of crossing the boundaries between computer science, applied physics, mathematics, and sociology. This high-level research comes from folks who are experts in one area, but conversant in a broad range of disciplines covering some mixture of physical, social, and computer science.

This represents a great opportunity for sociologists to move the discipline in new and exciting directions, and to make contributions in other areas. Sociologists who are conversant in the physical and computer sciences can produce work which has a meaningful impact beyond the focus of their own discipline. They, like their counterparts in physics or CS, serve as semantic brokers - those rare individuals who can unify the language of two disparate fields to push for developments which are both broader in scope and deeper in focus than the work that individuals in either field could do alone.

If Joroff is right, and I think he is, then in the not-too-distant-future the integrators - those whose understanding, insights, and contributions span a range of fields - will be the great names of our time. Let’s see to it that sociology is well-represented amongst those luminaries.

A common misconception is that social media is authored by a large group (for example that many many people contribute to a Wikipedia page). In practice, I think most social media objects are authored by individuals or small groups and are intended to be consumed by small or large groups (most Wikipedia pages attract only a handful of people to make regular contributions). In aggregate, these objects become collective goods — what most of the great stuff on the Internet is made of: archives, collections, discussions, data sets, and more collectively authored by potentially millions of people.

The emerging sweet spot may be the creation of objects through the activity of large groups (think of the movements of whole financial markets, or the contributions of behavior from millions of search engine users) that are then consumed by large groups (in the form of market data or optimized search results).

There are other dimensions of social media beyond the size of the producer and consumer populations and I will try to post about some others. The nature of the digital object being produced or consumed is an important dimension, as is the level of interaction possible. Over time I would like to construct a taxonomy of social media systems so that we can clearly distinguish between email, wiki pages and massively multiplayer role playing games, all of which seem to be labeled “social media”.

We’ll be updating this blog fairly often with pointers to interesting research and applications along with musings based on our own experiences. We hope you’ll find it informative, entertaining, enlightening, or some combination of the three.

Poke around, find out who we are, and pardon our dust as we finish constructing the site. We’ll be updating the various static pages in the next few weeks, but the blog is now live for posting and content updates, so stay tuned!