Cowboy and centralized research IT

Feb 08 2011 Published by under Research Data

The question of research-IT provisioning came up in my post on data-security horror stories. I saw some confusion from readers about it, and it's worth examining in detail for other reasons, so here goes.

So let's imagine Achaea University for a moment: immense, a diverse research agenda across many disciplines, lots of grants coming in, but some areas (often but hardly exclusively in the humanities) with no grant money incoming at all. How does Achaea U provision researchers with IT tools and services?

Achaea U doubtless has a central IT unit. At a minimum, it handles networking, campuswide administrative IT (payroll, HR, authentication/authorization, likely the course-management system, perhaps calendaring and email if those haven't been outsourced), and a lot of front-line student- and staff-facing IT (computer labs, campus wireless, helpdesk, webspace, basic web-accessible storage, etc). It may or may not have a learning-technology unit.

It almost certainly doesn't have a research-IT-specific unit. Such research computing services as it provides are of two types: repurposed other services (e.g. webspace), or pay-to-play services (e.g. specialized development teams). Big storage, if it exists, is almost certainly pay-to-play; you pay as long as you keep data on central IT's systems, and if you don't pay, central IT blows the data away. Such research-type services also tend to be "enterprisey" in their technical provisioning—which combined with pay-for-play means "serious sticker shock" for the average researcher, even the average well-funded researcher.

Services also tend to be lowest-common-denominator. If you have special needs, such as preservation past grant expiration or diamond-hard security? Tough noogies, chum. Central IT offers what central IT offers; you can take it or leave it. You can yell at central IT all you like that they don't know what the hell they're doing (and they may very well not; insular central IT units can and do gin up services that are convenient for them to provide, while not convenient at all to the intended user). Doesn't matter. Central IT offers what Central IT offers. Take it or leave it.

Most researchers leave it, which means no economy of scale, which means these services cost central IT even more than they need to—and since central IT is pay-to-play, well…

So Achaea U has a lot of other systems running research-related IT. For example, Achaea U does a fair bit of what's called "grid computing" (which has other guises too, but let that go for now). That's not run through central IT, because central IT was too big and ponderous and lowest-common-denominator to jump on that need (it's very hard, organizationally, for central IT to greenlight a service that not everybody on campus will use). Engineering or comp sci owns the grid, or it may have spun off into its own (likely pay-to-play, depending on the status of its internal grant funding) research/service enterprise.

And then we have the other end of the scale: a poorly-funded lone-wolf researcher limping along via a Linux server installed on a dusty beige consumer-grade box under his desk. If it breaks, he's humped, because it was set up years ago by a grad student who has since graduated, leaving no documentation behind, and he doesn't entirely know how it works. It hasn't broken. Yet. Is it backed up? Who the heck knows? Has it been hacked? Who the heck knows? Who the heck knows which networks it's even connected to, for that matter? The researcher sure doesn't. But he knows that his server (plus whatever free-to-him web services he tacks on to his processes) is cheaper by a factor of ten (maybe even a hundred) than equivalent computing provision from central IT! This, folks, is what I mean by "cowboy IT." Yee-ha! And there's a lot of it, scattered all over Achaea U! Yippee-ki-yi-yay!

It is, as I said, a continuum. Based on what's said in the Inside Higher Ed article, Dr. Yankaskas was very close to the cowboy-IT end. Somewhere in the middle, Achaea U has a few research-IT units that work on soft money for small or large groups of researchers. These units are more nimble, discipline-savvy, and responsive than Achaea U's central IT, and they're likely just as competent or more so (especially considering how little central IT knows about research-computing needs); the downside is that they're not as richly-funded and their funding is always in danger, so they probably cut some corners. The worse among them are no better than straight-up cowboy IT; part of the problem is that their staff may be selected by researchers who don't know jack about IT (as clearly happened in Dr. Yankaskas's case).

Plenty of Achaea U researchers, it must be said, can't even muster a cowboy-IT setup, when lack of outside funding combines with lack of skill. They are utterly shut out. Neither central IT nor research-computing units want them because they have no grant money to toss in the pot. The library may do what little it can, particularly for humanities scholars, but it's not enough.

So how do researchers get away with cowboy IT? Well, honestly, nobody's ever looked. It's that simple. And nobody looks because nobody much cares—until there's a huge, embarrassing screwup like the Dr. Yankaskas affair. (If this seems to resemble the laissez-faire IT environment that used to exist for social-security numbers in US universities? Quite right. Same causes.) Classic case of externalities: cowboy IT creates risks, sometimes serious risks to the researcher or even the institution, but mitigating the risks isn't perceived as important (and is known to be expensive) until there's a sudden crisis.

I expect the NSF data-management plan process to expose a shocking amount of cowboy IT in US science research, from the Achaea Universities among us to industry all the way down to the lone-wolves. I also expect the NSF will start to indicate gently that cowboy IT is not acceptable practice… and to become rather less gentle about it over time. This means that researchers will have to internalize risks they hadn't previously worried about, or they'll wind up like Dr. Yankaskas.

I don't entirely know what campus research-IT infrastructures will emerge from this. I wouldn't be celebrating if I worked for central IT; I have serious misgivings that central IT in its ongoing ignorance can even do this right. I'd rather see a mesh of the middles, growing collaboration among research-specific IT units to expand their services, service models, and funding sources to campus cowboys and have-nots. That's a tall order, though; funding models aren't clear, and these units think of themselves as independent fiefdoms, rarely valuing collaboration because of its added process overhead. It doesn't help that central IT will often fight to keep such a mesh from emerging, viewing it as a threat.

So we'll see. The bottom-line truth is that Achaea U will have to do better at research-IT provisioning in the next decade, or it'll start losing grant dollars to universities that work out how to do it right. Yippee-ki-yi-yay.

Share

19 responses so far

  • I work for giant fucking institution and we have not research-IT. I have the same guys who set up phones and image desktops and laptops, that are also tasked with helping us with research issues. It sucks.

    • Dorothea says:

      I hear you. But it won't change unless and until it becomes a pain point vis-a-vis securing grant funding. I just happen to think that day is coming...

      • The weird thing was that my small SLAC that I did my undergrad at, actually had a research IT liason for our school of math and science. It was immensely helpful.

        • Really?! Wow. I'm impressed. Usually if that exists, it's run by the school, not central IT.

          Many academic libraries already have a liaison-based service model; a few are looking at how we can get those librarians out of the library and into the school/department/unit they serve. I think that's an important and useful move for us, frankly, as the perceived importance of the physical library wanes for many instructors and researchers.

  • Markk says:

    Ok here is an alternative view. I think ALL research IT (whatever that is - operationally defined by your articles) should be cowboy IT. ALL of IT. The entire central IT facility for research (not admin) should be made local to the research groups. Its way (way way) cheaper, far more flexible, 99% of the time very effective and even when it isn't presents no risks not also present in centralized IT. These tools and practices are commodity or becoming so quickly, and if there are specialized needs then the research organization should contract them out. That is where expertise should start to lie.

    Issues with HIPPA and such are issues that must be addressed by any research group using personal information for anything. Naivete and such isn't an excuse, everybody should be literate enough about data and computers to recognize these issues, and if they are not, well that is sanction-able, I see. Good.

    The amount of money saved and the quickness and easy accountability is worth far more than any inevitable loss of data from shoddy practice and cost of integration.

    If we went this way, best practices would evolve quickly as needed and would be adopted by good groups and actually provide another ranking method for who gets the limited amount of cash. Yes there would be horror stories at first, but you know, I've worked in IT since the early '80's on the edge and at the center - for small contractors and Fortune 5 companies and there have always been horror stories.

    Just some devil's advocate food for thought.

    • Dorothea says:

      That may well be the way it goes! In which case "cowboy IT" is going to have to smarten up considerably, acting more like the mid-scale research-IT shops I mentioned.

      I've been careful to tell my digital-curation students that they may well not end up working for libraries. As data preservation becomes more of A Thing, research-IT shops may be hiring students like mine. (I already have students working for such shops at MPOW, for that matter!)

      The only problems with this from where I'm sitting is that some problems are needlessly difficult to solve with little IT fiefdoms (e.g. geographically-dispersed replication of digital data), and other problems may be much more cost-efficient to solve collectively. Figuring out which problems those are, and how to get them solved in a world where cowboy IT isn't going away, is a significant but IMO solvable challenge.

  • Heh heh. When you said cowboy IT, I thought you meant in the same sense as "watch out for that contractor, he's a cowboy" - ie. here's your system-today, gone tomorrow, oh what, you wanted a user interface? That'll be extra.

    • Dorothea says:

      Oh, that too, definitely. Only in the academic context I see more grad students than contractors. The result is the same: they're gone beyond recall shortly, and the systems they leave behind decay.

  • Bernard Peek says:

    Speaking as an IT Manager I share your pain. Frequently.

    We occasionally see the result of cowboy IT. Usually when someone with a lot of authority asks us to fix an Access database or a spreadsheet that's suddenly not working. So I've got to drop what I'm doing and assign someone to fixing a badly designed and completely undocumented application written in a language that we don't usually support.

    It's going to cost me $1000 of developer's time to investigate the problem. If it's bad I might have to spend another £20,000 fixing it. If it's really bad (I need to recover data from a broken disk) you can add another zero to that.

    My best solution is to make each department head aware that this isn't part of our normal service and that I expect them to pay my costs out of their budgets. A better solution of course is to make the IT department the first choice when a new application is required. OK, not every head of IT is a research biochemist too. But researchers need to get involved with the operation of their IT department to make sure that it meets their requirement. They also need to add realistic IT costs into their budgets. That may mean turning work away.

    • Yep. I know an IT pro who curses reflexively whenever someone says the word "Excel" to him. Again, externalities at work -- until you very wisely refused to let the cost-dumping happen to you.

  • Joann says:

    With respect to biomedical research, I am thinking the US granting agencies better wise up and not take leave to assume that one institution's facilities and research materials are the same as another's before the next catastrophe happens.

    I am specifically thinking of an example where a very major research university's improperly sourcing an off the shelf chemical compound for a human subject research protocol--wherein occurred the death of a young volunteer. The assumption is that only human grade compound formulations are to be used. One avenue of follow up afterwards, however, was that academic librarians located obscure, unknown citations describing previously observed reactions to the compound that would have been of interest to the IRB.

    • Dorothea says:

      Yes. I am working very hard to make the point at MPOW that good data management is going to become an issue of grant competitiveness. That, of course, depends on it actually becoming one!

  • Joann says:

    Some cowboy numbers from Science Magazine Feb.11, 2011

    http://www.sciencemag.org/content/331/6018/692/F1.medium.gif

    I.e. 50.2% surveyed archive their data on a lab computer--if it is saved at all.

  • ClintFeastWood says:

    Interesting blog. You want a Research IT group ? Be careful what you ask for. Where I work we have what is supposed to be a Research IT group but they are part of the larger computing facility and, correspondingly, seek to impose onto researchers a very narrow set of tools and services that are very limiting to anyone engaged in post-genomics research.

    At best the Research group can *maybe* find you some storage for your data (at high cost) but the network is shared with students passing MP3 and bitorrent packets so you might as well be on the general Internet. At worst the group functions as a bully sent by the CFO and his errand boy CIO, seeking to drastically reduce computing costs by "telling" researchers what they should be using even when no one on the Research IT staff has a relevant degree, applied for funding, or authored a publication. But this doesn't stop them from dictating solutions that are far more appropriate for use by undegrads than professional investigators wrestling with a never ending stream of data.

    When the economy took a downturn the university promoted the perception that IT is expensive so they've centralized as much as they can and dictate standards even when they aren't qualified to do so. To accomplish this they "in-sourced" a bunch of corporate castaways who know only "one way" to do anything and thats only if their sole source vendor relationships allow it. The central IT group was horrible before in terms of service and the new found authority handed to them by the CFO makes them insufferable.

    My advice to anyone is to do it yourself whenever possible. Yes it can be a pain to run your own setup but imagine a central IT guy walking in and telling you that you can NOT run Linux and that you must adopt .NET solutions for all your research. Or that you cannot share files to a collaborator in another building because its not "sanctioned" by the central IT security director (who knows only Microsoft solutions).

    My group knows more about running production servers than any of their "research engineers" who run for cover if you talk to them about tuning NFS parameters, using jumbo frames on a switch, establishing virtual networks, designing effective firewall policies, or setting up a high performance filesystem (e.g. Lustre) to accommodate next generation sequence data. I don't tell them what database solution to use for student records because its not my area of expertise - so why do they try to tell me what infrastructure and software I need to be a successful researcher ?

  • Monado says:

    Interesting subject! Have you considered writing a guideline for setting up a Research IT group? It could be very useful for other organizations and thus saleable if you wanted to bother.

    I'm eagerly awaiting your next article.

    If you are actively posting at least once a month, I'll add you to my blogroll.

  • Raincoat says:

    Any hope for a resurrection? You do a damn good blog. Twitter is great and all, but not the same--content or length.

  • scavenger says:

    ^ this

  • "Interesting subject! Have you considered writing a guideline for setting up a Research IT group?" - yeah, I agree. That could be a very good idea to make this real and could even bring good results.

  • We are presenting Almost a dozen dogecoins for the fortunate
    winner of a site operator with the selected prize number.
    The final results is going to be available in 4 weeks time.
    Your own prize code is: WNSDB6

Leave a Reply