Joho the Blogai Archives - Joho the Blog

April 5, 2018

[liveblog] Neil Gaikwad Human-AI Collaboration for Sustainable Market Design

I’m at a ThursdAI talk (Harvard’s Berkman Klein Center for Internet & Society and MIT Media Lab) being given by Neil Gaikwad (Twitter: @neilthemathguy, a Ph.D. at the MediaLab, in the Space Enabled Group.

NOTE: Live-blogging. Getting things wrong. Missing points. Omitting key information. Introducing artificial choppiness. Over-emphasizing small matters. Paraphrasing badly. Not running a spellpchecker. Mangling other people’s ideas and words. You are warned, people.

Markets and institutions are parts of complex ecosystem, Neil says. His research looks at data from satellites that show how the Earth is changing: crops, water, etc. Once you’ve gathered the data, you can use machine learning to visualize the changes. There are ecosystems, including of human behavior, that are affected by this. It affects markets and institutions. E.g., a drought may require an institutional response, and affect markets.

Traditional markets, financial markets, and gig economies all share characteristics. Farmers markets are complex ecosystems of people with differing information and different amounts of it, i.e. asymmetric info. Same for financial markets. Same for gig economies.

Indian markets have been failing; there have been 300,000 suicides in the last 30 years. Stock markets have crashed suddenly due to blackbox marketing; in some cases we still don’t know why. And London has banned Uber. So, it doesn’t matter which markets or institutions we look at, they’re losing our trust.

An article in New Scientist asked what we can do to regain this trust. For black box AI, there are questions of fairness and equity. But what would human-machine collaboration be like? Are there design principles for markets.?

Neil stops for us to discuss.

Q: How do you define the justice?

A: Good question. Fairness? Freedom? The designer has a choice about how to define it.

Q: A UN project created an IT platform that put together farmers and direct consumers. The pricing seemed fairer to both parties. So, maybe avoid intermediaries, as a design principle?

Neil continues. So, what is the concept of justice here?

1. Rawls and Kant: Transcendental institutionalism. It’s deontological: follow a principle for perfect justice. Use those principles to define a perfect institution. The properties are defined by a social contract. But it doesn’t work, as in the examples we just saw. What is missing. People and society. [I.e., you run the institution according to principles, but that doesn’t guarantee that the outcome will be fair and just. My example: Early Web enthusiasts like me thought the Web was an institution built on openness, equality, creative anarchy, etc., yet that obviously doesn’t ensure that the outcome will share those properties.]

2. Realized-focused institutionalism (Sen
2009): How to reverse this trend. It is consequentialist: what will be the consequences of the design of an institution. It’s a comparative assessment of different forms of institutions. Instead of asking for the perfectly justice society, Sen asks how justice can be advanced. The most critical tool for evaluating any institution is to look at how it actually realizes how people’s lives change.

Sen argues that principles are important. They can be expressed by “niti,” Sanskrit for rules and institutions. But you also need nyaya: a form of social arrangement that makes sure that those rules are obeyed. These rules come from social choice, not social contract.

Example: Gig economies. The data comes from mechanical turk, upwork, crowdflower, etc. This creates employment for many people, but it’s tough. E.g., identifying images. Use supervised learning for this. The Turkers, etc., do the labelling to train the image recognition system. The Turkers make almost no money at this. This is the wicked problem of market design: The worker can have identifications rejected, sometimes with demeaning comments.

The Market for Lemons” (Akerlog, et al., 1970): all the cars started to look alike and now all gig-workers look alike to those who hire them: there’s no value given to bringing one’s value to the labor.

So, who owns the data? Who has a stake in the models? In the intellectual property?

If you’re a gig worker, you’re working with strangers. You don’t know the reputation of the person giving me data. Or renting me the Airbnb apartment. So, let’s put a rule: reputation is the backbone. In sharing economies, most of the ratings are the highest. Reputation inflation. So, can we trust reputation? This happens because people have no incentive to rate. There’s social pressure to give a positive rating.

So, thinking about Sen, can we think about an incentive for honest reputation? Neil’s group has been thinking about a system [I thought he said Boomerang, but I can’t find that]. It looks at the workers’ incentives. It looks at the workers’ ratings of each other. If you’re a requester, you’ll see the workers you like first.

Does this help AI design?

MoralMachine has had 1.3M voters and 18M pairwise comparisons (i.e., people deciding to go straight or right). Can this be used as a voting based system for ethical decision making (AAAI 2018)? You collect the pairwise preferences, learn the model of preference, come to a collective preference, and have voting rules for collective decision.

Q: Aren’t you collect preferences, not normative judgments? The data says people would rather kill fat people than skinny ones.

A: You need the social behavior but also rules. For this you have to bring people into the loop.

Q: How do we differentiate between what we say we want and what we really want?

A: There are techniques, such as “Bayesian Truth Serum”nomics.mit.edu/files/1966”>Bayesian Truth Serum.

Conclusion: The success of markets, institutions or algorithms, is highly dependent on how this actually affects people’s lives. This thinking should be central to the design and engineering of socio-technical systems.

3 Comments »

April 2, 2018

"If a lion could talk" updated

“If a lion could talk, we could not understand him.”
— Ludwig Wittgenstein, Philosophical Investigations, 1953.

“If an algorithm could talk, we could not understand it.”
— Deep learning, Now.

1 Comment »

March 19, 2018

[liveblog] Kate Zwaard, on the Library of Congress Labs

Kate Zwaard (twitter: @kzwa) Chief of National Digital Strategies at the Library of Congress and leader of the LC Lab, is opening MIT Libraries’ Grand Challenge Summit..The next 1.5 days will be about the grand challenges in enabling scholarly discovery.

NOTE: Live-blogging. Getting things wrong. Missing points. Omitting key information. Introducing artificial choppiness. Over-emphasizing small matters. Paraphrasing badly. Not running a spellpchecker. Mangling other people’s ideas and words. You are warned, people.

For context she tells us that the LC is the largest library in the world, with 164M items. It has the world’s largest collection of film, maps, comic books, telephone directories, and more. [Too many for me to keep this post up with.]

  • You can wolk for two football fields just in the maps section. The world’s largest collection of recorded sound. The largest collection

  • Personal papers from Ben Franklin, Rosa Parks, Groucho Marx, Claude Shannon, and so many more.

  • Last year they circulated almost a million physical items.

  • Every week 11,000 tangible items come in through the Copyright office.

  • Last year, they digitized 4.7M iems, as well 730M documents crawled from the Web, plus much more. File count: 243M and growing every day.

These serve just one of the LC’s goal: “Acquire, preserve, and provide access to a universal collection of knowledge and the record of America’s creativity.” Not to mention serving Congress, and much more. [I can only keep up with a little of this. Kate’s a fantastic presenter and is not speaking too quickly. The LC is just too big!]

Kate thinks of the LC’s work as an exothermic reaction that needs an activation energy or catalyst. She leads the LC Labs, which started a year ago as a place of experimentation. The LC is a delicate machine, which makes it hard for it to change. The Labs enable experimentation. “Trying things that are easy and cheap is the only way forward.”

When thinking about what to do next, she things about what’s feasible and the impact. One way of having impact: demonstrating that the collection has unexplored potentials for research. She’s especially interested in how the Labs can help deal with the problem of scale at the LC.

She talks about some of Lab’s projects.

If you wanted to make stuff with LC data, there was no way of doing that. Now there’s LC for Robots, added documentation, and Jupyter Notebooks: an open source Web app that let you create open docs that contain code, running text, etc. It lets people play with the API without doing all the work from scratch.

But it’s not enough to throw some resources onto a Web page. The NEH data challenge asked people to create new things using the info about 12M newspapers in the collection. Now the Lab has the Congressional Data Challenge: do something with with Congressional data.

Labs has an Innovator in Residence project. The initial applicants came from LC to give it a try. One of them created a “Beyond Words” crowdsourcing project that asks them to add data to resources

Kate likes helping people find collections they otherwise would have missed. For ten years LC has collaborated wi the Flickr Commons. But they wanted to crowdsource a transcription project for any image of text. A repo will be going up on GitHub shortly for this.

In the second year of the Innovator in Residence, they got the artist Jer Thorp [Twitter: @blprnt] to come for 6 months. Kate talks about his work with the papers of Edward Lorenz, who coined the phrase “The Butterfly Effect.” Jer animated Lorenz’s attractor, which, he points out, looks a bit like a butterfly. Jer’s used the attractor on a collection of 3M words. It results in “something like a poem.” (Here’s Jer’s Artist in the Archive podcast about his residency.)

Jer wonders how we can put serendipity back into the LC and into the Web. “How do we enable our users to be carried off by curiousity not by a particular destination.” The LC is a closed stack library, but it can help guide digital wanderers. ”

Last year the LC released 25M catalog records. Jer did a project that randomly pulls the first names of 20 authors in any particular need. It demonstrates, among other things, the changing demographics of authors. Another project: “Birthy Deathy” that displays birthplace info. Antother looks for polymaths.

In 2018 the Lab will have their first open call for an Innovator in Residence. They’ll be looking for data journalists.

Kate talks about Laura Wrubel
‘s work with the Lab. “Library of Congress Colors” displays a graphic of the dominant colors in a collection.

Or Laura’s Photo Roulette: you guess the date of a photo.

Kate says she likes to think that libraries not just “book holes.” One project: find links among items in the archives. But the WARC format is not amenable to that.

The Lab is partnering with lots of great grops, including JSONstor and WikiData.

They’re working on using machine learning to identify place names in their photos.

What does this have to do with scale, she asks, nothng that the LC has done pretty well with scale. E.g., for the past seven years, the size of their digital collection has doubled every 32 months.

The Library also thinks about how to become a place of warmth and welcome. (She gives a shout out to MIT Libraries’ Future of Libraries
report). Right now, visitors and scholars go to different parts of the building. Visitors to the building see a monument to knowledge, but not a living, breathing place. “The Library is for you. It is a place you own. It is a home.”

She reads from a story by Ann Lamott.

How friendship relates to scale. “Everything good that has happened in my life has happened because of friendship.” The average length of employment of a current employee is thirty years. — that’s not the average retirement year. “It’s not just for the LC but for our field.” Good advice she got: “Pick your career by the kind of people you like to be around.” Librarians!

“We’ve got a tough road ahead of us. We’re still in the early days of the disruption that computation is going to bring to our profession.” “Friendship is what will get us through these hard times. We need to invite peopld into the tent.” “Everything we’ve accomplished has been through the generosity of our friends and colleagues.” This 100% true of the Labs. It’s ust 4 people, but everything they do is done in collaboration.

She concludes (paraphrasing badly): I don’t believe in geniuses, and i don’t believe in paradigm shirts. I believe in friendship and working together over the long term. [She put this far better.]

Q&A

Q: How does the Lab decide on projects?

A: Collaboratively

Q: I’m an archivist at MIT. The works are in closed stack, which can mislead people about the scale. How do we explain the scale in an interesting way.

A: Funding is difficult because so much of the money that comes is to maintain and grow the collection and services. It can be a challenge to carve out funding for experimentation and innovation. We’ve been working hard on finding ways to help people wrap their heads around the place.

Q: Data science students are eager to engage, e.g., as interns. How can academic institutions help to make that happen?

A: We’re very interested in what sorts of partnerships we can create to bring students in. The data is so rich, and the place is so interesting.

Q: Moving from models that think about data as packages as opposed to unpacking and integrating. What do you think about the FAIR principle: making things Findable, Accesible Interoperable, and Reusable? Also, we need to bring in professionals thinking about knowledge much more broadly.

I’m very interested in Hathi Trust‘s data capsules. Are there ways we can allow people to search through audio files that are not going to age into the commons until we’re gone? You’re right: the model of chunks coming in and out is not going to work for us.

Q: In academia, our focus has been to provide resources efficiently. How can weave in serendipity without hurting the efficiency?

A: That’s hard. Maybe we should just serve the person who has a specific purpose. You could give ancillary answers. And crowdsourcing could make a lot more available.

[Great talk.]

3 Comments »

February 11, 2018

The story of lead and crime, told in tweets

Patrick Sharkey [twitter: patrick_sharkey] uses a Twitter thread to evaluate the evidence about a possible relationship between exposure to lead and crime. The thread is a bit hard to get unspooled correctly, but it’s worth it as an example of:

1. Thinking carefully about complex evidence and data.

2. How Twitter affects the reasoning and its expression.

3. The complexity of data, which will only get worse (= better) as machine learning can scale up their size and complexity.

Note: I lack the skills and knowledge to evaluate Patrick’s reasoning. And, hat tip to David Lazer for the retweet of the thread.

Comments Off on The story of lead and crime, told in tweets

The brain is not a computer and the world is not information

Robert Epstein argues in Aeon against the dominant assumption that the brain is a computer, that it processes information, stores and retrieves memories, etc. That we assume so comes from what I think of as the informationalizing of everything.

The strongest part of his argument is that computers operate on symbolic information, but brains do not. There is no evidence (that I know of, but I’m no expert. On anything) that the brain decomposes visual images into pixels and those pixels into on-offs in a code that represents colors.

In the second half, Epstein tries to prove that the brain isn’t a computer through some simple experiments, such as drawing a dollar bill from memory and while looking at it. Someone committed to the idea that the brain is a computer would probably just conclude that the brain just isn’t a very good computer. But judge for yourself. There’s more to it than I’m presenting here.

Back to Epstein’s first point…

It is of the essence of information that it is independent of its medium: you can encode it into voltage levels of transistors, magnetized dust on tape, or holes in punch cards, and it’s the same information. Therefore, a representation of a brain’s states in another medium should also be conscious. Epstein doesn’t make the following argument, but I will (and I believe I am cribbing it from someone else but I don’t remember who).

Because information is independent of its medium, we could encode it in dust particles swirling clockwise or counter-clockwise; clockwise is an on, and counter is an off. In fact, imagine there’s a dust cloud somewhere in the universe that has 86 billion motes, the number of neurons in the human brain. Imagine the direction of those motes exactly matches the on-offs of your neurons when you first spied the love of your life across the room. Imagine those spins shift but happen to match how your neural states shifted over the next ten seconds of your life. That dust cloud is thus perfectly representing the informational state of your brain as you fell in love. It is therefore experiencing your feelings and thinking your thoughts.

That by itself is absurd. But perhaps you say it is just hard to imagine. Ok, then let’s change it. Same dust cloud. Same spins. But this time we say that clockwise is an off, and the other is an on. Now that dust cloud no longer represents your brain states. It therefore is both experiencing your thoughts and feeling and is not experiencing them at the same time. Aristotle would tell us that that is logically impossible: a thing cannot simultaneously be something and its opposite.

Anyway…

Toward the end of the article, Epstein gets to a crucial point that I was very glad to see him bring up: Thinking is not a brain activity, but the activity of a body engaged in the world. (He cites Anthony Chemero’s Radical Embodied Cognitive Science (2009) which I have not read. I’d trace it back further to Andy Clark, David Chalmers, Eleanor Rosch, Heidegger…). Reducing it to a brain function, and further stripping the brain of its materiality to focus on its “processing” of “information” is reductive without being clarifying.

I came into this debate many years ago already made skeptical of the most recent claims about the causes of consciousness by having some awareness of the series of failed metaphors we have used over the past couple of thousands of years. Epstein puts this well, citing another book I have not read (and another book I’ve consequently just ordered):

In his book In Our Own Image (2015), the artificial intelligence expert George Zarkadakis describes six different metaphors people have employed over the past 2,000 years to try to explain human intelligence.

In the earliest one, eventually preserved in the Bible, humans were formed from clay or dirt, which an intelligent god then infused with its spirit. That spirit ‘explained’ our intelligence – grammatically, at least.

The invention of hydraulic engineering in the 3rd century BCE led to the popularity of a hydraulic model of human intelligence, the idea that the flow of different fluids in the body – the ‘humours’ – accounted for both our physical and mental functioning. The hydraulic metaphor persisted for more than 1,600 years, handicapping medical practice all the while.

By the 1500s, automata powered by springs and gears had been devised, eventually inspiring leading thinkers such as René Descartes to assert that humans are complex machines. In the 1600s, the British philosopher Thomas Hobbes suggested that thinking arose from small mechanical motions in the brain. By the 1700s, discoveries about electricity and chemistry led to new theories of human intelligence – again, largely metaphorical in nature. In the mid-1800s, inspired by recent advances in communications, the German physicist Hermann von Helmholtz compared the brain to a telegraph.

Maybe this time our tech-based metaphor has happened to get it right. But history says we should assume not. We should be very alert to the disanologies, which Epstein helps us with.

Getting this right, or at least not getting it wrong, matters. The most pressing problem with the informationalizing of thought is not that it applies a metaphor, or even that the metaphor is inapt. Rather it’s that this metaphor leads us to a seriously diminished understanding of what it means to be a living, caring creature.

I think.

 

Hat tip to @JenniferSertl for pointing out the Aeon article.

Comments Off on The brain is not a computer and the world is not information

February 1, 2018

Can AI predict the odds on you leaving the hospital vertically?

A new research paper, published Jan. 24 with 34 co-authors and not peer-reviewed, claims better accuracy than existing software at predicting outcomes like whether a patient will die in the hospital, be discharged and readmitted, and their final diagnosis. To conduct the study, Google obtained de-identified data of 216,221 adults, with more than 46 billion data points between them. The data span 11 combined years at two hospitals,

That’s from an article in Quartz by Dave Gershgorn (Jan. 27, 2018), based on the original article by Google researchers posted at Arxiv.org.

…Google claims vast improvements over traditional models used today for predicting medical outcomes. Its biggest claim is the ability to predict patient deaths 24-48 hours before current methods, which could allow time for doctors to administer life-saving procedures.

Dave points to one of the biggest obstacles to this sort of computing: the data are in such different formats, from hand-written notes to the various form-based data that’s collected. It’s all about the magic of interoperability … and the frustration when data (and services and ideas and language) can’t easily work together. Then there’s what Paul Edwards, in his great book A Vast Machine calls “data friction”: “…the costs in time, energy, and attention required simply to collect, check, store, move, receive, and access data.” (p. 84)

On the other hand, machine learning can sometimes get past the incompatible expression of data in a way that’s so brutal that it’s elegant. One of the earlier breakthroughs in machine learning came in the 1990s when IBM analyzed the English and French versions of Hansard, the bi-lingual transcripts of the Canadian Parliament. Without the machines knowing the first thing about either language, the system produced more accurate results than software that was fed rules of grammar, bilingual dictionaries, etc.

Indeed, the abstract of the Google paper says “Constructing predictive statistical models typically requires extraction of curated predictor variables from normalized EHR data, a labor-intensive process that discards the vast majority of information in each patient’s record. We propose a representation of patients’ entire, raw EHR records based on the Fast Healthcare Interoperability Resources (FHIR) format. ” It continues: “We demonstrate that deep learning methods using this representation are capable of accurately predicting multiple medical events from multiple centers without site-specific data harmonization.”

The paper also says that their approach affords clinicians “some transparency into the predictions.” Some transparency is definitely better than none. But, as I’ve argued elsewhere, in many instances there may be tools other than transparency that can give us some assurance that AI’s outcomes accord with our aims and our principles of fairness.

 


 

I found this article by clicking on Dave Gershgon’s byline on a brief article about the Wired version of the paper of mine I referenced in the previous paragraph. He does a great job explaining it. And, believe me, it’s hard to get a writer — well, me, anyway — to acknowledge that without having to insert even one caveat. Thanks, Dave!

Comments Off on Can AI predict the odds on you leaving the hospital vertically?

January 11, 2018

Artificial water (+ women at PC Gamer)

I’ve long wondered — like for a couple of decades — when software developers who write algorithms that produce beautiful animations of water will be treated with the respect accorded to painters who create beautiful paintings of water. Both require the creators to observe carefully, choose what they want to express, and apply their skills to realizing their vision. When it comes to artistic vision or merit, are there any serious differences?

In the January issue of PC Gamer , Philippa Warr [twitter: philippawarr] — recently snagged
from Rock, Paper, Shotgun points to v r 3 a museum of water animations put together by Pippin Barr. (It’s conceivable that Pippin Barr is Philippa’s hobbit name. I’m just putting that out there.) The museum is software you download (here) that displays 24 varieties of computer-generated water, from the complex and realistic, to simple textures, to purposefully stylized low-information versions.

[ IMAGE ]

Philippa also points to the Seascape
page by Alexander Alekseev where you can read the code that procedurally produces an astounding graphic of the open sea. You can directly fiddle with the algorithm to immediately see the results. (Thank you, Alexander, for putting this out under a Creative Commons license.) Here’s a video someone made of the result:

Philippa also points to David Li’s Waves where you can adjust wind, choppiness, and scale through sliders.

More than ten years ago we got to the point where bodies of water look stunning in video games. (Falling water is a different question.) In ten years, perhaps we’ll be there with hair. In the meantime, we should recognize software designers as artists when they produce art.

 


 

Good work, PC Gamer, in increasing the number of women reviewers, and especially as members of your editorial staff. As a long-time subscriber I can say that their voices have definitely improved the magazine. More please!

Comments Off on Artificial water (+ women at PC Gamer)

December 17, 2017

[liveblog] Mariia Gavriushenko on personalized learning environments

I’m at the STEAM ed Finland conference in Jyväskylä where Mariia Gavriushenko is talking about personalized learning environments.


Web-based learning systems are being more and more widely used in large part because they can be used any time, anywhere. She points to two types: Learning management systems and game-based systems. But they lack personalization that makes them suitable for particular learners in terms of learning speed, knowledge background, preferences in learning and career, goals for future life, and their differing habits. Personalized systems can provide assistance in learning and adapt their learning path. Web-based learning shouldn’t just be more convenient. It should also be better adapted to personal needs.


But this is hard. But if you can do it, it can monitor the learner’s knowledge level and automatically present the right materials. In can help teachers create suitable material and find the most relevant content and convert it into comprehensive info. It can also help students identify the best courses and programs.


She talks about two types of personalized learning systems: 1. systems that allow the user to change the system or 2. the sysytem changes itself to meet the users needs. The systems can be based on rules and context or can be algorithm driven.


Five main features of adaptive learning systems:

  • Pre-test

  • Pacing and control

  • Feedback and assessment

  • Progress tracking and reports

  • Motivation and reward


The ontological presentation of every learner keeps something like a profile for each user, enabling semantic reasoning.


She gives an example of this model: automated academic advising. It’s based on learning analytics. It’s an intelligent learning support system based on semantically-enhanced decision support, that identifies gaps, and recommends materials and courses. It can create a personal study plan. The ontology helps the system understand which topics are connected to others so that it can identify knowledge gaps.


An adaptive vocabulary learning environment provides cildren with an adaptive way to train their vocabulary, taking into account the individuality of the learner. It assumes the more similar the words, the harder they are to recognize.


Mariia believes we will make increasing use of adaptive educational tech.

Comments Off on [liveblog] Mariia Gavriushenko on personalized learning environments

[liveblog] Ulla Richardson on a game that teaches reading

I’m at the STEAM ed Finland conference in Jyväskylä where Ulla Richardson is going to talk about GraphoLearn, an adaptive learning method for learning to read.

NOTE: Live-blogging. Getting things wrong. Missing points. Omitting key information. Introducing artificial choppiness. Over-emphasizing small matters. Paraphrasing badly. Not running a spellpchecker. Mangling other people’s ideas and words. You are warned, people.


Ulla has been working on the Jyväskylä< Longitudinal Study of Dyslexia (JLD). Globally, one third of people can’t read or have poor reading skills. One fifth of Europe also. About 15% of children have learning disabilities.


One Issue: knowing which sound goes with which letters. GraphoLearn is a game to help students with this, developed by a multidisciplinary team. You learn a word by connecting a sound to a written letter. Then you can move to syllables and words. The game teaches by trial and error. If you get it wrong, it immediately tells you the correct sound. It uses a simple adaptive approach to select the wrong choices that are presented. The game aims at being entertaining, and motivates also with points and rewards. It’s a multi-modal system: visual and audio. It helps dyslexics by training them on the distinctions between sounds. Unlike human beings, it never displays any impatience.

It adapts to the user’s skill level, automatically assessing performance and aiming at at 80% accuracy so that it’s challenging but not too challenging.


13,000 players have played in Finland, and more in other languages. Ulla displays data that shows positive results among students who use GraphoLearn, including when teaching English where every letter has multiple pronunciations.


There are some difficulties analyzing the logs: there’s great variability in how kids play the game, how long they play, etc. There’s no background info on the students. [I missed some of this.] There’s an opportunity to come up with new ways to understand and analyze this data.


Q&A


Q: Your work is amazing. When I was learning English I could already read Finnish, so I made natural mispronunciations of ape, anarchist, etc. How do you cope with this?


A: Spoken and written English are like separate languages, especially if Finnish is your first language where each letter has only one pronunciation. You need a bigger unit to teach a language like English. That’s why we have the Rime approach where we show the letters in more context. [I may have gotten this wrong.]


Q: How hard is it to adapt the game to each language’s logic?


A: It’s hard.

Comments Off on [liveblog] Ulla Richardson on a game that teaches reading

December 16, 2017

[liveblog] Mirka Saarela and Sanna Juutinen on analyzing education data

I’m at the STEAM ed Finland conference in Jyväskylä. Mirka Saarela and Sanna Juutinen are talking about their analysis of education data.

NOTE: Live-blogging. Getting things wrong. Missing points. Omitting key information. Introducing artificial choppiness. Over-emphasizing small matters. Paraphrasing badly. Not running a spellpchecker. Mangling other people’s ideas and words. You are warned, people.


There’s a triennial worldwide study by the OECD to assess students. Usually, people are only interested in its ranking of education by country. Finland does extremely well at this. This is surprising because Finland does not do particularly well in the factors that are taken to produce high quality educational systems. So Finnish ed has been studied extensively. PISA augments this analysis using learning analytics. (The US does at best average in the OECD ranking.)


Traditional research usually starts with the literature, develops a hypothesis, collects the data, and checks the result. PISA’s data mining approach starts with the data. “We want to find a needle in the haystack, but we don’t know what the needle looks like.” That is, they don’t know what type of pattern to look for.


Results of 2012 PISA: If you cluster all 24M students with their characteristics and attitudes without regard to their country you get clusters for Asia, developing world, Islamic, western countries. So, that maps well.


For Finland, the most salient factor seems to be its comprehensive school system that promotes equality and equity.

In 2015 for the first time there was a computerized test environment available. Most students used it. The logfile recorded how long students spent on a task and the number of activities (mouse clicks, etc.) as well as the score. They examined the Finnish log file to find student profiles, related to student’s strategies and knowledge. Their analysis found five different clusters. [I can’t read the slide from here. Sorry.] They are still studying what this tells us. (They purposefully have not yet factored in gender.)


Nov. 2017 results showed that girls did far better than boys. The test was done in a chat environment which might have been more familiar for the girls? Is the computerization of the tests affecting the results? Is the computerization of education affecting the results? More research is needed.


Q&A


Q: Does the clustering suggest interventions? E.g., “Slow down. Less clicking.”

A: [I couldn’t quite hear the answer, but I think the answer is that it needs more analysis. I think.]


Q: I work for ETS. Are the slides available?


A: Yes, but the research isn’t public yet.

Comments Off on [liveblog] Mirka Saarela and Sanna Juutinen on analyzing education data

Next Page »