I’m at a ThursdAI talk (Harvard’s Berkman Klein Center for Internet & Society and MIT Media Lab) being given by Neil Gaikwad (Twitter: @neilthemathguy, a Ph.D. at the MediaLab, in the Space Enabled Group.

Markets and institutions are parts of complex ecosystem, Neil says. His research looks at data from satellites that show how the Earth is changing: crops, water, etc. Once you’ve gathered the data, you can use machine learning to visualize the changes. There are ecosystems, including of human behavior, that are affected by this. It affects markets and institutions. E.g., a drought may require an institutional response, and affect markets.

Traditional markets, financial markets, and gig economies all share characteristics. Farmers markets are complex ecosystems of people with differing information and different amounts of it, i.e. asymmetric info. Same for financial markets. Same for gig economies.

Indian markets have been failing; there have been 300,000 suicides in the last 30 years. Stock markets have crashed suddenly due to blackbox marketing; in some cases we still don’t know why. And London has banned Uber. So, it doesn’t matter which markets or institutions we look at, they’re losing our trust.

An article in New Scientist asked what we can do to regain this trust. For black box AI, there are questions of fairness and equity. But what would human-machine collaboration be like? Are there design principles for markets.?

Neil stops for us to discuss.

Q: How do you define the justice?

A: Good question. Fairness? Freedom? The designer has a choice about how to define it.

Q: A UN project created an IT platform that put together farmers and direct consumers. The pricing seemed fairer to both parties. So, maybe avoid intermediaries, as a design principle?

Neil continues. So, what is the concept of justice here?

1. Rawls and Kant: Transcendental institutionalism. It’s deontological: follow a principle for perfect justice. Use those principles to define a perfect institution. The properties are defined by a social contract. But it doesn’t work, as in the examples we just saw. What is missing. People and society. [I.e., you run the institution according to principles, but that doesn’t guarantee that the outcome will be fair and just. My example: Early Web enthusiasts like me thought the Web was an institution built on openness, equality, creative anarchy, etc., yet that obviously doesn’t ensure that the outcome will share those properties.]

2. Realized-focused institutionalism (Sen
2009): How to reverse this trend. It is consequentialist: what will be the consequences of the design of an institution. It’s a comparative assessment of different forms of institutions. Instead of asking for the perfectly justice society, Sen asks how justice can be advanced. The most critical tool for evaluating any institution is to look at how it actually realizes how people’s lives change.

Sen argues that principles are important. They can be expressed by “niti,” Sanskrit for rules and institutions. But you also need nyaya: a form of social arrangement that makes sure that those rules are obeyed. These rules come from social choice, not social contract.

Example: Gig economies. The data comes from mechanical turk, upwork, crowdflower, etc. This creates employment for many people, but it’s tough. E.g., identifying images. Use supervised learning for this. The Turkers, etc., do the labelling to train the image recognition system. The Turkers make almost no money at this. This is the wicked problem of market design: The worker can have identifications rejected, sometimes with demeaning comments.

“The Market for Lemons” (Akerlog, et al., 1970): all the cars started to look alike and now all gig-workers look alike to those who hire them: there’s no value given to bringing one’s value to the labor.

So, who owns the data? Who has a stake in the models? In the intellectual property?

If you’re a gig worker, you’re working with strangers. You don’t know the reputation of the person giving me data. Or renting me the Airbnb apartment. So, let’s put a rule: reputation is the backbone. In sharing economies, most of the ratings are the highest. Reputation inflation. So, can we trust reputation? This happens because people have no incentive to rate. There’s social pressure to give a positive rating.

So, thinking about Sen, can we think about an incentive for honest reputation? Neil’s group has been thinking about a system [I thought he said Boomerang, but I can’t find that]. It looks at the workers’ incentives. It looks at the workers’ ratings of each other. If you’re a requester, you’ll see the workers you like first.

Does this help AI design?

MoralMachine has had 1.3M voters and 18M pairwise comparisons (i.e., people deciding to go straight or right). Can this be used as a voting based system for ethical decision making (AAAI 2018)? You collect the pairwise preferences, learn the model of preference, come to a collective preference, and have voting rules for collective decision.

Q: Aren’t you collect preferences, not normative judgments? The data says people would rather kill fat people than skinny ones.

A: You need the social behavior but also rules. For this you have to bring people into the loop.

Q: How do we differentiate between what we say we want and what we really want?

A: There are techniques, such as “Bayesian Truth Serum”nomics.mit.edu/files/1966”>Bayesian Truth Serum.

Conclusion: The success of markets, institutions or algorithms, is highly dependent on how this actually affects people’s lives. This thinking should be central to the design and engineering of socio-technical systems.

1 Comment »

I’m at the STEAM ed Finland conference in Jyväskylä where Mariia Gavriushenko is talking about personalized learning environments.

Web-based learning systems are being more and more widely used in large part because they can be used any time, anywhere. She points to two types: Learning management systems and game-based systems. But they lack personalization that makes them suitable for particular learners in terms of learning speed, knowledge background, preferences in learning and career, goals for future life, and their differing habits. Personalized systems can provide assistance in learning and adapt their learning path. Web-based learning shouldn’t just be more convenient. It should also be better adapted to personal needs.

But this is hard. But if you can do it, it can monitor the learner’s knowledge level and automatically present the right materials. In can help teachers create suitable material and find the most relevant content and convert it into comprehensive info. It can also help students identify the best courses and programs.

She talks about two types of personalized learning systems: 1. systems that allow the user to change the system or 2. the sysytem changes itself to meet the users needs. The systems can be based on rules and context or can be algorithm driven.

Five main features of adaptive learning systems:

• Pre-test

• Pacing and control

• Feedback and assessment

• Progress tracking and reports

• Motivation and reward

The ontological presentation of every learner keeps something like a profile for each user, enabling semantic reasoning.

She gives an example of this model: automated academic advising. It’s based on learning analytics. It’s an intelligent learning support system based on semantically-enhanced decision support, that identifies gaps, and recommends materials and courses. It can create a personal study plan. The ontology helps the system understand which topics are connected to others so that it can identify knowledge gaps.

An adaptive vocabulary learning environment provides cildren with an adaptive way to train their vocabulary, taking into account the individuality of the learner. It assumes the more similar the words, the harder they are to recognize.

Mariia believes we will make increasing use of adaptive educational tech.

Comments Off on [liveblog] Mariia Gavriushenko on personalized learning environments

I’m at the STEAM ed Finland conference in Jyväskylä. Maarit Rossi, who teaches math teaching around the world, is talking on the topic: “AI forces us to change maths education.”

Finnish teachers are doing a great, great job, she says. “But we are doing it too quietly.”

Education is too similar to industrial assembly lines. Students sit passively in rows. Students find math education to be boring, meaningless, and frightening. Typically this happens sometime in 5-7th grade. Teaching math has not changed in 100 years. It is a global problem.

Meanwhile, tech is changing really quickly. (She shows a photo from 1956 of workers shoving a 5 megabyte drive onto a truck.)

These days we are talking about personalizing math education. Easily available programs solve math problems. In the USA, people say the students are “cheating.” No, they’re being educated wrong. We need to be asking if we’re teaching students 10 critical skills, including cognitive flexibility, nebotiation, coordinating with others, emotional intelligence, critical thinking, creaetivity, complex problem solving, service orientation [and a couple of others I didn’t have time to copy down].

A modern math curriculum addresses attitudes, metacognition (e.g., self-regulation), skills, concepts, and processes. Instead, we focus on the concepts (e.g., algebraic, statistical, etc.).

A classroom has to be a safe place where you can make mistakes.

There are four pillars: practice, learning by doing, social learning, and interdisciplinary math. She gives some examples. Students estimate the price of a week’s shopping for a family of four. Maaritt has students work in groups of four. After that, they go to the nearest shop to find the actual prices; the students have to divide up the task to get it done in time. (You can have them do online shopping if there isn’t nearby shop.) Students estimate and round the numbers, tasks that are usually taught separately.

For higher grades, the students deal with real data from an African refugee camp. The students have to estimate how much food is needed to keep everyone alive for two weeks. “This is meaningful to them.”

It’s important for math to have double the lesson length. If it’s only one hour, it is not enough. “The students love it when they have the opportunity to think, to discover, to find themselves.”

Re-arrange the classroom. Cluster the tables rather than rows. The students can teach one another. “It is important that the feel successful.”

“And of course we use computers. And apps. And phones.”

“Math is also interesting because it can model many things.” If they have an embodied sense of a cubic meter, for example, they learn how to convert them to other measures. Or model the size of the solar system outside.

She has students estimate collections of objects, e.g. a bowl of noodles. Then they round. Then they count. Groups come up with strategies for counting, including doing it in ways that enable the count to be interrupted and resumed.

Physical exercise makes brains work better.

Classifying is important. She asks students to take sheets of paper and make the biggest triangle they can, and another of a different shape. They put all the triangles in the middle of the room. Then she asks them to see if they can cluster them by similarities.

“Students need to use their own language” rather than only hearing the teacher talk. This is how they learn to understand.

[My notes about the last few minutes, and the questions, go cut off via brain-computer glitch. Sorry.]

TAGS:

Comments Off on [liveblog] Maarit Rossi on teaching math that matters

I’m at the STEAM ed Finland conference in Jyväskylä. Maria Kankaanranta, Leena Hiltunen, Kati Clements and Tiina Mäkelä are on the faculty of the School of Education at the University of Jyväskylä The are going to talk about SMART education.

SMART means self-directed, motivated, adaptive, reseource enriched, and technology-embedded learning. (They credit South Korean researchers for this.) This is a paradigm shift: From education a specific times to any time. From lectures to motivated ed methods. From teaching the 3Rs to epanding the ed capacity. From traditional textbooks to enriched resources. From a physical space to anywhere there is the enabling tech.

One project (Horizon 2020) works across disciplines to connect students, parents, teachers, and companies. Companies expect universities to develop the skills they need, but you really have to begin with primary school. The aim of the project is to create a pedagogical framework and design principles for attractive and engaging STEM learning environments. She presents a long list of pedagogical design principles that guide the design of this kind of hybrid learning enviroments. It includes adaptive learning, self-regulation, project-based learning, novelty, but also conventionality: “you don’t have to abandon everything.”

What beyond MOODLE can we do? The EU has funded instruments for procurement of innovation. The presenters have worked on IMAILE & LEA (LearnTech Accelerator). IMAILE ran for 48 months in four countries. To address problems, the project pointed to two existing solutions: YipTree and AMIGO (e-books publisher from Spain). YipTree provides individual personalized learning paths (adaptive materials), student motivation by a virtual tutor and by other students, gamificiation, quick assessment tools, and notifications when a student is having difficulties. They tested this in two schools per country. YipTree did well.

They have been training teachers in computational thinking, programming, and robotics. They use online, mobile apps to make it available and free for all teachers and students. They’re using different training models to motivate and encourage teachers to adopt these apps. E.g., they’re “hijacking” schools and workplaces to train them where they are. Teachers really want human engagement.

Schools have access to tech resources but they’re under-used because the teachers don’t know what’s available and possible. This presentation’s project is helping teachers with this.

Conclusion: Smart ed is not easy. It takes time. It requires getting out of your comfort zone. It requires training, tools, research, and a human touch.

Q&A

Q: Does your model take into account students with disabilities?

A: Yes. Part of this is “access for all.” Also, IMAILE does. Imperfectly. They collaborate with a local school for the impaired.

Comments Off on [liveblogging] SMART education

I’m at the STEAM ed Finland conference in Jyväskylä where Ulla Richardson is going to talk about GraphoLearn, an adaptive learning method for learning to read.

Ulla has been working on the Jyväskylä< Longitudinal Study of Dyslexia (JLD). Globally, one third of people can’t read or have poor reading skills. One fifth of Europe also. About 15% of children have learning disabilities.

One Issue: knowing which sound goes with which letters. GraphoLearn is a game to help students with this, developed by a multidisciplinary team. You learn a word by connecting a sound to a written letter. Then you can move to syllables and words. The game teaches by trial and error. If you get it wrong, it immediately tells you the correct sound. It uses a simple adaptive approach to select the wrong choices that are presented. The game aims at being entertaining, and motivates also with points and rewards. It’s a multi-modal system: visual and audio. It helps dyslexics by training them on the distinctions between sounds. Unlike human beings, it never displays any impatience.

It adapts to the user’s skill level, automatically assessing performance and aiming at at 80% accuracy so that it’s challenging but not too challenging.

13,000 players have played in Finland, and more in other languages. Ulla displays data that shows positive results among students who use GraphoLearn, including when teaching English where every letter has multiple pronunciations.

There are some difficulties analyzing the logs: there’s great variability in how kids play the game, how long they play, etc. There’s no background info on the students. [I missed some of this.] There’s an opportunity to come up with new ways to understand and analyze this data.

Q&A

Q: Your work is amazing. When I was learning English I could already read Finnish, so I made natural mispronunciations of ape, anarchist, etc. How do you cope with this?

A: Spoken and written English are like separate languages, especially if Finnish is your first language where each letter has only one pronunciation. You need a bigger unit to teach a language like English. That’s why we have the Rime approach where we show the letters in more context. [I may have gotten this wrong.]

Q: How hard is it to adapt the game to each language’s logic?

A: It’s hard.

Comments Off on [liveblog] Ulla Richardson on a game that teaches reading

I’m at the STEAM ed Finland conference in Jyväskylä. Mirka Saarela and Sanna Juutinen are talking about their analysis of education data.

There’s a triennial worldwide study by the OECD to assess students. Usually, people are only interested in its ranking of education by country. Finland does extremely well at this. This is surprising because Finland does not do particularly well in the factors that are taken to produce high quality educational systems. So Finnish ed has been studied extensively. PISA augments this analysis using learning analytics. (The US does at best average in the OECD ranking.)

Traditional research usually starts with the literature, develops a hypothesis, collects the data, and checks the result. PISA’s data mining approach starts with the data. “We want to find a needle in the haystack, but we don’t know what the needle looks like.” That is, they don’t know what type of pattern to look for.

Results of 2012 PISA: If you cluster all 24M students with their characteristics and attitudes without regard to their country you get clusters for Asia, developing world, Islamic, western countries. So, that maps well.

For Finland, the most salient factor seems to be its comprehensive school system that promotes equality and equity.

In 2015 for the first time there was a computerized test environment available. Most students used it. The logfile recorded how long students spent on a task and the number of activities (mouse clicks, etc.) as well as the score. They examined the Finnish log file to find student profiles, related to student’s strategies and knowledge. Their analysis found five different clusters. [I can’t read the slide from here. Sorry.] They are still studying what this tells us. (They purposefully have not yet factored in gender.)

Nov. 2017 results showed that girls did far better than boys. The test was done in a chat environment which might have been more familiar for the girls? Is the computerization of the tests affecting the results? Is the computerization of education affecting the results? More research is needed.

Q&A

Q: Does the clustering suggest interventions? E.g., “Slow down. Less clicking.”

A: [I couldn’t quite hear the answer, but I think the answer is that it needs more analysis. I think.]

Q: I work for ETS. Are the slides available?

A: Yes, but the research isn’t public yet.

Comments Off on [liveblog] Mirka Saarela and Sanna Juutinen on analyzing education data

I’m at the STEAM ed Finland conference in Jyväskylä. Harri Ketamo is giving a talk on “micro-learning.” He recently won a prestigious prize for the best new ideas in Finland. He is interested in the use of AI for learning.

We don’t have enough good teachers globally, so we have to think about ed in new ways, Harri says. Can we use AI to bring good ed to everyone without hiring 200M new teachers globally? If we paid teachers equivalent to doctors and lawyers, we could hire those 200M. But we apparently not willing to do that.

One challenge: Career coaching. What do you want to study? Why? What are the skills you need? What do you need to know?

His company does natural language analysis — not word matches, but meaning. As an example he shows a shareholder agreement. Such agreements always have the same elements. After being trained on law, his company’s AI can create a map of the topic and analyze a block of text to see if it covers the legal requirements…the sort of work that a legal assistant does. For some standard agreements, we may soon not need lawyers, he predicts.

The system’s language model is a mess of words and relations. But if you zoom out from the map, the AI has clustered the concepts. At the Slush Sanghai conference, his AI could develop a list of the companies a customer might want to meet based on a text analysis of the companies’ web sites, etc. Likewise if your business is looking for help with a project.

Finland has a lot of public data about skills and openings. Universities’ curricula are publicly available.[Yay!] Unlike LinkedIn, all this data is public. Harri shows a map that displays the skills and competencies Finnish businesses want and the matching training offered by Finnish universities. The system can explore public information about a user and map that to available jobs and the training that is required and available for it. The available jobs are listed with relevancy expressed as a percentage. It can also look internationally to find matches.

The AI can also put together a course for a topic that a user needs. It can tell what the core concepts are by mining publications, courses, news, etc. The result is an interaction with a bot that talks with you in a Whatsapp like way. (See his paper “Agents and Analytics: A framework for educational data mining with games based learning”). It generates tests that show what a student needs to study if she gets a question wrong.

His newest project, in process: Libraries are the biggest collections of creative, educational material, so the AI ought to point people there. His software can find the common sources among courses and areas of study. It can discover the skills and competencies that materials can teach. This lets it cluster materials around degree programs. It can also generate micro-educational programs, curating a collection of readings.

His platform has an open an API. See Headai.

Q&A

Q: Have you done controlled experiments?

A: Yes. We’ve found that people get 20-40% better performance when our software is used in blended model, i.e., with a human teacher. It helps motivate people if they can see the areas they need to work on disappear over time.

Q: The sw only found male authors in the example you put up of automatically collated materials.

A: Small training set. Gender is not part of the metadata in Finland.

A: Don’t you worry that your system will exacerbate bias?

Q: Humans are biased. AI is a black box. We need to think about how to manage this

Q: [me] Are the topics generated from the content? Or do you start off with an ontology?

A: It creates its ontology out of the data.

Q: [me] Are you committing to make sure that the results of your AI do not reflect the built in biases?

A: Our news system on the Web presents a range of views. We need to think about how to do this for gender issues with the course software.

Comments Off on [liveblog] Harri Ketamo on micro-learning