Youth Forum Usability Results -- from Michele Anciaux Aoki 2/4/2004

Schools:

• Hamilton International Middle School (HIMS) 6-8
• John Stanford International School (JSIS) K-5
• University Preparatory Academy (U Prep) private 6-12

Feb 3 10:00-10:30 am - Mark Fisher (HIMS Social Studies) - computer lab
Feb 3 10:30-11:00 am - Mauna Edrozo 8th grade Student (HIMS)
Feb 3 11:00-11:30 am - Kayomi McDonald (HIMS Japanese)

Feb 4 8:30-9:30 am - Nani Castor Peck (JSIS 5th grade)
Feb 4 10:45-11:15 am - Etsuko Kimura (U Prep Japanese)
Feb 4 11:15-11:45 am - Dandy Porter (U Prep Social Studies)
Feb 4 2:10-2:40 pm - Mihoko Tsang (JSIS Japanese)

Feb 5 10:00-10:30 am Mary Bernson (UW Jackson School of International Studies, East Asia Resource Center)

Usability Test Scenarios

General Site

Script: We're going to begin the usability test by just opening the site and looking at how it's organized. I'll ask you to do a few simple tasks. Feel free to "guess" at what to do or how to do it. There are no right answers. Also, if you can, verbalize what you're thinking while you're doing it. I'll be taking notes so I can go back and see where the confusing things are in the site. This will help us make it much easier to use.

Open the site: http://youthforum.askasia.org/ What's the first thing you're inclined to do when you view this page?	(Capture general impressions.) Mark: Find out what's available on the site; register, just read what's there Mauna: Look around the menu to get oriented, see the Orientation, Project Showcase, Skills Toolbox, Register, etc. Kayomi: When she saw Japanese option, wanted to view it first Nani: Cultural exchange; means of communicating; liked list of careers Etsuko: Fine; would be inclined to Register first (or check that out in case you have to pay), review the rules (looked at Japanese) Mihoko: Impressions -- very young, pop, easy to read, images are interesting; not sure what all is on the site
If you go to HELP ORIENTATION, what do you expect to find there?	Mark: How to use the page Mauna: Explains about the site Kayomi: What site is about; brief description Nani: How to get ideas about the YF; orienting to different things available on the site Etsuko: Basic site navigation skills, such as scrolling, printing, downloading files Mihoko: Introduction to the site
If you go to PROJECT SHOWCASE, what do you expect to find there?	(Do they realize they can go here without logging in?) Mark: Yes, find examples of products, find out about the projects Mauna: Series of projects that they might choose from Kayomi: Explanation of the projects Nani: Ideas of projects for exchange Etsuko: See content-based cultural projects Mihoko: Pretty similar in Japanese ("Introduction to Projects") so hard to tell the difference between this and Orientation
If you go to SKILLS TOOLBOX, what do you expect to find there?	Mark: Find examples of how to use the site; how to get the projects done Mauna: Things to use for school, e.g., Dictionaries, Exercises, Memory Games for Language Kayomi: Strange (in Japanese); not sure what would be there Nani: Not sure, maybe skills for working with projects or enhancing exchanges; shows me what's available (e.g., video conferencing, photos, text) Etsuko: Different skills needed to do the projects (whereas Help is more general) Mihoko: In Japanese this is a literal translation of "Skills Toolbox" (not sure how meaningful it is); maybe this is Helpdesk, resources
Find a Gallery project about the Nebuta Festival.	(Do they go to Project Showcase?) Mark: Yes Mauna: Yes Kayomi: Not sure at first (looking for a Search box); later found it after explanation Nani: Yes Etsuko: Yes Mihoko: Yes
Which slide shows people marching in the festival?	(Do they click on Next until they get to it?) Mark: Yes, but struggled to figure out how to find it Mauna: Yes Kayomi: Yes Nani: First went to Number of Slides, then found the Next button Etsuko: First pointed to Number of Slides, then found NEXT; suggested using NEXT SLIDE Mihoko: Went to Number of slides before using NEXT
How many projects are listed in the Project Showcase? http://youthforum.askasia.org/pub-bin/showcase.pl	(Do they click on Project Showcase in the top nav? Mark: Yes (2) Mauna: Clicked around the slides for this project first, then finally went to the Project Showcase Kayomi: Yes Nani: Yes Etsuko: Couldn't quite tell from how projects were listed (one had just title, and one both title and description) Mihoko: Not obvious where to find the list of projects
Find the World of Maps and Numbers. http://youthforum.askasia.org/skills/skills_temp.html	(Do they go to Skills Toolbox?) Mark: Unsure, first went to Orientation, then finally to Skills Toolbox Mauna: Yes Kayomi: Yes Nani: Yes Etsuko: Not clear what these topics are Mihoko: Went to Orientation
Go back to the home page	(Do they click Home link on top nav?) Mark: Yes Mauna: Yes Kayomi: Yes Nani: Yes Etsuko: No, not obvious (in Japanese would say "Return to the home page") Mihoko: First hit back key, then found Home link

 

Members Area

Registering and Logging in.

Script: Now that we've looked around the public areas of the site, we're going to log in and check out the Members Area.

From the home page, Register on the YF.	(Do they have any problems with the steps?) Mark: Tried to fill in Please log in first, struggled, then figured out how to register Kayomi: No problems Nani: Wanted to enter on LOG IN (rather than clicking REGISTER Etsuko: Wanted to fill in the LOG IN; not clear where to register; Japanese form doesn't indicate that the Fax is optional; note clear what to put in Comments; Japanese form doesn't match English (i.e. Email/PW, then re-enter Email/PW) Mihoko: No problems
Now that you're registered, wait just a minute while I log in as Administrator and approve your registration.	(or can have them log in as test1@school.org aaa) Mauna: Yes
Now that your registration is approved, LOG IN.	(Do they have any problems with log in?) Mark: No problems Mauna: No problems Kayomi: No problems Nani: No problems Etsuko: OK (but Email field is too short, to it was hard to debug the problem with how she entered it) Mihoko: No problems
Now you see the My Account page of the YF displayed. What are you inclined to do first?	(Capture general impressions.) Mark: See what projects might be available, check out Project Showcase Mauna: Sees Projects: In Progress or Finished; My Account; Contact, etc.; DIDN'T see the Teachers' Modules and Available Projects right away until I pointed them out Kayomi: Didn't see Teachers' Modules Nani: It would be fun to see what projects and lesson plans are available Etsuko: Understands projects, but overall the page is confusing, not complete, needs orientation; thought that Available Projects were the Showcase of Japanese projects already started (for comparison) Mihoko: Personalized page to post projects (but currently don't have any); expects more info for the registered user; didn't notice new links in nav for Teachers' Modules and Available Projects

Applying for a Project (from Available Projects, Forum Project)

Script: Now that you're in the Members Area, you can apply for projects.

Let's say you'd like to apply for a project. How would you expect to do that?	(Do they go to Available Projects? Teachers' Modules) Kayomi: Project Showcase?, or look for link "Apply for Project"; finally found Available Projects Nani: Go to Available Projects Etsuko: Go to Available Projects Mihoko: Not obvious
Let's start by reviewing the AVAILABLE PROJECTs. http://youthforum.askasia.org/members/available_projects.pl	(Do they go to Available Projects?) Mark: Yes Mauna: Hard to find Kayomi: Hard to find Nani: Yes Etsuko: Yes Mihoko: Yes
How would you choose between INDEPENDENT and COLLABORATIVE projects?	(Do they understand the difference?) Mark: Independent - directions are there, work within classroom; Collaborative - work with other classroom in US or Japan Mauna: Independent - one person; Collaborative - Other people work together Kayomi: Looking at the project names it was hard to tell what the difference would be Nani: Collaborative - more interactive, one party to next Etsuko: Independent - do by yourself or schoolwide; Collaborative - group or bunch of schools Mihoko: Independent - only 1 person or classroom-based; Collaborative - exchange between two schools (Note: she was looking at project names and couldn't see a distinction)
Sign up for An Interview Project from the INDEPENDENT Projects list.	(Are they able to use the sign-up form? Do they click OK to get back to home page?) Mark: Yes (Comments: See an overview of the project to decide if he wants to have it sent to him or not; unsure about clicking on it; seemed disoriented when he got back to home page) Mauna: Yes, but not sure about clicking OK to get back to the home page Kayomi: Hard to find project in a list Nani: Liked project description, did find OK Etsuko: Description was not obvious (translation of project description was not too good); Japanese translated Subject as "Theme" -- didn't make sense (should be kyoya) Mihoko: Yes, hesitated on what to do with OK

Adding Content to a Forum Project

Script: Now, let's add some content to that project.

Select the project that you'd like to work on.	(Do they click on the project from the PROJECTS: IN PROGRESS list?) Mark: struggled, needed instructions Mauna: Yes Kayomi: Yes Nani: Yes Etsuko: Not obvious how to select (the OK bugged her -- not sure what to do with) Mihoko: Yes
Add a memo saying that you began this project on February X.	(Do they fill out the memo form correctly?) Mark: Yes Mauna: Yes Kayomi: Yes Nani: Didn't fill out From field initially Etsuko: Skilled From field; translation of Memo not good in Japanese; was surprised that this was not a personal memo (but could be read by students) Mihoko: Skilled From field, then went back when she got an error
Now you're ready to add some content to this project. What do you do?	(Do they click OPEN THIS PROJECT?) Mark: Read the Project Report first Mauna: Yes Kayomi: Looked on top menu, didn't see OPEN THIS PROJECT link until it was pointed out Nani: Yes Etsuko: Yes Dandy: Wanted to read find print (instructions) first Mihoko: Not obvious
The project is now open, what do you do to add content?	(Do they click SEND NEW MESSAGE?) Mark: Tried to open Teachers' Modules; not obvious how to add new content Mauna: Yes Kayomi: Hard to find Nani: Yes Dandy: Maybe ADD NEW MESSAGE instead? Mihoko: Skipped over; finally found it
Fill out the NEWMESSAGE form. (We can give them some sample text, e.g., Beatrice's Goat to copy and paste into the message.)	(Are they able to SUBMIT MESSAGE?) Mark: Yes Mauna: Yes Nani: Yes Mihoko: Yes
After seeing your message displayed, go back to the index to see it on the Date and Subject tabs.	(Do they click BACK TO INDEX?) Mark: Clicked Project Report; didn't think about using BACK TO INDEX Mauna: Yes Nani: Not sure this is intuitive Mihoko: Not intuitive
Check to see how the message is displayed on the DATE and SUBJECT tabs.	(Do they click on the tabs?) Mark: Hard to figure out Mauna: Yes Nani: Yes Mihoko: Yes
Select the message to edit it.	(Do they click on the date link? when the message opens, do they click EDIT THIS MESSAGE?) Mark: Not clear what to do Mauna: Went to Project Report; hard to figure out how to select the message Nani: Not clear how to select a message to edit (from the data link) Mihoko: Hard to find date link
Add a line to the message and save it.	(Do they click SAVE CHANGES?) Mark: Looking for DONE button Mauna: Yes Nani: Yes Mihoko: Yes
After seeing your message displayed, go back to the index to see it on the Date and Subject tabs.	(Do they click BACK TO INDEX?) Mark: Yes Mauna: Yes, but not intuitive Mihoko: Yes, but not intuitive
Select the message to reply to it.	(Do they click on the date link? When the message opens, do they click REPLY TO THIS MESSAGE?) Mark: Had trouble figuring out how to select the message Mauna: Yes Nani: Hard to select the message to reply to Mihoko: Yes
This project uses a template called a FORUM Template. Any questions about how it works?	(Collect their general impressions.) Mark: Could be helpful for collaborating Mauna: Hard to understand Index (thinks it's an Index to the whole site); hard to select a message to edit Kayomi: Not clear what a Forum project is for -- when would you use it? Nani: Likes having folder (Tabs) up front Mihoko: Once you get used to it, it's easy, but need training (need help for students too)

 

Logging out of the YF

Script: You've set up your project. Now you're ready to leave the YF site.

How would you exit the site?

(Do they click HOME, then LOG OUT?) Mark: Searched for LOG OUT on current page; didn't think to go back to home page Mauna: Couldn't find it; finally went to home page Kayomi: Missing LOG OUT on each page Nani: Yes, Yes Dandy: Not obvious, finally figured out to go Home first Mihoko: Confusing

 

Working on a Newsletter type of Project

Script: You now know how to work on a Forum type of project. Let's see how to work with another type of project.

How do you get to the Members Area?	(Do they remember to log in?) Mauna: Yes
Let's review the AVAILABLE PROJECTs. http://youthforum.askasia.org/members/available_projects.pl	(Do they go to Available Projects?) Mauna: Yes Kayomi: Yes Nani: Yes Mihoko: Yes
Sign up for the project [Newsletter] A Day in the Life from the INDEPENDENT Projects list.	(Are they able to use the sign-up form? Do they click OK to get back to home page?) Mauna: Yes Kayomi: Yes Nani: Yes Mihoko: Yes
Go ahead and open the project to start working on it.	(Do they click OK to get back to home page, then click on the project from the PROJECTS: IN PROGRESS list, then click OPEN THIS PROJECT?) Mauna: Yes Nani: Yes Mihoko: Yes

Adding Content to a Newsletter Project

Script: Now, let's add some content to this project.

The project is now open, what do you do to add content?	(Do they click ADD A NEW STORY?) Mauna: Yes Kayomi: Hard to find Nani: Yes Mihoko: Yes
Fill out the ADD NEW STORY form. (Give them some sample text to enter.)	(Are they able to ADD STORY?) Mauna: Yes, but couldn't remember what the project is about since it's not displayed where you enter the new story Nani: Yes Dandy: Not visible, so she exited the page without saving it! Mihoko: Yes
After seeing your story displayed, where do you go next?	(Do they click TABLE OF CONTENTS?) Mauna: Yes (like Table of Contents better than "Index") Kayomi: Goes to main top nav; TABLE OF CONTENTS is not descriptive Nani: Yes, but wasn't quite sure if that was the place to go Mihoko: Yes, but difficult to orient to it
Suppose you need to edit a story? Select the story and edit it.	(Do they click on the title link? When the story opens, do they click EDIT STORY?) Mauna: Yes Nani: Yes, but it was hard it select it Mihoko: Yes
Add a line to the story and save it. (Add an attachment if we can do that.)	(Do they click SAVE CHANGES?) Mauna: Yes Nani: SAVE CHANGES was not visible, so struggled to find it (until she scrolled down) Mihoko: Yes
After seeing your story displayed, go back to Table of Contents.	(Do they click TABLE OF CONTENTS?) Mauna: No Nani: Yes Mihoko: Yes
This project uses a template called a NEWSLETTER Template. Any questions about how it works?	(Collect their general impressions.) Mauna: OK Kayomi: Could use it for a project Nani: Yeah, very nice Mihoko: Need some examples, so hard to imagine

Working on a Gallery type of Project

Script: You now know how to work on Forum and Newsletter types of projects. Let's see how to work with a Gallery Type of project. (Do this while still logged on in the Members Area.)

Let's review the AVAILABLE PROJECTs again. http://youthforum.askasia.org/members/available_projects.pl	(Do they go to Available Projects?) Mauna: Yes
Sign up for the project [Gallery] Holidays We Like from the INDEPENDENT Projects list.	(Are they able to use the sign-up form? Do they click OK to get back to home page?) Mauna: Yes
Go ahead and open the project to start working on it.	(Do they click OK to get back to home page, then click on the project from the PROJECTS: IN PROGRESS list, then click OPEN THIS PROJECT?) Mauna: Yes, but slow to figure it out

Adding Content to a Gallery Project

Script: Now, let's add some content to this project.

The project is now open, what do you do to add content?	(Do they click ADD A NEW SLIDE AFTER THIS?) Mauna: Yes
Fill out the NEW SLIDE form. (Give them some sample text to enter. If possible, do attachments.)	(Are they able to ADD it?) Mauna: Yes
How many slides are now listed?	(Should be 1 after adding first slide.) Mauna: Yes
Add another NEW SLIDE. (Give them some sample text to enter. If possible, do attachments.)	(Do they click ADD A NEW SLIDE AFTER THIS?) Mauna: Yes (Note: we added photo that was .bmp and it didn't display on the page; .gif did though)
Now how many slides are listed?	(Should be 2 after adding second slide.) Mauna: Yes
How can you choose which slide to view?	(Do they use JUMP TO drop-down box or PREVIOUS or NEXT arrows?) Mauna: Yes, first arrow, then Back to Beginning, and finally JUMP TO drop-down
This project uses a template called a VIRTUAL GALLERY Template. Any questions about how it works?	(Collect their general impressions.) Mauna: OK

 

General Comments
 

General questions about the site?

Mark: Does it give a timeline to follow, some type of lesson plan with teaching activities and when to do them? Does it offer suggestions and available resources? Mauna: Students would like to use it; navigation is a little off, but overall the site is good; projects good. One idea is to have one student start a story, then another add to it, and so on. Nani: Pretty easy to use; fun way to work with kids; could have kids put project together then view it on LCD projector Dandy: Likes that you can click links or icons (on navigation); Might say "Switch to Japanese" so that English speakers know what Japanese characters mean. Wants to know the "why" of doing projects. Wants examples that show the benefits -- what would students be able to do? Has a penpal project; very important to set up the criteria for communications in advance. Dandy and Etusko: The navigation icons are not that clear or intuitive (i.e. for Help or Project Showcase). Need to make them more obvious (e.g., ? for help). It would be very helpful to create some curriculum-based pen pal projects (they don't exist now!). Maybe demo on Feb. 28 at WATJ conference at U Prep. Mihoko: For K-1, possibly use more for teacher to prepare projects for students or to scan in their work

Michele's Observations

On Registration form, would be nice to have larger space for email so the entire field is visible when you type and need to review mistakes. On Project Registration form, the Project Description info is not well-labeled and blends in with site information. No one really stopped to read it. On project pages, it would be nice to repeat the Project Description. It seemed to be hard for people to keep in mind what the project prompt was. Translations: Several Japanese speakers said that Japanese used for Project Showcase was "Introduction to Projects" which made it easy to confuse with Orientation; better translation might be "Sample Projects" Japanese for Virtual Gallery is  "Media Museum" -- does that convey the same thing? On Newsletter template, set it up so that the ADD STORY button is above the screen fold. Can you only add one attachment to a Story? We added a picture, but when we added a file, the picture got lost. For Forum projects, maybe add prompt by Subject line about making the Subject line meaningful.