Need on-screen keyboard with mouse functions

A home for all kinds of Puppy related projects
Post Reply
Message
Author
standard7452
Posts: 20
Joined: Mon 25 Feb 2013, 23:11

Need on-screen keyboard with mouse functions

#1 Post by standard7452 »

Hello, my name is Will and apologize in advance if this is not in the right part of the forum.

I pored over this to try to make it as concise as possible, as I do not want anyone to tl;dr it.

I am looking for help with crafting a solution for a few friends and people I volunteer my time for who have no use of their arms and legs.

These folks are in wheelchairs and interact with their environment on a "single switch" basis, which means they can use their heads or chins to press a button.

Solutions exist for Windows that allow for them to interact with the computer, but the software and hardware costs are astronomical.

My goal is to give people who can only access the computer with a single switch (quadriplegics/people with cerebral palsy/motor control skills) a computer of their own that they can take with them, so that they can communicate with their friends and family both online, and when they go home via the PC.

Sounds like I am dreaming I know, but after 3 years of working with these folks, I have determined what I need to help them.

I need an on screen keyboard that also has the ability to manipulate the mouse.

I hate to ask for help, but I really need it. I am not a programmer, but I am willing to learn whatever I need to.

To make it short, I need an onscreen keyboard (like onboard) that also has mouse functions (like GOK).

I have found the xdotool command to move the mouse appropriately, but I have no idea how to integrate everything together.

I am probably stupid, and someone could probably look at this and give me a 5 minute solution.

I just don't know where to start or who to ask.

Thanks for your time.

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

#2 Post by greengeek »

Would they have the ability/dexterity to distinguish between a short press and long press of the button?

User avatar
Ted Dog
Posts: 3965
Joined: Wed 14 Sep 2005, 02:35
Location: Heart of Texas

Can they talk?

#3 Post by Ted Dog »

There are speech controlled software for computers, saw some pets I think MU was working on.. Also would be interested in helping you if special drivers or complex open source compiling is needed. As a person with a slow form of Muscular Dystrophy myself, I'm constantly displeased with the extreme high cost and lack of quality hardware. There is a brand of joystick (thrust master i think) that can be programed as a keyboard and mouse and cost hundreds of dollars less.
I'm thinking a keyboard overlay with a hot zone for switching between mousemode(as a joystick) and keyboard mode, should be due-able. There are on screen keyboards that come with http://www.mythbuntu.org/ by default try that out and see if that type of onscreen keyboard would work,

User avatar
Ted Dog
Posts: 3965
Joined: Wed 14 Sep 2005, 02:35
Location: Heart of Texas

GOK with percise puppy

#4 Post by Ted Dog »

GOK is availble for Ubuntu apt-get GOK, so it would be available for precise-puppy new version is in Beta testing phase. Also, since I'm in the MDA program already, if you would give me models/controls you already work with I could get the same, if available, from my local support network.

User avatar
mikeslr
Posts: 3890
Joined: Mon 16 Jun 2008, 21:20
Location: 500 seconds from Sol

Onboard at least can be created for one Pup, maybe others

#5 Post by mikeslr »

Hi stardard7452,

Hoping to attract more attention, I posted on the "users" subforum: http://murga-linux.com/puppy/viewtopic. ... 498#689498. And got to thinking, and tinkering. So I know that Onboard, at least, can be built to run on LxPup-aptget-test. See the following link for the process for creating an SFS --a squash file you load/unload rather than install. http://murga-linux.com/puppy/viewtopic. ... 611#689611

I also provide instructions for creating (I hope) an installable pet. My guess is that it should work on any precise-pups to which python has been installed. First install python and then install an onboard pet or load an onboard sfs which will overwrite any conflicting python libraries. Also, now that we know what debs are necessary, once python is installed, it should be possible to download the Maverick version of those debs and create a pet or sfs which would run under Lupu/Lucids. But see below, and note that precise-pangolin pets/sfses are not always backward compatible: won't run on Lupu/Lucid --different libraries were used.

Regarding gok, I'll take a stab at it, but hope someone who actually knows what he's doing will also. I couldn't find a listing for gok for Ubuntu pangolin, only for Ubuntu Maverick. While I mention that my experience is that apps built for Lupu/Lucid --which correspond more or less to Maverick-- will almost alway run in a precise-pup, apps built for Maverick don't always run under Lupu/Lucids. Only about 2/3 the binaries used in building Lupu/Lucid were from Ubuntu. And sometimes, even though something may not appear on the dependency list of a Ubuntu app, such lists presumes certain base files built into Ubuntu which may not have been included in pups: i.e., Ubuntu presumes a full gnome environment, which Pups don't use because of its bloat. But one never knows until one tries.

Good luck on your project. Don't hesitate to email me if the instructions on the above post need further explanation.

mikesLr

User avatar
8-bit
Posts: 3406
Joined: Wed 04 Apr 2007, 03:37
Location: Oregon

#6 Post by 8-bit »

See my post here for a link to another post containing an on screen keyboard PET for Puppy.

User avatar
mikeslr
Posts: 3890
Joined: Mon 16 Jun 2008, 21:20
Location: 500 seconds from Sol

Please continue discussions HERE

#7 Post by mikeslr »

Hi All,

Nice work 8-bit. Hope it suites standard7452's needs, at least in part.

I found Ubuntu's repo listing for gok under pangolin. https://launchpad.net/ubuntu/precise/i3 ... 0-1ubuntu1 Suggest we concentrate on it as it was what standard7452 actually said he needed.

Sorry to report Onboard wasn't as successful as I hoped. Although it works fine under jejy69's lxpup-aptget-test, the SFS created under it did not run in pemasu's upup372 even though it had python installed. Starting it from a terminal at /usr/bin indicated that "gi.repository" could not be found. Googling revealed that gi.repository was something like --sorry it was late-- an experimental now being combined under something else. Pfind "python" in lxpup...test showed over 500 files> too many to manually copy into a package for use in other Pups. Typing python in synaptic generated a list which ran off the page, maybe thirty. Doable, but the description of many on the list suggests they wouldn't be needed or duplicate others on the list. Really needs someone who understands python.

mikesLr

User avatar
Ted Dog
Posts: 3965
Joined: Wed 14 Sep 2005, 02:35
Location: Heart of Texas

#8 Post by Ted Dog »

I fell asleep getting the mouse camera sofware to work,

Python was never a strong point since I never valued its appeal as yet another silly hack crutch that adds bloat on bloat.

But I have the source code and the dependencies are thankfully less than ten. Gnome is not even required, just gtk. so this is hopeful together with 8bits onsreen keyboard would make puppy more appealing to disabled folks.

User avatar
Keef
Posts: 987
Joined: Thu 20 Dec 2007, 22:12
Location: Staffordshire

#9 Post by Keef »

Another onscreen keyboard option is WriteAid
It needs Java, but I've tried it with Shinobar's JRE sfs (jre-1.7.0.09.sfs) and it is working well in Wary.
Also has a 'rolling' mode, so if someone has a problem using the mouse, it will scroll through all the letters, and they can be selected with any key press. The speed can be altered.
The selected text appears in the bottom window, but is not very big though - no option to increase the size.
Attachments
WriteAid.jpg
(18.71 KiB) Downloaded 3749 times

User avatar
mikeslr
Posts: 3890
Joined: Mon 16 Jun 2008, 21:20
Location: 500 seconds from Sol

Xdotools to simulate mouse

#10 Post by mikeslr »

Hi standard7452 and All;

Spent a couple of days trying to jury-rig gok into Puppy's environment. I can't do it, and I suspect it can't be done after having built both an SFS using pangoln required debs and another using the maverick required debs. When both sfses loaded but did not run, I typed "gok" in a terminal at /usr/bin --where its executable is located-- and received a report of a conflict in gtk's. I think ubuntu switched over to gtk+ 3, while Puppy is still using gtk +2.

With a couple of possibilities having been suggested to provide an onscreen keyboard, I wondered if there was a way to simulate mouse movements using such keyboard. Googling I discovered xdotools. From what I gather, xdotools uses X itself. So it should be, more or less, distro independent. To install in jejy69's precise-apttest required download of only two debs. The following is the link to Ubuntu pangolin's repo.
http://packages.ubuntu.com/precise/xdotool
According to above, libc6 is required. But Puppy already has it so I'd first test for compatibility before installing ubuntu's, especially as ubuntu's libc6 itself has dependenceis, which may have dependencies and so on until you end up with having built Ubuntu itself.
More information and download link can be found here:
http://ashu-geek.blogspot.com/2012/01/h ... ol-on.html
I couldn't figure out how to configure it, but then I never use hot-keys or similar. But the following is documentation for it:

http://www.semicomplete.com/projects/xd ... tool.xhtml

And apparently there's a gui making it easy to configure.

http://xdotool-gui.sourceforge.net/

I regret that I could not figure out how to upload the debs to the forum.

As muggins's virtual keyboard, http://www.murga-linux.com/puppy/viewto ... 712#261712 suggested by 8-bit also is for the X environment (ergo distro independent) and provides a keyboard with numeric keypad, function keys, and special keys, I suggest an experiment of a combination of it and xdotool. Again, my apologies for not having the expertise to conduct such experiment.


mikesLr

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

#11 Post by greengeek »

I got the impression from standard's post that the user would not be able to make keyboard taps so I think that means xdotool may not be a usable option. I think that the "rolling mode" keef mentions would probably be a pre-requisite if the user can only press a separate single button.

If they could use a joystick there would be lots more possibilities but I got the feeling that's not possible. I hope we do hear back from standard as this seems like a very valuable project. I'm still wondering if the user would be able to control "press duration" of the button as that might open some other options.

User avatar
mikeslr
Posts: 3890
Joined: Mon 16 Jun 2008, 21:20
Location: 500 seconds from Sol

Explanation of Xdotools employment

#12 Post by mikeslr »

Hi greengeek,

You may be right that physical limitations could inhibit the use of xdotool's full capabilities. But what I had in mind was PRE-CONFIGURED pet assigning to xvkbd's Function and other special keys mouse movements generated by arguments assigned through xdotools. Perhaps not perfect, and certainly looses xdotool's flexible customization. But, if possible, and someone were to built such pet, it would thereafter be easy to deploy, and anyone limited to single switch capabilities would only have to learn which special keys produced which mouse movements. Additionally, with the wealth of special keys available via xvkbd's GUI, the need for mouse movements could be limited. For example, a key to expand the menu, menu scroll keys up, down, left & right, and perhaps keys to place the cursor at the start of the address box of a browser, and the start of the search box of a search engine.
Actually, I just realized that the last two suggestions aren't necessary. I've gotten so used to relying on a mouse that I forgot --just tested-- that at least in opera and firefox the tab key will move thru webpage elements. Perhaps other "Standard" applications also "retain" such tab functionality.

By the way, unless I'm misrecollecting or misunderstood, GOK is structured to use 2 mice so may, in fact, present problems for those limited to single switch capabilities.

mikesLr

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

#13 Post by greengeek »

I'm thinking something like this Magoo pet:
http://www.murga-linux.com/puppy/viewto ... 558#668558
combined with WriteAid in rolling mode might somehow be configurable to offer reasonable access to basic functionality through a single switch.

I'm trying to get java installed so I can try WriteAid, buy not having much success yet. (I usually prefer to stay away from java due to security issues but this seems like a worthy project...)

I guess the basic screen would have to scroll through the Magoo tabs (allowing the user to click on the preferred application to open), then WriteAid (or similar) would have to scroll through the tabs within the app, and bring up a keyboard for scrolling through characters to be entered. I wonder if there is an onscreen keyboard that scrolls through an app list as well as keyboard characters?

I wonder if there is some way to set up xdotool so that it keeps scrolling through the Magoo tabs as soon as the PC powers up. Then maybe a single click of a button (grafted into mouse button 1) would activate the chosen tab as it became "scroll focused".

(question: would xdotool interpret WriteAid key depressions the same way it would interpret actual keyboard depressions? eg: if so then xdotool could be configured to interpret some of the WriteAid onscreen characters ("non A-Z" and "non numeric") as up/down/R/L mouse movements)
Last edited by greengeek on Sun 10 Mar 2013, 01:03, edited 3 times in total.

User avatar
8-bit
Posts: 3406
Joined: Wed 04 Apr 2007, 03:37
Location: Oregon

#14 Post by 8-bit »

It would be nice if stardard7452 checked back in so we would know our efforts are being seen and this is just not a futile effort.

User avatar
mikeslr
Posts: 3890
Joined: Mon 16 Jun 2008, 21:20
Location: 500 seconds from Sol

Sent email to xdotools group

#15 Post by mikeslr »

Hi All,

Patience 8-bit. Sad to say but some people have a real life which from time to time interferes with Puppy activity.

greengeek raised a good point, especially as I have been making suggestions without solid experiential support. So I've emailed the xdotools usergroup the following:

"Hello everyone.

First off, I'm way over my head but have gotten involved in a worthwhile project: enabling those with cerebral palsy, quadriplegics or otherwise limited to "single switch" mobility to use computers. The project was suggested by Will, known on the Puppy Linux Forum as standard7452. The thread he initiated, currently only 11 posts, can be quickly read here: http://murga-linux.com/puppy/viewtopic.php?t=84617. There are a couple more posts on a thread I opened hoping to attract more technically astute users of Puppy Linux. Although we have identified several onscreen keyboards which could be employed, emulation of mouse functions or a work-around remain elusive. I had suggested that perhaps xdotools might provide such functionality, http://murga-linux.com/puppy/viewtopic. ... 708#690708, in combination with Xvkbd: Virtual keyboard, http://www.murga-linux.com/puppy/viewto ... 712#261712 but greengeek --suggesting a different approach-- asked a question pertinent to both: whether xdotools would interpret the "pressing" of onscreen keys the same way as actual keyboard key depressions?
That prompted this inquiry to people who may know.
Please feel free to join the discussion on the Puppy Forum (registration required --everyone is welcome, but especially those with technical knowledge) or to respond to this email if for no other reason than to save us some time attempting the impossible. If you have alternative suggestions they would be welcomed as well.

Thanking you in advance for your courtesy and cooperation,

mike kessler"

mikesLr

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

#16 Post by greengeek »

Thats great - it would be useful to have someone familiar with xdotool to offer some advice.

I just had a tinker with xvkbd - it's quite good. I think it could be very useful, especially if it was also possible to write a program / script that is capable of moving the mouse pointer across the entire screen in a row by row scanning motion (over the top of any/all open windows).

If the mouse pointer could be made to scroll slowly across the top line of the screen, then step down 5mm and scroll back, then step down 5mm and scroll across again etc etc it would eventually cover the entire screen and allow a mouseclick to access all functions that xvkbd (or it's focused window) was presenting. (Very slow of course, but I'm just looking for a "proof of concept" idea at the moment. Something might be better than nothing...)

EDIT: xdotool does not seem to respond to WriteAid character depressions, but does respond to xvkbd character depressions. So, if a way could be found to automatically scroll through the xvkbd keys onscreen, it would be possible to program some of them to do special functions like File, save, file open, move mouse etc etc.

standard7452
Posts: 20
Joined: Mon 25 Feb 2013, 23:11

Delay

#17 Post by standard7452 »

Sorry for the delay in responding....I was dumb and didn't subscribe to my own thread, and I have 2 8 month old twins who have had the stomach bug.

The scanning function of gok/onboard allow the user to use a single action input to start "scanning" and logically determine which column, row, button they need.

So imagine the following....a person who has no control of arms/legs, and only a single range of motion in their neck. What we do now is put a spst momentary switch, let's say under their chin.

Currently, the only way to make the switch work now (in windows) is to buy a usb interface with a driver to recognize the switch as an input device.

What I did, was disassembled the keyboard to find the ic that runs let's say the pause/break key...a key that I doubt would come into conflict with normal internet surfing or email and connect the switch across the appropriate terminals to make essentially a "one button Keyboard"

Currently the interfaces are like $200, so for the price of a junk keyboard the solution works.

Now, I need that key to start the scanning process in something like GOK or Onboard.

The fundamental problem I found with GOK is that the original developers assigned all of the scanning functionality to any one of the mouse buttons.

This creates a conflict with the pointer, because every time you click to make it scan it also clicks wherever the pointer is.

Onboard appears to let you select another keyboard key as the scanning activator, but does not have mouse controls.

Its hard to imagine but I could post a video of someone using a very expensive paid application to show what I am talking about.

Thank you for your response, and sorry for the delay.

Will

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

scanning

#18 Post by greengeek »

standard7452 wrote:Now, I need that key to start the scanning process in something like GOK or Onboard.
Would it be problematic if the scanning started automatically as soon as the PC booted?

Also - does this system have to be designed so that the single-switch user is the only person available to start the required processes, or is it likely that there will be another person available to boot and commence the necessary programmes?

standard7452
Posts: 20
Joined: Mon 25 Feb 2013, 23:11

#19 Post by standard7452 »

Everyone thank you for your help.

I appreciate your time and patience while I try to explain this right because you guys are so close to solving this problem.

Its just so difficult to fully explain in one shot.

Onboard Onscreen Keyboard provides an option so that a keyboard key can be used to activate a scan of the keyboard layout. Here is a list of what happens for this:

1. The user presses a predefined keyboard button on the physical keyboard once and lets off.
2. The entire first row of the onscreen keyboard is selected so lets say the number row (` through =), meaning that if the user wants a key in that row they need to press the predefined button a second time.
3. From there, each key is selected from left to right. The user must press the predefined key again to make their final selection.
4. The process starts over for the next character or function the user needs.

With Onboard this part already exists, but there is no way to use the mouse.

Now think of "Mouse Keys" for windows or the use numberpad to control the mouse in Ubuntu.

if you hold the 8, the mouse cursor moves up relative to its position, 4 goes left, 6 goes right and 2 goes down.

if I could add six buttons to any onscreen keyboard that has the scanning feature mentioned above that ran a script for xdotool to move the mouse in a direction until the user hits the predefined physical keyboard button again. and also have a button for right, left and doubleclicks that would be it.

so like buttons on xvkbd that ran something like xdotool mousemove 0,5 on a loop until the user hit the predefined keyboard button.

hope this makes sense, if not I can make a video to explain it.

sorry and thanks!

Will Best

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

#20 Post by greengeek »

standard7452 wrote:if I could add six buttons to any onscreen keyboard that has the scanning feature mentioned above that ran a script for xdotool to move the mouse in a direction until the user hits the predefined physical keyboard button again. and also have a button for right, left and doubleclicks that would be it.
So is this similar to what you mean?

Programmatically assign the following keys with the attached functions as shown:
F4 = "slowly scroll left till user hits switch"
F5 = "slowly scroll up till user hits switch"
F6 = "slowly scroll down till user hits switch"
F7 = "slowly scroll right till user hits switch"

Post Reply