The System is envisioned to use a variety of different voice, visual, automated sensors, machine to machine, and direct tactile inputs and outputs. These are all designed to reduce labor requirements and errors, while improving the test, baselines, ease-of-use, and the overall end user experience. Basically, the System is using technology to replicate (and improve) a person’s vision, hearing, speech, memory, and intelligent correlation.

 

This page covers some of the visual tools and services using cameras. This page is primarily focused on the Interactive Visual and Audible aspects of the System, but it does cover some of the other areas where Visual Tools and Services can be used in System.

 

Some of the high-level Visual Tools and Services include:

• Automated Visual Monitoring and Evaluation (visual call for help, mobility/dexterity, activity, distress, temperature, electronic fence, etc.)

• Visual data capture

• Active and Passive Examinations (human input/output, machine-2-machine, data collection baseline, etc.)

• Medical Analysis

  • Intelligent Analog Interpretation and Logging

  • Baseline Skin

  • Skin Issue Detection

  • Skin Issue Comparison

  • Ears, Nose, and Throat

  • Eyes, Lips, Finger/Toe Nails

  • Facial Recognition

  • Mobility/Dexterity tests

  • Special Medical Imaging Devices

• Visual Checks (Guardian, Call Centers, Friends)

• Barcode/QR Code Services (Identity Services)

• Remote Presence

• Telemedicine

• Robotics and Robotic HealthMate

• Security and Control Systems

• Localization/Internationalization/Translation

•  “Large” Screen Displays (vision tests, message boards, information boards)

• Memory (and memory retention) Tests  - linked to schedule

 

Automated Visual Monitoring and Evaluation

 

Visual Tools and Services related to automated Medical Analysis, will look for identifiable issues either due to something they can directly detect, or detecting changes by comparing current images with previous baseline images. While a lot of potential issues can be identified through Visual Analysis, Visual Analysis is one dimension, and any newly identified items would typically be followed up with questions applicable to the condition through the Intelligent Adaptable Examination Decision Tree (e.g. pain – if so what kind, how much, more at certain times), itch, how long they had it, texture description, color, etc.), and/or with additional testing or taking of samples. For existing known issues, there could be questions as to how it may have changed (similar to the questions if it was a new issue). Detailed baselines are important, and the System creates and stores baselines during examinations.

 

The System will make extensive use of color high resolution cameras, and applications that can provide intelligent image recognition and comparison of the high resolution images.  The focus is on tablets and smartphones that are readily available to everyone, but can also include handheld, fixed location, remotely managed, camera glasses, and special camera setups.

 

Devices can actually see and compare modeled images better than we can. An important aspect of the System is taking baselines and then using them for current and future comparisons. Basically intelligent visual systems can be better eyes combined with detailed past memory.

 

There are a large number of tasks that do not require Doctors or RNs. These tasks can be performed by Medical Assistants, specifically trained individuals, or just plain individuals. Each task will need to be evaluated to determine the capability of breaking down and automating some of the effort.

 

Intelligent Analog Interpretation and Logging

 

This tool is focused on being collect and store input by just looking at it or taking a picture. There are many tests that currently require human visual conformation, followed by human entry of the information. While this is still an option (and can also be extended to verbal input), there are additional capabilities available through intelligent image processing. Wearing camera glasses, the examiner can just look at the action or the result, and it can be automatically captured, interpreted, and recorded, with less chance of error. Examples include mobility, dexterity, reading blood pressure, lung capacity, height, weight, temperature, legacy devices, etc.

 

Baseline Skin Checks

 

There are three methods envisioned to collect and create skin baselines:

1. Use of a handheld camera to take high resolution pictures of each area of a body. The System would provide instructions on where and how to take pictures of each section of the body and the individual would not wear any makeup. Based on the pictures, they would be sewn together into a model that would be stored and can be used in the future to compare to other baseline models. A printable set of reference markers (measured scale and possibly a cross sign) may need to be attached when the picture is taken to help create the baseline model.

2. If there were only certain areas of concern (or patient’s choice), the System can be informed, and it would adjust its instructions to only cover those areas.

3. Specialized camera systems (e.g.  an automated “Airport type standup  Image Scanning Device” and/or a “laydown tanning type  bed” ) could also be used take the images and create a baseline model.

 

Notes:

• Certain areas may require additional special pictures to look for different conditions in cracks, crevices, between toes & fingers, hair, etc.

• Pictures can be encrypted, obscure certain parts for general review (if they are not suspect) as well as stored on a HIPAA certified site as a secure download

• Individuals can choose what areas they want baseline, and could wear clothes to cover any areas they wish 

• Multiple full and partial baselines are maintained for comparison and progression analysis

• IR imaging could also be taken, baseline, analyzed and compared

 

Skin Issue Detection

 

Once a full or partial baseline is captured an automated analysis can be made of all sections looking for a variety of irregularities and then use intelligent image recognition and databases to compare them to different conditions (e.g. irregular moles, rashes, sores, infection, potential pest or fungus, allergy, bite, discoloration on feet, fingers or toes, growths, swelling, changes in texture/dimpling, etc.). Potential issues would be further checked or  additional information collected from the patient (as needed) for things that visual identification cannot check (e.g. pain, texture, changes, how long, etc.).

 

Identified areas of concern would be flagged for further action (e.g. Immediate Doctor review, follow-up examine with specialist, additional images taken and stored, marked as not an issue, flagged for special watch at next exam, and/or immediate interface with a Doctor through telemedicine).  The primary care physician (and any specialists associated with the patients) would be notified with a link to the report as well as the data. Potential issues that had been previously identified as non-issues would be listed as “additional information”.  Previous non-issues that “had changed” would be identified for further review.

 

The person performing the examination and the patient would be provided with recommendations for any actions by the patient or caregiver and/or follow-up in the final report.

 

Notes:

• The person performing the testing may be restricted to certain information or images (TBR), and/or require additional signed permission to view and collect additional information

 

Baseline Model Comparison

 

Baseline Issue Comparison takes the analysis of the model to another level. It compares the current baseline with previous baselines. It is envisioned that the System can detect differences between the current and past baselines to see how they have changed. These include skin, eyes, nose, throat, toes, coloration, growths, weight changes, sores not healing, reoccurring conditions, something thought to be benign but just appeared, etc., basically anything that may have changed that could be of concern.  Where appropriate the System can create time lapse changes via video for further review.

 

Ears, Nose, and Throat (tongue, gums, and cheeks), Lips, Finger/Toe Nails

 

Ears, Nose, and Throat (tongue, gums, and cheeks), Lips, Finger/Toe Nails have their own special models and things that need to be checked for. They have their own health, as well as providing information about the overall health of a person.

 

Certain checks can be made with an ordinary smartphone/tablet camera, but this may be an area that would require a special Bluetooth or Internet enabled camera for more formal examinations.  The camera is envisioned as existing, or it would require very little development. It could cover most of the hard to reach areas, and could be mass produced to lower costs.

 

The actual collection, storage, analysis, rechecks, and actionable follow-ups are basically the same as the skin models and previous baselines, but there are slight changes in the modeling and comparisons.

 

Eyes

 

As with other parts of the body, eyes have their own health, as well as providing information about the overall health of a person. Eyes were separated out from some of the other areas, because they can have many different temporary and longer term conditions, which need to be factored in (redness, tired, shape, color, clarity, puffiness, etc.). Most of these can be detected with regular smartphone/tablet or smartphone glasses camera. Additional capabilities could use a special Bluetooth camera for more formal examinations. A simple example could be the detection of pinkeye directly by the individual or a telemedicine “Ask a Nurse” App.

 

Finger and Toenail Analysis

 

Changes in nails can point to Health Issues and they can be important indicators of diseases and other health problems throughout the body. It is important to examine nails as well as baseline visual information about an individual’s nails to check for changes.

 

These include general color (e.g. yellow, blue, white/grainy), specific colors at different parts of the nail (e.g. red strip at the top of a nail bed, red/brown splinter lines under the nails), shape (e.g. curving to look like an upside spoon), angle with the cuticle, etc. These different conditions can be possible indicators (or early indicators) for things like psoriasis, eczema, vitamin deficiencies, cirrhosis of the liver, congestive heart failure, hyperthyroidism, malnutrition, diabetes, HIV, silver poisoning, rheumatoid arthritis, peptic ulcer, and many others.

 

Having a quick picture of the nails that can be taken virtually anywhere and automatically analyzed and compared against other collected data on the patient, could be very important in helping to identify any potential tests, or examinations that might be prudent.

 

Facial Recognition

 

Facial recognition can tell a lot about a person’s health. It can detect slight changes that could be associated with pain or illness anywhere in the body. These can be used in general residence monitoring, initial general evaluation, and during directed mobility and dexterity testing to look for pain or discomfort.

 

Mobility/ Dexterity/Activity

 

It is envisioned that the System can detect many changes and issues in the area of mobility and dexterity.

General Home Monitoring can be performed by fixed cameras, robotic health companions, or motion sensors to identify any changes in mobility (speed, posture). The System can measure the amount of time the person is active in a day (and factor out any in-home assistance).

With cameras (enabled remote viewable or not), General Home Monitoring can identify trend on speed taken for certain common tasks (walking across a room, going to the bathroom,  sitting down, standing up, getting out of bed).

Dexterity may or may not be able to be accomplished in a regular home environment, either through speed and range of motion, but also facial recognition to identify any distress that may occur during the movement. Time of day can also be trended to check based on time of day or if they had just had physical therapy.

 

Clinic or Specific Home Directed Testing can include some of the normal tasks that a person would do or special motions that they should make, and can uses cameras to determine baselines and changes in baselines. These can include walking across a room, standing up, sitting down, lifting an arm, reaching behind their back, bending over, etc.  These can be timed and/or measured automatically by intelligent image recognition, and use facial recognition to help determine any pain or discomfort levels.

 

There are many side benefits

• alerts (not leaving the bathroom, lack of mobility, not getting up, reduced activity, pain/discomfort))

• remote visual monitoring of the individual (based on rules and captured with audit trails)

 

Televiewing Visual Checks (Guardian, Call Centers, community groups, Friends)

 

Video visual checks can be very valuable tools in determining the continued and general health of individuals. We do this daily with people we interact with, and with existing two-way video calls, it can be used by guardians, friends, call centers, and community groups to check up on people.

 

The changes that the system is proposing is to make these schedule events, and can include instructions as to things that need to be gone over for their schedule/tasks,  check on certain issues, and possibly some simple questions to determine their general mental and physical health. This feature should help a variety of groups to be able to better support individuals they are monitoring and/or caring for.

 

Security Monitoring

 

Part of a person’s “whole health” is feeling secure and knowing that you can be found and taken care of in an emergency.  Cameras can be used as part of a security system that can be used alone and/or in conjunction with motion detectors, and scheduling information for expected events.  Security cameras in a residence or facility, when coupled with intelligent visual recognition, can perform many functions:

• Discourage and protect against predators

• Electronic Fence to notify if an individual has gone into or left an area they shouldn’t have

• Make sure the individual is moving within a prescribed time at different times of the day

• Make sure the person if taking certain medicines or treatments

• Monitor a person’s general health (mobility, dexterity, facial recognition, and IR monitoring)

• Special motions that would trigger a call for help

• Use in checking in on a person and telemedicine

 

Notes:

• For privacy reasons, security cameras don’t necessarily have to be monitored or all of the information stored.  Devices and intelligent visual recognition software can perform the monitoring, analysis, storage, and alerts as needed.  Any access to monitoring or stored data would result in an auditable event.

• If an emergency does exist (no response or movement), escalations and notifications can be triggered, access granted, and an audit trail of the action recorded.

 

Barcode/QR Code Services (Identity Services)

 

Identity Services are the fundamental backbone of being able to link items/people to information. It is envisioned that this is accomplished through visible or non-visible stickers, wrist bands, monitor displays, printed paper, and short and long range RFID. This section specifically covers the visual machine reading of identity information.

• General Identity: There are two parallel implementations of Identity Services. One is a public system that allows the general public to see “non-trusted” information that the person wants to share. The other one is based on identification that institutions and organizations issue and they control the data  (much like current hospital bands)

  • Public System: Simple scans using a smartphone, tablet, or PC camera, and would also support any network connected camera or scanner

  • Private System: Same Scanning capability, but tailored/optimized to handle active use of the linked relationship  in a hospital, examine, or other higher volume instructional use

• Prescription/Treatment Verification: Simple application that can scan a treatment or medicine and the person, to see when it was done last, when it is scheduled, verifies doctor and person, dosage, picture of the person, and creates an event log of the person who provides it.  It can be linked to the schedule application to provide reminders and create escalations as necessary.

• Data/Device Linkage: These are applications to link patients, devices, items, and data. 

  • Allows someone to scan the person and a device to link the collected data between the patient, the device, and the control point to collect and store the data (long or short term monitoring/test with a person)

  • Allows a group of devices to be associated with a person  (emergency room, general examination)

  • Allows the collection of data and then associate it with a person or other object

• Audit: All events on person, place, or thing create an event that can be searched through a permission based application to create a number of audit trails, as well as performance metrics

• Facility Management: Equipment, Rooms, Floors, and Buildings can all be monitored and scheduled for actions. By providing visual scanning of each object and anyone performing tasks on the object (local or remote) verifies and closes actions, creates event/audit trails, and creates escalations as needed.

• Public Message Boards: This is a simple application to visually associate a person (QR Code and/or RFID) with a public message board to receive and send messages, as well as provide general updates on their condition (typically used for Homeless or Disaster Relief)

 

Notes:

• The public system has a private side for additional information and controls that are only accessible to the individual and approved administrators of their account, plus shared data (e.g. selected monitored data, Logs, Personal Services, Technical Support, Alerts, etc.) for maintaining their individual health)

• Certain controlled information can be shared between the Public System and Private System

• Data Access is well defined and controllable. They are all just objects and the can be linked, unlinked, and transferred in many different combinations

 

Special Imaging Devices

 

This page covered some special imaging devices for obtaining images associated with general examinations. There are many other “imaging devices” (X-Rays, MRI, foot pressure, bone density, conductivity, etc.) that exist and have their own image analysis. It is envisioned that over time, the system could provide tools to help manage that equipment, have direct data collection, and provide analysis against current conditions and past baselines, reporting, and alerts… using some common tools (described above) that would be available to developers.

 

Localization/Internationalization/Translation

 

One of the important and critical aspects of the System is to provide the interface to the System and all of the Components in the individual’s native language, and to the extent possible adjust for any cultural and regional differences.  This is to increase the use of the System to other areas, plus allow less trained individuals to perform more tasks.

 

The System is envisioned to support:

• Many different languages through the use of Objected Oriented Language Support which provides different sets of visual and audio objects depending on a configuration parameter to provide the interface in the caregivers native language

• Support for real-time cross-language translation of verbal to text, text to verbal, and text to text

• The support of Abstract Interfaces to support certain cultural and regional differences

 

Remote Presence/Robotics

 

Remote Presence is verbal, visual, and mobile. It can perform a wide variety of tasks either as being connected to a remote human or acting autonomously. Visually it can do pretty much anything a human can do, and with mobility, it can move to almost any area and perform interaction, monitoring, test, and other tasks.

 

Telemedicine <                    >

 

In a simple form, and readily available, telemedicine is as close as your smartphone. It allows you to look things up on the Internet, access intelligent examination instructions, or cect to an health professional. . If  connected, you can have two way communications with a Doctor or other Health Professional, and provide them visual access to the patient through the phone, transfer any data from connected devices, and have access to additional local or online information.

 

From here there are many current, emerging, and envisioned technologies that greatly improve these capabilities.

• Have a subscription service (or part of insurance or network  “24 Hour Talk to a Nurse”) to provide telemedicine into your home

• Use of any combination of two or more smartphones, tablets, heads up display glasses, or Monitor/TVs to provide a Doctor or Health Provider access to more data and better viewing of a patient from anywhere

Telemedicine has many benefits

• Direct and as needed support to rural, mobile, relief, etc. clinics anywhere in the world (paid and volunteer)

• Remote On-Call Telemedicine support for Hospitals and Clinics to handle overflows (on-call Doctors can work from a designated support site, their homes, their offices, or practically anywhere they have a network connection. This also supports collaboration between hospitals and clinics that could support each other in an over flow condition

• Emergency Care

• Pre/Post-Op Consultations

• Quick checkups or check for a specific issues

• Remote check on a patient on a schedule or needed

 

Memory (and memory retention) Tests 

 

Memory needs to be reinforced, as well as tested for several different ailments that can affect us. Some can be commercial- off-the-shelf, while others can and should be customized for thee individual using sight and sound. These can be automated or manually initiated tests used in everyday life, and/or part of a regularly scheduled examination or “remote check-in”.

 

The System is envisioned to support:

• Two-way voice interface to the tests

• Customizable questions, sound, pictures, and/or video (created through a APP, probably by a guardian) with ability to try to help or escalate for follow-up

• Customizable memory reinforcement

• Standard Memory retention tests

• Memory strengthening puzzles and exercises

 

Note: A Guardian or Call Center could check in with the individual and either have the individual go through one with them