Using Image Processing Techniques (Face Detection, Face Recognition) in SAP UI5 Applications


I would like to show you some of Image Processing Techniques that you can consume in your SAP UI5 Apps , we are going to Build SAP UI5 App detecting all Cams that connecting to your PC and using two Techniques Face Detection and Face Recognition via Face-Api.js Lib on Live Captured Digital Images.

The face-api.js JavaScript module implements convolutional neural networks to solve for face detection and recognition of faces and face landmarks. The face-api.js created by Vincent Mühler.

Face detection is a computer technology being used in a variety of applications that identifies human faces in digital images, Face Recognition is a technology capable of identifying or verifying a person from a digital image.

As we know that there is no Camera Control in SAPUI5 Lib so let’s firstly create one Custom Control for that like this Camera blog post but with some different modifications to be applicable for Face Detection and Face Recognition Techniques.

Step1 :-  

Add folder openui5 to your webapp folder with the files from Source Files  to your Custom UI5 App and declare the resource root path in Descriptor File (Manifest).

	"resourceRoots": {
			"": "./openui5/camera/"

Step2 :-

In Camera Renderer File, we will do some modification as we mentioned later to be applicable for Face Detection and Face Recognition by adding two Canvas Like the Below Code.

	          function() {
	              "use strict";

	               * @namespace
	              var CameraRenderer = {};

	               * Renders the HTML for the control, using the provided {@link sap.ui.core.RenderManager}.
	               * @param {sap.ui.core.RenderManager} oRm RenderManager object
	               * @param {sap.ui.core.Control} oControl An object representation of the control that will be rendered
	              CameraRenderer.render = function(oRm, oControl) {
	              	 var arr = [];
	              	 arr = oControl.getId().split("camera");
	              	 if(arr.length > 0)
	              	  oRm.write("<div class='blockcam'");
                      oRm.write("<div style='position: relative; flex-direction: row; align-items: center; justify-content: space-around;'>");
                      // Video of the Cam
                          "<video id='video_control" + arr[1] +"' width='%w' height='%h' style='width: %pwpx; height: %phpx; position: absolute;' onplay='onplay'></video>"
                              .replace("%w", oControl.getVideoWidth())
                              .replace("%h", oControl.getVideoHeight())
                              .replace("%pw", oControl.getWidth())
                              .replace("%ph", oControl.getHeight())
                      // Draw the Rectangle around the face on Camera Control
                          "<canvas id='overlay" + arr[1] +"' width='%w' height='%h' style=' width: %pwpx; height: %phpx; position: absolute;'></canvas>"
                              .replace("%w", oControl.getVideoWidth())
                              .replace("%h", oControl.getVideoHeight())
                              .replace("%pw", oControl.getWidth())
                              .replace("%ph", oControl.getHeight()));
                        // Responsible for save the detected face to make Recognition Process.    
                          "<canvas id='temp" + arr[1] +"' width='%w' height='%h' hidden='true' style=' width: %pwpx; height: %phpx; position: absolute;'></canvas>");        

	              return CameraRenderer;

              }, /* bExport= */ true);

Step3 :-

In Camera.js File ,we will do some modification as well by adding new Property for Camss Device Id and Event for Face Detection.

	var oCamera = Control.extend("", {
			 * Control API
			metadata: {
				properties: {

					 * Width of the preview window in pixels
					"width": {
						type: "string",
						defaultValue: "448"

					 * Height of the preview window in pixels
					"height": {
						type: "string",
						defaultValue: "300"

					 * Width of the video capture window in pixels
					"videoWidth": {
						type: "string",
						defaultValue: "448"

					 * Height of the video capture window in pixels
					"videoHeight": {
						type: "string",
						defaultValue: "300"
					 * Camera ID (Connecting to PC)
					"deviceid": {
						type: "string"
				events: {
					 * Face Decetion Event.
					 * Responsible to Fire the Face Detection Process
					"facedetect": {}

Fire facedetect Event during Video Play for Camera at onAfterRendering Function.

	onAfterRendering: function(Oevent) {
				var that = this;
				var oVideo = this._getVideo();
				if (oVideo && !this._displayingVideo) {
					// set the camera stream on the canvas.
					// Ask the user for camera access.
							video: {
								facingMode: "environment",
								width: 448,
								height: 300,
						    deviceId: { exact: that.getDeviceid()}
							}, // Back camera
							audio: false
						.then(function(stream) {
							// We have a camera. Let's store the stream for later use
							that._stream = stream;
							oVideo.srcObject = stream;;
							that._displayingVideo = true;
							$('video').bind('play', function(e) {
							    // Fire Face Detection Event.


						.catch(function(err) {"Problems accessing the camera: " + err);

Step4 :-

Now we are ready to add the new custom control to our View and detect all Camss available.

	onInit: function(oControl) {
			// that refer to this.
			that = this;
			// Json Model refer to Faces Detected
			var facemodel = new JSONModel({
				"FacesSet": []
			var list = that.byId("facelist");
			list.setModel(facemodel, "faces");
			// Cameras Array 
			camss = [];
			// Get All Camss that connected to your PC
			navigator.mediaDevices.enumerateDevices().then(function(cameras) {
				var index = 1;
				for (var i in cameras) {
					if (cameras[i].kind === "videoinput") {
							"deviceid": cameras[i].deviceId,
							"label": cameras[i].label
                             // Add Custom Camera Control for each Cam detected
                             // camss_contents is Grid Control 
							new Camera({
								id: "camera" + index,
								width: "auto",
								height: "auto",
								deviceid: cameras[i].deviceId,
								label: cameras[i].label,
								class: "camera",
								facedetect: function(oEvent) {


Step5 :- 

Add New Folder (LibFace) to your webapp Folder  and copy the files from Source Files

,Declare Face-api Lib via your component File.

], function(UIComponent, Device, models, faceapi ) {

Step6 :-

Add New Folder (Models) to your webapp Folder to add Face-Api Lib Models by copy the below files from Source Files.

Step7 :- 

Consume Face-api Lib functions with Live Capture digital images for Camss via FaceDetect Event.

	onFacedetect: function(Control) {
			// Load Model Face Recognition.
			faceapi.loadFaceRecognitionModel('models/').then(function() {
				// Load Model Face Detection.
				faceapi.loadMtcnnModel('models/').then(function() {
					// Draw the Face Detection Border in Canvas of Camera


Step8 :-

In Calldrawing function,we will detect all Faces , getting the landmarks for them and compare them with my Face Photo in Grayscale Mode.

	calldrawing: function(Control) {
			var arr = [];
			arr = Control.getId().split("camera");
			if (arr.length > 0) {
				// Overlay is canvas that we added to the custom Camera control
				var canvas = document.getElementById("overlay" + arr[1]);
				var context = canvas.getContext('2d');
				context.clearRect(0, 0, canvas.width, canvas.height);
				var minConfidence = 0.8;
				var maxResults = 10;
                // Video Control is the Video tag for the Camera
				var myImg = document.getElementById("video_control" + arr[1]);
				var forwardParams = {
					scaleSteps: [0.4, 0.2, 0.1, 0.05]
				// Detect All Faces in the Camera
				var results = faceapi.mtcnn(myImg, forwardParams);
				var minConfidence = 0.9;
				// defs is Array for all faces in video
				results.then(function(defs) {

					for (var i = 0; i < defs.length; i++) {
						var faceDetection = defs[i].faceDetection;
						var faceLandmarks = defs[i].faceLandmarks;
						// ignore results with low confidence score
						if (faceDetection.score < minConfidence) {
						// Draw the Faces in Canvas
						faceapi.drawDetection('overlay' + arr[1], faceDetection);
						faceapi.drawLandmarks('overlay' + arr[1], faceLandmarks);
						var oCanvas = document.getElementById('temp' + arr[1]);
						var context = oCanvas.getContext('2d');
						var oVideo = document.getElementById('video_control' + arr[1]);
						context.clearRect(0, 0, oCanvas.width, oCanvas.height);
						oCanvas.width = faceDetection._box._width;
						oCanvas.height = faceDetection._box._height;
						context.drawImage(oVideo, faceDetection._box.x, faceDetection._box.y, faceDetection._box._width, faceDetection._box._height, 0,
							0, faceDetection._box._width, faceDetection._box._height);
						// Convert the Face Detected to GrayScale	
						// display the face that captured
						//add to face list
						var list = that.byId("facelist");
						var facemodel = list.getModel("faces");

					//================Compute the Distance between the two faces ==============================
					var result1;
					// Getting the landmarks for each Image
					// My Face
					var descriptor1 = faceapi.computeFaceDescriptor(that.byId("image1").getId()); 
					// Saved Detected Face
					var descriptor2 = faceapi.computeFaceDescriptor('temp' + arr[1]); 
					descriptor1.then(function(x) {
						result1 = x;
						descriptor2.then(function(result2) {
							// Recognition Function Core
							var distance = faceapi.euclideanDistance(result1, result2);
							if (facemodel) {
								// Distance Less than 0.39 mean not matched
								if (distance <= 0.39) {
									// Push Face Detected with Success Matching to Face JSON Model 
										"type": "Active",
										"datenow": new Date(),
										"description": Control.getId(),
										"info": "Matched" + distance.toFixed(2),
										"infoState": sap.ui.core.MessageType.Success,
										"icon": oCanvas.toDataURL(),
										"highlight": sap.ui.core.MessageType.Success
											//	"infoState": ""

								} else {
									// Push Face Detected with Failed Matching to Face JSON Model 
										"type": "Active",
										"datenow": new Date(),
										"description": Control.getId(),
										"info": "Unknown" + distance.toFixed(2),
										"infoState": sap.ui.core.ValueState.Error,
										"icon": oCanvas.toDataURL(),
										"highlight": sap.ui.core.MessageType.Error



					setTimeout(that.calldrawing, 1000, Control);
		// Convert Face image to GrayScale
		makeGrayscale: function(input) {
			//Get the context for the loaded image
			var inputContext = input.getContext("2d");
			//get the image data;
			var imageData = inputContext.getImageData(0, 0, input.width, input.height);
			//Get the CanvasPixelArray
			var data =;

			//Get length of all pixels in image each pixel made up of 4 elements for each pixel, one for Red, Green, Blue and Alpha
			var arraylength = input.width * input.height * 4;
			//Go through each pixel from bottom right to top left and alter to its gray equiv

			//Common formula for converting to grayscale.
			//gray = 0.3*R + 0.59*G + 0.11*B
			for (var i = arraylength - 1; i > 0; i -= 4) {
				//R= i-3, G = i-2 and B = i-1
				//Get our gray shade using the formula
				var gray = 0.3 * data[i - 3] + 0.59 * data[i - 2] + 0.11 * data[i - 1];
				//Set our 3 RGB channels to the computed gray.
				data[i - 3] = gray;
				data[i - 2] = gray;
				data[i - 1] = gray;

			inputContext.putImageData(imageData, 0, 0);

Finally we get it.


We just learnt how to create your custom control and consume external Image Processing JavaScript Lib in our Custom SAP UI5 App, May be in the future we will be able to connect our SAPUI5 App to Cams on same network by IPs and open the door for New security module responsible for monitoring in Companies and make face detection and recognition with the Employees photos in HCM Module.

In Image Processing science , There are alot of techniques that you can add to your UI5 App to cover your Client requirements , Alot of trusted Open Source Thirdparties and Libs avaialble on the internet, you can consume one of them.


I hope this post will be helpful for all.

All the best.

Original Article:

Related blogs


Please enter your comment!
Please enter your name here