Search

Geek Girl Joy

Artificial Intelligence, Simulations & Software

Month

April 2018

Benchmarking PHP

Lately i’ve been working on a lot of things that require me to “benchmark” the time it takes for parts of my code to run. This means that I have been using¬†microtime() a lot!

So rather than waste time rewriting the same code over and over again I decided to write an “AppTimer” class to make things a little simpler!

More importantly, I’m going to let you have it for free, and you don’t even have to give me your email address! ūüėõ

Here’s the project repo on GitHub: AppTimer on GitHub

I’ve created a few examples so you can see how to use it. ūüôā

BasicExample.php

The most basic way to use the AppTimer class is to create a $Timer object and then call the AppTimer Start() method. Then you just run your code or call your function.

Once your code is is done executing use the AppTimer Stop() method.

The AppTimer Report() method will return a human readable string with how long it took your code to run.

You can explicitly destroy the AppTimer object after using it if you are working in a memory constrained environment as demonstrated below.

<?php
include('AppTimer.Class.php');     // Include AppTimer class file

$Timer = new AppTimer();           // Create Timer
$Timer->Start();                   // Start() Timer

// Code you want to time

$Timer->Stop();                    // Stop() Timer
echo $Timer->Report() . PHP_EOL;   // Report()


// Destroy $Timer Object
$Timer = NULL; // Reclaim memory immediately by overwriting NULL on Timer object's memory space                  
unset($Timer); // Let Garbage Collection know it can eat the variable
?>

The result of this code should look something like this:

0.0001 Seconds

 

AutoStartAndReportExample.php

I included a few “features” in the AppTimer to make it more useful. The first being ‘Auto Start’ which means that the Timer will start automatically as soon as it is created in memory rather than requiring you to use the AppTimer Start() method.

The way to use Auto Start is to pass a 1 or true boolean value to the class constructor as I demonstrate below.

Additionally I have implemented an ‘Auto Report’ feature on the AppTimer¬†Stop() method which will automatically call the AppTimer Report() method for you when you pass a 1 or true boolean value to the Stop method, which I also demonstrate below.

Further, the AppTimer Report() method can be used before you call the AppTimer Stop() method to get an immediate report of the elapsed time.

<?php
 
include('AppTimer.Class.php');   // Include AppTimer class file
 
 
// Use true or 1 to Auto Start the Timer when the object is instantiated
$Timer = new AppTimer(1);

usleep(1000000); // wait for 1 seconds
echo $Timer->Report() . PHP_EOL; // Use Report() before Timer is stopped to get elapsed time 
usleep(3000000); // wait for 3 seconds
 
echo $Timer->Stop(1) . PHP_EOL;  // Use true or 1 to Auto Report
echo $Timer->Report() . PHP_EOL; // Report() wont change after Stop() 


// Destroy $Timer Object
$Timer = NULL; // Reclaim memory immediately by overwriting NULL on Timer object's memory space                  
unset($Timer); // Let Garbage Collection know it can eat the variable

The result of this code should look something like this:

1.0002 Seconds
4.0005 Seconds
4.0005 Seconds

First you will notice that the program waited 1 second and then gave a report that 1.0002 Seconds had passed. Also notice that 4.0005 Seconds appears twice. This is because I use the Stop(1) method to stop and Auto Report how long the $Timer object was in operation.

I then called Report() method manually to show that the time wont change after the timer is stopped.

 

FunctionCallbackExample.php

The AppTimer class does reduce and simplify things however in the end, you will still basically end up either using Auto Start or the Start() method as well as Stop() & Report or Auto Report…

The smarter and or more experienced among my readers will probably be figuring they can wrap these methods up into their own timer function and then pass their code to the function via a callback (for my non technical readers this means to do less work & improve code quality at the same time).

The thing is you would have to implement that system every time you start a new project… it’s still better/faster, though is in no way optimal!

The A+ students in the audience are thinking to themselves… “can’t you just do it for me?”… to which my response is “Yes! Yes I can!”. ūüėõ

In this example I have implemented a function called ExampleCountFunction() which has two arguments ($count_to & $message) and if no arguments are passed both variables have default values.

First, instantiate a $Timer object then use the AppTimer CallbackTimer() method as I show below to call your function and time how long it takes for it to run.

Notice I don’t use Start(), Stop(), Auto Start or Report() with CallbackTimer(). It will handle everything for you and return the results of the Report() when it is done.

The first argument is the name of your function as a String data type. The second argument (which is optional) is an array of variables/arguments for your function.

In this example I demonstrate using the CallbackTimer() method with and without passing arguments to the function.

<?php
 
include('AppTimer.Class.php');   // Include AppTimer class file

// Example Function to Time
function ExampleCountFunction($count_to = 1000000, $message = 'No Message Was Specified.'){
	$report_at = round($count_to / 2);
	// iterate $i from 0 to $count_to and echo $message at 1/2 way
	for($i=0;$i<$count_to;$i++){
		if($i == $report_at){
			echo $message . PHP_EOL;
		}
	}
}


$Timer = new AppTimer(); // New Timer Object


// No need to Start, Auto Start, or Report() with CallbackTimer()
echo $Timer->CallbackTimer('ExampleCountFunction') . PHP_EOL; // No Arguments
echo $Timer->CallbackTimer('ExampleCountFunction', array(6543210, 'Geek Girl Joy')) . PHP_EOL; // Has Arguments



// Destroy $Timer Object
$Timer = NULL; // Reclaim memory immediately by overwriting NULL on Timer object's memory space                  
unset($Timer); // Let Garbage Collection know it can eat the variable

The result of this code should look something like this:

No Message Was Specified.
0.2561 Seconds
Geek Girl Joy
1.6415 Seconds

From these results you can see that the first time I used the CallbackTimer() method I didn’t specify any arguments so ExampleCountFunction() used it’s defaults.

The function counted to 1 Million and echoed it’s $message variable (‘No Message Was Specified’) 1/2 way through counting.

Once ExampleCountFunction() completed its task the callback timer stopped the timer and reported the duration.

I then use the same $Timer object again to run the same function ExampleCountFunction() as before but the second time I used an array in the second argument of CallbackTimer() method to pass argument values to my function.

The result is ExampleCountFunction() takes longer to run because instead of counting to 1 Million I have it counting to 6,543,210.

Additionally, the $message output is different because I included a string in the second index position of the CallbackTimer() arguments array which lines up with the second argument ExampleCountFunction($count_to, $message).

Values in the array are passed in order to the corresponding argument on your function.

 

ObjectMethodCallbackExample.php

Some of you Object Oriented programmers will probably not want to include my software (the AppTimer  class) with your software but you may still want to use it to test your code.

The¬†CallbackTimer() method will also work with an Object and call it’s methods for you! This means that you can just instantiate an AppTimer object along with your Object and you never need to mix my code into your Class files.

The way you do it is to use an array for the CallbackTimer() $callback variable as I demonstrate below.

The first index position of the $callback array should be the variable reference to your object. The second position should be the name of the method you want to time and of course you can pass arguments to your methods as needed by using a second arguments array as demonstrated.

<?php
 
include('AppTimer.Class.php');   // Include AppTimer class file

// Example Class Object
class ExampleClassObject {
		
    // Method without args
    function HelloWorld() {
        echo 'Hello World' . PHP_EOL;
    }
    
    // Method with args
    function Add($a, $b) {
        if(is_numeric($a) && is_numeric($b)){
		    echo "Add($a, $b) = " . ($a + $b) . PHP_EOL;
		}
    }
}




$Timer = new AppTimer(); // New Timer Object
$Example = new ExampleClassObject(); // Create instance of your object


// Time an object method without args
$HelloWorld_method_run_time = $Timer->CallbackTimer(array($Example, 'HelloWorld'));


// Time an object method with args
$Add_method_run_time = $Timer->CallbackTimer(array($Example, 'Add'), array(9, 1));


// Destroy $Timer Object
$Timer = NULL; // Reclaim memory immediately by overwriting NULL on Timer object's memory space                  
unset($Timer); // Let Garbage Collection know it can eat the variable


// Echo Results
echo "\$Example->HelloWord() Run Time: $HelloWorld_method_run_time" . PHP_EOL;
echo "\$Example->Add(9, 1) Run Time: $Add_method_run_time" . PHP_EOL;

The result of this code should look something like this:

Hello World
Add(9, 1) = 10
$Example->HelloWord() Run Time: 0.0001 Seconds
$Example->Add(9, 1) Run Time: 0.0001 Seconds

 

As you can see both methods run, the first without arguments and the second with arguments.

The reports are assigned to variables.

After the $Timer object is destroyed I echo the results as interpolated strings.

 

GetTimeExample.php

One final feature that I would like to demonstrate separately is the AppTimer GetTime() method.

Basically what this does is let you pass the GetTime() method a number (as seconds) and get a formatted string back.

Note that GetTime() will overwrite the private $start & $stop properties of the $Timer object so you should not use GetTime() on an object that you are using to time.

<?php
 
include('AppTimer.Class.php');   // Include AppTimer class file


$Timer = new AppTimer(); // New Timer Object

echo $Timer->GetTime(89003) . PHP_EOL; // pass the number of seconds

// Destroy $Timer Object
$Timer = NULL; // Reclaim memory immediately by overwriting NULL on Timer object's memory space                  
unset($Timer); // Let Garbage Collection know it can eat the variable

The result of this code should look exactly like this:

25 Hours 43 Minutes 24 Seconds

As you can see AppTimer supports Hours, Minutes and Seconds.

I didn’t intend this class to be a full chronometer just a simple timer so I didn’t add additional conversion for Days, Weeks, Months, Years, Decades etc… In any case, we now have a tool to benchmark our PHP code! ūüėČ

With that, I will see you all in my next post.

 

Much Love,

~Joy

Advertisements

Self Driving Cars Are Stupid?

I sat down to have dinner with my in-laws recently and I had an interesting conversation about self driving vehicles with my father in law. He took the position that he didn’t think they were safe (which isn’t saying much right now) and are therefore a bad idea.

 

I can’t argue with his overall conclusion, self driving cars are not ready to drive your kids to school for you. Hell, you’re not even supposed to take your eyes off the road and the testers agree to keep their hands free in the event you suddenly need to take over for the bot.

 

However my father in law takes a particularly negative position on self driving cars and believes that it isn’t going to happen and that it ‚Äúcannot happen‚ÄĚ even going so far as to call it ‚Äú…a stupid idea‚ÄĚ.

Now, my father in law is not a dumb person by any means! He was an electrical engineer for the U.S. postal system for many years and isn’t shy when it comes to taking things apart and putting them back together.

 

Why then, do I believe he is wrong when he says ‚Äúdamn the statistics‚ÄĚ?¬†

Damn the Statistics?

No! Basically it boils down to how distributed systems like autonomous vehicles work. It’s probably more appropriate to look at autonomous capable vehicles as services like Siri or Alexa, it’s basically a cloud supported product if you want to use it.

You might be using Alexa locally on your phone but Alexa and Siri are really cloud resources that use distributed processing to achieve your goals faster and with greater accuracy. Further still, each time an improvement is made to a bot like Alexa or Siri those improvements are available to all users at the next update. Sometimes the updates directly affect the local software sometimes they affect the server, however the point still stands that essentially everyone gets the update.

 

What this means is that systems designed like this are always improving. 

Bots Improve With Training

This brings us to the subject of training the driving bot. Regardless of it’s function, a bot is only as good as the training it receives.

In my Getting Started with Neural Networks series as well as my Pathfinder Neural Network¬†and Machine Learning From A Database articles I demonstrate several ‚Äúhard/hand coded‚ÄĚ training sets which basically mean they are static and unchanging and were more or less created manually.

 

In my Lets teach AI how to read using OCR I demonstrated how to programmatically generate training data from a finite set of possibilities on the fly rather than having to manually create it, however the training data is still essentially fixed or static before the bot ever sees it.

 

However in real world situations when you are dealing with a complex problem like building a self driving vehicle you cannot actually build the training data by hand in a practical sense and it cannot really be static either because it is the nature of driving to be unpredictable and you need a bot that can improve over time and learn how to properly respond to new and varied conditions. What you would really want is to be able to show the bot examples of all driving conditions and have it work out all the rules for itself.

Additionally, you don’t want to go careening of a cliff if/when your internet connection inevitably ¬†drops out (see Murphy’s law) temporarily because you went behind a particularly tall mountain or hit a stopped bus because a building blocked your signal so autonomous vehicles will/do run a local copy of the ‚Äúcurrent build‚ÄĚ of the bot which just means that your bot has all the most up to date training.

 

So, while you drive an autonomous capable vehicle around it is always watching the environment, what the weather conditions are, how many cars are around, it noticed how you subtly turn the steering wheel to avoid an approaching pothole, it’s keeping track of where people are, other cars, which lane you are in‚Ķ ¬†and so forth.

Which brings us to the part where the real math-magic happens…

 

Data Agricultural & Server Farming

All of that data generated by each of the autonomous vehicles is combined with all of the data from all the other autonomous vehicles back at the manufacturer’s “server farm”. There may be a few intermediate processing steps on the data to ensure that only prime ‘Grade A’ quality training examples make it through to be included with the next batch of training data.

Once the training data is ready the bot is trained on the new data. The exact nature of training depends on the type of bot, technology “stack” and algorithms that are being used. ¬†For the purposes of this article I simply mean that the bot is allowed to “view” or incorporate the new information into it’s self or knowledge base.

 

Depending on the exact nature of the training, the process may be as quick as processing the data into a database, or require many hours (or even potentially days) of review by a neural network or other learning algorithm. In just about all cases though, at least as far as ones I can think of, more training equates with a better bot.

 

Once training is complete the bot is likely put through a battery of manual and automatic testing and evaluation where scenarios are either digital simulations or real world private tracks, likely both. Once the bot passes the internal tests and is approved for release “into the wild” it is pushed out to the autonomous vehicles.

 

In truth this would be happening all the time rather than at a single specific time every day but you get the idea.

 

But, doesn’t this just mean incremental improvements? Yes and no, I mean sure if your bot were to rely solely on you for it’s training it could never learn to be a good driver, even if you were the best driver there ever was because you only drive a few hours at most each day and that’s nowhere near enough training, even over the course of an entire human lifetime, but if you combine all the data from all the deployed autonomous vehicles you start to get meaningful improvements in capability.

Exponential Growth

Let’s look at some numbers. ūüôā

It’s kinda hard to actually determine how many individual self driving vehicles are currently on the road but according to this link (https://www.dmv.ca.gov/portal/dmv/detail/vr/autonomous/testing) as of right now there are 52 companies licensed to test autonomous vehicles in my home state of California and here is the list of companies for your reference (https://www.dmv.ca.gov/portal/dmv/detail/vr/autonomous/permit).

 

According to this wiki article on Waymo¬† (Alphabet‚Äôs (Google) autonomous vehicle company – not a sponsor ūüėõ ) just ordered ‚Äúthousands of minivans and 20,000 sedans‚ÄĚ… and since any company building these bots will only have access to data from their own vehicles we only need consider a single company to get a rough overview of what’s possible so let’s use¬†¬†a total of 25,000 deployed vehicles¬†since we can safely assume the total number they have deployed will be at least this many.¬†Further, let’s say that each only gets 2 hours of drive time a day.¬†For the sake of simplicity let’s say all the vehicles drive every day (365 days a year).¬†Now let’s also assume they never increase the number of vehicles and only keep the 25,000 operational.

 

Based on these conditions we can proceed with our thought experiment!

 

(25,000 (vehicles) * 2 (hours a day)) = 50,000 hypothetical driving hours each day in total generated by the Waymo system. To put that into perspective that means (50,000 hours / 24 (number of hours in a day)) = 2083.33 (days) / 365 (days in a year) = 5.70 years of bot experience (training data) generated every day!

If we extend our thought experiment out to one year:  (50,000 (hours of training data) * 365 (days)) = 18,250,000 hours of training data generated each year and every year!

18,250,000 hours is too large to grasp right? Lets simplify this! (18,250,000 (hours of training data generated in 1 year) / 24 (hours in a day)) = 760,416.66 (days). Still too big to wrap our minds around so lets keep simplifying! (760,416.66 (days of training data) / 365 (days in a year)) = 2,083.33 years of training data generated.

What this means is that in 1 year the extrapolated/hypothetical Waymo system as defined could easily generate 2 Millenniums (20 Centuries) worth of a single driver’s experience in just 1 year depending on their data collection and processing methods and they could double it just by changing to 4 hours a day of drive time per vehicle!

Notice however I am not making any claims on how much actual data this is. It’s customary to speak of “data quantity” in terms of “bits”, “bytes”, “kilobytes”,¬†“megabytes”,¬†“gigabytes” etc… and “data transmission speed” ¬†in terms of “bps” (bits per second),¬†“kbps” (kilobits per second),¬†¬†“mbps” (megabits per second),¬†“gbps” (gigabits per second) etc… and without examining the specifics of a given¬†system we could only speculate how much data and what the throughput of that system might be. Depending on the developer’s methodology (how many data points they collect and the resolution at which the data is collected) they may use all or only some of the data generated in the actual training of the bots.

 

Full Circle

I’ve generically covered the general concept of how these self driving cars are being developed because I really wanted to stress the amount of effort going into training the bots that will one day be truly and fully autonomous. The reason being that while I do agree with my father in law that these systems can be dangerous right now, that will not always be the case. Very quickly (some industry experts say less than 5 years) these systems will be fully realised and deployable.

 

Even if it were to take 30, 60 or even 100 years more (unlikely as that may be) it would still mean that eventually these systems would be ready and affordable to boot!

Which brings us to the last as well as sobering topic & the chief component of my father in laws argument…

 

Safety

Let’s not sugar coat this, It would suck to be hurt or killed by an autonomous vehicle! Further, my heart genuinely goes out to the families of those who have had or will have their lives altered by an accident involving an autonomous vehicle, but in truth is it not also equally terrible when an accident occurs and it is a human driver at the wheel?

The fact is that human life is fragile and we make mistakes all the time that could cost us our lives or the life of someone around us!

According to ASIRT (Association for Safe International Road Travel – not a sponsor) 3,287 deaths occur globally each day (1.3 Million Annually) as a result of vehicle accidents where humans were at the helm.

So to say that human drivers are “far superior” or “preferable” to an intelligent machine is only a matter of perspective in so far as that is actually the case.¬†Every day that the hypothetical system we discussed above doesn’t kill someone or even get in an accident represents 5.7 years of “good driver record” and an entire year (365 days) without incidents is like a single person driving continuously for over 2000 years and not once having an accident.¬†The odds against there being an accident will never be zero but how many ‘safe driving years’ is necessary before we could call the bot “safe enough”? Would you say 1 million years of drive time without incident would do it?

Let’s revise our thought experiment briefly and see how long it would take.

First, lets say 8 hours drive time per vehicle every day and instead of 25K lets say they start with 50k and every year they double the number of vehicles in their testing & training fleet.

It would only take 7 years to train this bot and based on my projections (assuming the system was accident free) in the last few months of 2024 they would/will have generated a bot with 1 million ‘person/driver years’.

Here’s a chart for your reference:

Given the state of machine learning technologies today and the fact that our hypothetical self driving car developer needs less than half of the number of all the cars in Los Angeles county¬†to pull it off, I’d wager that not only it is possible for self driving cars to actually be better drivers than humans given enough time and training, its simply an eventuality!

So can we do it? Without a doubt no matter how high we set the bar for safety we can and will build bots that exceed our requirements.

As to whether or not you think self driving cars are stupid, well… I guess that’s just a matter of personal opinion.

 

With that, everyone have a great day and I will see you all in my next post.

 

Much Love,

~Joy

Blog at WordPress.com.

Up ↑