Search

Geek Girl Joy

I’m Feeling Generous

It’s raining in Los Angeles today and I’m feeling generous! So generous in fact that I have decided to give you a free gift.

How would you like 256 pre-trained neural networks and the associated training files?

Last year while working on my Ancestor Simulations Series I wrote a 4 part mini-series on Elementary Cellular Automata and here are the links:

Elementary Cellular Automata

Elementary Cellular Automata 2

Elementary Cellular Automata 3

Elementary Cellular Automata 4

In these articles I provide a fairly thorough introduction to what Elementary Cellular Automata are and how they function.

I also demonstrate a program I wrote in PHP to algorithmically compute the Wolfram 1D CA Rules and then render them as PNG images. You can find my algorithmic Wolfram 1D CA implementation on GitHub here.

After publishing those articles I announced that I was releasing the 256 Pre-Trained Wolfram CA Neural Networks to my followers on Patreon.

Now, I have decided to make those Neural Networks available to you for free!

Here’s what you get with this download:

  • 256 training sets (.data files) to review, modify  and train neural networks to perform all 256 (0 – 255) rules  conveniently organized into separate files and named after the rule that  it trains the network on.
  • 256 PRE-TRAINED neural networks (.net files) capable of performing the computation as outlined in my Elementary Cellular Automata series except via neural networks.
  • 5 separate programs (.php files):
  1. generate_rule_set.php: This program will recreate the entire set of 256 .data training files.
  2. train_all.php:  This program will train all 256 FANN .net neural networks.
  3. train_specific.php: This program lets you enter the  rule numbers into an array prior to running to train the specific rules  as FANN .net file neural networks.
  4. ann_output_all_rule_set_images.php: This program will render all the rules as images using the neural networks.
  5. ann_output_specific_rule_images.php: This program will render the specified rules as images using the specified neural networks.
  • The entire set of rules (0 – 255) per-rendered as 1025x1025px PNG images.

You will need PHP and FANN installed for this to work. I have a public tutorial on how to setup your test environment for free here: 

Getting Started with Neural Networks Series

 

You can get your copy in one of two ways:

Download Neural Networks from Patreon  Get Code.

Download Neural Networks from GitHub Get Code.

If you like this project please “star” it on GitHub.

Since I’ve been generous to you I’d like you to also be generous with other people today and if you are so inclined to be generous to me please consider supporting me on Patreon for as little as $1 a month.

This post was made possible by the generous contributions of my sponsors on Patreon

 

 

Much Love,

~Joy

Advertisements

How to Build a Spider Bot

We live in the age of information and there is a seemingly endless abundance of new websites being launched all the time creating entrepreneurial opportunities for innovative developers to ‘data mine’ all that aggregate info.

You might be building a specialized knowledge base for a unique Machine Learning problem or you may be working on a new algorithm for analyzing and ranking website content. In fact the uses for mined web content extend far beyond the scope of this article, however if you find yourself in need of the ability to acquire a large portion of a website (or sites) or simply review the content on them (like a search engine) then you need a ‘crawler’ or ‘spider’.

There are plenty of spiders available already (free and paid) that you can run on your equipment or there are services available as well but since this isn’t a review I won’t be mentioning any names.

If you offer such a product and would like to sponsor a post feel free to contact me.

Also I think it’s fun to see how things work and you don’t really get to do that when you can’t look at the code and change how it works.

So today we’re going to look at building a basic crawler, in fact I would go so far as to say this is more of a ‘page scraper’ than a fully realized spider, the main difference being features and higher levels of autonomous operation.

I am going to demonstrate basic scraping/crawling but I will leave more advanced processing and parsing for you to implement as they are use case specific and I want to keep this post as simple as possible.

I may do another post on how to improve this basic implementation if it seems like you guys like this topic so let me know in the comments if I should do more posts on this subject.

 

A Word of Warning

Before we get started I want to warn you that site owners take steps to prevent data miners from scraping all their content.

Even free sites that act as “knowledge hubs” usually do not want their content copied by a robot because the person operating the bot may be one of their competitors.

I am not providing this code to you so that you can be irresponsible or malicious.

I say this from personal experience, it sucks having a website you need block your external IP Address! 😛

It sucks even worse if they get your ISP involved and it sucks the most if government entities get involved!

What you choose to do with this spider is entirely up to you, play nicely or YOU WILL get blacklisted or maybe even worse! DO NOT use this code to violate any laws and don’t steal intellectual property!

When in doubt, find out if the site has a “robots.txt” file which will offer some insight into how the site views spiders and what they will and wont allow, you can find it at siteroot/robots.txt

The robots.txt file for my site can be found here: https://geekgirljoy.wordpress.com/robots.txt

Having a robots.txt file is not a requirement for having a website and just because a site doesn’t have one doesn’t mean they are OK with you crawling their entire site.

However if you crawl links or sections they disallow then your spider will likely be discovered and blocked.

 

Our Strategy

Now let’s talk about the strategy we will use for the spider bot.

 

Target Specific Content

First, we will not be implementing any auto URL extraction and crawling based on found hyperlinks because it’s not a good idea to just auto crawl any URL without first knowing what’s there.

Web developers are smart and they have developed counter measures to unwanted spiders, see “Honeypot”, false links and never ending cascading auto generated URLs that do nothing but link to deeper auto generated URLS… a dumb bot cannot understand this and will happily continue to crawl these links sending up HUGE red flags and tripping alarms all over the place that a bot is operating at your IP Address.

If you’re smart you might be thinking that you could set a maximum URL “depth” which is not a bad idea, however this isn’t a guarantee that your bot won’t be found, it only guarantee that if your bot gets stuck in a honeypot it will eventually stop crawling it, though perhaps after it has already been discovered.

Think about it, one of the methods to confuse your bot is to place a hidden link on a page that no human would ever see and with no visible link to click on, then the only way someone could reasonably get there is if they either reviewed the site code or an automated bot found the link, so repeated attempts to access bad links would only occur outside of the normal operation of the site.

As such, in this example URLs to be crawled will be hand coded into an array, this ensures that the spider will only crawl pages you have approved.

You could extend this spider to collect the URLs and simply review & approve them before the bot crawls them, but we won’t cover that today.

 

Record Keeping

As for what we will do with the data we collect, it makes sense to simply store a local copy for our later use offline rather than processing the data while the spider is crawling.

Mainly because you can always use your local copy without any penalty, just save all extracted data separately from the original.

This prevents the need for repeated crawling of the same content which reduces the server resources your spider will use enabling it to operate slightly more covertly. 🙂

As to how you store that data, it’s entirely up to you.

I have used raw text, json and even MySQL, however today we will simply clone the page and keep it’s extension, which is probably the most straight forward method.

 

Be Patient

Websites are for people not spiders and even if a site is totally cool with you crawling their entire site, you should exercise caution and not crawl too frequently or quickly so that you do not utilize an inordinate amount of their server resources, if you do, YOU WILL BE BLOCKED!

As such we will implement a wait period that should help reduce our load on the web server we are crawling.

Additionally, humans don’t read all webpages in the same amount of time so we will have the wait period change after each crawl rather than the same each time so that the bot leaves less of a repeating crawl pattern. 

 

Simple Spider Code

 

<?php
// Instructions:
// Update the $url_array with the pages you want to crawl then run from command line.
// References:
// mkdir() - http://php.net/manual/en/function.mkdir.php
// count() - http://php.net/manual/en/function.count.php
// file_get_contents() - http://php.net/manual/en/function.file-get-contents.php
// fopen() - http://php.net/manual/en/function.fopen.php
// fwrite() - http://php.net/manual/en/function.fwrite.php
// fclose() - http://php.net/manual/en/function.fclose.php
// basename() - http://php.net/manual/en/function.basename.php
// mt_rand() - http://php.net/manual/en/function.mt-rand.php
// sleep() - http://php.net/manual/en/function.sleep.php
// List of URLs to Crawl
$url_array = array('http://www.sitename.com/page1.html',
                     'http://www.sitename.com/page2.html',
                     'http://www.sitename.com/page3.html'
                     // add more links here as needed
                     );
@mkdir('crawled/', 0777, true); // quietly make a subfolder called 'crawled'
// Loop Through $url_array
foreach($url_array  as $key=>$page){
    // Do Crawl	
    echo  'Crawling (' . $key . ' of ' . count($url_array) . ')' . $page . PHP_EOL;
    $data = file_get_contents($page);
 
    // Save a clone of the crawled page in the crawled subfolder 
    $file = fopen('crawled/' . basename($page), 'w');
    fwrite($file, $data);
    fclose($file);        
    
    // Wait	- DO NOT REMOVE
    // This keeps your spider from looking like a bot
    // It makes it look like you are spending a few minutes reading each
    // page like a person would. It keeps the spider from using excessive
    // resources which will get you blacklisted.
    $sleep = mt_rand(150, 300); // Between 2.5 & 5 Minutes
    echo 'Sleeping for ' . $sleep . PHP_EOL;
    while($sleep > 0){
		sleep(1);
		$sleep-=1; // take one second away
		echo $sleep . ' seconds until next crawl.' . PHP_EOL;
    }
}
echo 'Program Complete'  . PHP_EOL;

 

Beware Dirty Data and Canary Traps

When you scrape data from the wild (the internet) you will encounter “Dirty Data” which simply means it’s incomplete, misspelled or in some way contains inaccuracies from what it should actually be.

Further, I mentioned above that some sites will go to great lengths to confound your efforts to crawl their content, closely related to Dirty Data but intentional is the concept of a “Canary Trap” which means the data purveyor deliberately ‘tainted’ their information in an effort to either confirm you obtained your information from them or to simply make it more difficult to use the data without “Cleansing” it first.

The simplest solution to both problems is to obtain the same content from as many different sources as possible (if possible) and compare the differences.

Any significant variance definitely indicates the presence of Dirty Data and may indicate the presence of a Canary Trap embedded in one or more of the data sources.

 

If you would like to obtain a copy of this project on GitHub you can find it here.

 

I hope you enjoyed this post, if so please consider supporting me on Patreon.

 

 

Much Love,

~Joy

Benchmarking PHP

Lately i’ve been working on a lot of things that require me to “benchmark” the time it takes for parts of my code to run. This means that I have been using microtime() a lot!

So rather than waste time rewriting the same code over and over again I decided to write an “AppTimer” class to make things a little simpler!

More importantly, I’m going to let you have it for free, and you don’t even have to give me your email address! 😛

Here’s the project repo on GitHub: AppTimer on GitHub

I’ve created a few examples so you can see how to use it. 🙂

BasicExample.php

The most basic way to use the AppTimer class is to create a $Timer object and then call the AppTimer Start() method. Then you just run your code or call your function.

Once your code is is done executing use the AppTimer Stop() method.

The AppTimer Report() method will return a human readable string with how long it took your code to run.

You can explicitly destroy the AppTimer object after using it if you are working in a memory constrained environment as demonstrated below.

<?php
include('AppTimer.Class.php');     // Include AppTimer class file

$Timer = new AppTimer();           // Create Timer
$Timer->Start();                   // Start() Timer

// Code you want to time

$Timer->Stop();                    // Stop() Timer
echo $Timer->Report() . PHP_EOL;   // Report()


// Destroy $Timer Object
$Timer = NULL; // Reclaim memory immediately by overwriting NULL on Timer object's memory space                  
unset($Timer); // Let Garbage Collection know it can eat the variable
?>

The result of this code should look something like this:

0.0001 Seconds

 

AutoStartAndReportExample.php

I included a few “features” in the AppTimer to make it more useful. The first being ‘Auto Start’ which means that the Timer will start automatically as soon as it is created in memory rather than requiring you to use the AppTimer Start() method.

The way to use Auto Start is to pass a 1 or true boolean value to the class constructor as I demonstrate below.

Additionally I have implemented an ‘Auto Report’ feature on the AppTimer Stop() method which will automatically call the AppTimer Report() method for you when you pass a 1 or true boolean value to the Stop method, which I also demonstrate below.

Further, the AppTimer Report() method can be used before you call the AppTimer Stop() method to get an immediate report of the elapsed time.

<?php
 
include('AppTimer.Class.php');   // Include AppTimer class file
 
 
// Use true or 1 to Auto Start the Timer when the object is instantiated
$Timer = new AppTimer(1);

usleep(1000000); // wait for 1 seconds
echo $Timer->Report() . PHP_EOL; // Use Report() before Timer is stopped to get elapsed time 
usleep(3000000); // wait for 3 seconds
 
echo $Timer->Stop(1) . PHP_EOL;  // Use true or 1 to Auto Report
echo $Timer->Report() . PHP_EOL; // Report() wont change after Stop() 


// Destroy $Timer Object
$Timer = NULL; // Reclaim memory immediately by overwriting NULL on Timer object's memory space                  
unset($Timer); // Let Garbage Collection know it can eat the variable

The result of this code should look something like this:

1.0002 Seconds
4.0005 Seconds
4.0005 Seconds

First you will notice that the program waited 1 second and then gave a report that 1.0002 Seconds had passed. Also notice that 4.0005 Seconds appears twice. This is because I use the Stop(1) method to stop and Auto Report how long the $Timer object was in operation.

I then called Report() method manually to show that the time wont change after the timer is stopped.

 

FunctionCallbackExample.php

The AppTimer class does reduce and simplify things however in the end, you will still basically end up either using Auto Start or the Start() method as well as Stop() & Report or Auto Report…

The smarter and or more experienced among my readers will probably be figuring they can wrap these methods up into their own timer function and then pass their code to the function via a callback (for my non technical readers this means to do less work & improve code quality at the same time).

The thing is you would have to implement that system every time you start a new project… it’s still better/faster, though is in no way optimal!

The A+ students in the audience are thinking to themselves… “can’t you just do it for me?”… to which my response is “Yes! Yes I can!”. 😛

In this example I have implemented a function called ExampleCountFunction() which has two arguments ($count_to & $message) and if no arguments are passed both variables have default values.

First, instantiate a $Timer object then use the AppTimer CallbackTimer() method as I show below to call your function and time how long it takes for it to run.

Notice I don’t use Start(), Stop(), Auto Start or Report() with CallbackTimer(). It will handle everything for you and return the results of the Report() when it is done.

The first argument is the name of your function as a String data type. The second argument (which is optional) is an array of variables/arguments for your function.

In this example I demonstrate using the CallbackTimer() method with and without passing arguments to the function.

<?php
 
include('AppTimer.Class.php');   // Include AppTimer class file

// Example Function to Time
function ExampleCountFunction($count_to = 1000000, $message = 'No Message Was Specified.'){
	$report_at = round($count_to / 2);
	// iterate $i from 0 to $count_to and echo $message at 1/2 way
	for($i=0;$i<$count_to;$i++){
		if($i == $report_at){
			echo $message . PHP_EOL;
		}
	}
}


$Timer = new AppTimer(); // New Timer Object


// No need to Start, Auto Start, or Report() with CallbackTimer()
echo $Timer->CallbackTimer('ExampleCountFunction') . PHP_EOL; // No Arguments
echo $Timer->CallbackTimer('ExampleCountFunction', array(6543210, 'Geek Girl Joy')) . PHP_EOL; // Has Arguments



// Destroy $Timer Object
$Timer = NULL; // Reclaim memory immediately by overwriting NULL on Timer object's memory space                  
unset($Timer); // Let Garbage Collection know it can eat the variable

The result of this code should look something like this:

No Message Was Specified.
0.2561 Seconds
Geek Girl Joy
1.6415 Seconds

From these results you can see that the first time I used the CallbackTimer() method I didn’t specify any arguments so ExampleCountFunction() used it’s defaults.

The function counted to 1 Million and echoed it’s $message variable (‘No Message Was Specified’) 1/2 way through counting.

Once ExampleCountFunction() completed its task the callback timer stopped the timer and reported the duration.

I then use the same $Timer object again to run the same function ExampleCountFunction() as before but the second time I used an array in the second argument of CallbackTimer() method to pass argument values to my function.

The result is ExampleCountFunction() takes longer to run because instead of counting to 1 Million I have it counting to 6,543,210.

Additionally, the $message output is different because I included a string in the second index position of the CallbackTimer() arguments array which lines up with the second argument ExampleCountFunction($count_to, $message).

Values in the array are passed in order to the corresponding argument on your function.

 

ObjectMethodCallbackExample.php

Some of you Object Oriented programmers will probably not want to include my software (the AppTimer  class) with your software but you may still want to use it to test your code.

The CallbackTimer() method will also work with an Object and call it’s methods for you! This means that you can just instantiate an AppTimer object along with your Object and you never need to mix my code into your Class files.

The way you do it is to use an array for the CallbackTimer() $callback variable as I demonstrate below.

The first index position of the $callback array should be the variable reference to your object. The second position should be the name of the method you want to time and of course you can pass arguments to your methods as needed by using a second arguments array as demonstrated.

<?php
 
include('AppTimer.Class.php');   // Include AppTimer class file

// Example Class Object
class ExampleClassObject {
		
    // Method without args
    function HelloWorld() {
        echo 'Hello World' . PHP_EOL;
    }
    
    // Method with args
    function Add($a, $b) {
        if(is_numeric($a) && is_numeric($b)){
		    echo "Add($a, $b) = " . ($a + $b) . PHP_EOL;
		}
    }
}




$Timer = new AppTimer(); // New Timer Object
$Example = new ExampleClassObject(); // Create instance of your object


// Time an object method without args
$HelloWorld_method_run_time = $Timer->CallbackTimer(array($Example, 'HelloWorld'));


// Time an object method with args
$Add_method_run_time = $Timer->CallbackTimer(array($Example, 'Add'), array(9, 1));


// Destroy $Timer Object
$Timer = NULL; // Reclaim memory immediately by overwriting NULL on Timer object's memory space                  
unset($Timer); // Let Garbage Collection know it can eat the variable


// Echo Results
echo "\$Example->HelloWord() Run Time: $HelloWorld_method_run_time" . PHP_EOL;
echo "\$Example->Add(9, 1) Run Time: $Add_method_run_time" . PHP_EOL;

The result of this code should look something like this:

Hello World
Add(9, 1) = 10
$Example->HelloWord() Run Time: 0.0001 Seconds
$Example->Add(9, 1) Run Time: 0.0001 Seconds

 

As you can see both methods run, the first without arguments and the second with arguments.

The reports are assigned to variables.

After the $Timer object is destroyed I echo the results as interpolated strings.

 

GetTimeExample.php

One final feature that I would like to demonstrate separately is the AppTimer GetTime() method.

Basically what this does is let you pass the GetTime() method a number (as seconds) and get a formatted string back.

Note that GetTime() will overwrite the private $start & $stop properties of the $Timer object so you should not use GetTime() on an object that you are using to time.

<?php
 
include('AppTimer.Class.php');   // Include AppTimer class file


$Timer = new AppTimer(); // New Timer Object

echo $Timer->GetTime(89003) . PHP_EOL; // pass the number of seconds

// Destroy $Timer Object
$Timer = NULL; // Reclaim memory immediately by overwriting NULL on Timer object's memory space                  
unset($Timer); // Let Garbage Collection know it can eat the variable

The result of this code should look exactly like this:

25 Hours 43 Minutes 24 Seconds

As you can see AppTimer supports Hours, Minutes and Seconds.

I didn’t intend this class to be a full chronometer just a simple timer so I didn’t add additional conversion for Days, Weeks, Months, Years, Decades etc… In any case, we now have a tool to benchmark our PHP code! 😉

With that, I will see you all in my next post.

 

Much Love,

~Joy

Self Driving Cars Are Stupid?

I sat down to have dinner with my in-laws recently and I had an interesting conversation about self driving vehicles with my father in law. He took the position that he didn’t think they were safe (which isn’t saying much right now) and are therefore a bad idea.

 

I can’t argue with his overall conclusion, self driving cars are not ready to drive your kids to school for you. Hell, you’re not even supposed to take your eyes off the road and the testers agree to keep their hands free in the event you suddenly need to take over for the bot.

 

However my father in law takes a particularly negative position on self driving cars and believes that it isn’t going to happen and that it “cannot happen” even going so far as to call it “…a stupid idea”.

Now, my father in law is not a dumb person by any means! He was an electrical engineer for the U.S. postal system for many years and isn’t shy when it comes to taking things apart and putting them back together.

 

Why then, do I believe he is wrong when he says “damn the statistics”? 

Damn the Statistics?

No! Basically it boils down to how distributed systems like autonomous vehicles work. It’s probably more appropriate to look at autonomous capable vehicles as services like Siri or Alexa, it’s basically a cloud supported product if you want to use it.

You might be using Alexa locally on your phone but Alexa and Siri are really cloud resources that use distributed processing to achieve your goals faster and with greater accuracy. Further still, each time an improvement is made to a bot like Alexa or Siri those improvements are available to all users at the next update. Sometimes the updates directly affect the local software sometimes they affect the server, however the point still stands that essentially everyone gets the update.

 

What this means is that systems designed like this are always improving. 

Bots Improve With Training

This brings us to the subject of training the driving bot. Regardless of it’s function, a bot is only as good as the training it receives.

In my Getting Started with Neural Networks series as well as my Pathfinder Neural Network and Machine Learning From A Database articles I demonstrate several “hard/hand coded” training sets which basically mean they are static and unchanging and were more or less created manually.

 

In my Lets teach AI how to read using OCR I demonstrated how to programmatically generate training data from a finite set of possibilities on the fly rather than having to manually create it, however the training data is still essentially fixed or static before the bot ever sees it.

 

However in real world situations when you are dealing with a complex problem like building a self driving vehicle you cannot actually build the training data by hand in a practical sense and it cannot really be static either because it is the nature of driving to be unpredictable and you need a bot that can improve over time and learn how to properly respond to new and varied conditions. What you would really want is to be able to show the bot examples of all driving conditions and have it work out all the rules for itself.

Additionally, you don’t want to go careening of a cliff if/when your internet connection inevitably  drops out (see Murphy’s law) temporarily because you went behind a particularly tall mountain or hit a stopped bus because a building blocked your signal so autonomous vehicles will/do run a local copy of the “current build” of the bot which just means that your bot has all the most up to date training.

 

So, while you drive an autonomous capable vehicle around it is always watching the environment, what the weather conditions are, how many cars are around, it noticed how you subtly turn the steering wheel to avoid an approaching pothole, it’s keeping track of where people are, other cars, which lane you are in…  and so forth.

Which brings us to the part where the real math-magic happens…

 

Data Agricultural & Server Farming

All of that data generated by each of the autonomous vehicles is combined with all of the data from all the other autonomous vehicles back at the manufacturer’s “server farm”. There may be a few intermediate processing steps on the data to ensure that only prime ‘Grade A’ quality training examples make it through to be included with the next batch of training data.

Once the training data is ready the bot is trained on the new data. The exact nature of training depends on the type of bot, technology “stack” and algorithms that are being used.  For the purposes of this article I simply mean that the bot is allowed to “view” or incorporate the new information into it’s self or knowledge base.

 

Depending on the exact nature of the training, the process may be as quick as processing the data into a database, or require many hours (or even potentially days) of review by a neural network or other learning algorithm. In just about all cases though, at least as far as ones I can think of, more training equates with a better bot.

 

Once training is complete the bot is likely put through a battery of manual and automatic testing and evaluation where scenarios are either digital simulations or real world private tracks, likely both. Once the bot passes the internal tests and is approved for release “into the wild” it is pushed out to the autonomous vehicles.

 

In truth this would be happening all the time rather than at a single specific time every day but you get the idea.

 

But, doesn’t this just mean incremental improvements? Yes and no, I mean sure if your bot were to rely solely on you for it’s training it could never learn to be a good driver, even if you were the best driver there ever was because you only drive a few hours at most each day and that’s nowhere near enough training, even over the course of an entire human lifetime, but if you combine all the data from all the deployed autonomous vehicles you start to get meaningful improvements in capability.

Exponential Growth

Let’s look at some numbers. 🙂

It’s kinda hard to actually determine how many individual self driving vehicles are currently on the road but according to this link (https://www.dmv.ca.gov/portal/dmv/detail/vr/autonomous/testing) as of right now there are 52 companies licensed to test autonomous vehicles in my home state of California and here is the list of companies for your reference (https://www.dmv.ca.gov/portal/dmv/detail/vr/autonomous/permit).

 

According to this wiki article on Waymo  (Alphabet’s (Google) autonomous vehicle company – not a sponsor 😛 ) just ordered “thousands of minivans and 20,000 sedans”… and since any company building these bots will only have access to data from their own vehicles we only need consider a single company to get a rough overview of what’s possible so let’s use  a total of 25,000 deployed vehicles since we can safely assume the total number they have deployed will be at least this many. Further, let’s say that each only gets 2 hours of drive time a day. For the sake of simplicity let’s say all the vehicles drive every day (365 days a year). Now let’s also assume they never increase the number of vehicles and only keep the 25,000 operational.

 

Based on these conditions we can proceed with our thought experiment!

 

(25,000 (vehicles) * 2 (hours a day)) = 50,000 hypothetical driving hours each day in total generated by the Waymo system. To put that into perspective that means (50,000 hours / 24 (number of hours in a day)) = 2083.33 (days) / 365 (days in a year) = 5.70 years of bot experience (training data) generated every day!

If we extend our thought experiment out to one year:  (50,000 (hours of training data) * 365 (days)) = 18,250,000 hours of training data generated each year and every year!

18,250,000 hours is too large to grasp right? Lets simplify this! (18,250,000 (hours of training data generated in 1 year) / 24 (hours in a day)) = 760,416.66 (days). Still too big to wrap our minds around so lets keep simplifying! (760,416.66 (days of training data) / 365 (days in a year)) = 2,083.33 years of training data generated.

What this means is that in 1 year the extrapolated/hypothetical Waymo system as defined could easily generate 2 Millenniums (20 Centuries) worth of a single driver’s experience in just 1 year depending on their data collection and processing methods and they could double it just by changing to 4 hours a day of drive time per vehicle!

Notice however I am not making any claims on how much actual data this is. It’s customary to speak of “data quantity” in terms of “bits”, “bytes”, “kilobytes”, “megabytes”, “gigabytes” etc… and “data transmission speed”  in terms of “bps” (bits per second), “kbps” (kilobits per second),  “mbps” (megabits per second), “gbps” (gigabits per second) etc… and without examining the specifics of a given system we could only speculate how much data and what the throughput of that system might be. Depending on the developer’s methodology (how many data points they collect and the resolution at which the data is collected) they may use all or only some of the data generated in the actual training of the bots.

 

Full Circle

I’ve generically covered the general concept of how these self driving cars are being developed because I really wanted to stress the amount of effort going into training the bots that will one day be truly and fully autonomous. The reason being that while I do agree with my father in law that these systems can be dangerous right now, that will not always be the case. Very quickly (some industry experts say less than 5 years) these systems will be fully realised and deployable.

 

Even if it were to take 30, 60 or even 100 years more (unlikely as that may be) it would still mean that eventually these systems would be ready and affordable to boot!

Which brings us to the last as well as sobering topic & the chief component of my father in laws argument…

 

Safety

Let’s not sugar coat this, It would suck to be hurt or killed by an autonomous vehicle! Further, my heart genuinely goes out to the families of those who have had or will have their lives altered by an accident involving an autonomous vehicle, but in truth is it not also equally terrible when an accident occurs and it is a human driver at the wheel?

The fact is that human life is fragile and we make mistakes all the time that could cost us our lives or the life of someone around us!

According to ASIRT (Association for Safe International Road Travel – not a sponsor) 3,287 deaths occur globally each day (1.3 Million Annually) as a result of vehicle accidents where humans were at the helm.

So to say that human drivers are “far superior” or “preferable” to an intelligent machine is only a matter of perspective in so far as that is actually the case. Every day that the hypothetical system we discussed above doesn’t kill someone or even get in an accident represents 5.7 years of “good driver record” and an entire year (365 days) without incidents is like a single person driving continuously for over 2000 years and not once having an accident. The odds against there being an accident will never be zero but how many ‘safe driving years’ is necessary before we could call the bot “safe enough”? Would you say 1 million years of drive time without incident would do it?

Let’s revise our thought experiment briefly and see how long it would take.

First, lets say 8 hours drive time per vehicle every day and instead of 25K lets say they start with 50k and every year they double the number of vehicles in their testing & training fleet.

It would only take 7 years to train this bot and based on my projections (assuming the system was accident free) in the last few months of 2024 they would/will have generated a bot with 1 million ‘person/driver years’.

Here’s a chart for your reference:

Given the state of machine learning technologies today and the fact that our hypothetical self driving car developer needs less than half of the number of all the cars in Los Angeles county to pull it off, I’d wager that not only it is possible for self driving cars to actually be better drivers than humans given enough time and training, its simply an eventuality!

So can we do it? Without a doubt no matter how high we set the bar for safety we can and will build bots that exceed our requirements.

As to whether or not you think self driving cars are stupid, well… I guess that’s just a matter of personal opinion.

 

With that, everyone have a great day and I will see you all in my next post.

 

Much Love,

~Joy

Are you a Bookmark Ninja?

Welcome back, I mentioned in my last few posts that I would be doing a review of Bookmark Ninja soon and today we’re going to talk about how Bookmark Ninja is the ultimate online bookmark manager and organizer!

However, before we do, in accordance with FTC Guidlines I’d like to make you aware that this post does contain sponsored content. That being said I pride myself on being open and honest and my statements in this article are my true and actual opinions.

Now with that out of the way… in my post Happy New Years 2018 + Special Announcement I discussed the origins of this post:

“It all started when I contacted Jozsef Torzan…”

Needless to say one thing led to another and I was well on my way to reviewing Bookmark Ninja 🙂

If you need the TL;DR here it is: You will not find a better option when it comes to the simplicity, ease and affability of storing your bookmarks with Bookmark Ninja!

If you have time to keep reading, here’s why…

Imagine never having to sync all your bookmarks again! Never having to worry about about your research links and preying that your disaster recovery software shadow copied the latest bookmarks before your system died…

Imagine not having to send yourself links from your work email and instead having them immediately available to you whether you are on your phone, tablet, home system or AFK… With Bookmark Ninja your links are available no matter where you are or what device you are on!

Honestly, I wish I had thought of the idea myself because its such a great concept but since I can’t build it I might as well evaluate it. 😛

Installation

Let’s start with installation, this one is a little hard to score because while bookmark Ninja is a web app, there is sort of an instillation step that you have to do if you want to bookmark a URL.

You only need to do it once and it basically amounts to creating a browser bookmark.

Once you have a bookmark created in your browser of choice you replace the URL with some Javascript they give you and you don’t even have to copy it, there is a “Copy to Clipboard” button so they have really taken the PITA out of it.

After which you are ready to start saving new bookmarks to your Ninja account by navigating to the URL and then clicking the saved bookmark.

It’s super easy and takes 10 seconds of effort, the thing is that this might seem a little weird to a novice user but the fact that it’s a one time thing and super quick to do (and they DO provide all instructions necessary) makes it hard to reduce points for this.

Additionally, the reason for doing this has more to do with a limitation of how browser’s & web apps work and is not really reflective of a failing on behalf of the Bookmark Ninja developer team. Frankly, if I were to sit down and build a system like Bookmark Ninja I would likely use the exact same method so I think I will give this an 9 out of 10 because while it does take a second to do, its really not that big of a deal.

9/10

 

Cross-platform Compatibility

I conducted my evaluation of Bookmark Ninja using Linux and Windows OS desktops as well as my personal Android cellphone, an iPhone in addition to an Amazon Fire Tablet and in all cases I was pleased with how seamless the experience was!

I was a little reluctant initially to try Bookmark Ninja on anything but my desktop because of how well it worked on my computer, however after getting over my inceptive trepidation I was pleasantly rewarded!

Bookmark Ninja is a browser based web application so you can be confident in knowing that it will basically always work the same no matter what devices you happen to use.

10/10

 

Portability

Let’s face it, using cloud services inherently means that you are handing over your data for safe keeping with a 3rd party that you NEED to be able to trust. Should you decide that you need to walk away with your data, whatever your reasons may be, that you can.

Further, you may simply want to store offline backup copies for your own use and or sanity.

Bookmark Ninja has you covered and makes this super easy! You can do this by accessing the Tools menu where you will find a couple of options.

The first method is to just have a list of selected bookmarks emailed to you and this seems like a nice way to have a relatively short list of URLs sent to you (or somebody) for reference. I believe you can export all your bookmarks this way too but that’s potentially a really long email and not the most efficient for migration though excellent for sharing URLs with friends, colleagues & customers via email!

The second method is to download an html file. This file can contain all your bookmarks and is ideal for migrating your data or simply maintaining an offline personal backup because essentially all modern browsers these days can import bookmarks via an HTML file.

Further, your organization structure is maintained via tabs becoming folders and any tags you create and add to a bookmark are preserved via this method as well. Effectively this means that there is ABSOLUTELY NO PENALTY for storing your bookmarks with Bookmark Ninja.

In fact, the file I got back from them did a better job of organizing my bookmarks when I imported it into Firefox than I have ever personally done!

I am inclined to give this a 10 out of 10 as well however to be fair I’m going to remove half a point because I know there is 1% of you out there going “but I wanted…” and insert your specific file format (txt, json, csv, xml, xls, xlsx, ods, pdf, sql, etc.) and while I agree conceptually it would be nice to have other file formats available should you want them, 99% of users are really going to prefer the HTML format because all browsers will import it and that 1% that wants something different have the unique technical skills required & the tools available to easily read, parse and convert the data to fit their needs from the base HTML file anyway.

Perhaps Bookmark Ninja could decide to offer other formats should their user base ask for it but to be fair to Bookmark Ninja and most users I am going to give Portability a 9.5 out of a possible 10.

9.5/10

 

Account Recovery

I also examined the password recovery system. I didn’t actually lose my credentials but I went through the process anyway to see how well it worked.

It’s been my experience that building a good account recovery system is easily overlooked by even skilled developers, which can be a real pain for end users!

Here again, Bookmark Ninja gets it exactly right!

The Ninja server sent me an email promptly with a link to reset my password and in just a couple of minutes I was back accessing my bookmarks.

Overall very smooth and exactly as expected.

10/10

 

UI/UX

If you spend any amount of time at all reviewing either Jozsef’s Blog or the Bookmark Ninja Blog its easy to see that creating a seamless user friendly UI & UX is a core concern of the Bookmark Ninja Dev team and recently Jozsef even blogged about some updates to their UI & UX (The UI And UX Design Of The Advanced Search) where he describes the latest set of improvements to Bookmark Ninja and I believe the Ninja Dev’s will continue to innovate so long as there remains anything they can improve upon for their customers!

Bookmark Ninja sports a modern Flat Design which many of my regular readers will know is not my personal favorite, (though I’ve implemented it plenty of times myself 😛 ) however if you are going for the flat look in your own projects you would do well to follow in the footsteps of Bookmark Ninja because they strike the right balance of simplicity and usability.

Everything you are going to want to do is available and easily found without any superfluous visual cruft I.E. the “Goldilocks Principle” as it applies to design where you strive for “not too much & not too little”, but “just… right” which in practice can be difficult to achieve without a sincere and concerted effort.

The menus and interfaces themselves are various shades of whites & grays which present a clean experience and at times is mildly reminiscent of Apple, though personally I found myself wanting just a bit more color and contrast on the screen to make navigating menus and pages easier and more pleasant.

Ninja does offer you the ability to apply color to category headings which is a great feature that helps to differentiate categories especially while visually sorting and finding bookmarks on the Dashboard interface. My only gripe here is that there is a small list of colors to choose from and either adding more colors to that list or adding an RGB color wheel (or RGB HEX field) would expand personalization and further enhance the user experience.

Bookmarks & Categories are treated as objects and you can simply drag and drop to move them around which is rather wonderful in how easy and direct that makes organizing and managing your links.

Before I proceed to render my score however I feel like an additional point should be addressed.

One of my regular readers had read my article where I mentioned that I was reviewing Bookmark Ninja and decided to check it out for themselves and actually subscribed to the service.  Since they had subscribed I decided to “pick their brain” a little and get their impression of Bookmark Ninja.

Generally they gave Ninja high marks as well but they raised a valid though minor point that I think speaks to the importance of making UX a serious focus when developing a product and how even when you get it right it can be difficult to address every issue.

The point they raised to me was simply having a little confusion over the proper formating when adding tags to a bookmark. Specifically they were uncertain if they needed to use hash marks # in front of the tag name or not.

Ultimately this makes sense as we all understand what the concept of a “tag” is but the actual implementation can very depending on what “tag” meant to the design team.

Of course I directed them to the Ninja blog search for all references to tags, where they show examples without hash marks.

Further, I believe tags are likely searched in the database using some form of “greedy/inclusive” wildcard or %LIKE% search (assuming SQL or an equivalent if not) so if you create tags using hash marks it shouldn’t really matter as %mytag% or *mytag* should match #mytag however I think it would probably be simpler not to use them.

So getting back to the importance of UX which stands for (User eXperience) I believe this type of confusion could be resolved quite simply by adding a “tool tip” component on the form fields or by having a dedicated help button somewhere nearby that either displays relevant help or links directly to the related articles on the blog.

To be fair, the blog covers every topic under the sun and information is organized so that is easily found. The issue here is one of not having to stop your current action to go find information, hence my suggestion of adding tooltips or perhaps even a “local context”  help popover, modal or even popup to offer prompts, examples or explanations.

Overall I was highly satisfied with the UI & UX offered by Bookmark Ninja and I’m going to give them an 9 out of 10.

9/10

 

Support

When I think of support as an issue by which to rate or score a service like Bookmark Ninja I tend to define it as an issue that requires me (the user) to submit a support request because I cannot resolve the issue by myself.

Using this definition I never actually ran into any issues that would allow me to request support.

Which if you think about it is actually a great indication of how well Bookmark Ninja is designed (though see the next section Reliability) and I was unable to think of something convincing that wasn’t actually occurring that would allow me to submit a support request.

Further I wanted to avoid any “special treatment” that might unduly affect my overall opinion of their service though I am unaware if anyone on the Dev team other than Jozsef who was actually aware of who specifically was reviewing their service.

As a result I am unable to objectively say how responsive the Bookmark Ninja team is with support requests, however I can honestly say that if my email correspondence with Jozsef is any indication of how he and his team conduct support then they will be prompt and courteous without fail!

I’m going to give this one a 10 out of 10 as well but with the expressed caveat that this is inferred.

If you decide to give Bookmark Ninja a try and find that you need support you can send an email to  support@bookmarkninja.com. If you end up submitting a support request go ahead and let me know how your experience went and I would be happy to include a summery here for the benefit of my readers.

10/10

 

Reliability

Despite it not being an issue worth submitting a support request over and fully acknowledging that I suppose I could have, I did encounter a weird behavior that occurred occasionally and seems to have more to do with browser security and executing Javascript code than anything Bookmark Ninja did wrong.

Essentially, every so often when I would use my browser bookmark to access my Bookmark Ninja account to create a new bookmark, the actual URL I wanted to save wasn’t transfered to the popup form and instead I would get an About:Blank link.

This was easily fixed by a second try or by manually copying the link and pasting it in the field. I don’t feel like this was a major pain at all and I am not even sure if there is anything the Ninja Dev’s can do to resolve this but because it did occur enough that I did notice it, I am going to ding Reliability 1 point.

9/10

 

Cost

Bookmark Ninja costs $1.99 a month (admit it, thats less than you spend on coffee every day!) and is billed annually, just $23.88 gets you an entire year of service!

10/10

 

So out of a possible 80 Bookmark Ninja scored 76.5 (that’s 95.625%) or an ‘A’ if I were handing out letter grades 😛

Bookmark Ninja offers an affordable way to simplify accessing and managing important links at work, home and on the go!

If you would like to try Bookmark Ninja you can get your Free 30 day Trial by using this link.

I don’t receive any compensation if you choose to use Bookmark Ninja however I do like and recommend their product and should they ever decide to implement an affiliate program I would certainly be willing to endorse Bookmark Ninja because I know my readers will benefit from it!

With that, everyone have a great day and I will see you all in my next post.

 

Much Love,

~Joy

Lets build an Auto Coding GIT Commit Bot

My friend Oliver approached me recently with an idea for a project that we could both work on together and after he explained the basic premise I was on board!

The initial concept was wouldn’t it be fun to have a bot that could publish your git commits for you? Which of course the only correct answer is “HELL YEAH!”!

Oliver, built a prototype and then had the great idea of using a Raspberry Pi to act as a sort of “Auto Commit Robot” and to test his newly minted bot he modified it to create randomly generated files to commit, and that’s when it hit us… There was a bigger story here…

One of the first uses for the bot we considered was the the idea that although most employers are reasonable and will look at the quality of your code, there are still some that look at how frequently you commit as a large indicator of how ‘driven’ or ‘voracious’ of a programmer you are, which if you think about it makes absolutely no sense at all!

So Oliver suggested, what if you were to just commit the random text he was generating to a private repo but have your commit data publicly available on your GitHub account, this would allow you to push commits every day (or at least on a schedule) and establish a history of regularly pushing commits which is exactly what he built! When he publishes his half of this project I will link to it here.

I highly recommend when it’s published that you head over to read his post and give it a like too! He’s also really nice so feel free to leave him a comment as well, I am sure he would appreciate it! 😉

Now this is where I come in. My portion of this project stems from the fact that not everyone has a Raspberry Pi (or Linux machine since for this purpose they are basically are one in the same) lying around and also not everyone pays GitHub to maintain private repos, so a lot of people are just using GitHub to store free public repositories.

So, my challenge is to turn his bot into a cross platform solution so that it also works with public repos because anyone looking past just the number of commits would discover the ruse of randomly generated data.

So I modified my scenario a little bit so that my challenge was clearer.

Consider the following: It’s the the new year and you are looking to “get a head start on commits”… or maybe you have a bet with a friend that you can make more commits to your GitHub than they can between now and next month… or maybe you just want to watch the Stock Market fall… Whatever the reason is you decide that you happen to need an auto commit bot…

Once I had a scenario I knew that in order to solve this challenge the data that Oliver’s bot was pushing to the repo had to pass a basic “visual test” ( if not a run test ) even if the results of the code was useless.

With that in mind I set to work designing my methodology.

    1. First, I make the following assumption… ANY line of code that is capable of passing an eval() MUST be good because it “runs” so if I can generate the code segments and make it look like real code (more or less) even if it essentially does nothing, well it’s that much harder to question the validity of the commit, isn’t it?
    2. Further, use shell_exec() to execute the git commands in a cross-platform way.

I could hypothetically generate unit tests using the ASSERT function (thinking in PHP here 😛 ) if I wanted to go that route, though I’ll leave that for you to add to your own implementation!

  1. In any case I set to work coding generate-functions-example.php

    generate-functions-example.php

     


<?php 


function RandomString($length) {		
    $output = '';
    $alphabet = 'abcdefghijklmnopqrstuvwxyz';
    $alphabet_length = strlen($alphabet);

    for ($i = 0; $i <  $length; $i++) {
        $output .= $alphabet[mt_rand(0, $alphabet_length - 1)];
    }
	
	return $output;
}


function SelectLanguageFunction($param) {		
	$output = '';
	$action = mt_rand(0, 38);
	
	// abs
	 if($action == 0){
		$output .= 'abs(' . $param . ')';
	 }
	 // acos
	elseif($action == 1){
		$output .= 'acos(' . $param . ')';
	 } 
	 // acosh
	elseif($action == 2){
		$output .= 'acosh(' . $param . ')';
	 }
	 // asin
	elseif($action == 3){
		$output .= 'asin(' . $param . ')';
	 }
	 // asinh
	elseif($action == 4){
		$output .= 'asinh(' . $param . ')';
	 }
	  // atan
	elseif($action == 5){
		$output .= 'atan(' . $param. ')';
	 }
	 // atan2
	elseif($action == 6){
		$output .= 'atan2(' . $param . ', ' . "($param + " . mt_rand(3, 7) . ')  * pi())';
	 }
	 // atan
	elseif($action == 7){
		$output .= 'atan(' . $param . ')';
	 }
	 // atanh
	elseif($action == 8){
		$output .= 'atanh(' . $param . ')';
	 }
	 // bindec
	elseif($action == 9){
		$output .= 'bindec(' . $param . ')';
	 }
	 // ceil
	elseif($action == 10){
		$output .= 'ceil(' . $param . ')';
	 }
	 // cos
	elseif($action == 11){
		$output .= 'cos(' . $param . ')';
	 }
	 // cosh
	elseif($action == 12){
		$output .= 'cosh(' . $param . ')';
	 }
	 // decbin
	elseif($action == 13){
		$output .= 'decbin(' . $param . ')';
	 }
	 // dechex
	elseif($action == 14){
		$output .= 'dechex(' . $param . ')';
	 }
	 // decoct
	elseif($action == 15){
		$output .= 'decoct(' . $param . ')';
	 }
	 // deg2rad
	elseif($action == 16){
		$output .= 'decoct(' . $param . ')';
	 }
	 // exp
	elseif($action == 17){
		$output .= 'exp(' . $param . ')';
	 }
	 // expm1
	elseif($action == 18){
		$output .= 'expm1(' . $param . ')';
	 }
	 // floor
	elseif($action == 19){
		$output .= 'floor(' . $param . ')';
	 }
	 // fmod
	elseif($action == 20){
		$output .= 'fmod(' . $param . ', 0)';
	 }
	 // hexdec
	elseif($action == 21){
		$output .= 'hexdec(' . $param . ')';
	 }
	 // hypot
	elseif($action == 22){
		$output .= 'hypot(' . $param . ', ' . $param . ' * 2)';
	 }
	  // is_finite
	elseif($action == 23){
		$output .= 'is_finite(' . $param . ')';
	 }
	  // is_infinite
	elseif($action == 24){
		$output .= 'is_infinite(' . $param . ')';
	 }
	 // is_nan
	elseif($action == 25){
		$output .= 'is_nan(' . $param . ')';
	 }
	 // log10
	elseif($action == 26){
		$output .= 'log10(' . $param . ')';
	 }
	 // log1p
	elseif($action == 27){
		$output .= 'log1p(' . $param . ')';
	 }
	 // log
	elseif($action == 28){
		$output .= 'log(' . $param . ')';
	 }
	 // max
	elseif($action == 29){
		$output .= 'max(' . $param . ')';
	 }
	 // min
	elseif($action == 30){
		$output .= 'min(' . $param . ')';
	 }
	 // octdec
	elseif($action == 31){
		$output .= 'octdec(' . $param . ')';
	 }
	 // pow
	 elseif($action == 32){
		$output .= 'pow(' . $param . ',' . mt_rand(2, 7) . ')';
	 }
	 // rad2deg
	 elseif($action == 33){
		$output .= 'rad2deg(' . $param . ')';
	 }
	 // round
	 elseif($action == 34){
		$output .= 'round(' . $param . ')';
	 }
	 // sin
	 elseif($action == 34){
		$output .= 'sin(' . $param . ')';
	 }
	 // sinh
	 elseif($action == 35){
		$output .= 'sinh(' . $param . ')';
	 }
	 // sqrt
	 elseif($action == 36){
		$output .= 'sqrt(' . $param . ')';
	 }
	 // tan
	 elseif($action == 37){
		$output .= 'tan(' . $param . ')';
	 }
	 // tanh
	 elseif($action == 38){
		$output .= 'tanh(' . $param . ')';
	 }
	 return $output;		 
}


function SelectOpperation() {
	$action = mt_rand(0, 5);
	
	// Addition + 
	 if($action == 0){
		 return '+';
	 }
	// Subtraction -
	elseif($action == 1){
		 return '-';
	 }
	// Multiplication *
	elseif($action == 2){
		 return '*';
	 }
	// Division /
	elseif($action == 3){
		 return '/';
	 }
	// Modulus %
	elseif($action == 4){
		 return '%';
	 }
	// Exponentiation **
	elseif($action == 5){
		 return '**';
	 }
}




function GenerateRandomFunction() {	
	
	// name the function
    $function_name = ucfirst(RandomString(mt_rand(2, 7)));

	// how many parameters should the function take?
	$number_of_parameters = mt_rand(1, 3);
	$parameters = array();
	for ($i = 0; $i < $number_of_parameters; $i++) {
		 $param_name = RandomString(mt_rand(2, 5));
		 $parameters[] = "$$param_name";
    }
	
	// build function comment
	$funk_header = '/**' . PHP_EOL; // open comment
	$funk_header .= ' * Does Something' . PHP_EOL;  // function description
	$funk_header .= ' *' . PHP_EOL; 
	foreach($parameters as $parameter){
		$funk_header .= ' * @param ' . $parameter . ' Add Description' . PHP_EOL; // parameter description
	}
	$funk_header .= ' *' . PHP_EOL; 
	$funk_header .= ' */' . PHP_EOL;  // close comment
		
	// build function
	$funk = 'function ' . $function_name . '(' . implode(', ', $parameters) . '){ ' . PHP_EOL;
	
	// do something with the parameters
	$funk .= str_repeat(' ', 4) . 'return ';
	foreach( $parameters as $key=>$parameter){
		$funk .= SelectLanguageFunction($parameter); // do something with the parameters
		if($key < $number_of_parameters - 1){
			$funk .= ' '. SelectOpperation() .' '; // add opperation
		}
	}
	$funk .= ';'; // add semicolon

	// build end of function
	$funk .= PHP_EOL . '}' . PHP_EOL . PHP_EOL;
	
	return  $funk_header . $funk;
}
/////////////////////////////////////////////


$number_of_files = 10; // how many files to generate

// create the files
for($f = 1; $f <= $number_of_files; $f++)
{
	$file = fopen('functions/' . RandomString(mt_rand(3, 7)) . '.php','w');
	fwrite($file, '<?php' .  PHP_EOL);
	for($i = 0; $i < mt_rand(3, 17); $i++){
		fwrite($file, GenerateRandomFunction() . PHP_EOL);
	}
	fclose($file);
}

This code could be simplified by making use of arrays of language functions rather than hard coding them as if/else statements however I’ll leave that for you to implement if you want.

If you run this code the you will get 10 PHP files that contain a random number of functions (that will eval()) in a sub-folder called ‘functions’ and each function will have it’s own comment header that lists the parameters and included a place for a description of the function and the parameters.

At first glance the generated code appears complicated and useful, here is an example function generated:

/**
 * Does Something
 *
 * @param $cl Add Description
 * @param $jw Add Description
 * @param $qobwa Add Description
 *
 */
function Icbekz($cl, $jw, $qobwa){ 
    return atan($cl) + hypot($jw, $jw * 2) ** atan2($qobwa, ($qobwa + 3)  * pi());
}

You could use this function like this:


echo Icbekz(4.3, 7.1, 22);

With the resulting output being: 3.470139188433

Of course this works but the output is essentially useless except for exactly what we want it for, generating functional non-sense code!

At this point whatever you set the $number_of_files variable to is how many new files you will get in the functions folder every time the script is run.

Schedule this PHP script to run on a Linux machine using a Chron Job or on a Mac using a Scheduled Job or Windows using Scheduled Task so that it runs weekly, daily or even hourly depending on your needs.

Now all we need is a cross platform script to use GIT to upload the new files to your repo once the other script is complete.

update-repo.php

// !IMPORTANT! Manually Pull/Create your repo before using this script 

// This script assumes that you keep your repos in your 
// user folder in a folder called repos.
// Adjust the $local_path variable as needed
$repo_name = 'test';
$local_path = '$HOME'."/repos/$repo_name/"; // linux & mac path example
// $local_path = "%userprofile%\repos\$repo_name"; // win path example
////////////////////////////////////////////////////////////////////////////////////
/// Put anything you want to occur before the push here.
////////////////////////////////////////////////////////////////////////////////////

// comment this line after the script has run once
// it updates the config file for the repo to tell git to store the
// credentials after you enter them once
shell_exec("cd $local_path;git config credential.helper store");

// Pull Repo
shell_exec("cd $local_path;git pull");

// THE FIRST TIME THIS SCRIPT RUNS YOU WILL BE PROMPTED
// TO MANUALLY ENTER CREDENTIALS HERE.

// Add all new files
shell_exec("cd $local_path;git add *");

// Push new data
shell_exec("cd $local_path;git push origin master");

///////////////////////////////////////////////////////////////////////////////////
/// Put anything you want to occur after the push here.
///////////////////////////////////////////////////////////////////////////////////

Schedule this Script to run after the generator and all the newly generated code will be pushed to your GitHub repo on a regular interval! 😉

You can find this project on my GitHub account for free here: Auto Commit Bot

I have also created a version that will generate complete class files and you can see examples of the output on my GitHub in the classes subfolder for this project. All of my $1 or greater supporters on Patreon can download a copy of this project that includes the Class Generator.

Don’t forget to follow my up-to-date & ongoing research over on my public BookmarkNinja tab!

With that, everyone have a great day and I will see you all in my next post.

Much Love,

~Joy

Happy New Years 2018 + Special Announcement

Hello everyone, HAPPY NEW YEARS!

Just wanted to check in since its been a few weeks… OK about a month 😛 since my last post so lets dish!

I caught (then got over) a cold and Xavier did the same 😛

I did TONS of year end house cleaning, with a little consulting mixed in there and I even found time to built a prototype app for a friend of mine who Identified an interesting potential market niche. He is going to demo the prototype soon and if all goes well we MAY use Electron which is a desktop cross-platform build tool to deploy it, though I would say we are far from a product we can actually bring to market just yet. I hope to discuss more about this in future posts so for now I will leave this as is.

I’ve also recently started collaborating with another developer (note to self: EMAIL OLIVER!) on a set of posts centering on a bot concept he had and we should be ready to publish that soon so I also don’t want to give too much about this topic away… and Ive also since started collaborating with this same developer on my writer/coder bot, I’m building out the bot API and he will be working on a front-end interface.

In regard to the bot, I managed to build out a functional language model (Bi-Grams & Tri-Grams). I’m hoping early this year to expand it up to and including Deca-Grams by adding and improving the abstraction in the bot class.

I expect we will have some real fun with this bot in the coming year because outside of writing and code, I want to see how well my bot could learn things like music generation and perhaps chat or natural conversation… we’ll see. 😉

There was also a New Years camping trip which was relaxing but not much to tell.

Which brings me to a special announcement…

Special Announcement

I have my first first sponsored content coming soon and I am really looking forward to telling you guys all about it! Of course I will reserve my review for the actual review post which I plan to publish in a month or two (depending on how everything goes) but I have a good reason for telling you now which I’ll get into in a moment.

It all started when I contacted Jozsef Torzan, a blogger I follow who hadn’t published in a few weeks and I was curious as to why he stopped and to see if he was okay.

Turns out everything was fine and he was deep in development on a whole series of updates and new features on his web service Bookmark Ninja (Ninja) which quite literally is the ultimate online bookmark manager and organizer!

In any case, after Jozsef described the new features I just had to try Ninja for myself and offered to do a review, Jozsef agreed and graciously extended to me a trial membership!

One of Ninja’s new features is the ability to publicly share a “tab” which is a collection of your bookmarks via a URL with anyone! This means I can give you guys a sneak peak into what I am working on by saving all my “current research” here and you also get a real world example of  how Bookmark Ninja can be used, a win/win if I ever saw one! 🙂

So, for the next month or two I would like you to use the link below to go to one of my public tabs which I am updating constantly so check this link frequently if you want a sneek peek to see the kinds of things I am researching for my work and posts.

Bookmark & Share This Link: https://www.bookmarkninja.com/share/25819465

All my posts are made possible by the generous contributions and hard work of my sponsors!

If you like my work please consider supporting me over on Patreon for as little as $1 a month

With that, everyone have a great day and I will see you all in my next post.

Much Love,

~Joy

Thanksgiving

It’s Thanksgiving again in the United States, a time when people come together to celebrate all the good things they have in life and these celebrations frequently include toasts over expensive champaign and speeches expressing thanks and love to those present.

It marks the unofficial start of “office party season” and often enough people regard  Thanksgiving simply as “turkey day” with the actual “celebration” centering on overeating followed by the inevitable stupor that results as a sort of pre-shopping ritual in preparation for the main event “Black Friday” and now “Cyber Monday” in more recent years.

Now in truth there is nothing wrong with trying to get a “deal” on gifts for loved ones, I am not chastising anyone for shopping!

I do have one question for you however… when was the last time you focused on the GIVING portion of Thanksgiving?

According to the UCIH  (The Unites States Interagency Counsel on Homelessness) my home state of California has a homelessness rate of 21%! That means that for every 10 people 2 are homeless and likely “unsheltered”, that’s absolutely insane!

Some of those 118,142 people are families, some of them are veterans, some are just people who lost everything as a result of the last economic downturn and simply never recovered!

Now before you start say things about “handups” rather than “handouts” I have another question for you… when was the last time you actually reached out and gave someone a “hand up”?

Regardless of your political beliefs, there is a really great quote by former U.S. President Ronald Regan that I like:

“We can’t help everyone, but everyone can help someone.”

~Ronald Reagan

Homelessness is NOT a partisan issue, poverty affects everyone regardless of your religious or political beliefs or lack there of!

Poverty affects the weakest members of our society and robs them of their dignity as human beings! It turns neighbors into social pariahs, people we don’t look at, people we step over, people we cross the street to avoid… “throw away people”.

It’s easy to be thankful for the nice things we have and it’s easy to share those nice things with the people we love, its a lot harder to look around our local communities, countries and the world at large and see people worse off than we are and exclaim “there but for the grace of god go I“!

With another economic bubble on the horizon we cannot “sit idly by” and do nothing!

My regular readers know I believe in the principle of “practicing what I preach” so I am going to donate half of all the money I receive on Patreon for the month of November towards solving the poverty issue my community is experiencing. I do not currently know which charity I will be donating to but I will post an update on Patreon with all the details once I have researched the available charities.

If you support me on Patreon for the month of November your patronage will be included in my donation however even if you can’t afford to help me make this donation I urge you to look around your community for people who are worse off than yourself and offer them a helping hand.

Now be completely clear, Patreon takes a fee for their services and there are additional transfer fees I incur when transferring money from my Patreon account to my bank or PayPal so when I  say “half” I mean I will donate “half” of the actual amount I receive once all transfer fees etc. have been paid and the money is mine to actually donate.

Happy Thanksgiving everyone!

Much Love,

~Joy

 

 

My Bots Are Coming

Xavier is at Grandma’s today…and I have a little time so let’s catchup! 🙂

I’ve been posting less on my blog, usually about once a week because my landlord is having some renovations done on my house and it makes sitting down to think very difficult with all the banging, not that I’m complaining though, it will be very nice when everything is done. 🙂

Anyway, when I have found time to sit and work I’ve been coding rather than writing, hence I create fewer posts.

So with that out of the way lets talk about what I have been up to!

Back on my post A Halloween Tale I mentioned that I had been working on an interactive animated crystal ball (like my Magic 8 Ball post but better!) but I had that system failure so I was forced to… “improvise” a post by using the bot project I have been working on quietly in the background to write a “spooky” story. Well, frankly that’s what I’ve been working on. 😛

Near the end of my post The impending Death of the Programmer I told you guys:

“Now imagine if you could build an API around the systems I describe in this article, it would generate trillions in profit and end the career of the typical programmer!

Once I complete my current dev cycle I will begin experimenting with implementing the system I describe above for myself and my followers.”

Well let’s just say I’ve been a busy little bee! 😉

I’m still training the bot to properly model written/spoken language rules before expanding into the more complex realm of code and software development, however I want to reiterate here, the important thing and the “take away” for today’s post:

“…my bots are coming…”

Now as I am sure you can understand there is quite a lot of work for me to do still so I am going to keep this post short and get back to it as they say!

 

Please Fund, Subscribe, Like, Share & Comment:

This post was made possible by the generous contributions of my sponsors on Patreon & anonymous contributions to my Bitcoin Tip Jar.

 

 

I will see you all in my next post!

Much Love,

~Joy

Create a free website or blog at WordPress.com.

Up ↑