Home
  • You are here:  
  • Bruce Moore's Tech Blog
Although not statistically complex, the ACLU use of R for litigation support in the family separation case was big.

RStudio::conf 2019

What a difference 12 years makes. My first R-related conference was the 2007 UseR! at Iowa State University in Ames, Iowa. It was about 300 people and almost all attendees were from academia. Fast forward to the 2019 RStudio::conf, with 1700 people, almost all of whom are from industry. The UseR! conference now draws about 1000, the RFinance conference draws hundreds, and the BioConductoR conference draws large crowds. The constant theme over this decade has been documentation and reproducibility of research; in 2007, a pharma developed and put the MS Word version of sweave into the open source world because they needed a MS Word version and wanted to contribute. At the 2011 conference in Warwick, RStudio and IDEs that made the use of sweave and then knitr much easier. 2015 in Aalborg brought discussions of Docker and portability/archivability into the discussion.

Reproducible Research Continues to be a Theme

Reproducible research has been a theme in R for a decade, and the RStudio::conf 2019 continued the theme with workshops and conference sessions on Markdown and Bookdown among other reproducibility- and repeatability-related topics. I have continued to stick to LaTeX due to inertia and the early lack of citation capability in Markdown, but that has changed, and it is time to make the switch for new work. Many of the new books in R are being produced in Bookdown rather than LaTeX.

Garret Grolemund’s talk R Markdown: the Bigger Picture pointed out the scope of the repeatability and reproducibility problems in research today and how using Markdown can help in documenting what you did and how you did it.

The proportion of published articles describing experiments that cannot be repeated due to lost code, documentation or data is a huge problem.
A large portion of current published research cannot be replicated. R Markdown can help with documenting data and procedures.

Using R for Family Reunification

Although much of the RStudio::conf presentations were oriented to the RStudio products and open source packages, many were not. One of the most compelling was Brooke Watson presentation on using R to clean up the total garbage data provided by the U.S. Government on the locations of children and parents who were separated at the border. Regardless of politics, it is clear that the government was unprepared for this policy and did not (does not?) have procedures in place to keep track of kids and their parents who have been separated.

Brooke Watson's (@BrookLYNevery1) presentation on family reunification showed a jumble of files provided during discovery.
The Department of Homeland Security provided jumbled data to the ACLU for child separation lawsuits.
Brooke Watson's (@BrookLYNevery1) presentation on family reunification showed that the Department of Homeland Security does not have systems in place to track children separated from their parents..
The Department of Homeland Security provided bad data to the ACLU during discovery in the child separation lawsuit.

RStudio Package Stickers Continue to be a Hit

The RStudio package stickers were a major hit in Brussels, and continue to be a big hit at R conferences. This has to be one of the best marketing ideas I have ever seen. Whenever they put out a new batch of stickers, there was always a scrum as people searched for ones that they didn’t have.

The RStudio hexagonal package stickers are clearly one of the best marketing ideas I have ever seen.
The RStudio hexagonal marketing stickers continue to be a hit at conferences.

R and Data Science are No Longer PhD Things

The Education track on the last day was poorly attended, but I think in many ways it was perhaps the most important for understanding the future, the future of R and the future of Data Science. Mary Rudis (@mrshrbrmstr) spoke on teaching R at the community college level and on improving data literacy in the general population. Her presentation talked about certificate programs that do not require a prior bachelor’s degree. Her presentation was followed by Carl Howe’s (@cdhowe) presentation on Teaching the Next Million R Users. Carl is RStudio’s Director of Education or something like that and it is clear that RStudio is looking not just at the professional Data Scientist, but at increasing data literacy in the general population.

Panel Discussion of Data Science as a Career

The conference concluded with a panel discussion that was effectively how to manage a career in Data Science. It has a lot of useful information, but failed to talk about how to survive being the messenger in “shoot the messenger,” which all Data Scientists need to know how to do. How I would answer that question will be the subject of a future blog post.

Knitting is Making a Comeback

When I was a kid, my mom knitted all of the time–it was the major form of her gift giving, and I still have most of the sweaters from my late teen and college years that theoretically still fit. Mom was quite proficient and could do intricate patterns while carrying on a spirited conversation, though if the conversation was too spirited, she might have to rip out a row and re-do it. During the education track on the last day, the woman in front of me did a major portion of a child’s sock during the session. It was pretty cool.

Knitting is making a comeback. A child somewhere is going to get some cool socks for a birthday.
The woman in front of me made major progress on knitting a childs sock during the education track on Friday.
Details
Hits: 1791
Turning on DNS caching in a router can improve DNS performance dramatically.

Choosing a DNS Server

For several years, I have used the Norton domain name server (DNS) servers as a way to avoid known phishing and malware websites. Unfortunately, Norton discontinued this service earlier this year so I had to look for new DNS servers, as I have been unable to find information on the security approach of the servers provided by my ISP. I also do not like the idea of my ISP selling my DNS lookup information to the highest bidder.

While using my ISP’s DNS servers while I looked around, I discovered that the Norton servers had become really slow, and that browsing performance was much better using a faster DNS server. In looking for DNS benchmarking software, I found NameBench which is available as on Windows via Cygwin and on OS X via MacPorts.

The first run using NameBench was to compare two local caching DNS servers built in to routers, and to compare OpenDNS with Google DNS servers. Figure 1 shows that configuring a caching nameserver on you local router makes a HUGE difference; if your router does not offer this feature, get one that does.

Figure 1. The local caching name server built in to one of the routers (192.168.1.1) is clearly much faster than direct calls to the name servers, and is clearly a critical feature.
The namebench DNS benchmarking tools shows that a local caching DNS server is critical.

After looking at the importance of a caching DNS server, it is then worth comparing the performance of other public DNS servers, as shown in Figure 2. For this test, I used only DNS servers that do not appear to track requests; this is why the Google DNS servers were excluded from this test.

Figure 2. OpenDNS was the fastest of the non-tracking DNS servers in this test, but it does not provide blocking for phishing and malware without a paid subscription.
The namebench DNS benchmarking tools shows that OpenDNS is the fastest non-tracking server for this location on this day.

The sections that follow describe installing NameBench and a review of some of the open name servers available. NameBench is relatively old and appears to have some Python 2.7 dependencies that may present problems in some environments.

Installing NameBench

Installing Namebench is easy if you have Cygwin or MacPorts installed, but difficult if you do not.

Installing NameBench on Ubuntu

To install NameBench on Ubuntu, use the command

sudo apt-get update sudo apt-get install namebench

It is that easy. Start it from the launcher.

Installing NameBench on OS X

On OS X, you will need to first install MacPorts, which is not trivial; if you do not have MacPorts installed, look for another DNS benchmarking tool. To install it use

sudo port selfupdate sudo port install namebench

Installing NameBench on Windows

NameBench is available under the Cygwin setup.exe installer.

Comparison of Selected Open DNS Services

Comodo

Comodo is a security and SSL certificate provider that also provides an open DNS service with malware and phishing blocking. For my connection, it is slower than OpenDNS, but with DNS caching turned on in my router, this is not a big issue.

OpenDNS

OpenDNS was one of the early non-ISP DNS services, and was purchased by Cisco in August, 2015. Although it offers adult content filtering free, getting malware and phishing filtering is a for-fee service.

Cloudflare

Cloudflare is a non-tracking DNS service, and claims to be the fastest. It does not provide phishing and malware filtering.

Verisign

Verisign is a well-known SSL certificate vendor that also offers DNS and other services. Verisign claims not to sell your DNS lookup data, but does not make any statements about blacklisting phishing and malware domains.

Google

Google offers a fast public DNS service, but makes no statements disclaiming tracking, nor does it have blacklist for malware and phishing domains.

Conclusions

This article only gives a sampling of the options for domain name services. Before choosing a DNS service, make sure to test the performance at your location.

Details
Hits: 1843

New Privacy (GDPR) Features in Joomla 3.9

The European Union’s (EU) General Data Protection Regulation (GDPR) went into effect in May, 2018 and has resulted in major changes in privacy administration within the Information Technology world in general and web business operations in particular. Joomla release 3.9 (November, 2019) indroduced a number of changes that immediately provide tools to help with compliance and APIs so that extension developers will be able to easily provide compliance tools.

What is GDPR?

Before talking about the new privacy features in Joomla 3.9, it makes sense to talk give an overview of GDPR. First, neither this article nor Moore Software Services provide legal advice on whether you are subject to GDPR, whether or not your site is compliant, or any other legal advice. This article describes new privacy features in Joomla; whether these are sufficient for your compliance needs is beyond the scope of this article. GDPR became effective in May, 2018 and resulted in a large number of emails where businesses asked customers to confirm consent for data tracking related to email newsletters and other customer relationships. There are many good web articles on the history of GDPR so there is not point in repeating the history here. Suffice it to say that the regulation is overdue and that most businesses have struggled to comply with the basic requirements of the regulation.

Requirements of GDPR

The regulation is complex and has many pages, but is based upon a few simple principles:

  • Consent
  • Reporting to user of what data a company has pertaining to the user
  • Must remove data about a user upon user’s request
  • Must report data breaches to users with 72 hours of discovery
  • Must maintain records of processing of user data.

These represent significant changes from the way most web businesses have traditionally operated and will require significant work for many firms.

The Penalties for Failure to Comply are Draconian

Failure to comply is fundamentally a bankruptcy issues for most companies and is the greater of

  • €20 million
  • 4 percent of annual global revenue.

Ignoring GDPR is not an option.

Do US-based Companies Need to Comply with GDPR?

The short answer is yes for most companies and web businesses. If any of your customers are EU citizens and use the site at home or while in the US, you probably need to comply. If any of your customers are non-EU citizens, but use your site while in the EU, you must comply. Consult an attorney. Whether it will be enforced heavily for smaller businesses is an open question, but given that the California Consumer Privacy Act of 2018, the Chicago Personal Data Collection and Protection Ordinance and other US jurisdictions have implemented similar legislation, it is probably a good idea to work toward GDPR compliance; GDPR appears to be the most strict, so complying with it may make it easier for you to comply with the hodgepodge of regulations developing outside the EU.

Joomla 3.9 Helps with GDPR Compliance

The new privacy features and APIs in Joomla help with tracking consent, responding to user requests for information, and maintaining processing records. The sections that follow take a user-interface approach to the new features rather than a functional approach. There are five major user interface additions for the new GDPR privacy functions:

  • Privacy Dashboard (under the Users menu)
  • Privacy User Action Log (under the Users menu)
  • Privacy menu item types (under the Menu menu)
  • Privacy plugins (under the Extensions->Plugins menu)
  • Privacy Global Configuration options (under the System->Global Configuration menu)

The GDPR Privacy Dashboard is Under the User Menu

The User menu adds new Privacy and User Action Log options as shown in Figure 1. Going to the Privacy menu option shows the dashboard (see Figure 2) where you can get an overview of the information requests an other compliance status items for your site. The Requests option in the dashboard (see Figure 3) shows the number and status of user’s requests for a report on the data pertaining to the user plus a work flow for processing requests.

The most powerful addition in Joomla 3.9 is the addition of API features for extensions to integrate with the core privacy functions. Figure 4 shows the privacy-enabled extensions reporting back what privacy features they have implemented.

The last option in the privacy dashboard is the report on the status of user consents (see Figure 5).

Figure 1. Joomla 3.9 introduces the Privacy and User Actions Log menu options to help with GDPR compliance.
Joomla 3.9 introduces the Privacy and User Actions Log menu options to help with GDPR compliance.
Figure 2. Joomla 3.9 introduces the Privacy Dashboard to give a summary of compliance issues on your site.
Joomla 3.9 introduces the Privacy Dashboard to give a summary of compliance issues on your site.
Figure 3. Joomla 3.9 adds a workflow for processing user requests for their data on your site, and escalation tools for overdue requests.
Joomla 3.9 adds a workflow for processing user requests for their data on your site, and escalation tools for overdue requests.
Figure 4. Joomla 3.9 introduces privacy APIs and easy reporting on what privacy capabilities each extension has implemented.
Joomla 3.9 introduces privacy APIs and easy reporting on what privacy capabilities each extension has implemented.
Figure 5. Joomla 3.9 introduces a list tracking consent for data collection for each site user.
Joomla 3.9 introduces a list tracking consent for data collection for each site user.

The GDPR User Action Log is Under the User Menu

GDPR requires that you keep a log of how user information is processed. The User Action Log (see Figure 6) under the User menu provides this capability. It will probably be very helpful for problem diagnosis in addition to compliance.

Figure 6. Joomla 3.9 implements a log of processing actions for each user&rsquo.s data.
Joomla 3.9 implements a log of processing actions for each user&rsquo.s data.

New GDPR End-User Forms are New Menu Item Types

To implement the user interface for the new privacy capabilities, a new menu item category, Privacy (see Figure 7) has been introduced along with three new menu items types (see Figure 8).

Figure 7. Joomla 3.9 adds the Privacy category to menu items types.
Joomla 3.9 adds the Privacy category to menu items types.
Figure 8. Joomla 3.9 adds three menu item types for GDPR privacy funtions.
Joomla 3.9 adds three menu item types for GDPR privacy funtions.

GDPR Plugins for Logging

To make the new privacy functions work, Joomla 3.9 adds several new plugins (see Figure 9), one of which requires some configuration. The consent plugin shown in Figure 10 requires administrators to enter a short version of the privacy policy along with a link to an article containing the long version of the privacy policy. You can also set up the user consent to expire, requiring a new consent. This is helpful whenever the privacy policy changes; you set the expiration and then all users will be forced to update their consent.

Figure 9. Joomla 3.9 adds several plugins to implement new GDPR privacy compliance capabilities.
Joomla 3.9 adds several plugins to implement new GDPR privacy compliance capabilities.
Figure 10. Joomla 3.9 introduces a plugin to track user consent for data storage.
Joomla 3.9 introduces a plugin to track user consent for data storage.

GDPR Privacy Options in Global Configuration

The final user interface change for the Joomla 3.9 privacy enhancements is a Global Configuration category Privacy that now contains one item for the number of days before a user request for data is escalated to URGENT status.

Figure 11. Joomla 3.9 adds a global configuration option for the number of days before a user data request is escalated to URGENT status.
Joomla 3.9 adds a global configuration option for the number of days before a user data request is escalated to URGENT status.

Conclusions

The privacy extensions in Joomla 3.9 do not provide everything you need for compliance with GDPR and other privacy regulations, but they do provide a way for extensions developers to add capabilities and make it easier for webmasters. Over the next two or three years, extension developers that do not implement privacy features will have a much more difficult time selling their extensions.

Details
Hits: 1398

Upgrading to Joomla 3.8, 3.9, 3.10 and 4.0

Since the release of Joomla 3.8 in September of 2017, the Joomla community has been preparing for two major technical upgrades: moving from PHP 5.6 to 7.x and from Joomla 3.x to Joomla 4.0. PHP 5.6 will cease to get security updates after December 31, 2018, while PHP 7.0 will cease to get security updates after December 3, 2018, so it is imperative that webmasters move their sites to PHP 7.1 or 7.2 in November, 2018. Similarly, the security fixes in Joomla 3.8.13, make it wise to get to at least that software level as of this writing and move to 3.9 and subsequently 3.10 as soon as possible. All of these upgrades have higher than normal testing requirements; this article aims to help with preparation for these migrations.

Joomla 3.8 included a number of changes to provide support for Joomla 7.x and is required in order to upgrade to PHP 7.1 or PHP 7.2.

Moving to Joomla 3.8

Because of the changes to support PHP 7.x, Joomla 3.8 ended up requiring changes to many templates, template frameworks and extensions. Moving early, as I did, required a lot of work, but today most vendors have updated everything so there is not much work required. Here are the steps for upgrading to Joomla 3.8:

  • Backup Joomla
  • Update your template framework. If there are no updates after September 2017, you will probably be forced to change templates and template frameworks.
  • Update your template. If there are not template updates after September 2017, you will probably be forced to change templates.
  • Update your extensions.
  • Update Joomla.
  • Test.

You will probably find that some things are broken. If so, you may need to replace functionality with a different extension, as most extensions that have support have already fixed any problems related to the upgrade to Joomla 3.8 and the related upgrade to PHP 7.x.

Moving to a New PHP Level

Most web hosting firms use CPanel to provide a graphical user interface (GUI) for system administrator. CPanel introduced MultiPHP support that allows each shared host to run a different PHP level and configuration which makes it possible to move to more recent software in an orderly way. If you have a Virtual Private Server (VPS), you have a lot of control over how PHP packages are provisioned, but if you are on a typical shared host you do not. Doing some research on your current configuration before starting. With this in minde, here are the suggested steps to migrate to a higher PHP level:

  • Verify that you are on at least Joomla 3.8.
  • First do research to identify the PHP modules that you need.
    • Look at the PHP requirements for Joomla, and make a list of the PHP packages are listed in Technical Requirements. “Zlib support” means that the Zlib PHP module mod_zlib must be installed. Make a list of all of the modules that Joomla requires. Do not forget mod_ssl for HTTPS support.
    • Look at the PHP requirements for the Joomla extensions that you use. Some may require PHP Exif support for image metadata, other Joomla extensions my have other unique requirements.
    • If you have a VPS, list the modules currently selected.
  • Provision the new PHP release in the EasyApache section of CPanel
    • If you have a VPS, use the list of PHP modules that you need to provision the new level of PHP.
    • If you do not have a VPS, you will need to call your hosting firm’s technical support to have them confirm that the various modules have been included.
  • In Cpanel, set the php.ini file values for the new PHP level to match the values in your current configuration.
  • In CPanel, use MultiPHP to change PHP levels.
  • Test. If you have problems, turn on debugging in the Joomla configuration to see some of the error messages. It may take two or three tries to get all of the PHP modules that you need.

Moving to Joomla 3.9, 3.10 and 4.0

Moving to Joomla 3.9 required another round of template and template framework updates for all sites along with updates to a number of extensions. Otherwise, this was an uneventful upgrade.

Moving to Joomla 3.10 is planned to be an uneventful upgrade as it is intended to be a transitional release to provide security fix support for two years while site owners and developers fix and test extensions on 4.0. Although 4.0 has some new function, most of the changes are updates to outdated and unsupported dependency libraries; it is likely that a lot of extensions will need maintenance to work with new levels of the various libraries. See the article Potential Backward Compatibility Issues in Joomla 4. In any case, do not move to 4.0 without testing extensively first.

Details
Hits: 1431
Upgrading a Raspberry Pi is easy with a little preparation

Upgrading a Raspberry Pi from Wheezy to Jessie

I recently realized that one of my Raspberry Pi’s running Network Caller ID (NCID) was running an unsupported version of Debian/Raspian and needed to be upgraded. The first instructions that Google listed omitted some steps that I knew should be included, so I have put together this page with the procedure that I used. The full (and best) document is the official guide on Debian.org.

Backup the Working System

Before doing any upgrade, you should do a backup of the system. In the case of a Raspberry Pi, it is easy to clone the SD, pull the original and upgrade the clone. Since this was a very minimal installation, I chose to backup only the /etc/ncid/ directory to the Subversion server that I use to keep modified configuration files. An ancient saying among system programs was “tape is cheap (no matter how much it costs).” You never need a backup until you do, and when you do, you really need it badly.

The upgrade process will remove packages that do not show up as having dependencies, and some packages may have unresolved dependencies. You may get into a situation where you have not recourse but to fall back and do a painstaking package-by-package upgrade.

Make Sure You Have Enough Free Space

Some upgrade programs are smart enough to check for the necessary free space to complete the upgrade. The Debian approach described here is not, and I know from experience that it will get ugly if you do not have enough free space. Make sure that you have plenty of free space before start; use the command

df -h

to find out how much free space you have available. If you are above 80% or so, you will probably have problems. Keep two windows open and at each step in the process run a df -h command to make sure that you have more space than apt-get updateindicates that it will use.

user@thisrpi /etc/apt/sources.list.d $ df -h Filesystem Size Used Avail Use% Mounted on /dev/root 13G 3.8G 8.4G 31% / devtmpfs 214M 0 214M 0% /dev tmpfs 44M 240K 44M 1% /run tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 87M 0 87M 0% /run/shm /dev/mmcblk0p5 60M 20M 40M 33% /boot

Make Sure You Have Time

The upgrade process may take a few hours, depending upon which Raspberry Pi model you have, the speed of your connection, and the complexity of your installation. Make sure that you have time to do this, and something else to work on while watching the console messages scroll past.

Upgrade the Current Release

The first step is to upgrade to current maintenance on the old release:

sudo apt-get update sudo apt-get upgrade

The next step will remove some unused dependencies. This sometimes breaks things if you have optional software installed outside the standard installer, and sometimes breaks dependencies even for software that was installed with the installer. Make sure you have a backup. This step is not absolutely necessary, but can reduce the upgrade time in later steps.

sudo apt-get dist-upgrade sudo apt-get autoremove sudo apt-get clean

Upgrade the /etc/apt Configuration Files

Next, upgrade the /etc/apt/sources.list file to change wheezy to jessie, then do the same for all files in the /etc/apt/sources.list.d directory.

Upgrade the System

This is the point of no return; make sure that you have a backup and sufficient free space. To upgrade the basic system, use the commands

sudo apt-get update

Some items may not update; raspberrypi.<collabora.com has been deprecated, so you will need comment it out in /etc/apt/sources.list.d/collabora.list. When you have resolved all repository issues so that you do not get errors, proceed with

sudo apt-get upgrade

When this is complete run the commands

sudo apt-get update sudo apt-get upgrade sudo apt-get dist-upgrade sudo apt-get autoremove

Reboot and Test

At this point, you should be ready to reboot the system. Test all of your applications to make sure that they are working properly–especially those installed outside the installer.

Details
Hits: 1651
  1. Configure Network Caller ID (NCID) to work with an OBi202
  2. Writing an Article Using the Joomla Frontend
  3. Upgrading from Ubuntu 14.04 to 16.04
  4. Using CiviCRM on Joomla