R Reporting Part 2: Choosing the Right Markup for the Task
This is the second of a series of article on how to use R, RStudio and TexMaker to prepare presentations and batch jobs for automated reporting on a web server or Microsoft SharePoint server. The series is based upon the presentation that I did at the February 27, 2016 Dallas R User Group Meetup. Because the presentation was primarily a demonstration, there really isn’t a presentation to distribute; this series covers the topics from the presentation/demonstration. The series will eventually include the following articles as I complete them over the next couple of weeks:
- R Reporting Part 1: Tools
- R Reporting Part 2: Choosing the Right Markup for the Task
- R Reporting Part 3: Using Rhtml for Batch Web Reporting
- R Reporting Part 4: Using Markdown for Interactive Presentations
- R Reporting Part 5: Using LaTeX/Beamer for PDF Presentations
- R Reporting Part 6: Using LaTeX for PDF Articles
- R Reporting Part 7: Converting R Documents to E-books
- R Reporting Part 8: Using LaTeX to Create Posters
The series of articles describes the process for daily batch jobs that generate the Daily Econometric Graphs web page which includes links to the same econometric charts in several formats, all generated through the same R code:
- Econometric graphs in PDF form for use as slides on a projector
- Econometric graphs in PDF form for printing
- Econometric graphs in EPUB format
- Econometric graphs in Kindle AZW3 format
- Econometric graphs in A0 poster format
All of the examples are based upon the knitr R package; you should reference the knitr documentation, as this article is not a replacement for the knitr documentation.
Example source is available in images/documents/econometric_source.zip.
Choosing the Right Markup Language for the Task
There are several markup languages in which you can embed R code to make it easier to write and maintain presentations, articles and web pages. This article discusses when each of the various tools work well, and when they do not. There are three basic markup languages that are available in RStudio and other R-based integrated development environments (IDEs), with a couple of variants for each:
- Markdown
- Presentation
- Document
- MS Word Document
- HTML
- LaTeX
- Presentation
- Article
- Poster
- E-book
Table 1 shows a simplified summary of the features for each of the markup languages. The sections that follow discuss the features of each of the markup languages in more detail. Generally speaking, Markdown is the easiest to learn and is good for documents that are not going to be printed, Rhtml is the best choice for something that will be included in a web page, while LaTeX is best for something that is large, and complex. LaTeX was originally developed for mathematical and scientific publishing and has tremendous capabilities for automated Table of Contents maintenance, cross referencing, citation, bibliography maintenance and indexing.
| Markup Language | Presentations with Static Graphics | Presentations with Interactive Graphics | Presentations with Running Table of Contents | Include Mathematics (Portable) | Include Mathematics (non-Portable) | Include Complex Tables | Print Articles with Table of Contents, cross reference and bibliograpy | Web Articles | E-book | Poster | MS Word .docx |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Markdown | Y | Y | Y | Y | Y | Y | |||||
| Rhtml | Y | Y | Y | Y | |||||||
| LaTeX | Y | Y | Y | Y | Y | Y | |||||
| LaTeX/Beamer | Y | Y | Y | Y | Y | Y | Y | Y |
Markdown
Markdown is by far the easiest of the markup languages and has some important advantages and disadvantages over the Rhtml and LaTeX:
- Markdown can output directly to a Microsoft Word document and is the best choice by far if this is your target; you can convert both Rhtml and LaTeX to
.docxbut it is not simple nor are the results necessarily pleasing. - Markdown can embed some of the HTML and JavaScript based interactive graphics. This is possible for Rhtml with some knowledge, but this is impossible in LaTeX.
- Markdown cannot do the automated Table of Contents, cross referencing, and other large-publication features possible in LaTeX.
Mathematics
Markdown can use MathJax to render mathematics using LaTeX mathematics syntax, but this is not necessarily portable from one machine to another. By default, MathJax retrieves JavaScript libraries from https://www.mathjax.org/; if the presentation machine does not have an Internet connection, the mathematics rendering will fail unless MathJax is installed locally on the machine. This is especially important to understand if you are using Markdown for a presentation at a conference; you may be able to transfer your presentation file to another machine, but the math may not render on that machine unless it has a working Internet connection.
If you are using Markdown for a presentation at a conference, you should generate a PDF version in case the conference machines do not have Internet connections. You can do this reasonable conveniently by changing the output type within the Markdown file.
Interactive Graphics
Markdown can embed most of the interactive graphic types, and is the best choice for interactive graphics unless you are specifically planning to incorporate the output into a web article. If you plan to include interactive graphics in a web article, make sure that you can use the iframe on the web server. For a variety of good security reasons, web site administrators do not give the authority to use this tag routinely; you do not want to get to the end of a long development project only to learn that you cannot get the necessary authorities to display your results.
Presentation Output
Markdown can output to three different presentation types:
- HTML/ioslides
- HTML/slidey
- PDF (LateX/Beamer)
You can put all three output types into you Markdown file and comment them out as needed. For interactive graphics, some types work well in one of the HTML formats, but not the other format, so you should probably experiment a little with your particular presentation. For a conference, you should always run a copy using Beamer output so that you have something that will always work on another machine.
MS Word Output
If you need to generate MS Word output (a .docx file), Markdown is by far the best choice, as you can easily change the output type to MS Word format and the output looks quite good with little tweaking.
Document Output
Markdown can generate PDF output–it uses LaTeX internally–but is probably not the best choice if you are doing a longer document and need cross referencing.
Rhtml
Rhtml allows you to create whatever you want in an HTML file and is the preferred choice if the plan is to include the document into a web article. All of the data analysis articles on tis web site were generated as Rhtml documents. You can include interactive graphics into an Rhtml document, but it requires an iframe tag which may not be available without special security authorities on the target web site.
Mathematics
Rhtml cans use MathJax for rendering math symbols using LaTeX syntax. For web use portability really is not an issue, so it is reasonable to use the MathJax CDN URLs to load the MathJax JavaScript files, though this can occasionally create problems if there your markup is somehow incompatible with a new version of MathJax. You can choose to install MathJax locally in this case.
Document Output
Although the table does not list Microsoft Word as an output type, you can import HTML into Word to generate a Word document.
LaTeX
LaTeX was created in the 1990’s to make it easier for academic math and science authors to create publication-quality output that included mathematics and to manage bibliographies, citations and cross references through the related BibTeX program. It is still far and away the preferred tool for all quantitative PhD dissertations and most publications; this is why RStudio and other tools make it easy to work with LaTeX. If you plan to use R in your career, you should develop at least a minimal proficiency in LaTeX, and especially the mathematics markup, as LaTeX math syntax is used by MathJax, which is the preferred way to markup mathematics and is used for Markdown and other tools.
LaTeX has document types for articles, dissertations, books, presentations, posters–everything that a math or science professor would need to do.
Beamer Presentation Document Type
The most widely used LaTeX presentation document type is Beamer, which has a feature for an easy to generate running table of contents that is difficult or impossible to do in Microsoft PowerPoint or in Markdown’s presentation formats. LaTeX/Beamer produces PDF output; if you plan to present on both 4x3 and 16x9 projectors, you will need to create PDF files for both sizes in order to get a good looking presentation on both projector types.
Beamer supports all of the LaTeX math syntax, but because the fonts are much larger, equations that work in articles and books may not work in a Beamer presentation. All of LaTeX’s cross referencing and citation capabilities work in Beamer.
Article Document Type
If you need to use extensive cross referencing and bibliography for a web article, the LaTeX article document type may have some advantages over the Rhtml format; there are translators that will take LaTeX input, compile it and generate an HTML file. Graphics do not necessarily work perfectly the first time, but this approach is probably the way to go if you need a lot of citations.
Poster Document Type
Many accedemic conferences have “poster sessions” where an author creates an A0-sized poster (roughly 33x47 inches or 841x1189 mm). If you need to generate a poster for a conference, a conference room display, a cafeteria display or any other poster, LaTeX’s poster capabilities may well be the easiest way to do this.
Summary
The remaining articles in the series discuss specific examples of the output types discussed above. The various articles show how to create the components of the Daily Econometric Graphs article on this web site.
- Details
- Written by Bruce Moore
- Hits: 2960
R Reporting Part 1: Tools
This is the first of a series of article on how to use R, RStudio and TexMaker to prepare presentations and batch jobs for automated reporting on a web server or Microsoft SharePoint server. The series is based upon the presentation that I did at the February 27, 2016 Dallas R User Group Meetup. Because the presentation was primarily a demonstration, there really isn’t a presentation to distribute; this series covers the topics from the presentation/demonstration. The series will eventually include the following articles as I complete them over the next couple of weeks:
- R Reporting Part 1: Tools
- R Reporting Part 2: Choosing the Right Markup for the Task
- R Reporting Part 3: Using Rhtml for Batch Web Reporting
- R Reporting Part 4: Using Markdown for Interactive Presentations
- R Reporting Part 5: Using LaTeX/Beamer for PDF Presentations
- R Reporting Part 6: Using LaTeX for PDF Articles
- R Reporting Part 7: Converting R Documents to E-books
- R Reporting Part 8: Using LaTeX to Create Posters
The series of articles describes the process for daily batch jobs that generate the Daily Econometric Graphs web page which includes links to the same econometric charts in several formats, all generated through the same R code:
- Econometric graphs in PDF form for use as slides on a projector
- Econometric graphs in PDF form for printing
- Econometric graphs in EPUB format
- Econometric graphs in Kindle AZW3 format
- Econometric graphs in A0 poster format
All of the examples are based upon the knitr R package; you should reference the knitr documentation, as this article is not a replacement for the knitr documentation.
Example source is available in images/documents/econometric_source.zip.
Software to Install
To use R for presentations and batch reports, there are a number of software applications that you will need install on your desktop and your web server. The sections that following describe the installation of the various packages that you will need. All of the software applications in this section are available on Linux, Windows and OS X.
R and Related Packages
First you will need to install R from CRAN. Install files and instructions are available on the various CRAN mirrors.
RStudio
RStudio is a popular integrated development environment (IDE) for R, although is not required for any of the presentations here, it makes a number of things very convenient. Other R IDEs are Emacs Speaks Statistics, which has the advantage of working with other statistical and programming languages. Eclipse users should look at StatET for R. The examples for this series are done in RStudio, but would work in the other environments with minimal modification.
Once you have R and RStudio installed, you will need to install knitr, and sweave for any presentation or reporting use, and you will need the fImport, ggplot2 packages to run the examples in this series of articles. Use the following command to install the packages:
install.packages(c("knitr","sweave","ggplot2"))
The HTML Editor of Your Choice
Although RStudio is a great IDE for R, it does not do a good job of HTML sytax highlighting and spell checking. Once you have the R portions of your code working well, you will want to use a dedicated HTML editor to do the writing, and HTML formatting in your Rhtml documents. You can use any editor that you want; Bluefish is available on Linux, Windows and OS X.
LaTeX
If you need heavy formating, tables of contents, figure cross references and bibliography management in your presentations and reports, you will want to use LaTeX. It was developed primarily for accedemic writing for math and science and thus does a very good job of handling mathmatical symbols and equations, automatic tables of contents, indexing, cross referencing, bibliography, and citation. It is a tag language like HTML.
In Linux, most package managers will allow the easy install of the TeXLive distribution, although not necessarily the most recent one. On Windows, MikTeX is the preferred way to install LaTeX, but you can also install it via Cygwin. On OS X, MacTeX and MacPorts are perhaps the most convenient ways to install the LaTeX distribution. You should install the Beamer package; it is not part of the default installation.
TexMaker
If will be using LaTeX for presentations and articles, you will want to use embed your R code in LaTeX documents; although RStudio has great capabilities for the R portion of this workflow, at the point that you start working on the writing tasks, you will want to begin using a LaTeX IDE. Texmaker runs on Linux, Windows and OS X; it allows you to run R using sweave and knitr in the same way that RStudio does, but has spell checking features that make working with the LaTeX document easier.
By default, Texmaker uses sweave as shown in Figure 1. For most uses today and particularly for the examples in this article, you will want to switch it to knitr by changing the Sweave command to
/usr/bin/Rscript -e "require('knitr'); knit('%.Rnw')"
as shown in Figure 2. The final configuration step is changing the Quick Build (F1) key to run Sweave/Knitr before running pdflatex as shown in Figure 3.
Secure File Transfer Utilities–scp, rsync or Something Else
For batch reporting through cron or some other scheduler, you will almost invariably need some way to transfer files between systems. Secure copy or scp is probably the most universal way to do this. It is installed by default on most Linux and OS X systems. On Windows, scp is available as part of Cygwin. In a corporate Windows environment, you should talk to you IT group about what tools to use on your network; in Windows environments, shared drives are a common way to handle file copies. Another alternative is rsync which routinely available on Linux; for OS X, it can be installed via MacPorts while on Windows it can be installed via Cygwin.
For scp and rsync, you will want to use ssh-keygen to allow secure connections without using passwords and potentially ssh-agent for additional security.
Optipng and Other Image Compression Tools
The PNG and other images that R generates are not compressed as fully as is possible. To speed up web pages, you will want to compress images before uploading them using optipng or some other compression optimization tool. Optipng is available routinely in Linux, Cygwin and MacPorts. To call it in R use
system("optipng images/figures/*.png")
where images/figures/*.png is the path to the image files that your R script created.
ImageMagick Image Resizing and Conversion Tools
For web applications, you will probably want additional image sizes for use in links that are specific for different social media sites. ImageMagick is the most convenient tool for converting and resizing images in a script. It is available for Linux, Windows (Cygwin) and OS X (MacPorts). To use it in R to create an 450 pixel image for use in Facebook or some other social media site in R code, you would use something like
system("convert images/figures/ncid_daily_plot-1.png -resize 450x images/figures/ncid_daily_plot-1_shrink.png")
system("optipng images/figures/*.png")
Calibre and latex2html for E-book Tools
To create ebooks in EPUB for most e-readers and AZW3 for Kindle e-readers, you will need latex2html (or some other LaTeX to HTML converter) and Calibre. LaTeX2html is not being actively maintained so it is not a good choice for a production environment, but it is available for all platforms.
Server Side Include Software on Web Server
If you are posting your R document to a webserver running a content management system like Wordpress, Joomla or Drupal, you will need an extension to enable server side includes. This will probably require higher administrative rights than is typically given to normal authors, so check with your CMS administrator before you start on a big project. On Joomla, Sourcerer is one of several extensions that allow server side includes. It uses the syntax
{source}
<?php include("images/interactive/econometric_charts_home_page.html"); ?>
{/source}
## Warning in file(filename, "r", encoding = encoding): cannot open file ## 'optipng images/figures/*.png': No such file or directory
## Error in file(filename, "r", encoding = encoding): cannot open the connection
- Details
- Written by Bruce Moore
- Hits: 6096
Current Month Phone Spam Call Blocking Effectiveness
Spam telephone calls from “Rachel at card holder services”, “your Google listing has not been optimized” and numerous others are an annoyance that has become a major problem. Stopping Rachel from Cardholder Services covers multiple ways to address phone spam, including
- Turning on call blocking at your telco
- Signing up for a call blocking service like NoMoRoBo
- Setting up your own call blocker using open source software (Network Caller ID).
This article is a daily job that reports the effectiveness of these methods during the current month on my residential phone line.
Month to Date Call Blocking Effectiveness
Figure 1 shows the current month to date call blocking effectiveness for NCID for both calls on the FTC List and the black list from previous spam calls, as well as calls that are either blocked by NoMoRoBo or passed through as spam. There is really no good way to identify calls that NoMoRoBo stops. Valid calls are omitted.
Installation to Date Call Blocking Effectiveness
Figure 2 shows call blocking effectiveness for NCID from the date of installation. In early 2016, the phone line was ported from Verizon to a VOIP provider; this broke the NCID installation, but also made NoMoRoBo more effective by delaying the first ring to the VOIP phone. This delay allows NoMoRoBo to block many calls before the VOIP phone line rings. With the improved NoMoRoBo efficiency, there was not a strong incentive to fix the NCID installation, and thus the missing data for much of 2016. Although NoMoRoBo is effective at blocking calls at one ring, the peak of 37 calls in October, 2018 caused me to begin blocking repeat offenders with filters at VOIP provider; this immediately reduced the number of calls that ring even once.
Related Articles
For additional information, you may be interested in other articles on NCID and stopping phone spam:
- Stopping Rachel from Cardholder Services covers multiple ways to address phone spam, including setting up an NCID server.
- Download and Format the FTC Robocall Complaint List for NCID shows how to download and format the FTC complaint list to give you a list of spammers before they call you.
- Using NCID on Two Phone Lines shows how to add a second modem to your NCID configuration.
Updated Thursday, April 16 at 06:00 PM.
- Details
- Written by Bruce Moore
- Hits: 5020
Daily Econometric Graphs
Good pricing and asset and liability management decisions require up-to-date relevant economic information. The econometric charts that follow are generated automatically in R using data pulled from the FRED2 data repository of the St. Louis Federal Reserve. For authoritative data, please refer directly to the FRED2 system.
- Econometric graphs in PDF form for use as slides on a projector
- Econometric graphs in PDF form for printing
- Econometric graphs in EPUB format
- Econometric graphs in Kindle AZW3 format
- Econometric graphs in A0 format
- Details
- Written by Bruce Moore
- Hits: 3307
Modifying a LaTeX (Beamer) Presentation
Most graduate students in disciplines that use a lot of math learn to use a text formatting tool called LaTeX ("la teck") because it is much easier to do math formulas and bibliographic entries when compared to Microsoft Word. You aren’t really a member of the the “club” unless you know LaTeX. I continue to use LaTeX for presentations because it offers a great running table of contents bar that I like, as shown in Figure 1. I’ve asked a number of Microsoft-certified people how to do that in PowerPoint, but no one seams to not a straight-forward way to do it, so I stick to LaTeX. Once you know how to use it, creating a presentation in LaTeX/Beamer is as fast or faster than doing the same thing in PowerPoint.
There are several Toastmasters Leadership Institute presentations that people have requested; this article is aimed at giving this group a way to edit and use the source to these presentations.
Install MikTex, TexMaker and Adobe Acrobat Reader
The first step in the process is to install several pieces of software:
- MikTeX, which is the most convenient Microsoft Windows distribution of LaTeX. For OS X, there are similar packages available.
- After you install MikTeX, make sure to install the Beamer package.
- TexMaker is an editor that is designed to make LaTeX easier to use. It has syntax highlighting, spell checking, and a lot of other tools built in to it. Because LaTeX works with plain ASCII files, you can use any editor, but I use TexMaker because it is available on Ubuntu, Windows and OS X, so I don’t have to learn multiple software tools.
- Adobe Acrobat Reader is the software that you will use to display your presentation on a projector once you have completed it.
- If you expect to work with PostScript images, you should also install Ghostscript and GhostView.
Modify the Presenter and Date
Once you have the software installed, open up the presentation_name.tex file in TexMaker. It will look like the screen capture in Figure 2–a little intimidating if you aren’t a programmer, but it is actually pretty easy to work. LaTeX has to “compile” your .tex file; press F1, and the compiled PDF will show up on the panel on the right. This is probably too small to really read it easily, so go to the directory where your .tex file is located and open the PDF file in Adobe Acrobat.
To modify the presentation, search on “Bruce Moore” and change all of the occurrences to your name; press F1 to recompile, and look at the presentation. If the date is incorrect, search on the date that is listed in the presentation and change it to the date that you need for your TLI.
Add a Slide
To add a slide for a bullet-ed list, add the following to the file:
\begin{frame}[fragile]{This is the Title on the Slide}
\begin{itemize}
\item This is an item in a bullet-ed list
\pause % Press the slide clicker to display the next bullet
\item This is the next item
\end{itemize}
\end{frame}
To add a slide with a graphic, add the following to the file and press F1 to compile:
\begin{frame}[fragile]{This is the Title on the Slide}
\includegraphics[scale=0.35]{graphic_file.png}
\end{frame}
If your graphic runs off the side of the page, adjust the scale parameter to make it smaller (or larger).
Compile Versions for 4x3 Projectors, 16x9 Projectors and Handouts
One downside for LaTeX/Beamer is that the PDF output is fixed dimension and won’t automatically adjust to the display resolution of your projector. You will need to compile versions of your presentation for both 4x3 projectors (generally 1024x768 resolution) and 16x9 projectors (generally HD or 1920x1080). To do this comment (% sign at beginning of the line) or uncomment (remove % at beginning of the line) the appropriate line at the very top of the file:
%\documentclass[xcolor=dvipsnames,aspectratio=169]{beamer} % Print version for presentation
%\documentclass[xcolor=dvipsnames,aspectratio=43]{beamer} % Print version for presentation
\documentclass[xcolor=dvipsnames,handout]{beamer} % Print version for handout printing
Use F1 to compile the source and then rename the output PDF file to something like presentation_4x3.pdf, presentation_16x9.pdf or presentation_handout.pdf as appropriate.
If the Presentation Will Not Compile
If the presentation won’t compile, first check for missing \end{frame} or \end{itemize} tags in your source. If that does not work, try adding [fragile] to the \begin{frame}{title} tag if it is missing. If that fails, Google the specific compile error codes and if you still can’t figure it out and you have a few days before the TLI, send me an email and attach your modified presentation file.
Websites with Useful Information
There are a number of web sites with tutorials on LaTeX and Beamer:
- Details
- Written by Bruce Moore
- Hits: 5948