Page 1 of 2

Ruby version review 2016

While upgrading ruby gem dependencies for gitdocs I decided that I would not upgrade activerecord to v5.x in order to keep support for Ruby v2.0. This allow gitdocs to work with the default ruby on most desktop operating systems.

The current state of Ruby versions are:

  • 2.3.1 and 2.2.5 are stable and supported
  • 2.1.10 is in security maintenance, EOL soon
  • 2.0.0 is EOL

Which suggests that dropping 2.0.0 support would be a good idea. However, gitdocs is a user facing desktop/laptop kind of program so what Ruby versions can I expect in that environment?

Operating System Ruby Version
Debian 8.6 "jessie" (stable) 2.1.5
Debian "sid" (unstable) 2.3.0
Fedora 24-23 2.4.8
OS X El Capitan 2.0.0
OS X Sierra 2.0.0
Windows with RubyInstaller 2.2.x (the suggested default)

(N.B.: I have included Windows here and gitdocs might run on Windows, but it has not been tested.)

Because of OSX I am going to hold off on dropping Ruby v2.0.0 support for now.

Notes for KWLUG October 2016: Tor Virtualization

At the KWLUG meeting on October 3 2016 Nik Unger spoke to us about his work emulating the Tor network at UWaterloo as part of the Cryptography, Security, and Privacy (CrySP) Research Group.

  • CrySP publishes and contributes to various software projects at https://crysp.uwaterloo.ca/software/
  • Nik's research focuses on secure messaging and private web browsing
  • he discussed the reasons why privacy is important
    • mentioned the counter-argument "If you have nothing to hide, you have nothing to fear"
    • however, there are many minor and incidental crimes (e.g., unknowingly buying a lobster which is little smaller than legal) which you could be guilty of
    • additional some rule break is necessary for social change. Many social and political changes we value could never have happend with perfect legal enforcement
    • privacy is a collective right as well as an individual right, where a perceived lack of privacy can chill speech and reduce individual autonomy
  • Privacy Enhancing Technologies(PET)
  • SSL/TLS protects communication content but not metadata
  • an anonymity network, like Tor, is intended to provide metadata protection
    • protects the user from their ISP or man-in-the-middle attacks
    • can optionally:
    • protect the identity of the service provider
    • prevent 3rd parties from knowing Tor is being used
  • the bigger an anonymity network is the better protection it provides
  • there are other anonymity networks too:
  • Nik went through an explanation of a path through the Tor network which consists of 3 relays:
    • Guard (knows the user and the middle relay)
    • Middle (knows the guard and the exit)
    • Exit (knows the middle and the destination
  • fewer relays provides obvious places for attack and more does not provide additional security

Virtualizing Tor for research

  • researchers want to run modified Tor to collect data or test changes but changing the real network can put real users at risk
  • Tor provides guidelines for research which suggestrs using virtual networks whenever possible
  • using real nodes can be a problem if researchers bring up a large number of nodes without declaring their ownership, result in a Sybil attack on the network
  • NetMirage is the project that Nik is currently working on
    • it is a C++ re-write of an older Tor virtualization system from CrySP, written in Python and Bash to improve performance
    • they found the performance bottle neck was actually the API for the Linux Kernel Namespaces
    • this is currently a hard problem in the kernel, and sovling might be a future part of their research
    • they choose not to use an existing python project called mininet because:
    • it was more complex then they needed
    • and does not communicate directly with the kernel API, which was a performance concern
  • Linux Kernel Namespaces (manpage) provide the building blocks for containerization
    • NetMirage uses the network namespace only
  • 500 node network is about the limit, due to the performance bottleneck during setup
  • once the setup is complete performance is only limited by kernel packet handling, so it is pretty fast
  • challenges they have encountered which building NetMirage include:
    • debugging is difficult because valgrind does not support the kernel namespaces
    • using name spaces requires using various kernel APIs which have different conventions

Notes For Indieweb Presentation at KWLUG

At the KWLUG meeting on September 12 2016 I gave a short introduction about the IndieWeb. I do not have any slides which are worth publishing but I will summarize what I presented and provide some links.

I think that the 2 of the core principles of the IndieWeb are: * owning your own data * use visible data (N.B.: There are other principles too too.)

The key technologies to achieve this are:

  • microformats, where semantic information is encoded directly into the HTML
  • webmention (e.g. a new simpler pingback), which allows notifications to be sent to other IndieWeb users

There are a variety of ways to combined these 2 ideas which are described in the building blocks page.

Now, with all of this Indiweb sites could talk to each other but are effectively forming a new social network that has not external connections. However, there are 2 ideas to address this:

  • first, POSSE (Publish (on your) Own Site, Syndicate Elsewhere)
    • this is preferred because your content still lives primarily on your own site
    • but it is pushed out, so you can still communicate with your friends and family using the systems that they prefer
    • you may also want to backfeed comments, replies, RSVPs, etc. from the silos that you POSSE to. This allow your site to present a more full view of the activity surrounding yours posts
  • second, PESOS (Publish Elsewhere, Syndicate (to your) Own Site)
    • the copy on your own site is not the canonical version with this method, which is not as good
    • still useful in making you less dependent upon the silo where you are publishing your content, as you always have another copy

Getting started

Get a name

  • traditional DNS (short and long if possible)
  • something distributed would be nice but not sufficient (e.g., NameCoin)

Choose a system to use

Follow some checklists to gauge your progress

Example scenarios for publishing posts

Here are the simple examples that I walked through in my presentation:

Posting an article about the KWLUG meeting

<article class="h-entry">
  <header>
    <h1 class="p-name">
      About KWLUG September 2016
    </h1>
    <p>
      By <a class="p-author h-card" href="http://andrewsullivancant.ca">Andrew Sullivan Cant</a>
      on <time class="dt-published" datetime="2016-09-12 23:30:00">2016-09-12</time>.
    </p>
  </header>
  <section class="p-summary">
    <p>
      Here are some notes about <a href="http://kwlug.org/node/1031">KWLUG September 2016</a>.
      Presenters include <href="/">me</href>, <a href="http://sobac.com/bjonkman/">Bob</href>, Kirk and more.
    </p>
  </section>
  <section class="e-content">
    <p>And here I can say all the things.</p>
  </section>
  <footer>
    <a class="p-url" href="http://andrewsullivancant.ca/2016/09/13/articles/about_kwlug_september_2016">
      permalink
    </a>
  </footer>
</article>

Steps to publish it

Making a comment on the meeting

<article class="h-entry">
  <a href="http://kwlug.org/node/1031" rel="in-reply-to" class="u-in-reply-to">
    KWLUG September 2016
  </a>
  <div>Hi I am a comment.</div>
</article>

Steps to publish it

Notes for KWLUG July 2016: Sync mail, calendars and contacts

Chris Irwin described his recent experiments with mail/calendar/contact synchronization to KWLUG on July 4 2016. He discussed his goals, the options he considered, and what he is using currently.

  • Chris is only creating a system for a single user or small family of users
    • multi-user support is not necessary
  • baikal is the calendar server with a very simple user interface
  • Chris has mostly moved back to google
    • on android, even if sync is disabled, it still puts things into the primary google calendar and not the self-hosted one
    • this makes apps misbehave, such as things like the Cineplex app which pushes events into an unexpected calendar
    • pros
    • first class on android
    • they are their own standard eco-system
    • cons
    • does not really support standards so other clients will not work
  • on the desktop he is using the following software:
  • Chris used to use offlineimap but encountered sync issues every few months
  • imapsync has been working well for 3 years now. Well enough that Chris has to remind himself about how to use it for the presentation
  • mail is stored locally in maildir to it can be accessed by multiple clients (e.g., mutt, Gnome evolution)
  • appears that Thunderbird might be getting maildir support soon too (it is mentioned in the CHANGELOG, but not officially announced)
  • Chris suggests starting with a simple mutt configuration
    • slowly add useful things as you come across things you need
  • he also puts maildir into git and sync it to his gitlab server
  • security concerns
    • protection from someone walking about with this laptop, and really nothing else
    • keeping in mind that google has clear copies of his emails
    • servers at home are full-disk encrypted and need a password on reboot
  • davmail proxy which will contact to Microsoft Exchange and provides IMAP, SMTP, CalDav, and LDAP
  • evolutions components are getting skinnier
    • hooks into the gnome notifications
    • supports google
    • the calendar looks nice and simple
    • they are still backed by the evolution database
  • NextCloud (formerly OwnCloud) could be the next step for more self-hosting
  • talking about gitlab hosting

Notes for KWLUG June 2016: RaspberryPi

Omar and Khalid Baheyeldin presenting about using the RaspberryPi at KWLUG on June 6 2016. Omar described his Pi based plant watering project, and Khalid reviewed the new RaspberryPi 3

Omar presents his Automated Plant Watering project

  • built a automated planter watering project as part of his schools science fair
  • moisture sensor + MCP3008 ADC to connection to the GPIO header
  • consider using a monitoring/controlling services
    • but the data would be sent by email, which was not responsive enough
    • and connecting to the service would not work through his schools firewall
  • Omar also needed to supply his only wireless network, as the school's network would not pass his traffic, and we were able to connect to his router to see the graphs the device was generating
  • used an LED and an electro-relay to demonstrate how the valve would be controlled
  • demonstrated the data collection just by touching the moisture sensor
  • software used

Khalid's Review of the RaspberryPi 3

  • using the CanaKit Ultimate starter kit
  • RaspberryPi 2, which is supported in Debian
  • RaspberryPi 3, now 64-bit and has integrated WiFi and Bluetooth LE
  • RaspberryPi 3 still has proprietary boot loader, network and GPU
  • Khalid uses Raspian but there are lost of other OSes to choose from
  • Weather Forecaster project
    • bottle web server
    • pywapi, and he had to fix a bug in the process
  • Internet Radio
    • mopidy
    • can control through a variety of clients (e.g., ncmpc, mpdroid
    • tried an IR remote but the one he bought would not work
    • kodi, formerly XMBC
    • not headless, need a screen to navigate
    • but one a stream is selected you can start and stop without looking
  • investigating the OpenHab home automation project

Notes for KWRuby April 2016: Heroku and Ansible

Eric Roberts and Jesse McGinnis presented 2 different options for deploying a Rails project at the KWRuby meeting on April 19th 2016.

Eric Roberts: Heroku

  • Eric's advice is to just use Heroku when starting a new project
  • it is cheap, easy and allows you to ignore the deployment details on new project
  • Heroku deploys with a git push
  • recently added support for doing deployments from PR into staging environments
  • the free tier now limits server to running 13 hours/day
    • this eliminates a previous hack of keeping a server running by pinging it regularly
    • still more than enough for most purposes

Jesse McGinnis: Deploy Rails with Ansible

  • Jesse has recently been using Ansible for doing server setup and deployments
  • integration with Vagrant for local development, but the configuration can get quirky as it gets complicated
  • Boltmade has been using it for VM setup and deploys
  • Ansible also includes an inventory system
    • handle dynamic and environment based configurations
    • Jesse has not needed to use it, so far
  • Ansible galaxy roles <=> Ruby gems
  • Jessie finds that there is still a temptation to start a new system with bash
    • the assumption is that THIS system will not get more complicated, but they nearly always do
    • his lesson was to just always use Ansible
    • its not much more complicated than bash to start with, and avoid needing a re-write once it gets too complicated for bash

Notes for KWLUG April 2016: Docker

Tim Laurence presented a docker workshop to KWLUG on April 4th 2016,

  • there will be an Ubuntu release party on April 23rd

  • this is a repeat of Tim's presentation to KWRuby in March (which I also have some notes on)

  • Tim reminded us that:

    • docker images are immutable
    • docker containers are mutable
  • docker images size list is confusing because images are based on shared images, so list does not show real disk usage

  • dockid is similar to a git commit hash, but it is just a random number

  • dockid is the canonical id, and the names are just for readability

  • rmi command to delete all images which are only used by the specified image

  • Docker < LXC < OpenVZ

  • run -rm runs a container deleted on exist

  • volumes can be mounted in multiple containers

  • in Dockerfile CMD needs an absolute path, where ADD does not

Notes for KWRuby March 2016: Docker

Tim Laurence presented to KWRuby on March 15 2016. He took us through a workshop on how to use the Docker system for managing containers.

Tim has also done some other related presentations:

And finally the notes for this presentation:

  • Tim has a CS background and has been working as a sysadmin for ~20 years
  • he has coming backing into some developer stuff recently, working mostly in python
  • Docker containers are composed of layered file system images
    • these are layered together to get the final operating system
    • layers can removed (i.e., block out) things from previous layers
    • layers are addressed by hash so they cannot change without notice
  • when running a container changes occur in the top layer, which is read-write
  • you can list all the layers and their total size
docker images
  • hub.docker.com is the default place to find and publish docker images
    • it is also possible to store a private repository for private images or for local caching
    • anyone can push and images into hub.docker.com
    • so there are official images and un-official ones
    • looks sort of like projects on github
  • docker log returns to the STDOUT or STDERR
  • can turn a container into and image
    • tim is not going to show that but you can do it if you absolutely need to
  • list the contains which are running
docker ps
  • list all the contains, of any status
docker ps -a
  • can mount host directories in the docker container, either Read-Write or ReadOnly
  • docker has recently added a plugin system, examples of plugins include:
    • making two systems appears local to each other
    • use other external storage systems (e.g., NAS type stuff)
  • can mount a simple directory with docker volume for local data
  • there are various ways to handle secrets, but no canonical way to do it
  • images can be created by using a Dockerfile
  • docker-compose
    • will run and coordinate various docker machines
    • it seems very similar to a Vagrant file

Notes for KWLUG March 2016: DVD Creation and KDEnlive

Raul Suarez: Creating a Photo DVD

  • Raul is using FreeMind to organize his presentation, as usual
  • goals
    • open source
    • easy to teach
    • burn photos and videos to DVD
  • did the work over a weekend
  • video came on an SD card in MTS format
  • plan to spend time curating and making the photo orientation consistent
  • options considered
  • options chosen
  • Raul found these were both straight forward and quickly did what he wanted
    • he did not need any tutorials for either tool
  • there were 3 different events recorded and he wanted chapter/section for easy navigation
  • it was easy to import some photos to play with, and display pretty easily
  • background music can be added as well, but this does not play in preview mode
    • using a long piece of music for the menu actually generates a video
    • choosing a short piece of music will make this smaller and faster
  • doing the VOB generation is in "go and have dinner" time range
  • by generating an ISO image so that his father could download and burn avoiding puttig physical media into the postal service
  • creating the ISO can be another "go and have dinner" wait
  • mount the ISO to preview
    • Bob mentioned that VLC might also have an option for playing ISO's directly
  • Raul's final product for 2 hours was ~40% of the DVD, using SD video

Bob Jonkman: Video Editing with KDEnlive

  • Bob is going to try to use video that he recorded from Raul's presentation for the demo
  • Cinelara is a more capable than KDEnlive, but also more complicated
  • his install tends to crash on transitions so he will avoid them
  • using Saw Square Noise music for the sound track, which is licensed CC BY 3.0
  • Bob started this while making a nice video of his GNU social presentation
  • would like the razor tool to disable itself after use, as he usual want to do one cut at a time
  • the edits are stored in XML and the source videos are not changed
  • can separate the video & audio good for things like combining and cutting between multiple cameras
  • 3 - 4 times as long as the actual video to generate the whole video
  • Bob has not gotten into the back-end details
  • can also do a slideshow clip option
  • can normalize audio
  • lots of fancy 3d options
  • through this project Bob has gained more appreciation of how hard video/film editing actually is

Notes for KWRuby February 2016: Technical Debt

Declan Whelan presented to KWRuby meeting on February 17. He spoke about how to deal with technical debt in agile software projects:

Definition

  • old code bases tend to have lots of debt which slows down development
  • Martin Fowler described the tech debt quadrants (i.e., deliberate vs inadvertent, reckless vs prudent)
  • the term was originally coined by Ward Cunningham when describing technical trade-offs to the financial people on a project
  • Declan's definition is: anything in the code that slows me down
  • code which does not map well to the problem domain
  • badly written code, however, should not be considered technical debt (e.g., not tests, not TDD, not SOLID
  • Simon pointed out that technical debt can slow down paying feature debt, and that it can change as features change over time
  • Domain Driven Design(DDD) can help my continuously driving the code back towards the problem domain

Measuring

  • there are various software metrics which can be used measure/estimate technical debt (e.g., various kinds of static analysis, churn analysis)
  • the Agile Alliance working from is trying to develop tools for analyzing finance and risk related to technical debt in a project
  • Garter wrote a technical debt report in 2010
    • estimated the global cost of technical debt was $500 billion, and expected it to rise to $1 trillion by 2015
  • a more informally definition could include
    • how happy are you in the code
    • what % of time is wasted
    • what would it take to fix

Dealing with it

  • increasing tech debt will increase the desire for a re-write and it is well establish that a re-write can be VERY risky
    • this is something that Declan has seen in large companies
  • it is a deeper problem than "our developers write crappy code"
  • What are the underlying reasons?
  • over time is gets hard to do fundamental fixes
    • this gets worse unless there is specific push back on this
  • our cognitive bias of recency bias, leads us to short term/fast solutions
    • one strategy is to not offer short term/faster options to client, because they will always chooser faster
  • forward looking leaders are important to choosing slower but longer term solutions
  • legal liability may eventually apply some pressure
    • Uncle Bob had discussed this before
    • Declan did an interviewed with [Martin Fowler] and Uncle Bob at Remote Agile Conference
    • maybe some big catastrophic event will precipitate this
  • avoid short term project which build on underlying products
    • this encourages the short term choices
    • instead align team/projects to maintaining the products over the long term
  • Test Driven Developer(TDD) is a useful tool for guiding your code towards the better long term choices
  • [Scrum](https://en.wikipedia.org/wiki/Scrum_(software_development) has been a problem because is frequently adopted as a management practise without also adopting a corresponding technical process (i.e., Agile engineering practices, XP) such as:

  • some suggestions from the audience for helping with technical debt

    • business agreement
    • static analysis
    • explicitly organized dealing with technical debt with epics, sprints, or chore stories
    • education

Next page