Recent blog entries

25 Jul 2016 Hobart   » (Journeyer)

Cgroups example - limiting memory to control disk writes (Debian)

I ran into a problem with an overactive process that left the rest of the system running slow. nice(1) did nothing to solve it, neither did ionice(1) rescheduling it to "Idle". If you run into something similar, cgroups may help

cgroups ("Control groups") were developed at Google around 2006 and showed up in Linux around 2.6.24. Searching for cgroups examples largely leads one to the RHEL Resource Management Guide. (Link goes to the latest version, most Google searches point to older copies.)

In my case, I had a long running (>1hr) process that wrote several hundred GB of output.

I looked at the processes' speed by piping it through pv(1), and also looked at top(1), iotop(1), and

$ watch cat /proc/meminfo  (watching the Dirty: line)

The process was doing buffered writes to disk, which was good (keeping the disk continuously fed for best throughput) but was filling up huge amounts of cache (1~2 dozen GB of Dirty pages.) When I paused it, sync(1) took over 5min to complete.

Debian 8.0 (Jessie) has cgroups by default but, the memory type are disabled by default.

# apt-get install cgroup-tools
# vi /etc/default/grub

(Add  cgroup_enable=memory  to kernel boot parameters, run  update-grub2  and reboot.)

# cgcreate -g memory:/foo
# echo 64M > /sys/fs/cgroup/memory/foo/memory.limit_in_bytes
# cgexec -g memory:/foo bash
#
(your task here)

The cgcreate(1) command is a fancy equivalent to doing a mkdir in the cgroup partition, which automatically is populated with the appropriate control files. Debian 8's kernel has both cgroup and cgroup2 support, but as systemd(8) is using version 1 and it appears the two cannot be used concurrently, that's what I used.

Pros:

  • Fast throughput - better than piping through dd oflag=direct or dd oflag=dsync
  • Solved the system-wide performance hit
  • Everything ran nicely and the watching meminfo (as above) showed dirty pages were being regularly flushed

Cons:

  • Your task might be hit by the OOM killer.
  • Your task can have malloc(3) calls fail, which makes most tools bail out.

This feels like a hack solution, but since cgroups can't limit just write buffered memory yet, and using cgroups actual disk-write limiter (blkio.throttle.write_bps_device) would require the above-mentioned slow dd(1) (which ran at 30% of the speed, at best) and none of the other tools actually worked, I'm sharing it. YMMV - and I'd love to hear of other solutions that actually work for people. A good test program to run is:

$ pv -S -s 80g < /dev/zero > zeroes.dat
(write 80GB to a file, with progress bar and live throughput details)

Syndicated 2016-07-25 07:29:17 from jon's blog

24 Jul 2016 elwell   » (Journeyer)

The writing's on the wall

For the last couple of jobs I've had, having some sort of status display has proved itself really useful. Things like an overloaded nagios dashboard help to drill down to see what system issues you may have, but on large systems there'll always be some component that's not green (however your service should work around these transparently to the end user).  In a smaller team without 24/7 operations staff and shift handovers, how do you know things aren't on fire when you walk into the office? - I'll ignore the fact that you probably read your email over breakfast.

Concerto

At $job[-1] we put a spare monitor on the office wall and ran concerto on a PC feeding it. The backend at that time was php, and made assumptions such as assuming that short tags were OK - I hacked on a branch to make it more standard with the scientific linux systems we were using at the time. Given this was (is?) a student project out of the Rensselaer Institute ir's hardly suprising as young developers want LATEST SHINY. They then went through a second system effect, rewriting from scratch and completely missing the launch of the raspberry pi, which could have made a killer combination.
The problem is that many browser based clients need the overhead of X11 and all the various hacks to remove mouse cursors and make them more 'kiosky'

info-beamer

Fast forwards a few years and I have mini-magnus on my desk showing status and a set of 3 unused monitors on a wall looking to display some info. A quick bit of research flagged up info-beamer which has been used in production at the CCC events for several years. I've been playing with it for a couple of days now in the standalone pi and hosted variant.

hosted info-beamer

The install of this is very smoothly done. Small zip file download to populate an SD card with the raspberry pi bootloader files and a squashfs  and it self-installs the rest of the distribution from S3 and prompts you to register the node to your info-beamer account (in a similar way to a chromecast).
Florian has made some really nice touches to the setup - little things like setting cec_osd_name if your TV screen uses it, a custom kernel logo, and suppression of all but the player-related boot messages. In a public area, this makes it look a lot more professional than most of the other solutions which show the operating system before launching a player should they reboot. I've only played with a couple of the sample packages, but the install process is slick, if initially confusing terminology between packages, setups and playlists.


I'd love to see this integrated with indico - the meeting software used at CERN. Hint :-)

standalone info-beamer

The personal use player is distributed as a binary executable - I can understand the reasons for that, but it doesn't feel right. It doesn't (yet?) come as a debian package - It should be trivial for me to do myself according to the docs. When that's done, I'll install and run using systemd rather than the daemontools method (which is used by the various syncer scripts used in the hosted version). My concern is that the logging is presently noisy (good for debugging, bad for SD lifetime) 
I'm also planning on managing these R-Pi nodes using Ansible (Puppet is overkill for this as I want something that can bootstrap up a fresh SD card without needing extra daemons running) so I suspect there'll be future blogging on that. 

I've not yet investigated sending values directly to the info-beamer listening port (hoping I can do something with an MQTT subscriber) but pushing json files from various subsystems seems to work well. Obligatory screenshot:
Prototype display testing
Overall, I'm very impressed. Given I'd started learning lua for some nodemcu work I should be able to develop something functional. I've asked the work graphic designer to assist, so hopefully it won't end up "engineer style"

Syndicated 2016-07-24 03:16:00 (Updated 2016-07-24 10:28:50) from Andrew Elwell

24 Jul 2016 LaForge   » (Master)

Going to attend Electromagnetic Field 2016

Based on some encouragement from friends as well as my desire to find more time again to hang out at community events, I decided to attend Electromagnetic Field 2016 held in Guildford, UK from August 5th through 7th.

As I typically don't like just attending an event without contributing to it in some form, I submitted a couple of talks / workshops, all of which were accepted:

  • An overview talk about the Osmocom project
  • A Workshop on running your own cellular network using OpenBSC and related Osmocom software
  • A Workshop on tracing (U)SIM card communication using Osmocom SIMtrace

I believe the detailed schedule is still in the works, as I haven't yet been able to find any on the event website.

Looking forward to having a great time at EMF 2016. After attending Dutch and German hacker camps for almost 20 years, let's see how the Brits go about it!

Syndicated 2016-07-23 14:00:00 from LaForge's home page

24 Jul 2016 LaForge   » (Master)

EC-GSM-IoT: Enhanced Coverage GSM for IoT

In private conversation, Holger mentioned EC-GSM-IoT to me, and I had to dig a bit into it. It was introduced in Release 13, but if you do a web search for it, you find surprisingly little information beyond press releases with absolutely zero information content and no "further reading".

The primary reason for this seems to be that the feature was called EC-EGPRS until the very late stages, when it was renamed for - believe it or not - marketing reasons.

So when searching for the right term, you actually find specification references and change requests in the 3GPP document archives.

I tried to get a very brief overview, and from what I could find, it is centered around GERAN extension in the following ways:

  • EC-EGPRS goal: Improve coverage by 20dB
    • New single-burst coding schemes
    • Blind Physical Layer Repetitions where bursts are repeated up to 28 times without feedback from remote end
      • transmitter maintains phase coherency
      • receiver uses processing gain (like incremental redundancy?)
    • New logical channel types (EC-BCCH, EC-PCH, EC-AGC, EC-RACH, ...)
    • New RLC/MAC layer messages for the EC-PDCH communication
  • Power Efficient Operation (PEO)
    • Introduction of eDRX (extended DRX) to allow for PCH listening intervals from minutes up to a hour
    • Relaxed Idle Mode: Important to camp on a cell, not best cell. Reduces neighbor cell monitoring requirements

In terms of required modifications to an existing GSM/EDGE implementation, there will be (at least):

  • changes to the PHY layer regarding new coding schemes, logical channels and burst scheduling / re-transmissions
  • changes to the RLC/MAC layer in the PCU to implement the new EC specific message types and procedures
  • changes to the BTS and BSC in terms of paging in eDRX

In case you're interested in more pointers on technical details, check out the links provided at https://osmocom.org/issues/1780

It remains to be seen how widely this will be adopted. Rolling this cange out on moderm base station hardware seems technicalyl simple - but it remains to be seen how many equipment makers implement it, and at what cost to the operators. But I think the key issue is whether or not the baseband chipset makers (Intel, Qualcomm, Mediatek, ...) will implement it anytime soon on the device side.

There are no plans on implementing any of this in the Osmocom stack as of now,but in case anyone was interested in working on this, feel free to contact us on the osmocom-net-gprs@lists.osmocom.org mailing list.

Syndicated 2016-07-23 10:00:00 from LaForge's home page

21 Jul 2016 sye   » (Journeyer)

did know our Raph had pull-out this fantastic stunt before !

http://www.petting-zoo.net/~deadbeef/archive/340.html


unrelated tip:

https://kr5ddit.com/post/1031/attn-procrasti-free-ssl-certs


Procrasti, I know a while back you had hassles getting a cert.
Have you considered these guys?

https://letsencrypt.org/about/

Let’s Encrypt is a free, automated, and open certificate authority (CA), run for the public’s benefit. Let’s Encrypt is a service provided by the Internet Security Research Group (ISRG).

The key principles behind Let’s Encrypt are:

Free: Anyone who owns a domain name can use Let’s Encrypt to obtain a trusted certificate at zero cost.
Automatic: Software running on a web server can interact with Let’s Encrypt to painlessly obtain a certificate, securely configure it for use, and automatically take care of renewal.
Secure: Let’s Encrypt will serve as a platform for advancing TLS security best practices, both on the CA side and by helping site operators properly secure their servers.
Transparent: All certificates issued or revoked will be publicly recorded and available for anyone to inspect.
Open: The automatic issuance and renewal protocol will be published as an open standard that others can adopt.
Cooperative: Much like the underlying Internet protocols themselves, Let’s Encrypt is a joint effort to benefit the community, beyond the control of any one organization.

20 Jul 2016 ade   » (Journeyer)

Art, AI and Google I/O 2016

What if there are interesting things waiting to be discovered where artificial intelligence meets art?