Why Keep Swap Memory in Linux?

To cache rarely accessed pages. It is still better to keep them in swap than paging them out (discarding).
But always have enough RAM to hold your expected load.

Thorough Perf Tuning Guide

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/performance_tuning_guide/index

Reference

https://haydenjames.io/linux-performance-almost-always-add-swap-space/

Written with StackEdit.

Notes on Memory Mapped Files using mmap

Introduction

mmap maps a file to a process virtual address space. The file blocks are loaded as needed, in units of the system page size. Each I/O must align with the page size. A block load is handled as a page fault.

mmap() can provide a private or shared mapping of a region.

mmap() to allocate heap memory

Interesting usage of mmap is to allocate heap memory in a process.

MAP_ANONYMOUS
or
MAP_ANON

This flag tells the system to create an anonymous mapping, not connected to a file. filedes and offset are ignored, and the region is initialized with zeros.

Anonymous maps are used as the basic primitive to extend the heap on some systems. They are also useful to share data between multiple tasks without creating a file.

On some systems using private anonymous mmaps is more efficient than using malloc for large blocks. This is not an issue with the GNU C Library, as the included malloc automatically uses mmap where appropriate.

Example

#include <stdio.h>
#include <unistd.h>
#include <sys/mman.h>
#include <fcntl.h>
#include "assert.h"
#include <string.h> 

int main() {
    size_t pageSize = getpagesize();

    printf("system page size=%zu bytes\n", pageSize);
    int fd = open("/tmp/xx", O_RDONLY, 0);
    char *region = mmap(NULL, 4*pageSize, PROT_READ|PROT_EXEC, MAP_ANON|MAP_PRIVATE, fd, 0);
    
    assert(region != MAP_FAILED);

    memset(region, 'y', pageSize);
    
    char read_buf[4096];
    memcpy(read_buf, region, pageSize);
    printf("char at 0 offset = %c\n", read_buf[0]);

    // cleanup
    int rc = munmap(region, 4*pageSize);
    assert(rc == 0);
    close(fd);
    return 0;
}

References

Written with StackEdit.

How to Get PSQL Client on Alpine Linux

How to Install?

$ apk --update add postgresql-client

How to Login?

$ psql -h host_name -d db_name -U user_name

Keep the user password ready 🙂

Reference

https://gist.github.com/petrkohut/2ed5dc83ca4afd508e9b

Written with StackEdit.

One Way Communication with Pipe in Linux

Pipe uses kernel buffer to store data. It is a unidirectional read/write buffer and connects two processes. Each end (read or write) Pipe mimics a file operation. A process can choose to read or write (but not both). The processes have to use at least two pipes for two-way communication.

Example Communication

Parent process can talk to a child with an EOF. Just close its write end of the pipe. The child would get EOF on reading.

Example

#define _GNU_SOURCE
#include <sys/types.h>
#include <sys/wait.h>
#include <stdio.h>
#include <sched.h>
#include <signal.h>
#include <unistd.h>
#include <assert.h>

int main()
{
    int contact[2];
    pipe(contact);

    int fd = fork();
    if (fd == 0) { // child
       char c;
       close(contact[1]); // close the write end.

       while (read(contact[0], &c, 1) > 0) // each call checks for EOF.
           printf("c=%c", c);
       close(contact[0]);
       exit(0);
    } else { // parent
        close(contact[0]); // close the read end.

        write(contact[1], "X", 1);
        close(contact[1]); // child sees EOF
        wait(NULL);
        exit(EXIT_SUCCESS);
    }
    return 1;
}

References

Written with StackEdit.

Why Docker is a Long Term Future for Platform?

What make Docker so popular and long-lasting?

Container is essentially OS level virtualization. Each application gets illusion of its own OS, having almost absolute control over it. Another advantage is that host OS knows about the container processes and hence can share its resources among hosted containers.
The concept of containers was started by FreeBSD, refined by Solaris and re-implemented by Linux.
Containers are better than two other levels of virtualization:
- ABI/platform level, where application integrates with the platform (Google App Engine), doesn’t scale well.
- Hardware level, where a virtual hardware runs the OS (e.g. virtual machines, hypervisors).
Docker containers run close to the real hardware, and host OS has knowledge of resource usage. Hence it’s an optimal sweet spot for virtualization.
Joyent SmartOS is built on OpenSolaris and provides Solaris features to Linux like Docker containers. It acheives that by allowing Linux APIs translated to Solaris APIs. Everything runs on the bare metal hence.
SmartOS containers get ZFS, Dtrace by default 🙂
SmartOS containers are very secure as they run in zones.

I hope to cover Smart OS design internals, Docker container on Linux details in next posts.

Reference

https://www.joyent.com/blog/docker-and-the-future-of-containers-in-production

Linux Memory Management Tricks

Tips to Improve Dynamic Memory Performance

Instead of using memset() to initialize malloc()’ed memory, use calloc(). Because when you call memset(), VM system has to map the pages in to memory in order to zero initialize them. It’s very expensive and wasteful if you don’t intend to use the pages right away.

calloc() reserves the needed address space but does not zero initialize them unless memory is used. Hence it postpones the need to load pages in to memory. It also lets the system initialize pages as they’re used, as opposed to all at once.

Lazy allocation: A global(normal variable or a buffer) can be replaced with a static and a couple of functions to allow its access.
memcpy() & memmove() needs both blocks to be memory resident. Use them if size of blocks is small(>16KB), you would be using the blocks right away, if blocks are not page aligned, blocks overlap.
But if you intend to postpone the use, you would increasing the working set of the application. For small amount of data, use memcpy().
To check heap dysfunctional behavior: $ MALLOC_CHECK_=1 ./a.out
It’ll give an address related to each violation of dynamic memory routines.
Electric fence : Works very well with gdb
Libsafe: for libc routines

Internet Not working eh? How Ubuntu Resolves DNS?

Problem

I upgraded my Ubuntu from 16.04 to 18. And the Internet stopped working. 😦

I tried rebooting interface, adding a new interface to ifconfig. But it did not help.

What helped was an answer on AskUbuntu

How it worked?

The answer is in how Ubuntu lookup domains like www.google.com.

Step 1
Check the domain in the local DNS, also called a DNS stub.

Step 2
If Step 1 fails (local DNS lookup is a miss), it contacts a public DNS such as 8.8.8.8 also called Google Public DNS.

Since after the upgrade public DNS was not set, the internet seemed inaccessible!

How to Update Public DNS?

open /etc/resolv.conf
Add DNS=8.8.8.8

Done 🙂

Written with StackEdit.I

Internals of Linux Process Signals

Linux Signals

A process in Linux is represented as task_struct.
A process A sends a signal to process B using system call kill() or kill -<sig num> <pid>.
Kernel updates the task_struct of the receiving process B. It changes the signal field with the passed value.
The scheduler checks the signals on a process before running the process. If the signal is set, it calls the signal handler.
A process can define its handlers for all signals except SIGKILL and SIGSTOP.
There is always a default signal handler for each signal.

References

Find a String in Files Excluding Some Files

—
tags: ‘shell, linux, command, development, grep,find’
categories: development

—

I use the following command to find a string in a Go repo. The vendor directory is not needed, so skip it.

$ find . -type f |grep -v vendor|cut -d":" -f2|xargs grep -n my_search_string

The command find all files in current directory. Next, it removes all files that have vendor in their path.

401:./.git/HEAD
402:./.git/info/exclude
403:./.git/logs/HEAD

The cut command picks the filename and passed to xargs. Each file is then processed by grep for the search string.

Fixing Terminal Row & Columns Display on Docker and Linux

Ever faced garbled screen on a terminal, text wrapping over and screen command string messed up!
It happens because the terminal is using default row and column value (e.g. columns = 80).

The following command fixes it (tested on Docker container’s terminal)

docker exec -e COLUMNS="`tput cols`" -e LINES="`tput lines`" -ti

How it Works?

tput initializes or reset a terminal by providing terminal info to the shell.

From man page of tput:

The tput utility uses the terminfo database to make the
values of terminal-dependent capabilities and information available to the shell

Let’s find the type of terminal
```
# echo $TERM
  xterm
```
Let’s see what’s suggested rows and columns for this terminal.
```
# tput cols
167
# tput lines
49
```
While accessing the container terminal, we passed the number of columns and rows to COLUMNS & LINES variable.
The terminal database is present at /usr/share/terminfo.

References

Written with StackEdit.

Thorough Perf Tuning Guide

Reference

Share this:

Introduction

mmap() to allocate heap memory

Example

References

Share this:

Share this:

Example Communication

Example

References

Share this:

Reference

Share this:

Tips to Improve Dynamic Memory Performance

Share this:

Problem

How it worked?

How to Update Public DNS?

Share this:

Linux Signals

References

Share this:

Share this:

How it Works?

References

Share this: