What is ETag?

A web client (a browser) requests a resource from a web server.
Multiple calls for a resource would hit the server everytime and
server sends the response with return code 200.

What if the requested data is unchanged for most of the calls? Could the client somehow help server with the previous value of the requested resource?

Etags:

  • The purpose of ETag is to Save bandwidth and utilize client side caching.
  • The client gets a hash of content for the first time request of a resource.
  • The next call from the client has the ETag=Received hash value.
  • Server checks if the value of the requested respurce has modified. So it recalculates the hash and compares it against the received hash.
  • If Etag is same, server returns 304.

Written with StackEdit.

Advertisements

Ceph RGW Storage Policies

The storage policy allows a custom placement for user buckets, on a RADOS pool. A user bucket creation only request needs to specify the location constraint. Since default bucket placement uses common data, and index pools, and pool is a physical isolation unit for data in Ceph. A custom bucket placement allows us to use different pools for data and index. It solves the problem of a too-big pool, maintenance on a pool and expansion of a pool.

S3 Bucket Location Constraint
Bucket PUT request could contain location constraint as follows:

<CreateBucketConfiguration  
<LocationConstraint>BucketRegion</LocationConstraint>  
</CreateBucketConfiguration>

Use Cases for a cluster

  1. A set of graded pools that use SSD, HDD, Hybrid storage media. We can prioritize user data latency requirements with a suitable pool.

  2. Replicated and EC pools for data could help us efficiently host hot and cold data under the same cluster.

  3. A template based cluster expansion where we create new pools for new storage needs.

  4. Physically isolate critical accounts

References

Method to place a bucket

  • Update the zonegroup configuration to add a new bucket placement rule.
  • Commit and update rgw period.
  • Update the zone configuration to add an entry for new placement, with pools for data, index and extra.
  • Commit and update rgw period.

Client code for bucket creation

import boto3

access_key = 'abcde'
secret_key = '12345'
endpoint_url = 'http://10.32.34.157'

conn = boto3.client(service_name='s3',
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
endpoint_url=endpoint_url)

connection_type = 'client'
conn.create_bucket(Bucket='ourbucket', 
                    CreateBucketConfiguration=
                    {'LocationConstraint': ":special-placement"})

Output

# radosgw-admin zonegroup get
{

"id": "696b9fca-bf82-4030-85fd-9b4281241c66",

"name": "in",

"api_name": "in",

"is_master": "true",

"endpoints": [],

"hostnames": [],

"hostnames_s3website": [],

"master_zone": "5221e79f-0ce1-4e9b-82d7-3b0095bba8ad",

"zones": [

{

"id": "5221e79f-0ce1-4e9b-82d7-3b0095bba8ad",

"name": ".in-chennai-1",

"endpoints": [],

"log_meta": "false",

"log_data": "false",

"bucket_index_max_shards": 0,

"read_only": "false",

"tier_type": "",

"sync_from_all": "true",

"sync_from": []

}

],

"placement_targets": [

{

"name": "default-placement",

"tags": []

},

{

"name": "special-placement",

"tags": []

}

],

"default_placement": "default-placement",

"realm_id": "5f04c150-b66b-4d05-ba31-38354eaf6ec0"

}
# radosgw-admin zone get

{

"id": "5221e79f-0ce1-4e9b-82d7-3b0095bba8ad",

"name": ".in-chennai-1",

"domain_root": ".in-chennai-1.rgw.meta:root",

"control_pool": ".in-chennai-1.rgw.control",

"gc_pool": ".in-chennai-1.rgw.log:gc",

"lc_pool": ".in-chennai-1.rgw.log:lc",

"log_pool": ".in-chennai-1.rgw.log",

"intent_log_pool": ".in-chennai-1.rgw.log:intent",

"usage_log_pool": ".in-chennai-1.rgw.log:usage",

"reshard_pool": ".in-chennai-1.rgw.log:reshard",

"user_keys_pool": ".in-chennai-1.rgw.meta:users.keys",

"user_email_pool": ".in-chennai-1.rgw.meta:users.email",

"user_swift_pool": ".in-chennai-1.rgw.meta:users.swift",

"user_uid_pool": ".in-chennai-1.rgw.meta:users.uid",

"system_key": {

"access_key": "",

"secret_key": ""

},

"placement_pools": [

{

"key": "default-placement",

"val": {

"index_pool": ".in-chennai-1.rgw.buckets.index",

"data_pool": ".in-chennai-1.rgw.buckets.data",

"data_extra_pool": ".in-chennai-1.rgw.buckets.non-ec",

"index_type": 0,

"compression": ""

}

},

{

"key": "special-placement",

"val": {

"index_pool": ".in-chennai-1.rgw.buckets.index",

"data_pool": ".in-chennai-1.rgw.special.storage",

"data_extra_pool": ".in-chennai-1.rgw.buckets.non-ec",

"index_type": 0,

"compression": ""

}

}

],

"metadata_heap": "",

"tier_config": [],

"realm_id": "5f04c150-b66b-4d05-ba31-38354eaf6ec0"

}
# radosgw-admin bucket stats --bucket=ourbucket

{

"bucket": "ourbucket",

"zonegroup": "696b9fca-bf82-4030-85fd-9b4281241c66",

"placement_rule": "special-placement",

"explicit_placement": {

"data_pool": "",

"data_extra_pool": "",

"index_pool": ""

},

"id": "5221e79f-0ce1-4e9b-82d7-3b0095bba8ad.646695.2",

"marker": "5221e79f-0ce1-4e9b-82d7-3b0095bba8ad.646695.2",

"index_type": "Normal",

"owner": "rgwadmin",

"ver": "0#3",

"master_ver": "0#0",

"mtime": "2018-06-16 00:24:22.644511",

"max_marker": "0#",

"usage": {

"rgw.main": {

"size": 7707,

"size_actual": 8192,

"size_utilized": 7707,

"size_kb": 8,

"size_kb_actual": 8,

"size_kb_utilized": 8,

"num_objects": 1

}

},

"bucket_quota": {

"enabled": false,

"check_on_raw": false,

"max_size": -1,

"max_size_kb": 0,

"max_objects": -1

}
}

MySQL & Python- Error: 2006 mysql has gone away

MySQL & Python: Error: 2006 mysql has gone away

This problem occurs for multiple reasons such as DB connection problem. In our code, we hit this issue due to a subtle problem with DB cursor.

The code was as following:

with conn as cur:
    try:
        print "hello"
    except:
        print "sorry"
    finally: 
        conn.close()

The above code would throw the error 2016 mysql has gone away exception. The problem lies in with conn as cur. This statement creates a cursor on the DB and the cursor object autmatically gets destroyed.
Here, we are closing the DB connection before the automatic destruction happened.

So since connection was invalid(closed), cursor deletion hit an exception.

The solution is to close the connection after cursor object deletion.

try:
    with conn as cur:
        print "hello"
except:
    print "sorry"
finally: 
    conn.close()

Written with StackEdit.

dyld: Library not loaded: Python.framework

I started getting this error after trying to install macvim as follows:

brew install macvim --override-system-vim

The error string is as following:

$ vi linkedlist.cc
dyld: Library not loaded: /usr/local/opt/python/Frameworks/Python.framework/Versions/3.6/Python
  Referenced from: /usr/local/bin/vim
  Reason: image not found
Abort trap: 6

I checked shared libs for vim using otool.

$ otool -L /usr/local/bin/vim
/usr/local/bin/vim:
	/usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
	/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
	/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 1561.40.112)
	/usr/local/opt/lua/lib/liblua.5.3.dylib (compatibility version 5.3.0, current version 5.3.4)
	/usr/local/opt/perl/lib/perl5/5.26.1/darwin-thread-multi-2level/CORE/libperl.dylib (compatibility version 5.26.0, current version 5.26.1)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)
	/usr/lib/libutil.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/local/opt/python/Frameworks/Python.framework/Versions/3.6/Python (compatibility version 3.6.0, current version 3.6.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1452.23.0)
	/usr/local/opt/ruby/lib/libruby.2.5.dylib (compatibility version 2.5.0, current version 2.5.1)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
	/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 822.31.0)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1452.23.0)

It appears that many shared libs versions are updated incorrectly.

To solve the problem, I tried to upgrade packages.

$ brew update
Already up-to-date.

$ brew upgrade

It started working after doing upgrades.

Written with StackEdit.

Why use Base64 Encoding?

What is Base64 encoding?

  • Given a stream of binary bits, it will encode 6-bits to a character from a set of 2 pow 6 (64 chracters).
  • Example “abcd”, the ASCII representation is 65666768.
  • [1000001][1000010][1000011][1000100]
  • Base64 would pics six continuous bits
  • 100000|| 110000|| 101000|| 011100||0100xx here xx would be 00 (padding)
  • gwocQ

Why use base64 encoding?

  • Transferring binary data in URLs
  • Tranferring binary data such as images as text
  • Transmit and store text that might cause delimiter collision.
    • Example is a random string followed by a delimiter (_) and a pattern and the code logic searches the delimiter to seperate the pattern.
    • The _ can appear in the generated random string too.
    • So encoding the random string in base64 would avoid such case.
    • Embed image in a XML

Golang Runtime and Concurrency

  • Golang uses a user-space component (runtime) linked to the executable.
  • The runtime is written in C.
  • It has implementation of scheduler, goroutine management and OS-threads management.
  • Per go process, there is a max limit of OS threads.
  • Go runtime schedules N goroutines on M OS threads
  • One goroutine runs exactly on one thread.
  • A goroutine can get blocked (e.g. on a syscall) and blocks the OS-thread too.

References