Friday, May 30, 2014

Quickly create large files in python (fast disk space allocation)

Quickly create large files in python (fast disk space allocation)

Sometimes you want quickly allocate some disk space to a certain file, without worrying about what the content is. Here are two ways to do it in python (in this case I just want to test whether os.stat handles large files correctly, and it turns out to be fine):

Double click to toggle code
import os

with open("newfile", "wb") as f: * 1024 * 1024 * 40 - 1)

with open("newfile1", "wb") as out:
    out.truncate(1024 * 1024 * 1024 * 40)

In some file systems, the first method (seek and write) may give you file appearing to be 40G, but actually only occupying 1 byte on disk, this is called a sparse file system.

Thursday, May 29, 2014

Python module structure example

Python module structure example

Here is the directory tree:
directory tree of the example module
Double click to toggle code

1 directory, 6 files is just an empty files that tells python that this folder can be used as a module. The content of the rest 2 files are listed here:
Double click to toggle code
import os
from .test2 import sayhi

class sysutil():
    def modpath():
        return os.path.dirname(os.path.realpath(__file__))
    def cwd():
        return os.getcwd()
    def newsayhi():
Double click to toggle code
def sayhi():
    print("I said hi.")

Notice how imports a function from, by putting a dot before (meaning the current directory), this is called intra-package reference.

Tuesday, May 27, 2014

Extra wifi access point with a second router

  1. Make sure that internet is working on the first router / IAD / modem
  2. Set up wifi SSID and password in the second router, make sure you can access it from your mobile device
  3. Disable DHCP on the second router
  4. Connect one the LAN ports of the first router to one of the LAN ports of the second
  5. Enjoy!

ubuntu packages to install and settings

ubuntu packages to install and settings

ubuntu packages to install and settings

# math equation editor

# chinese input

# tup the build system for c++
sudo apt-get install libfuse-dev
download and follow instructions.

# armadillo, c++ linear algebra library
# download, unzip, cd into dir
sudo aptitude install liblapack3 liblapack3gf liblapack-dev liblapacke liblapacke-dev liblapack-pic libatlas3-base libatlas3gf-base libatlas-base-dev libatlas-cpp-0.6-1 libatlas-cpp-0.6-dev libatlas-dev libarpack++2c2a libarpack++2-dev libopenblas-base libopenblas-dev cmake
cmake .
sudo make install

# zip applications
sudo apt-get install unace unrar zip unzip p7zip-full p7zip-rar sharutils rar uudeview mpack lha arj cabextract

# needed by deepin-scrot
sudo aptitude install python-xlib 

# play dvd
sudo aptitude install libdvdread4 libavcodec-extra
sudo /usr/share/doc/libdvdread4/

# clear type font rendering

# commandline email:

# gtk theme engines / icons

# code tidy up

# take photos from webcam

# kde dolphin, thumbnails for video files

# print to pdf

# newline convertion

# some r packages need this

# clang c/c++ complier


# mp3 tag editor

# screen shooter


# clipboard manager on commandline

# keyboard and mouse automation

# kde settings
information tips: no
session management: start with empty session
services: remove unnecessary ones
session start script: ~/bin/
font: msyh 10 regular
gtk style: qtcurve
custom shortcuts: import ~/.kaiyin_custom_kde_shortcuts.khotkeys and
global shortcuts: set everything to none, then import  ~/.kaiyin_global_kde_shortcuts.kksrc

# eclipse plugins and settings
# color theme / syntax highlighting
# ui theme
# vim simulator: vrapper
# vim style ctrl+n ctrl+p selecting autocompletion suggestions: in pref--general--keys, configure "line down" and "line up", in context of "text editing"
# file explorer view: RSE plugin, in marketplace, search for Remote system explorer

# qtcreator settings:
# in fakevim, check pass control keys. if you need any control key in fakevim, just disable it in the normal editor (settings--environment--keyboard)

# things to backup before reinstall:
# kde custom shortcuts
# qtcreator shortcuts
# kde color themes

Friday, May 16, 2014

Pydev test, fairly good, but not perfect

Pydev test, fairly good, but not perfect

I am trying out the pydev plugin for eclipse, so far the code completion looks ok, but the python console is not reliable at all. Here is a very simple test:

Double click to toggle code

from subprocess import Popen
from subprocess import PIPE
shell_cmd = ["xsel", "-b"]
p = Popen(shell_cmd, stdin=PIPE)
p.communicate("what the hell is this?".encode("ascii"))

This code is just trying to put some text into the system clipboard, and it runs ok in ipython3, yet results are weird in eclipse. The same occurs to pycharm, I have no clue why this is happening.

Wednesday, May 14, 2014

Genotype collapse function using python

Genotype Letter Code Number Code PLINK Code
HB 00

Homozygote B HB 02 00
NA 00 01

HE 00 01 00

heterozygote HE 01 10
HA 00 01 11 11
Homozygote A HA 00 11

02 NA 01 00

02 00

NA 00 01

01 00 01 00

00 00 01 11 11

00 01 10 11

00 00

01 00 01

10 00 01 00

11 00 01 11 11

def collapse(x, y):
    if x == "00" or y == "00":
        return "00"
    elif x == "01" or y == "01":
        return "01"
    elif x == "11" or y == "11":
        return "11"
    elif x == "10" and y == "10":
        return "00"
        raise Exception("Nonsense combination!")

# test collapse function
allfour = ["00", "01", "10", "11"]
for i in allfour:
    for j in allfour:
        res = collapse(i, j)
        print(res, end="  ")

# from random import randint
# xint = randint(0, 256)
# yint = randint(0, 256)
def collbyte(xbyte, ybyte):
    x = "{:08b}".format(xbyte)
    xstring = [x[i:i+2] for i in range(0, len(x), 2)]
    y = "{:08b}".format(ybyte)
    ystring = [y[i:i+2] for i in range(0, len(y), 2)]
    collstring = [collapse(xstring[i], ystring[i]) for i in range(4)]
    # fh.write("\n---------------------------------------------\n")
    # fh.write("byte1: {}, binString1: {}".format(xbyte, x) + "\n")
    # fh.write("byte2: {}, binString2: {}".format(ybyte, y) + "\n")
    # fh.write(str(xstring) + "\n")
    # fh.write(str(ystring) + "\n")
    # fh.write(str(collstring) + "\n")
    resbyte = int("".join(collstring), 2)
    return resbyte

import numpy as np
collgen = np.zeros((256, 256), dtype=np.unsignedinteger)
with open("/tmp/collgen.csv", "w") as fh:
    for i in range(256):
        for j in range(256):
            fh.write(str(collbyte(i, j)) + ", ")

np.savetxt("/tmp/collgen.csv", collgen, delimiter=",")

# def checkbytes(xbyte, ybyte):
#     if collbyte(xbyte, ybyte) != collgen[xbyte][ybyte]:
#         print("Error!")
# for i in range(256):
#     for j in range(256):
#         checkbytes(i, j)

Tuesday, May 13, 2014